A Hive small file management method

Abstract

The present invention relates to the field of file management technology, and in particular to a Hive small file management method, the method comprising: setting an initial cleaning cycle, regularly cleaning the generated small files to obtain regular cleaning results; real-time monitoring of the generation of small files, and determining whether the small files are abnormal based on the generation; determining the abnormality level of the small files based on the abnormality of the small files; determining a small file management strategy based on the abnormality level, and determining a management score based on running the small file management strategy; calculating an overall performance score based on the regular cleaning results combined with the management score; adjusting the initial cleaning cycle based on the overall performance score, or optimizing the management strategy. The present invention reduces the need for manual intervention, improves efficiency, avoids the accumulation of small files and performance degradation, and ensures the efficient operation of the management process through comprehensive management from regular cleaning to performance scoring to strategy optimization.

Description

Technical Field

[0001] The present invention relates to the field of file management technology, and in particular to a Hive small file management method. Background Art

[0002] The current small file management technology is unable to achieve comprehensive management from regular cleanup to performance scoring and policy optimization, resulting in excessive manual intervention and low management efficiency. It is unable to quickly identify and respond to problems. It is unable to effectively handle various small file anomalies, resulting in poor management stability and performance degradation caused by the accumulation of small files.

[0003] Chinese patent application publication number CN116069741A discloses a file processing method, which is applied to a file storage system, the method comprising: obtaining a first file directory of the file storage system, the first file directory comprising file identification information and file size information of N files, the size of each of the N files being less than or equal to a first target value, where N is an integer greater than 1; performing a first processing on the file identification information of the N files using a backtracking algorithm, wherein the first processing is used to divide the file identification information of the N files into piles to obtain M piles, where M is an integer greater than 1; wherein, among the M piles, the total size value of the files corresponding to all the file identification information of each pile of at least M-1 piles satisfies the following conditions: the difference with the first target value is less than or equal to a first threshold value, and is less than or equal to a storage space value corresponding to the minimum storage unit of the file storage system.

[0004] This shows that the current small file management technology cannot effectively handle various small file abnormalities, resulting in poor management stability, leading to the accumulation of small files and causing performance degradation. Summary of the invention

[0005] To this end, the purpose of the present invention is to provide a Hive small file management method to overcome the problem that the current small file management technology cannot effectively handle various small file abnormalities, resulting in poor management stability and causing performance degradation due to the accumulation of small files.

[0006] To achieve the above object, the present invention provides a Hive small file management method, comprising:

[0007] Set the initial cleanup cycle to regularly clean up the small files generated to obtain regular cleanup results;

[0008] Monitor the generation of small files in real time, and determine whether there are any abnormalities in the small files according to the generation situation;

[0009] Based on the existence of anomalies in small files, determine the anomaly level of small files;

[0010] Determine a small file governance policy according to the abnormality level, and determine a governance score based on running the small file governance policy;

[0011] Calculate an overall performance score based on the periodic cleaning results combined with the governance score;

[0012] The initial cleaning cycle is adjusted based on the overall performance score, or the governance strategy is optimized.

[0013] Furthermore, the process of real-time monitoring the generation of small files and determining whether there is an abnormality in the small files according to the generation includes:

[0014] Real-time monitoring of the actual number, source and frequency of small files generated;

[0015] Determine the number of items that meet a single determination condition among the actual generation quantity, the actual generation source, and the actual generation frequency;

[0016] Determining whether the small file is abnormal according to the number of items;

[0017] The generation situation includes: the actual generation quantity, the actual generation source and the actual generation frequency.

[0018] Furthermore, the process of determining the number of items that meet a single determination condition among the actual generation quantity, the actual generation source, and the actual generation frequency includes:

[0019] Determining whether the actual generated quantity meets the single determination condition according to the actual generated quantity and the preset standard generated quantity;

[0020] According to the actual generation source: when the file system is used abnormally, it is determined that the actual generation source meets the single determination condition;

[0021] It is determined whether the actual generation frequency meets the single determination condition according to the actual generation frequency and the preset standard generation frequency.

[0022] Furthermore, the process of determining the number of items that meet a single determination condition among the actual generation quantity, the actual generation source, and the actual generation frequency also includes:

[0023] Based on the fact that the absolute value of the first difference is greater than the preset first evaluation value and the actual production quantity is greater than the standard production quantity, it is determined that the actual production quantity meets the single determination condition;

[0024] Based on the second difference absolute value being greater than a preset second evaluation value and the actual generation frequency being greater than the standard generation frequency, determining that the actual generation frequency meets the single determination condition;

[0025] The first absolute value of the difference is the absolute value of the difference between the actual generation quantity and the standard generation quantity; the second absolute value of the difference is the absolute value of the difference between the actual generation frequency and the standard generation frequency.

[0026] Furthermore, the process of determining whether the small file is abnormal according to the number of items includes:

[0027] When the number of items that meet the single determination condition among the actual generation quantity, the actual generation source, and the actual generation frequency is 0, it is determined that there is no abnormality in the small file;

[0028] When the number of items that meet the single determination condition among the actual generation quantity, the actual generation source, and the actual generation frequency is greater than or equal to 1, it is determined that an abnormality exists in the small file.

[0029] Furthermore, the process of determining the abnormality level of small files based on the presence of abnormal small files includes:

[0030] When the number of items that meet the single determination condition is 1, the abnormality level is determined to be level one;

[0031] When the number of items that meet the single determination condition is 2, the abnormality level is determined to be level 2;

[0032] When the number of items that meet the single determination condition is 3, the abnormality level is determined to be level three.

[0033] Furthermore, the process of determining the small file management strategy according to the abnormality level includes:

[0034] Based on the abnormal level being level one, determining the small file management strategy as: merging small files;

[0035] Based on the abnormal level being level 2, determining the small file management strategy as: merging small files and optimizing partition design;

[0036] Based on the abnormality level being level three, the small file management strategy is determined to be: merging small files, optimizing partition design, and adjusting data writing mode.

[0037] Furthermore, the process of merging small files includes:

[0038] Merging small files according to data processing requirements and the actual generation quantity and the actual generation frequency to obtain a number of merged files;

[0039] Comparing the merged file with the small files corresponding to the merged file to determine data integrity and format compatibility;

[0040] A combined score is calculated based on the data completeness and the format compatibility.

[0041] Further, the process of determining the governance score based on running the small file governance policy includes:

[0042] Run the optimized partition design and obtain the partition score;

[0043] Run the adjustment data writing mode to obtain the adjustment score;

[0044] Based on the different abnormality levels, the governance score is calculated according to the combined score, the partition score and the adjusted score.

[0045] Furthermore, the process of calculating the overall performance score based on the periodic cleaning result and the governance score includes:

[0046] Obtain the cleaning log records after regular cleaning, the difference in the number of small files after cleaning, and the amount of storage space released;

[0047] Calculate a cleaning score based on the cleaning log record, the difference in the number of small files after cleaning, and the amount of storage space released;

[0048] The overall performance score is calculated based on the cleanup score and the governance score.

[0049] Compared with the prior art, the beneficial effects of the present invention are that, through comprehensive management from regular cleaning to performance scoring to strategy optimization, the need for manual intervention is reduced and efficiency is improved; the generation of small files can be monitored in real time, and react according to abnormal situations, and problems can be quickly identified and responded to; the governance strategy can be determined according to the abnormality level, and various small file abnormalities can be effectively handled, thereby improving the stability and reliability of governance; the overall performance score is calculated by comprehensively combining the regular cleaning results and the governance score, and the cleaning cycle is adjusted or the governance strategy is optimized accordingly, thereby continuously optimizing the governance performance; the optimization of regular cleaning and exception handling strategies can effectively utilize resources, avoid the accumulation of small files and performance degradation, and ensure the efficient operation of the governance process.

[0050] In particular, by monitoring the actual number, source and frequency of small files generated, we can understand the generation of small files in detail, which is helpful for accurately identifying problems; we can count the number of eligible items according to preset judgment conditions, quickly and easily evaluate abnormal situations, and improve the efficiency of identifying abnormalities; determine whether there are abnormalities in small files based on the number of judged items, so as to quickly discover and locate problems, which is conducive to timely handling of abnormalities; taking the actual number, source and frequency as comprehensive considerations, it is helpful to analyze whether small files are normal from multiple angles, and improve the accuracy and comprehensiveness of detection; we can respond quickly and take necessary measures to prevent further expansion of abnormalities and ensure the stability and reliability of governance; we can efficiently identify and respond to abnormal situations of small files, and improve the effectiveness and responsiveness of the monitoring system.

[0051] In particular, by calculating the absolute value of the difference between the actual number and frequency of generation and the preset standard, the generation of small files can be accurately evaluated, including the deviation of the number and frequency, and anomalies can be effectively identified; by combining multiple factors such as the actual number of generation, the actual frequency of generation and the source of generation for evaluation, it is possible to comprehensively analyze whether there are anomalies in small files from multiple angles, thereby enhancing the comprehensiveness and accuracy of the evaluation; by setting a preset evaluation value, the judgment process can be standardized, making the anomaly judgment process more standardized and reliable, which helps to reduce misjudgments and improve processing efficiency; according to different absolute value judgment conditions of the difference, it is possible to quickly determine whether the actual number of generation, frequency and source meet expectations, so as to quickly respond and adjust the governance strategy, and improve real-time response capabilities; distinguishing between file system anomalies and other abnormal causes (such as insertion operation anomalies, partition design anomalies, etc.) helps to more accurately locate the root cause of the problem, take targeted measures to deal with it, and improve the efficiency and accuracy of problem solving; combining precise calculations, multi-dimensional evaluations and standardized processing procedures, it is possible to efficiently identify, classify and handle small file anomalies, thereby ensuring the stability and reliability of governance.

[0052] In particular, by determining the number of items that meet a single judgment condition, it is clear whether a small file is abnormal, making the judgment process simple, clear, easy to understand and execute; determining the abnormality level according to the number of abnormal items makes the abnormal processing hierarchical, and can adopt corresponding governance strategies according to different severities, thereby improving the pertinence and effectiveness of the processing; according to different abnormality levels, detailed governance strategies are formulated, such as merging small files, optimizing partition design, and adjusting data writing mode, so that the governance process has a clear direction and steps, and improves the processing efficiency; from simple merging of small files to optimizing partition design and adjusting data writing mode, it covers all aspects from file management to system design and operation mode, providing a comprehensive solution, which helps to solve the problem fundamentally; according to different abnormality levels, different governance strategies are adopted, which can flexibly respond to various abnormal situations, avoid a one-size-fits-all approach, and increase the adaptability and flexibility of governance; by adding governance measures step by step, the governance performance can be gradually optimized without having to make large-scale adjustments at one time, and small file anomalies can be efficiently and flexibly identified and processed, improving the stability and performance of governance.

[0053] In particular, by combining data processing requirements and actual access frequency to merge small files, it is ensured that the merging process meets system requirements and improves the actual effect of the merger; by comparing the merged file and the original small file, the integrity of the data and the compatibility of the format are ensured, data loss or format incompatibility problems are avoided, and the reliability of governance is enhanced; by calculating the product of data integrity and format compatibility, an objective merging scoring indicator is provided, which is convenient for evaluating the merging effect and provides a basis for further optimization; setting the merging threshold according to data processing requirements, such as merging when the file size reaches 10MB or the access frequency reaches 50 times / min, the merging strategy can be dynamically adjusted according to actual conditions to improve the flexibility and adaptability of governance; by merging small files, file fragmentation is reduced, file access speed is increased, and governance performance is optimized; data integrity inspection and verification ensure that all data is correctly copied and stored, avoiding data loss or damage and improving data security; format compatibility inspection ensures that the merged file can be correctly identified and used by the target system or application, avoiding failures caused by format incompatibility; ensuring the effectiveness, reliability and flexibility of the merging process helps to improve the overall performance and stability of governance.

[0054] In particular, by evaluating the operating effects of partition design and data writing mode, partition scores and adjustment scores were obtained, so that the governance score has quantifiable indicators, which is convenient for evaluating the governance effect; the merge score, partition score and adjustment score are combined to calculate the governance score, and the effects of different governance measures are comprehensively considered, making the evaluation more comprehensive and objective; according to different anomaly levels, the merge effect, partition effect and data writing mode effect are comprehensively considered, and targeted governance strategies are formulated, which is conducive to solving anomaly problems at different levels; by monitoring indicators such as post-partition query speed, number of small files, data writing efficiency and data consistency, the operation of governance can be understood in real time, and the governance strategy can be adjusted in time, which improves the real-time and flexibility of governance; by calculating the governance score, the effect of governance measures can be intuitively reflected, which is convenient for management personnel to monitor and adjust, and improves the efficiency and accuracy of management.

[0055] In particular, by obtaining the cleaning log records after regular cleaning, the difference in the number of small files after cleaning and the amount of storage space released, combined with the cleaning score, and then calculating the overall performance score based on the governance score and the cleaning score, a comprehensive evaluation of the overall governance performance is achieved; the cleaning score takes into account multiple indicators such as the cleaning log records, the difference in the number of small files after cleaning and the amount of storage space released, and can comprehensively evaluate the effect of regular cleaning to ensure that the storage space is effectively released. The overall performance score comprehensively considers the governance score and the cleaning score, reflecting the comprehensive contribution of governance measures and regular cleaning to governance performance, and providing a comprehensive indicator for governance performance improvement; the calculation of the cleaning score and the overall performance score enables the cleaning effect and the governance effect to be quantified and visualized. The process of regularly executing the overall performance score helps to periodically evaluate the improvement of governance, provide data support and decision-making basis for continuous improvement, and enhance the stability and efficiency of governance.

[0056] In particular, the initial cleaning cycle is dynamically adjusted or the governance strategy is optimized through the difference between the overall performance score and the preset standard performance score, so that it can be continuously optimized and adjusted according to the actual situation to improve the governance effect. The third difference absolute value is used to quantify the gap in governance performance, which provides a clear quantitative standard to facilitate the judgment of whether the governance performance is qualified, thereby enhancing the objectivity and scientific nature of the evaluation; it can timely determine whether the governance strategy is effective, so as to maintain the existing strategy; when the score is unqualified, it can accurately determine the need to adjust the cleaning cycle or optimize the governance strategy to improve the accuracy and effectiveness of governance; it can flexibly adjust the cleaning cycle or optimize the governance strategy to better respond to changes in governance performance and maintain efficient operation of governance; through periodic performance evaluation and dynamic adjustment of strategies, a continuous improvement mechanism is formed to ensure that performance can be continuously optimized and improved to maintain long-term stability and efficiency; it can preventively maintain the current governance strategy when the performance is qualified, and can also correct it when the performance is unqualified. The dual mechanism guarantees the governance effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a flow chart of a Hive small file management method according to an embodiment of the present invention;

[0058] Figure 2 This is a flowchart for determining whether there is an abnormality in the Hive small file management method in an embodiment of the present invention;

[0059] Figure 3 This is a flow chart of determining the abnormality level in the Hive small file management method according to an embodiment of the present invention;

[0060] Figure 4 The present invention is a flowchart of merging small files in the Hive small file management method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0062] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0063] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0064] See also Figure 1 As shown, this embodiment provides a Hive small file management method, including:

[0065] Step S100, setting an initial cleaning cycle, and regularly cleaning the generated small files to obtain regular cleaning results;

[0066] Step S200, monitoring the generation of small files in real time, and determining whether there is an abnormality in the small files according to the generation;

[0067] Step S300, determining the abnormality level of small files based on the abnormality of small files;

[0068] Step S400, determining a small file governance strategy according to the abnormality level, and determining a governance score based on running the small file governance strategy;

[0069] Step S500, calculating an overall performance score based on the periodic cleaning result and the governance score;

[0070] Step S600, adjusting the initial cleaning cycle based on the overall performance score, or optimizing the governance strategy.

[0071] Specifically, the embodiments of the present invention reduce the need for manual intervention and improve efficiency through comprehensive management from regular cleaning to performance scoring to policy optimization; can monitor the generation of small files in real time, react according to abnormal situations, and can quickly identify and respond to problems; can determine governance strategies based on abnormal levels, effectively handle various small file abnormalities, and improve the stability and reliability of governance; calculate the overall performance score by combining regular cleaning results and governance scores, and adjust the cleaning cycle or optimize the governance strategy accordingly, thereby continuously optimizing governance performance; regular cleaning and optimization of exception handling strategies can effectively utilize resources, avoid small file accumulation and performance degradation, and ensure efficient operation of the governance process.

[0072] Specifically, see Figure 2 As shown, it is a flowchart for determining whether there is an abnormality; the process of real-time monitoring of the generation of small files and determining whether there is an abnormality in the small files according to the generation situation includes:

[0073] Step S211, real-time monitoring of the actual number, actual source and actual frequency of small files generated;

[0074] Step S212, determining the number of items that meet a single determination condition among the actual generation quantity, the actual generation source, and the actual generation frequency;

[0075] Step S213, determining whether the small file is abnormal according to the number of items;

[0076] The generation situation includes: the actual generation quantity, the actual generation source and the actual generation frequency.

[0077] Specifically, by monitoring the actual number, source and frequency of small files generated, the embodiments of the present invention can understand the generation of small files in detail, which is helpful for accurately identifying problems; it can count the number of items that meet the conditions according to preset judgment conditions, and can quickly and easily evaluate abnormal situations, thereby improving the efficiency of identifying abnormalities; it can determine whether there are abnormalities in small files based on the number of judged items, so as to quickly discover and locate problems, which is conducive to timely handling of abnormalities; taking the actual number, source and frequency as comprehensive considerations, it is helpful to analyze whether small files are normal from multiple angles, thereby improving the accuracy and comprehensiveness of detection; it can respond quickly and take necessary measures to prevent further expansion of abnormalities, and ensure the stability and reliability of governance; it can efficiently identify and respond to abnormal situations of small files, and improve the effectiveness and responsiveness of the monitoring system.

[0078] Specifically, the process of determining the number of items that meet a single determination condition among the actual generation quantity, the actual generation source, and the actual generation frequency in this embodiment includes:

[0079] Calculate a first difference absolute value according to the actual generated quantity and the preset standard generated quantity;

[0080] Wherein, the absolute value of the first difference = |actual production quantity - standard production quantity|;

[0081] Calculate a second difference absolute value according to the actual generation frequency and the preset standard generation frequency;

[0082] Wherein, the absolute value of the second difference = |actual generation frequency - standard generation frequency|;

[0083] The actual generation sources include: abnormal use of the file system and other abnormal reasons;

[0084] In this embodiment, the other abnormal reasons include: insertion operation abnormality, partition design abnormality, and data update and deletion operation abnormality;

[0085] When the absolute value of the first difference is less than or equal to a preset first evaluation value, it is determined that the actual generated quantity does not meet the single determination condition;

[0086] When the absolute value of the first difference is greater than the first evaluation value and the actual production quantity is less than the standard production quantity, it is determined that the actual production quantity does not meet the single determination condition;

[0087] When the absolute value of the first difference is greater than the first evaluation value and the actual production quantity is greater than the standard production quantity, it is determined that the actual production quantity meets the single determination condition;

[0088] When the absolute value of the second difference is less than or equal to a preset second evaluation value, it is determined that the actual generation frequency does not meet the single determination condition;

[0089] When the absolute value of the second difference is greater than the second evaluation value and the actual generation frequency is less than the standard generation frequency, it is determined that the actual generation frequency does not meet the single determination condition;

[0090] When the absolute value of the second difference is greater than the second evaluation value and the actual generation frequency is greater than the standard generation frequency, it is determined that the actual generation frequency meets the single determination condition;

[0091] When the actual generation source is: when the file system is used abnormally, determining that the actual generation source meets the single determination condition;

[0092] When the actual generation source is: other abnormal reasons, it is determined that the actual generation source does not meet the single determination condition;

[0093] In the specific implementation process, the number of standards generated is set to 300, the standard generation frequency is set to 20 / s; the first evaluation value is set to 50; the second evaluation value is set to 5; the specific settings are also affected by the specific usage scenario and hardware resources.

[0094] Specifically, the embodiment of the present invention can accurately evaluate the situation of small file generation, including the deviation of quantity and frequency, and effectively identify abnormal situations by calculating the absolute value of the difference between the actual generation quantity and frequency and the preset standard; the evaluation is combined with multiple factors such as the actual generation quantity, the actual generation frequency and the generation source, and it can comprehensively analyze whether the small file is abnormal from multiple angles, thereby enhancing the comprehensiveness and accuracy of the evaluation; by setting a preset evaluation value, the judgment process can be standardized, making the abnormal judgment process more standardized and reliable, which helps to reduce misjudgment and improve processing efficiency; according to different difference absolute value judgment conditions, it can quickly determine whether the actual generation quantity, frequency and source meet expectations, so as to quickly respond and adjust the governance strategy, and improve the real-time response capability; distinguishing between file system abnormalities and other abnormal causes (such as insertion operation abnormalities, partition design abnormalities, etc.) helps to more accurately locate the root cause of the problem, take targeted measures to deal with it, and improve the efficiency and accuracy of problem solving; combined with precise calculation, multi-dimensional evaluation and standardized processing flow, it can efficiently identify, classify and process small file abnormalities, thereby ensuring the stability and reliability of governance.

[0095] Specifically, the process of determining whether the small file is abnormal according to the number of items in this embodiment includes:

[0096] When the number of items that meet the single determination condition among the actual generation quantity, the actual generation source, and the actual generation frequency is 0, it is determined that there is no abnormality in the small file;

[0097] When the number of items that meet the single determination condition among the actual generation quantity, the actual generation source, and the actual generation frequency is greater than or equal to 1, it is determined that the small file is abnormal;

[0098] See also Figure 3 As shown, it is a flow chart for determining the abnormality level. The process of determining the abnormality level of small files based on the existence of abnormalities in small files includes:

[0099] Step S311, determining whether the number of items that meet the single determination condition is 1, if yes, determining that the abnormality level is level 1, if no, jumping to step S312;

[0100] Step S312, determining whether the number of items that meet the single determination condition is 2, if yes, determining that the abnormality level is level 2, if no, jumping to step S313;

[0101] Step S313, determining that the abnormality level is level three.

[0102] Specifically, the process of determining the small file management strategy according to the abnormality level includes:

[0103] Based on the abnormal level being level one, determining the small file management strategy as: merging small files;

[0104] Based on the abnormal level being level 2, determining the small file management strategy as: merging small files and optimizing partition design;

[0105] Based on the abnormality level being level three, the small file management strategy is determined to be: merging small files, optimizing partition design, and adjusting data writing mode.

[0106] Specifically, the embodiment of the present invention determines the number of items that meet a single judgment condition to clearly determine whether a small file is abnormal, so that the judgment process is simple, clear, easy to understand and execute; the abnormal level is determined according to the number of abnormal items, so that the abnormal processing is hierarchical, and corresponding governance strategies can be adopted according to different severities, thereby improving the pertinence and effectiveness of the processing; according to different abnormal levels, detailed governance strategies are formulated, such as merging small files, optimizing partition design, and adjusting data writing mode, so that the governance process has a clear direction and steps, and improves the processing efficiency; from simple merging of small files to optimizing partition design and adjusting data writing mode, it covers all aspects from file management to system design and operation mode, provides a comprehensive solution, and helps to fundamentally solve the problem; according to different abnormal levels, different governance strategies are adopted, which can flexibly respond to various abnormal situations, avoid a one-size-fits-all processing method, and increase the adaptability and flexibility of governance; by adding governance measures step by step, the governance performance can be gradually optimized without having to make large-scale adjustments at one time, and small file anomalies can be efficiently and flexibly identified and processed, improving the stability and performance of governance.

[0107] Specifically, see Figure 4 As shown, it is a flowchart of merging small files. The process of merging small files in this embodiment includes:

[0108] Step S411, merging small files according to data processing requirements and actual access frequencies to obtain a number of merged files;

[0109] Step S412, comparing the merged file with the small files corresponding to the merged file to determine data integrity and format compatibility;

[0110] Step S413: Calculate a combined score according to the data completeness and the format compatibility.

[0111] Specifically, in the specific implementation process, according to the data processing requirements, for example, when the file size of the small file reaches a specific threshold, the specific threshold in this implementation is 10MB, the small files are merged;

[0112] When the actual access frequency reaches 50 times / min, small files are merged to increase the access speed.

[0113] The data integrity is to ensure that all data is correctly copied and verified to avoid data loss or damage; the format compatibility is to ensure that the merged file format is compatible with the target system or application;

[0114] The combined score = (data integrity × integrity weight) + (format compatibility × compatibility weight);

[0115] In this implementation, the integrity weight is set to 0.7, and the compatibility weight is set to 0.6. The specific settings are also affected by the specific usage scenario and hardware resources.

[0116] Specifically, the embodiment of the present invention combines small files with data processing requirements and actual access frequency to ensure that the merging process meets system requirements and improves the actual effect of the merging; by comparing the merged file and the original small file, the integrity of the data and the compatibility of the format are ensured, the problem of data loss or format incompatibility is avoided, and the reliability of governance is enhanced; by calculating the product of data integrity and format compatibility, an objective merging scoring indicator is provided, which is convenient for evaluating the merging effect and provides a basis for further optimization; the merging threshold is set according to the data processing requirements, for example, merging is performed when the file size reaches 10MB or the access frequency reaches 50 times / min, and the merging strategy can be dynamically adjusted according to the actual situation to improve the flexibility and adaptability of governance; by merging small files, file fragmentation is reduced, file access speed is improved, and governance performance is optimized; the data integrity check and verification ensure that all data is correctly copied and stored, avoid data loss or damage, and improve data security; the format compatibility check ensures that the merged file can be correctly identified and used by the target system or application, avoiding failures caused by format incompatibility; the effectiveness, reliability and flexibility of the merging process are ensured, which helps to improve the overall performance and stability of governance.

[0117] Specifically, the process of determining the governance score based on running the small file governance policy in this embodiment includes:

[0118] Step S421, running the optimized partition design and obtaining the partition score;

[0119] Step S422, running the adjustment data writing mode to obtain the adjustment score;

[0120] Step S423, based on the different abnormality levels, the governance score is calculated according to the combined score, the partition score and the adjusted score.

[0121] Specifically, the optimized partition design is run, based on the difference in query speed after reading the partition and the number of small files after partitioning after running the governance measure;

[0122] Calculate the partition score according to the query speed after the partition and the difference between the number of small files after the partition;

[0123] The partition score = query speed after partitioning × the difference in the number of small files after partitioning;

[0124] The difference in the number of small files after partitioning is the difference in the number of small files before and after partitioning;

[0125] Run the adjusted data writing mode, and read the adjusted data writing efficiency and adjusted data consistency after running the governance measure;

[0126] Calculating the adjustment score according to the data writing efficiency and the adjusted data consistency;

[0127] The adjustment score = data writing efficiency × adjusted data consistency;

[0128] The governance score = combined score + partition score + adjustment score.

[0129] Specifically, the embodiment of the present invention evaluates the operating effects of the partition design and data writing mode to obtain a partition score and an adjustment score, so that the governance score has a quantifiable indicator, which is convenient for evaluating the governance effect; the merge score, partition score and adjustment score are combined to calculate the governance score, and the effects of different governance measures are comprehensively considered, making the evaluation more comprehensive and objective; according to different abnormality levels, the merge effect, partition effect and data writing mode effect are comprehensively considered, and targeted governance strategies are formulated, which is conducive to solving abnormal problems at different levels; by monitoring indicators such as post-partition query speed, number of small files, data writing efficiency and data consistency, the operation status of governance can be understood in real time, the governance strategy can be adjusted in time, and the real-time and flexibility of governance can be improved; by calculating the governance score, the effect of the governance measures can be intuitively reflected, which is convenient for management personnel to monitor and adjust, and the efficiency and accuracy of management are improved.

[0130] Specifically, the process of calculating the overall performance score according to the periodic cleaning result combined with the governance score in this embodiment includes:

[0131] Step S511, obtaining the cleaning log record after regular cleaning, the difference in the number of small files after cleaning, and the amount of storage space released;

[0132] Step S512, calculating a cleaning score according to the cleaning log record, the difference in the number of small files after cleaning, and the amount of storage space released;

[0133] Step S513: Calculate the overall performance score according to the cleaning score and the governance score.

[0134] Wherein, the cleaning score = cleaning log record × difference in the number of small files after cleaning × amount of storage space released; the difference in the number of small files after cleaning is the difference in the number of small files before and after cleaning;

[0135] The overall performance score = cleaning score + governance score.

[0136] Specifically, the embodiment of the present invention achieves a comprehensive evaluation of the overall governance performance by obtaining the cleaning log records after regular cleaning, the difference in the number of small files after cleaning, and the amount of storage space released, combining the cleaning score, and then calculating the overall performance score based on the governance score and the cleaning score; the cleaning score takes into account multiple indicators such as the cleaning log records, the difference in the number of small files after cleaning, and the amount of storage space released, and can comprehensively evaluate the effect of regular cleaning to ensure that the storage space is effectively released. The overall performance score comprehensively considers the governance score and the cleaning score, reflects the comprehensive contribution of governance measures and regular cleaning to governance performance, and provides a comprehensive indicator for governance performance improvement; the calculation of the cleaning score and the overall performance score enables the cleaning effect and the governance effect to be quantified and visually expressed, and the process of regularly executing the overall performance score helps to periodically evaluate the improvement of governance, provide data support and decision-making basis for continuous improvement, and enhance the stability and efficiency of governance.

[0137] Specifically, the process of adjusting the initial cleaning cycle based on the overall performance score, or optimizing the governance strategy in this embodiment includes:

[0138] Step S611, calculating a third difference absolute value according to the overall performance score and a preset standard performance score;

[0139] Step S612, determining to adjust the initial cleaning cycle or optimize the governance strategy according to the absolute value of the third difference combined with a preset third evaluation value.

[0140] Wherein, the absolute value of the third difference=|overall performance score-preset standard performance score|;

[0141] When the absolute value of the third difference is greater than the third evaluation value and the overall performance score is greater than the standard performance score, it is determined that the overall performance of small file management is qualified;

[0142] When the absolute value of the third difference is less than or equal to the third evaluation value, it is determined that the overall performance of small file management is unqualified, and the initial cleaning cycle is adjusted;

[0143] When the absolute value of the third difference is greater than the third evaluation value and the overall performance score is less than the standard performance score, it is determined that the overall performance of small file management is unqualified, and the management strategy is optimized.

[0144] In the specific implementation process, the specific settings of the standard performance score and the third evaluation value are also affected by the specific usage scenario and hardware resources.

[0145] Specifically, the embodiment of the present invention dynamically adjusts the initial cleaning cycle or optimizes the governance strategy through the difference between the overall performance score and the preset standard performance score, so that it can be continuously optimized and adjusted according to the actual situation to improve the governance effect, and uses the absolute value of the third difference to quantify the gap in governance performance, providing a clear quantitative standard, which is convenient for determining whether the governance performance is qualified, and enhancing the objectivity and scientificity of the evaluation; it can timely determine whether the governance strategy is effective, so as to maintain the existing strategy; when the score is unqualified, it can accurately determine the need to adjust the cleaning cycle or optimize the governance strategy, so as to improve the accuracy and effectiveness of governance; it can flexibly adjust the cleaning cycle or optimize the governance strategy, so as to better respond to changes in governance performance and maintain efficient operation of governance; through periodic performance evaluation and dynamic adjustment of strategies, a continuous improvement mechanism is formed to ensure that performance can be continuously optimized and improved, and long-term stability and efficiency can be maintained; it can preventively maintain the current governance strategy when the performance is qualified, and can also correct it when the performance is unqualified, and the dual mechanism guarantees the governance effect.

[0146] The calculation compensation parameters and the calculation adjustment parameters described in the present invention have two functions: one is to balance the left and right dimensions of the formula, and the other is to adjust the numerical results. No specific assignment is performed in this embodiment. In addition, the calculation formulas in this embodiment are used to intuitively reflect the adjustment relationship between the numerical values, such as positive correlation and negative correlation. Unless otherwise specified, the parameter values ​​that are not specifically limited to numerical values ​​are all positive.

[0147] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0148] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A Hive small file management method, characterized in that: include: The generated Hive small files are stored in the database to be cleaned, an initial cleaning cycle is set, and the database to be cleaned is cleaned in the initial cleaning cycle to obtain a regular cleaning result; Monitor the generation of Hive small files in real time, and determine whether there are any abnormalities in the Hive small files based on the generation; Based on the abnormality of Hive small files, determine the abnormality level of Hive small files; Determine a Hive small file management strategy according to the abnormality level, and determine a management score based on running the Hive small file management strategy; Calculate an overall performance score based on the periodic cleaning results combined with the governance score; Determine whether the management of the Hive small files is qualified based on the overall performance score combined with a preset standard performance score, and adjust the initial cleanup cycle or optimize the management strategy when the management is determined to be unqualified; The process of real-time monitoring the generation of Hive small files and determining whether the Hive small files are abnormal according to the generation includes: Real-time monitoring of the actual number, source, and frequency of small Hive files. Determine the number of items that meet a single determination condition among the actual generation quantity, the actual generation source, and the actual generation frequency; Determine whether the Hive small file is abnormal according to the number of items; The generation situation includes the actual generation quantity, the actual generation source and the actual generation frequency; the actual generation source includes abnormal use of the file system and other abnormal reasons.

2. The Hive small file management method according to claim 1 is characterized in that: The process of determining the number of items that meet a single determination condition among the actual generation quantity, the actual generation source, and the actual generation frequency includes: Determining whether the actual generated quantity meets the single determination condition according to the actual generated quantity and the preset standard generated quantity; When the actual generation source is the abnormal use of the file system, determining that the actual generation source meets the single determination condition; It is determined whether the actual generation frequency meets the single determination condition according to the actual generation frequency and the preset standard generation frequency.

3. The Hive small file management method according to claim 2 is characterized in that: The process of determining the number of items that meet a single determination condition among the actual generation quantity, the actual generation source, and the actual generation frequency also includes: Based on the fact that the absolute value of the first difference is greater than the preset first evaluation value and the actual production quantity is greater than the standard production quantity, it is determined that the actual production quantity meets the single determination condition; Based on the second difference absolute value being greater than a preset second evaluation value and the actual generation frequency being greater than the standard generation frequency, determining that the actual generation frequency meets the single determination condition; Among them, the first absolute value of the difference is the absolute value of the difference between the actual generation quantity and the standard generation quantity; the second absolute value of the difference is the absolute value of the difference between the actual generation frequency and the standard generation frequency; the setting of the first evaluation value and the second evaluation value is affected by the specific usage scenario and hardware resources.

4. The Hive small file management method according to claim 3 is characterized in that: The process of determining whether the Hive small file is abnormal according to the number of items includes: When the number of items that meet the single determination condition among the actual generation quantity, the actual generation source, and the actual generation frequency is 0, it is determined that there is no abnormality in the Hive small file; When the number of items that meet the single determination condition among the actual generation quantity, the actual generation source, and the actual generation frequency is greater than or equal to 1, it is determined that an abnormality exists in the Hive small file.

5. The Hive small file management method according to claim 4 is characterized in that: The process of determining the abnormality level of Hive small files based on the abnormality of Hive small files includes: When the number of items that meet the single determination condition is 1, the abnormality level is determined to be level one; When the number of items that meet the single determination condition is 2, the abnormality level is determined to be level 2; When the number of items that meet the single determination condition is 3, the abnormality level is determined to be level three.

6. The Hive small file management method according to claim 5 is characterized in that: The process of determining the Hive small file management strategy according to the abnormal level includes: Based on the abnormal level being level one, determining the Hive small file management strategy to be merging Hive small files; Based on the abnormal level being level 2, determining the Hive small file management strategy to merge Hive small files and optimize partition design; Based on the abnormal level being level three, the Hive small file management strategy is determined to be merging Hive small files, optimizing partition design, and adjusting data writing mode; The optimized partition design includes optimizing the partition fields and optimizing the partition quantity.

7. The Hive small file management method according to claim 6 is characterized in that: The process of merging Hive small files includes: Merge small Hive files based on data processing requirements and actual access frequency to obtain merged files; Compare the merged file with the Hive small files corresponding to the merged file to determine data integrity and format compatibility; A combined score is calculated based on the data completeness and the format compatibility.

8. The Hive small file management method according to claim 7 is characterized in that: The process of determining the governance score based on running the Hive small file governance policy includes: Run the optimized partition design and obtain the partition score; Run the adjustment data writing mode to obtain the adjustment score; Based on the different abnormality levels, the governance score is calculated according to the combined score, the partition score and the adjusted score.

9. The Hive small file management method according to claim 8 is characterized in that: The process of calculating the overall performance score based on the periodic cleaning result and the governance score includes: Obtain the cleanup log records after regular cleanup, the difference in the number of Hive small files after cleanup, and the amount of storage space released; Calculate the cleaning score according to the cleaning log record, the difference in the number of Hive small files after cleaning, and the amount of storage space released; The overall performance score is calculated based on the cleanup score and the governance score.

Images