Data storage optimization method and system

Abstract

The present invention discloses a method and system for data storage optimization, which relates to the technical field of data storage and includes the following steps: analyzing data, understanding the characteristics of data, selecting a compression algorithm suitable for the characteristics of data, and compressing data; obtaining multiple data information during data compression, including computing resource allocation information and compression performance information during compression, and after obtaining, processing the computing resource allocation information and compression performance information. The present invention introduces a real-time monitoring and analysis process, obtains multiple data information during data compression, generates a compression risk coefficient, and then compares it with a preset threshold to determine whether there is an abnormality in the data compression process. This comprehensive analysis and evaluation method improves the overall control of the data compression process, not only focusing on performance information, but also considering the allocation of computing resources, which is more conducive to accurately evaluating the effectiveness and potential risks of compression.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a method and system for optimizing data storage. Background Art

[0002] Data storage optimization refers to the use of a series of effective strategies and technologies to improve the performance, efficiency and reliability of data storage systems. This process aims to minimize resource usage, including storage space, computing resources and network bandwidth, while ensuring fast and secure access to data. The goals of data storage optimization include improving query speed, reducing storage costs, increasing system scalability, and ensuring data integrity and reliability.

[0003] Common data storage optimization goals and strategies include the following: Improve data read and write performance through reasonable data model design, use of indexes, caching technology, etc. This includes reducing latency when accessing data and improving system response speed. Minimize the space required for data storage and reduce storage costs through compression technology, partitioning, denormalization, etc. This is especially important for large-scale data sets and cloud storage environments. In order to adapt to the growing amount of data, the system should have good scalability. Distributed storage, sharding, partitioning and other technologies can help the system effectively handle large-scale data.

[0004] Data storage optimization aims to establish an efficient, reliable, secure and cost-effective data management system to meet the diverse needs of data storage in different application scenarios.

[0005] When optimizing data storage, using data compression technology can reduce storage space requirements and increase data transmission speed to a certain extent. When compressing data, if there are anomalies in the data compression process but there is no obvious manifestation, then the anomalies in the compression process will continue to exist. When this happens, it may cause data to be damaged or lost during the compression process, affecting the integrity of the data. Damaged data may not be restored correctly, resulting in information loss or data inconsistency, thereby affecting business decisions and the accuracy of data analysis. Secondly, potential anomalies may gradually accumulate in the system, causing system instability. Unhandled anomalies may cause system crashes or large amounts of data loss after a period of time, which will have a serious negative impact on the overall system stability.

[0006] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not constitute the prior art that is already known to one of ordinary skill in the art. Summary of the invention

[0007] The purpose of the present invention is to provide a method and system for data storage optimization, by introducing a real-time monitoring and analysis process, by obtaining multiple data information during data compression, including computing resource allocation information and compression performance information, wherein the CPU clock frequency information and the real-time data backlog information are processed to generate a CPU clock frequency anomaly concealment index and a data backlog index, so that the system can perceive the abnormal hidden dangers in the data compression process in real time, avoiding the situation where the abnormality lurks for a long time without being detected, and helping to take measures in time to maintain the integrity of the data and the stability of the system. By comprehensively analyzing the computing resource allocation information and compression performance information during data compression, a compression hidden danger coefficient is generated, and then it is compared with a preset threshold to determine whether there is an abnormality in the data compression process. This comprehensive analysis and evaluation method improves the overall control of the data compression process, not only focusing on performance information, but also taking into account the allocation of computing resources, and is more helpful to accurately evaluate the effectiveness and potential risks of compression to solve the problems in the above-mentioned background technology.

[0008] In order to achieve the above object, the present invention provides the following technical solution: a method for optimizing data storage, comprising the following steps:

[0009] Analyze the data, understand the characteristics of the data, select a compression algorithm suitable for the data characteristics, and compress the data;

[0010] Acquire multiple data information during data compression, including computing resource allocation information and compression performance information during compression, and after acquisition, process the computing resource allocation information and compression performance information;

[0011] Comprehensively analyze the processed computing resource allocation information and compression performance information during data compression to generate a compression risk coefficient, and evaluate data compression through the compression risk coefficient;

[0012] Compare and analyze the compression hidden danger coefficient generated during data compression with the pre-set compression hidden danger coefficient reference threshold to determine whether there are abnormal hidden dangers during data compression;

[0013] When there are abnormal hidden dangers during data compression, an analysis set is established with the compression hidden danger coefficient corresponding to the starting point and several compression hidden danger coefficients generated subsequently. The compression hidden danger coefficients in the analysis set are comprehensively analyzed with the pre-set compression hidden danger coefficient reference threshold to generate a hidden danger risk index. The data compression process is further judged by the hidden danger risk index. When there is a high risk during data compression, compression is stopped. When the data is compressed at a low risk, compression continues.

[0014] Preferably, when compressing data, the computing resource allocation information during data compression includes CPU clock frequency information, and after the CPU clock frequency information is obtained, the CPU clock frequency information is processed to generate a CPU clock frequency anomaly concealment index; when compressing data, the compression performance information during data compression includes real-time data backlog information, and after the real-time data backlog information is obtained, the real-time data backlog information is processed to generate a data backlog index.

[0015] Preferably, the logic of obtaining the CPU clock frequency anomaly concealment index is as follows:

[0016] Get the real-time CPU clock frequency within T time during data compression, and mark the real-time CPU clock frequency as Clock CPU x , x represents the number of the real-time CPU clock frequency during the data compression within the time T, x = 1, 2, 3, 4, ..., p, p is a positive integer;

[0017] The CPU clock frequency standard deviation and the CPU clock frequency expected deviation are calculated by using the real-time CPU clock frequency obtained within T time during data compression. The calculation formula is: In the formula, Clock CPU 标 Indicates the standard deviation of CPU clock frequency, Clock CPU 平 It represents the average value of the real-time CPU clock frequency obtained within T time during data compression. In the formula, Clock CPU 偏差 Indicates the expected deviation of CPU clock frequency, Clock CPU 预期 Indicates the expected CPU clock frequency, Clock CPU x' represents the real-time CPU clock frequency obtained within T time and lower than the expected CPU clock frequency during data compression, x' represents the number of the real-time CPU clock frequency obtained within T time and lower than the expected CPU clock frequency during data compression, x'=1, 2, 3, 4, ..., p', p' is a positive integer, and p'≤p;

[0018] Calculate the CPU clock frequency anomaly concealment index. The calculation expression is: Clock CPU =μ1*Clock CPU 标 +μ2*Clock CPU 偏差 , where Clock CPU represents the CPU clock frequency anomaly concealment index, μ1 and μ2 are the CPU clock frequency standard deviation ClockCPU 标 Expected deviation from CPU clock frequency Clock CPU 偏差 The weight factor of , all values ​​are greater than 0.

[0019] Preferably, the logic for obtaining the data backlog index is as follows:

[0020] Get the real-time data backlog rate within T time when data is compressed, form a time series, and use the function Back to convert the real-time data backlog rate into a time series. Rate (t) to make an indication;

[0021] The data backlog index is calculated by the real-time data backlog rate obtained within time T. The calculation expression is: In the formula, Back Rate represents the data backlog index, [t x , t y ] indicates the period of time during which data compression has been completed, [t m , t n ] represents the period of waiting for data compression during data compression, t x <t y , t m <t n .

[0022] Preferably, the CPU clock frequency anomaly concealment index Clock after processing during data compression is obtained CPU and Data Backlog Index Rate Then, the CPU clock frequency anomaly concealment index Clock CPU and Data Backlog Index Rate Perform formula analysis to generate compression risk coefficient YS ττ , based on the formula:

[0023]

[0024] , where f1 and f2 are the CPU clock frequency anomaly concealment index Clock CPU and Data Backlog Index Rate The preset proportional coefficient, and f1 and f2 are both greater than 0.

[0025] Preferably, the compression hidden danger coefficient generated during data compression is compared and analyzed with a preset compression hidden danger coefficient reference threshold, and the result of the comparison and analysis is as follows:

[0026] If the compression hidden danger coefficient is greater than or equal to the compression hidden danger coefficient reference threshold, an abnormal signal is generated. When an abnormal signal is generated during data compression, the data compression process is further analyzed;

[0027] If the compression hidden danger coefficient is less than the compression hidden danger coefficient reference threshold, a normal signal is generated.

[0028] Preferably, when there is an abnormal hidden danger during data compression, the compression hidden danger coefficient corresponding to the starting point and several compression hidden danger coefficients generated subsequently are used to establish an analysis set, and the analysis set is calibrated as F, then F = {YS ττ v}={YS ττ 1 , YS ττ 2 ,…,YS ττ s}, v = 1, 2, 3, 4, ..., s, s is a positive integer;

[0029] The compression hidden danger coefficient in the analysis set is comprehensively analyzed with the pre-set compression hidden danger coefficient reference threshold to generate the hidden danger risk index, based on the formula: In the formula, ξξ represents the hidden danger risk index, YS ττ v' represents the compression hidden danger coefficient greater than the compression hidden danger coefficient reference threshold, v' represents the number of the compression hidden danger coefficient greater than the compression hidden danger coefficient reference threshold, v'=1, 2, 3, 4, ..., s', s' is a positive integer, s represents the total number of compression hidden danger coefficients in the analysis set.

[0030] Preferably, the hidden danger risk index generated by the compression hidden danger coefficient greater than the compression hidden danger coefficient reference threshold and the compression hidden danger coefficient reference threshold in the analysis set is compared with the preset hidden danger risk index reference threshold, and the result of the comparison analysis is as follows:

[0031] If the hidden danger risk index is greater than or equal to the hidden danger risk index reference threshold, a high risk signal is generated. When a high risk signal is generated during data compression, compression is stopped;

[0032] If the compression hidden danger coefficient is less than the compression hidden danger coefficient reference threshold, a low risk signal is generated. When a low risk signal is generated during data compression, the data continues to be compressed.

[0033] A data storage optimization system includes a data characteristic analysis and compression algorithm selection module, a data information acquisition and processing module, a comprehensive analysis and compression hidden danger assessment module, a compression hidden danger comparison analysis module, and a risk index generation and compression processing module;

[0034] The data characteristic analysis and compression algorithm selection module analyzes the data, understands the characteristics of the data, selects a compression algorithm suitable for the data characteristics, and compresses the data;

[0035] A data information acquisition and processing module acquires multiple data information during data compression, including computing resource allocation information and compression performance information during compression, and processes the computing resource allocation information and compression performance information after acquisition;

[0036] The comprehensive analysis and compression risk assessment module comprehensively analyzes the processed computing resource allocation information and compression performance information during data compression, generates a compression risk coefficient, and evaluates data compression through the compression risk coefficient;

[0037] The compression hidden danger comparison and analysis module compares and analyzes the compression hidden danger coefficient generated during data compression with the pre-set compression hidden danger coefficient reference threshold to determine whether there are abnormal hidden dangers during data compression;

[0038] The risk index generation and compression processing module, when there are abnormal hidden dangers during data compression, establishes an analysis set with the compression hidden danger coefficient corresponding to the starting point and several compression hidden danger coefficients generated subsequently, conducts a comprehensive analysis of the compression hidden danger coefficients in the analysis set and the pre-set compression hidden danger coefficient reference threshold, generates a hidden danger risk index, and further judges the data compression process through the hidden danger risk index. When there is a high risk during data compression, stop compression; when the data is compressed at a low risk, continue compression.

[0039] In the above technical solution, the technical effects and advantages provided by the present invention are:

[0040] The present invention introduces a real-time monitoring and analysis process, obtains multiple data information during data compression, including computing resource allocation information and compression performance information, wherein the CPU clock frequency information and real-time data backlog information are processed to generate a CPU clock frequency anomaly concealment index and a data backlog index. This enables the system to perceive abnormal hidden dangers in the data compression process in real time, avoids the situation where abnormalities lurk for a long time without being detected, and helps to take timely measures to maintain data integrity and system stability.

[0041] The present invention generates a compression risk coefficient by comprehensively analyzing the computing resource allocation information and compression performance information during data compression, and then compares it with a preset threshold to determine whether there is an abnormality in the data compression process. This comprehensive analysis and evaluation method improves the overall control of the data compression process. It not only pays attention to performance information, but also takes into account the allocation of computing resources, which is more helpful to accurately evaluate the effectiveness and potential risks of compression.

[0042] The present invention generates a hidden danger risk index by further analyzing the compression hidden danger coefficient in the analysis set. The introduction of this index provides a more refined data anomaly risk assessment, and corresponding strategies can be taken according to the level of risk. When there is a high risk of data compression, a suggestion to stop compression is made to ensure the security and integrity of the data, providing a specific action plan for system maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0044] Figure 1 The present invention is a method flow chart of a data storage optimization method and system.

[0045] Figure 2 A module schematic diagram of a method and system for optimizing data storage according to the present invention. DETAILED DESCRIPTION

[0046] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these example embodiments are provided so that the description of the present disclosure will be more comprehensive and complete, and the concept of the example embodiments will be fully conveyed to those skilled in the art.

[0047] The present invention provides Figure 1 A method for optimizing data storage is shown, comprising the following steps:

[0048] Analyze the data, understand the characteristics of the data, select a compression algorithm suitable for the data characteristics, and compress the data;

[0049] When conducting an in-depth analysis of the characteristics of the data, focus on key features such as repetitive patterns, frequently occurring characters, and statistical information. Recognition of repetitive patterns helps select compression algorithms that can effectively capture and utilize repetitive structures in the data, such as the Lempel-Ziv family of algorithms. Focusing on frequently occurring characters allows the selection of algorithms that can achieve higher compression efficiency by encoding frequently occurring characters, such as Huffman coding. At the same time, analysis of statistical information can provide insights into data distribution and frequency, providing guidance for selecting appropriate compression algorithms to ensure optimal compression performance in actual applications. By deeply understanding these characteristics, compression algorithms can be selected more accurately to achieve efficient and optimized storage of data.

[0050] Common algorithms include the Lempel-Ziv series (such as LZ77, LZ78), Huffman coding, arithmetic coding, Run-Length Encoding (RLE), Burrows-Wheeler Transform (BWT), etc.

[0051] Acquire multiple data information during data compression, including computing resource allocation information and compression performance information during compression, and after acquisition, process the computing resource allocation information and compression performance information;

[0052] When data is compressed, the computing resource allocation information during data compression includes CPU clock frequency information. When optimizing data storage, CPU clock frequency information refers to the data related to recording and understanding the clock frequency of the CPU (central processing unit) during data compression. The CPU clock frequency refers to the number of clock pulses executed by the CPU per second, usually in Hertz (Hz). In data compression, the information of the CPU clock frequency is monitored and considered in order to understand the operation speed and performance status of the CPU during the compression process. After obtaining the CPU clock frequency information, the CPU clock frequency information is processed to generate a CPU clock frequency anomaly concealment index;

[0053] When data is compressed, large fluctuations in CPU clock frequency may have a negative impact on data integrity, thereby affecting the accuracy of business decisions and data analysis. The following is a detailed explanation:

[0054] Uncertainty in the execution of the compression algorithm: Fluctuations in the CPU clock frequency may cause the execution speed of the compression algorithm to be unstable, resulting in changes in the time taken to execute the compression process at different times. If the CPU clock frequency decreases at a certain moment, and the execution time of the compression algorithm is relatively long, the compression process may not be completed within the required time, so that part of the compressed data cannot be processed correctly, affecting the overall data integrity.

[0055] Problems caused by dynamic frequency modulation: Some systems use dynamic frequency modulation technology to automatically adjust the CPU clock frequency according to load and power consumption. In the case of frequent adjustments to the clock frequency, some steps in the compression process may be executed at different frequencies, thus affecting data consistency. Such fluctuations may cause mismatches between different compression and decompression steps, thus affecting the accuracy of data restoration.

[0056] Inconsistency in data transmission and storage: During the compression process, the data transmission speed is closely related to the CPU's computing speed. If the CPU clock frequency fluctuates greatly, the data transmission speed during the compression process may be unstable, which in turn affects the consistency of storage and transmission. This may cause the data compression and decompression in the storage system to be out of sync, ultimately leading to data inconsistency.

[0057] Risk of information loss: In the case of large fluctuations in clock frequency, there is a risk of data loss during the compression process. This loss may be due to incomplete or incorrect calculation steps performed when the clock frequency is low, making it impossible to correctly restore the compressed data. This may cause serious problems for applications and businesses that have extremely high requirements for data integrity.

[0058] When the CPU clock frequency is lower than the expected CPU clock frequency, it may cause data to be damaged or lost during the compression process, which in turn affects the integrity of the data. This has a potential negative impact on the accuracy of business decisions and data analysis. The following is a detailed explanation:

[0059] Increased execution time: When the CPU clock frequency is lower than expected, the execution time of the compression algorithm will be extended accordingly. This may result in the entire compression process not being completed within the specified time, so that part of the data in the compression process is processed, while other parts may not be fully processed due to the extended execution time, causing data inconsistency.

[0060] Mismatched compression steps: Compression algorithms usually consist of multiple steps, and the execution time of each step may be affected by the CPU clock frequency. If different compression steps are executed at different clock frequencies, it may lead to mismatches between the steps during decompression, making it impossible to restore the data correctly, thus affecting the integrity of the data.

[0061] Unstable transmission speed: Data transmission speed is usually closely related to the CPU's computing speed. When the CPU clock frequency is lower than expected, it may cause instability in data transmission speed during the compression process. This may cause data in the storage and transmission system to be out of sync, ultimately affecting the consistency and accuracy of the data.

[0062] Data corruption and information loss: Calculation steps performed at lower clock frequencies may be incomplete or erroneous, leading to an increased risk of data corruption and information loss. Corrupted data may not be restored correctly, negatively impacting the accuracy of business decisions and data analysis.

[0063] Therefore, by monitoring the CPU clock frequency during data compression, abnormal problems such as large fluctuations in the CPU clock frequency and a CPU clock frequency lower than the expected frequency can be discovered in a timely manner.

[0064] The logic for obtaining the CPU clock frequency anomaly concealment index is as follows:

[0065] Get the real-time CPU clock frequency within T time during data compression, and mark the real-time CPU clock frequency as Clock CPU x, x represents the number of the real-time CPU clock frequency during the data compression within the time T, x = 1, 2, 3, 4, ..., p, p is a positive integer;

[0066] It should be noted that the operating system usually provides performance monitoring tools that can be used to monitor various CPU performance parameters in real time, including clock frequency. Some common performance monitoring tools include:

[0067] Windows Task Manager: In the "Performance" tab, you can view the real-time CPU clock frequency;

[0068] Linux system monitoring commands (such as top, htop): provide real-time monitoring of CPU frequency;

[0069] The CPU clock frequency standard deviation and the CPU clock frequency expected deviation are calculated by using the real-time CPU clock frequency obtained within T time during data compression. The calculation formula is: In the formula, Clock CPU 标 Indicates the standard deviation of CPU clock frequency, Clock CPU 平 It represents the average value of the real-time CPU clock frequency obtained within T time during data compression. In the formula, Clock CPU 偏差 Indicates the expected deviation of CPU clock frequency, Clock CPU 预期 Indicates the expected CPU clock frequency, Clock CPU x' represents the real-time CPU clock frequency obtained within T time and lower than the expected CPU clock frequency during data compression, x' represents the number of the real-time CPU clock frequency obtained within T time and lower than the expected CPU clock frequency during data compression, x'=1, 2, 3, 4, ..., p', p' is a positive integer, and p'≤p;

[0070] Calculate the CPU clock frequency anomaly concealment index. The calculation expression is: Clock CPU =μ1*Clock CPU 标 +μ2*Clock CPU 偏差 , where Clock CPU represents the CPU clock frequency anomaly concealment index, μ1 and μ2 are the CPU clock frequency standard deviation Clock CPU 标 Expected deviation from CPU clock frequency Clock CPU偏差 The weight factor of is greater than 0. The weight factor is used to balance the proportion of each data in the formula, thereby improving the accuracy of the calculation results;

[0071] From the calculation expression of the CPU clock frequency anomaly concealment index, it can be seen that the larger the performance value of the CPU clock frequency anomaly concealment index generated within the T time during data compression, the greater the potential risk of data damage or loss caused by the data compression process. Conversely, the smaller the potential risk of data damage or loss caused by the data compression process.

[0072] When compressing data, the compression performance information during data compression includes real-time data backlog information. The real-time data backlog information is obtained, and the real-time data backlog information is processed to generate a data backlog index.

[0073] The data backlog rate refers to the real-time performance indicator of the compression system when processing data, which indicates the proportion of data in the queue waiting to be processed. Specifically, the lower the data backlog rate, the faster the system processes data, while an increase in the data backlog rate means that the compression system is under load, possibly because the processing speed cannot keep up with the data input speed. In data storage optimization, it is important to maintain a low data backlog rate to ensure that data can pass through the compression system quickly and efficiently, thereby reducing storage requirements and increasing data transmission speeds.

[0074] A high data backlog may have a negative impact on the integrity of the data, thereby affecting the accuracy of business decisions and data analysis. The following is a detailed explanation:

[0075] Unprocessed data is damaged: When the data backlog rate is high, it means that the compression system cannot process the input data in time, causing the data to wait in the processing queue. If unprocessed data is backlogged in the queue for a long time, this may cause the data to be damaged or modified before compression, thus affecting the integrity of the original data.

[0076] Lost compressed data: During the data compression process, if the data backlog rate is too high, the compression system may not be able to process all the data in time. In extreme cases, data may be discarded or not fully processed, resulting in the compressed data set containing lost information. Such data loss will affect the restored data and reduce the integrity of the data.

[0077] Inconsistent decompression: High data backlogs may cause the decompression system to be unable to restore compressed data in a timely manner. If the decompression speed cannot keep up with the compression speed, it may cause inconsistent data during decompression. This may include partially compressed data not being restored, or the restored data being inconsistent with the original data, which affects data integrity.

[0078] Risk of data loss: The increase in data backlog may make the system unable to respond to data input in a timely manner, which may lead to data loss. In some real-time processing scenarios, data loss may cause serious problems for business decision-making and data analysis because the system does not have a complete data set for accurate analysis.

[0079] Therefore, monitoring the data backlog rate during data compression can timely discover abnormal problems with high data backlog rate.

[0080] The logic for obtaining the data backlog index is as follows:

[0081] Get the real-time data backlog rate within T time when data is compressed, form a time series, and use the function Back to convert the real-time data backlog rate into a time series. Rate (t) to make an indication;

[0082] Some application performance monitoring tools can provide application-level performance indicators, including data backlog rate. Such tools can obtain real-time backlog rate information by monitoring the status of queues or processing tasks within the application. For example, New Relic is a full-stack application performance monitoring tool that can provide real-time performance analysis, error tracking, and transaction tracking. By monitoring the queue and task status of the application, New Relic can provide real-time information about the data backlog rate. For another example, AppDynamics is an application performance management tool that can monitor and analyze the performance of the application, including the status of queues and tasks, and can provide real-time performance indicators to help users understand the operation status of the application, including information such as data backlog rate.

[0083] The data backlog index is calculated by the real-time data backlog rate obtained within time T. The calculation expression is: In the formula, Back Rate represents the data backlog index, [t x , t y ] indicates the period of time during which data compression has been completed, [t m , t n ] represents the period of waiting for data compression during data compression, t x <t y , t m <t n ;

[0084] From the calculation expression of the data backlog index, it can be seen that the larger the performance value of the data backlog index generated within T time during data compression, the greater the potential risk of data damage or loss caused by the data compression process. Conversely, the smaller the potential risk of data damage or loss caused by the data compression process.

[0085] Comprehensively analyze the processed computing resource allocation information and compression performance information during data compression to generate a compression risk coefficient, and evaluate data compression through the compression risk coefficient;

[0086] Get the CPU clock frequency anomaly concealment index Clock after data compression CPU and Data Backlog Index Rate Then, the CPU clock frequency anomaly concealment index Clock CPU and Data Backlog Index Rate Perform formula analysis to generate compression risk coefficient YS ττ , based on the formula:

[0087]

[0088] , where f1 and f2 are the CPU clock frequency anomaly concealment index Clock CPU and Data Backlog Index Rate The preset proportional coefficient, and f1 and f2 are both greater than 0;

[0089] From the calculation formula, we can see that the larger the CPU clock frequency anomaly concealment index generated during data compression within T time, the larger the data backlog index, that is, the compression hidden danger coefficient YS generated during data compression within T time ττ The larger the performance value, the greater the potential risk of data damage or loss caused by the data compression process. Conversely, the smaller the potential risk of data damage or loss caused by the data compression process.

[0090] Compare and analyze the compression hidden danger coefficient generated during data compression with the pre-set compression hidden danger coefficient reference threshold to determine whether there are abnormal hidden dangers during data compression;

[0091] The compression potential risk coefficient generated during data compression is compared and analyzed with the pre-set compression potential risk coefficient reference threshold. The results of the comparison and analysis are as follows:

[0092] If the compression hidden danger coefficient is greater than or equal to the compression hidden danger coefficient reference threshold, an abnormal signal is generated. When an abnormal signal is generated during data compression, it indicates that there may be abnormal hidden dangers of data damage or loss in the data compression process, which requires further analysis;

[0093] If the compression hidden danger coefficient is less than the compression hidden danger coefficient reference threshold, a normal signal is generated. When a normal signal is generated during data compression, it indicates that the data compression process is running stably and efficiently.

[0094] When there are abnormal hidden dangers during data compression, the compression hidden danger coefficient corresponding to the starting point and several compression hidden danger coefficients generated subsequently are used to establish an analysis set, and the compression hidden danger coefficients in the analysis set are comprehensively analyzed with the pre-set compression hidden danger coefficient reference threshold to generate a hidden danger risk index. The hidden danger risk index is used to further judge the data compression process. When there is a high risk during data compression, compression is stopped, and when the data is compressed at a low risk, compression is continued;

[0095] When there are abnormal hidden dangers during data compression, the compression hidden danger coefficient corresponding to the starting point and several compression hidden danger coefficients generated subsequently are used to establish an analysis set, and the analysis set is calibrated as F, then F = {YS ττ v}={YS ττ 1 , YS ττ 2 ,…,YS ττ s}, v = 1, 2, 3, 4, ..., s, s is a positive integer;

[0096] The compression hidden danger coefficient in the analysis set is comprehensively analyzed with the pre-set compression hidden danger coefficient reference threshold to generate the hidden danger risk index, based on the formula: In the formula, ξξ represents the hidden danger risk index, YS ττ v' represents the compression potential risk coefficient greater than the compression potential risk coefficient reference threshold, v' represents the number of the compression potential risk coefficient greater than the compression potential risk coefficient reference threshold, v'=1, 2, 3, 4, ..., s', s' is a positive integer, s represents the total number of compression potential risk coefficients in the analysis set;

[0097] The hidden danger risk index generated by the compression hidden danger coefficient greater than the compression hidden danger coefficient reference threshold in the analysis set and the compression hidden danger coefficient reference threshold is compared with the pre-set hidden danger risk index reference threshold. The results of the comparison analysis are as follows:

[0098] If the hidden danger risk index is greater than or equal to the hidden danger risk index reference threshold, a high risk signal is generated. When a high risk signal is generated during data compression, it indicates that there is a high possibility of abnormal hidden dangers of data damage or loss in the data compression process. At this time, the compression is stopped and relevant maintenance personnel are arranged to perform maintenance and repair work on the data compression process in a timely manner;

[0099] If the compression hidden danger coefficient is less than the compression hidden danger coefficient reference threshold, a low-risk signal is generated when the data is compressed, indicating that there may only be accidental minor anomalies in the data compression process, which will not cause serious impact on the data compression process. At this time, continue to compress the data.

[0100] The present invention introduces a real-time monitoring and analysis process, obtains multiple data information during data compression, including computing resource allocation information and compression performance information, wherein the CPU clock frequency information and real-time data backlog information are processed to generate a CPU clock frequency anomaly concealment index and a data backlog index. This enables the system to perceive abnormal hidden dangers in the data compression process in real time, avoids the situation where abnormalities lurk for a long time without being detected, and helps to take timely measures to maintain data integrity and system stability.

[0101] The present invention generates a compression risk coefficient by comprehensively analyzing the computing resource allocation information and compression performance information during data compression, and then compares it with a preset threshold to determine whether there is an abnormality in the data compression process. This comprehensive analysis and evaluation method improves the overall control of the data compression process. It not only pays attention to performance information, but also takes into account the allocation of computing resources, which is more helpful to accurately evaluate the effectiveness and potential risks of compression.

[0102] The present invention generates a hidden danger risk index by further analyzing the compression hidden danger coefficient in the analysis set. The introduction of this index provides a more refined data anomaly risk assessment, and corresponding strategies can be taken according to the level of risk. When there is a high risk of data compression, a suggestion to stop compression is made to ensure the security and integrity of the data, providing a specific action plan for system maintenance.

[0103] The present invention provides Figure 2 A data storage optimization system shown includes a data characteristic analysis and compression algorithm selection module, a data information acquisition and processing module, a comprehensive analysis and compression hidden danger assessment module, a compression hidden danger comparison analysis module, and a risk index generation and compression processing module;

[0104] The data characteristic analysis and compression algorithm selection module analyzes the data, understands the characteristics of the data, selects a compression algorithm suitable for the data characteristics, and compresses the data;

[0105] A data information acquisition and processing module acquires multiple data information during data compression, including computing resource allocation information and compression performance information during compression, and processes the computing resource allocation information and compression performance information after acquisition;

[0106] The comprehensive analysis and compression risk assessment module comprehensively analyzes the processed computing resource allocation information and compression performance information during data compression, generates a compression risk coefficient, and evaluates data compression through the compression risk coefficient;

[0107] The compression hidden danger comparison and analysis module compares and analyzes the compression hidden danger coefficient generated during data compression with the pre-set compression hidden danger coefficient reference threshold to determine whether there are abnormal hidden dangers during data compression;

[0108] The risk index generation and compression processing module, when there are abnormal hidden dangers during data compression, establishes an analysis set with the compression hidden danger coefficient corresponding to the starting point and several compression hidden danger coefficients generated subsequently, conducts a comprehensive analysis of the compression hidden danger coefficients in the analysis set and the pre-set compression hidden danger coefficient reference threshold, generates a hidden danger risk index, and further judges the data compression process through the hidden danger risk index. When there is a high risk during data compression, stop compression, and when the data is compressed at a low risk, continue compression;

[0109] A data storage optimization method provided in an embodiment of the present invention is implemented by the above-mentioned data storage optimization system. The specific methods and processes of the data storage optimization system are detailed in the embodiment of the above-mentioned data storage optimization method, which will not be repeated here.

[0110] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0111] The above description is only by way of illustration of certain exemplary embodiments of the present invention. It is undoubted that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

[0112] It should be noted that, in this article, if there are relational terms such as first and second, etc., they are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0113] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0114] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0115] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0116] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0117] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0118] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0119] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for optimizing data storage, characterized in that: The following steps are involved: Analyze the data, understand the characteristics of the data, select a compression algorithm suitable for the data characteristics, and compress the data; Acquire multiple data information during data compression, including computing resource allocation information and compression performance information during compression, and after acquisition, process the computing resource allocation information and compression performance information; The computing resource allocation information during data compression includes CPU clock frequency information. After the CPU clock frequency information is obtained, the CPU clock frequency information is processed to generate a CPU clock frequency anomaly concealment index. When data is compressed, the compression performance information during data compression includes data real-time backlog information. After the data real-time backlog information is obtained, the data real-time backlog information is processed to generate a data backlog index. The logic for obtaining the CPU clock frequency anomaly concealment index is as follows: Get the real-time CPU clock frequency within T time during data compression, and mark the real-time CPU clock frequency as Clock CPU x , x represents the number of the real-time CPU clock frequency during the data compression within the time T, x = 1, 2, 3, 4, ..., p, p is a positive integer; The CPU clock frequency standard deviation and the CPU clock frequency expected deviation are calculated by using the real-time CPU clock frequency obtained within T time during data compression. The calculation formula is: In the formula, Clock CPU 标 Indicates the standard deviation of CPU clock frequency, Clock CPU 平 It represents the average value of the real-time CPU clock frequency obtained within T time during data compression. In the formula, Clock CPU 偏差 Indicates the expected deviation of CPU clock frequency, Clock CPU 预期 Indicates the expected CPU clock frequency, Clock CPU x' represents the real-time CPU clock frequency obtained within T time and lower than the expected CPU clock frequency during data compression, x' represents the number of the real-time CPU clock frequency obtained within T time and lower than the expected CPU clock frequency during data compression, x'=1, 2, 3, 4, ..., p', p' is a positive integer, and p'≤p; Calculate the CPU clock frequency anomaly concealment index. The calculation expression is: Clock CPU =μ1*Clock CPU 标 +μ2*Clock CPU 偏差 , where Clock CPU represents the CPU clock frequency anomaly concealment index, μ1 and μ2 are the CPU clock frequency standard deviation Clock CPU 标 Expected deviation from CPU clock frequency Clock CPU 偏差 The weight factor of , all values ​​are greater than 0; The logic for obtaining the data backlog index is as follows: Get the real-time data backlog rate within T time when data is compressed, form a time series, and use the function Back to convert the real-time data backlog rate into a time series. Rate (t) to make an indication; The data backlog index is calculated by the real-time data backlog rate obtained within time T. The calculation expression is: In the formula, Back Rate represents the data backlog index, [t x , t y ] indicates the period of time during which data compression has been completed, [t m , t n ] represents the period of waiting for data compression during data compression, t x <t y , t m <t n ; Comprehensively analyze the processed computing resource allocation information and compression performance information during data compression to generate a compression risk coefficient, and evaluate data compression through the compression risk coefficient; Get the CPU clock frequency anomaly concealment index Clock after data compression CPU and Data Backlog Index Rate Then, the CPU clock frequency anomaly concealment index Clock CPU and Data Backlog Index Rate Perform formula analysis to generate compression risk coefficient YS ττ , based on the formula: , where f1 and f2 are the CPU clock frequency anomaly concealment index Clock CPU and Data Backlog Index Rate The preset proportional coefficient, and f1 and f2 are both greater than 0; Compare and analyze the compression hidden danger coefficient generated during data compression with the pre-set compression hidden danger coefficient reference threshold to determine whether there are abnormal hidden dangers during data compression; When there are abnormal hidden dangers during data compression, an analysis set is established with the compression hidden danger coefficient corresponding to the starting point and several compression hidden danger coefficients generated subsequently. The compression hidden danger coefficients in the analysis set are comprehensively analyzed with the pre-set compression hidden danger coefficient reference threshold to generate a hidden danger risk index. The data compression process is further judged by the hidden danger risk index. When there is a high risk during data compression, compression is stopped. When the data is compressed at a low risk, compression continues.

2. A method for optimizing data storage according to claim 1, characterized in that: The compression potential risk coefficient generated during data compression is compared and analyzed with the pre-set compression potential risk coefficient reference threshold. The results of the comparison and analysis are as follows: If the compression hidden danger coefficient is greater than or equal to the compression hidden danger coefficient reference threshold, an abnormal signal is generated. When an abnormal signal is generated during data compression, the data compression process is further analyzed; If the compression hidden danger coefficient is less than the compression hidden danger coefficient reference threshold, a normal signal is generated.

3. A method for optimizing data storage according to claim 2, characterized in that: When there are abnormal hidden dangers during data compression, the compression hidden danger coefficient corresponding to the starting point and several compression hidden danger coefficients generated subsequently are used to establish an analysis set, and the analysis set is calibrated as F, then F = {YS ττ v }={YS ττ 1. YS ττ 2. ..., YS ττ s }, v = 1, 2, 3, 4, ..., s, s is a positive integer; The compression hidden danger coefficient in the analysis set is comprehensively analyzed with the pre-set compression hidden danger coefficient reference threshold to generate the hidden danger risk index, based on the formula: In the formula, ξξ represents the hidden danger risk index, YS ττ v' represents the compression hidden danger coefficient greater than the compression hidden danger coefficient reference threshold, v' represents the number of the compression hidden danger coefficient greater than the compression hidden danger coefficient reference threshold, v'=1, 2, 3, 4, ..., s', s' is a positive integer, s represents the total number of compression hidden danger coefficients in the analysis set.

4. A method for optimizing data storage according to claim 3, characterized in that: The hidden danger risk index generated by the compression hidden danger coefficient greater than the compression hidden danger coefficient reference threshold in the analysis set and the compression hidden danger coefficient reference threshold is compared with the pre-set hidden danger risk index reference threshold. The results of the comparison analysis are as follows: If the hidden danger risk index is greater than or equal to the hidden danger risk index reference threshold, a high risk signal is generated. When a high risk signal is generated during data compression, compression is stopped; If the compression hidden danger coefficient is less than the compression hidden danger coefficient reference threshold, a low risk signal is generated. When a low risk signal is generated during data compression, the data continues to be compressed.

5. A system for optimizing data storage, characterized in that: A method for data storage optimization for implementing any one of claims 1 to 4, comprising a data characteristic analysis and compression algorithm selection module, a data information acquisition and processing module, a comprehensive analysis and compression hidden danger assessment module, a compression hidden danger comparison and analysis module, and a risk index generation and compression processing module; The data characteristic analysis and compression algorithm selection module analyzes the data, understands the characteristics of the data, selects a compression algorithm suitable for the data characteristics, and compresses the data; A data information acquisition and processing module acquires multiple data information during data compression, including computing resource allocation information and compression performance information during compression, and processes the computing resource allocation information and compression performance information after acquisition; The comprehensive analysis and compression risk assessment module comprehensively analyzes the processed computing resource allocation information and compression performance information during data compression, generates a compression risk coefficient, and evaluates data compression through the compression risk coefficient; The compression hidden danger comparison and analysis module compares and analyzes the compression hidden danger coefficient generated during data compression with the pre-set compression hidden danger coefficient reference threshold to determine whether there are abnormal hidden dangers during data compression; The risk index generation and compression processing module, when there are abnormal hidden dangers during data compression, establishes an analysis set with the compression hidden danger coefficient corresponding to the starting point and several compression hidden danger coefficients generated subsequently, conducts a comprehensive analysis of the compression hidden danger coefficients in the analysis set and the pre-set compression hidden danger coefficient reference threshold, generates a hidden danger risk index, and further judges the data compression process through the hidden danger risk index. When there is a high risk during data compression, stop compression; when the data is compressed at a low risk, continue compression.

Images