An electronic file management method and system

Abstract

This invention discloses an electronic records management method and system, specifically relating to the field of electronic records management technology. By real-time monitoring of system resource usage and read / write throughput fluctuations during data migration, and utilizing extracted I / O wait time deviations and abnormal network bandwidth usage frequency characteristics, an overall resource usage anomaly index is calculated. Combined with throughput fluctuation values, fuzzy logic analysis is used to dynamically adjust the storage levels of cold and hot data. By optimizing storage resource allocation during data migration, cold data avoids occupying high-performance storage space or hot data being incorrectly migrated to low-speed media, thereby improving system response speed, overall operating efficiency, and user experience.

Description

Technical Field

[0001] This invention relates to the field of electronic records management technology, and specifically to an electronic records management method and system. Background Technology

[0002] Electronic records management refers to the systematic creation, storage, retrieval, maintenance, and utilization of electronic records through computers and related information technologies. Electronic records include various digital formats of documents, such as text, images, audio, and video. The management process typically involves the classification, indexing, encryption, backup, and recovery of electronic files to ensure their security, integrity, and accessibility. Traditional records management usually relies on paper documents for recording and storage, a method with many limitations. First, paper records occupy a large amount of storage space, and as the number of records increases over time, management becomes more difficult. Furthermore, paper records are susceptible to physical damage, such as fire, floods, and human-caused damage, leading to loss or destruction. Simultaneously, retrieval efficiency is low; record searching and sharing often require significant time and human resources, greatly limiting the practical application value of the records.

[0003] Most archival data in an electronic records management system is "cold data," meaning data that hasn't been accessed for a long time. However, as data continues to grow, the system may need to perform constant data migration, moving cold data to slower storage media to conserve storage resources. During this process, when the system is performing a large number of data migration operations, unreasonable migration strategies or excessively slow migration speeds may cause the data migration process to stall, even affecting normal archival access. Furthermore, an unreasonable migration strategy may lead to confusion in the storage hierarchy between cold and hot data. That is, cold data may fail to migrate in time, continuing to occupy high-performance storage resources, while hot data, i.e., frequently accessed data, may be mistakenly migrated to low-speed storage media. The result is a significant slowdown in the response speed of frequently accessed data, leading to a sharp decline in the overall responsiveness of the system and the user experience. Summary of the Invention

[0004] The purpose of this invention is to provide an electronic record management method and system to address the shortcomings of the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: an electronic record management method, comprising the following steps:

[0006] S1: When performing data migration operations, determine several monitoring time intervals based on the scale of data migration and system capacity, and obtain read and write throughput data in real time within each monitoring time interval through data transmission monitoring tools, and obtain resource contention and usage data in real time within each monitoring time interval through system performance monitoring tools.

[0007] S2: After preprocessing the resource contention and usage data within each monitoring time interval, feature extraction is performed to extract the I / O wait time deviation features and network bandwidth usage frequency abnormal features between different tasks.

[0008] S3: After comprehensively analyzing the I / O wait time deviation characteristics and network bandwidth usage frequency abnormal characteristics extracted from different tasks, determine the weight assignment of resource contention usage data in each monitoring time interval, and calculate the overall resource usage abnormality index by weighted average of the weight assignment of resource contention usage data in each monitoring time interval.

[0009] S4: After analyzing the fluctuation frequency of read and write throughput data in each monitoring time interval, determine the throughput fluctuation value of read and write throughput data in each monitoring time interval.

[0010] S5: Based on the obtained overall resource usage anomaly index and throughput fluctuation value, analyze them using fuzzy logic and dynamically adjust the storage level of cold data and hot data.

[0011] Preferably, in S2, an I / O waiting time deviation index is generated based on the extracted I / O waiting time deviation characteristics between different tasks. The method for obtaining the I / O waiting time deviation index is as follows:

[0012] Real-time acquisition of I / O wait time data for n tasks within a time period M. The task's I / O latency data is normalized to a proportional value, expressed as: ; It represents the proportion of the total latency to the I / O latency of the i-th task, based on the normalized latency of each task. The contribution of each task to the overall system entropy is calculated using the following expression: ; It is the entropy contribution of the i-th task, and the entropy contribution of all tasks. Adding them together, we get the total entropy of the entire system, expressed as: H is the total entropy of the system, the standardized total entropy, expressed as: ; It is the standardized entropy value. It is the maximum entropy value in the system, based on the standardized entropy value. The I / O latency deviation index is calculated using the following expression: In the formula, This is the I / O latency deviation index.

[0013] Preferably, in S2, a network bandwidth usage frequency anomaly index is generated based on the extracted network bandwidth usage frequency anomaly features. The method for obtaining the network bandwidth usage frequency anomaly index is as follows:

[0014] Collect network bandwidth usage data for the system over m different time periods, including bandwidth utilization rate and bandwidth occupancy rate; standardize the network bandwidth usage data and label it as Z; based on the standardized data, calculate the covariance matrix of the network bandwidth usage data, expressed as: In the formula, This is the bandwidth usage data for the i-th data point. Let X be the mean of all data points, Cov(X) be the covariance matrix of the bandwidth usage data, and T be the matrix transpose. Eigenvalue decomposition is performed on the covariance matrix to obtain eigenvalues ​​and eigenvectors. The eigenvalues ​​represent the variance of the data explained by each principal component, and the eigenvectors represent the directions of the principal components. The expression is: In the formula, It is the i-th eigenvector, representing the direction of the i-th principal component. The i-th eigenvalue is used to select principal components based on their magnitudes. The cumulative variance contribution rate is calculated using the following expression: ; k is the number of principal components selected, and q is the total number of features. Using the selected principal components, the score for each data point along the principal component direction is calculated, expressed as: In the formula, This is the score of the i-th principal component. Based on the principal component scores and their weights, the network bandwidth usage frequency anomaly index for each time period is calculated, expressed as: In the formula, The weight of the i-th principal component is equal to the eigenvalue. The proportion of the total feature values, SC is the network bandwidth usage frequency anomaly index.

[0015] Preferably, in S3, the waiting time deviation index and the network bandwidth usage frequency anomaly index are converted into a first feature vector. The first feature vector is used as the input of the machine learning model. The machine learning model uses the prediction of the weight assignment labels of the resource contention usage data in each monitoring time interval as the prediction objective, and minimizes the sum of prediction errors of the weight assignment labels of the resource contention usage data in all monitoring time intervals as the training objective. The machine learning model is trained until the sum of prediction errors converges and the model training stops. The weight assignment of the resource contention usage data in each monitoring time interval is determined according to the model output. The machine learning model is a multinomial regression model. The overall resource usage anomaly index is obtained by weighted averaging the weight assignments of the resource contention usage data in each monitoring time interval.

[0016] Preferably, in S4, after analyzing the fluctuation frequency of read / write throughput data within each monitoring time interval, the throughput fluctuation value of read / write throughput data within each monitoring time interval is determined, specifically as follows:

[0017] Determine the size Y of the sliding window, collect the system's read and write throughput data in each time interval, and denot it as... Where x is the index of the time interval, the average throughput is calculated within each sliding window. The length of the sliding window is d, and its expression is: ; Calculate the throughput change between adjacent sliding windows, i.e., calculate the absolute rate of change, expressed as: In the formula, This is the absolute rate of change; based on the calculated rate of change in throughput. The throughput fluctuation value is calculated using the following expression: In the formula, j is the number of sliding windows, and LM is the throughput fluctuation value.

[0018] Preferably, in S5, based on the obtained overall resource usage anomaly index and throughput fluctuation value, after analyzing them using fuzzy logic, the storage levels of cold data and hot data are dynamically adjusted, specifically as follows:

[0019] The overall resources obtained are used as the input items of the fuzzy logic using the anomaly index QS and the throughput fluctuation value LM, and the storage hierarchy priority R of cold data and hot data is used as the output item of the fuzzy logic.

[0020] Define a fuzzy set based on the overall resource usage anomaly index, throughput fluctuation value, and storage hierarchy priority of cold and hot data, and construct fuzzy rules.

[0021] The fuzzy inference method is used to process the overall resource usage anomaly index QS and throughput fluctuation value LM of the input items, and the storage level priority R of cold data and hot data is calculated by the preset fuzzy rules. The fuzzy inference result is converted into a precise numerical output R by the defuzzification method, which represents the storage level priority of cold data and hot data.

[0022] Based on the storage priority R of the defuzzified cold and hot data, they are divided into high priority, medium priority, and low priority. The storage level of cold and hot data is automatically adjusted according to the priority level of R.

[0023] The present invention also provides an electronic records management system, including a monitoring module, a feature extraction module, a comprehensive analysis module, a throughput fluctuation analysis module, and a fuzzy logic decision-making module;

[0024] Monitoring module: When performing data migration operations, it determines several monitoring time intervals based on the scale of data migration and system capacity, and obtains read and write throughput data in real time within each monitoring time interval through data transmission monitoring tools, and obtains resource contention and usage data in real time within each monitoring time interval through system performance monitoring tools.

[0025] Feature extraction module: After preprocessing the resource contention and usage data within each monitoring time interval, feature extraction is performed to extract the I / O wait time deviation features and network bandwidth usage frequency abnormal features between different tasks.

[0026] The comprehensive analysis module analyzes the I / O wait time deviation characteristics and network bandwidth usage frequency anomaly characteristics extracted from different tasks. It then determines the weight of the resource contention usage data in each monitoring time interval and calculates the overall resource usage anomaly index by weighted averaging of the weighted values ​​of the resource contention usage data in each monitoring time interval.

[0027] Throughput Fluctuation Analysis Module: After analyzing the fluctuation frequency changes of read and write throughput data within each monitoring time interval, the throughput fluctuation value of read and write throughput data within each monitoring time interval is determined.

[0028] Fuzzy logic decision module: Based on the obtained overall resource usage anomaly index and throughput fluctuation value, the module uses fuzzy logic to analyze the data and then dynamically adjusts the storage levels of cold and hot data.

[0029] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0030] 1. This invention effectively solves the problems of resource waste and performance degradation caused by unreasonable strategies during data migration by dynamically adjusting cold and hot data in an electronic records management system. By monitoring the system's read / write throughput and resource contention in real time, extracting I / O wait time deviations and abnormal network bandwidth usage frequency characteristics, and combining this with machine learning models to analyze resource usage anomaly indices, the system can automatically optimize storage resource allocation based on the current load. This avoids cold data occupying high-speed storage space and hot data being mistakenly migrated to low-speed storage media, thereby improving the response speed of frequently accessed data and the overall efficiency of the system.

[0031] 2. By using fuzzy logic to dynamically adjust the storage levels of cold and hot data, the system can prioritize the storage location of hot data when resources are scarce, while efficiently migrating cold data to low-speed storage when resources are sufficient. This optimizes storage space utilization, not only improving the flexibility of data management but also reducing the load on high-performance storage devices, improving user experience, and ensuring that the system maintains stable performance when facing massive data growth. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0033] Figure 1 This is a flowchart of the method of the present invention.

[0034] Figure 2 This is a system module diagram of the present invention. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] Example 1, please refer to Figure 1 and Figure 2 As shown in this embodiment, an electronic record management method includes the following steps:

[0037] S1: When performing data migration operations, determine several monitoring time intervals based on the scale of data migration and system capacity, and obtain read and write throughput data in real time within each monitoring time interval through data transmission monitoring tools, and obtain resource contention and usage data in real time within each monitoring time interval through system performance monitoring tools.

[0038] S2: After preprocessing the resource contention and usage data within each monitoring time interval, feature extraction is performed to extract the I / O wait time deviation features and network bandwidth usage frequency abnormal features between different tasks.

[0039] S3: After comprehensively analyzing the I / O wait time deviation characteristics and network bandwidth usage frequency abnormal characteristics extracted from different tasks, determine the weight assignment of resource contention usage data in each monitoring time interval, and calculate the overall resource usage abnormality index by weighted average of the weight assignment of resource contention usage data in each monitoring time interval.

[0040] S4: After analyzing the fluctuation frequency of read and write throughput data in each monitoring time interval, determine the throughput fluctuation value of read and write throughput data in each monitoring time interval.

[0041] S5: Based on the obtained overall resource usage anomaly index and throughput fluctuation value, analyze them using fuzzy logic and dynamically adjust the storage level of cold data and hot data.

[0042] In S1, during data migration, several monitoring time intervals are determined based on the data migration scale and system capacity. Read / write throughput data for each monitoring time interval is acquired in real-time using a data transmission monitoring tool, and resource contention and usage data for each monitoring time interval is acquired in real-time using a system performance monitoring tool. Specifically:

[0043] Determining the monitoring interval is crucial when performing data migration operations, as a reasonable monitoring interval can balance the real-time nature of the data with the system load, while ensuring that the collected monitoring data is detailed enough to reflect changes in system performance.

[0044] Determine the initial monitoring frequency based on the scale of the data migration. Calculate the total data volume of this migration task (e.g., TB, GB). Determine the type and size of the data files; larger files and more complex file structures typically require more frequent monitoring. Estimate the total duration of the data migration based on the data volume, network bandwidth, and storage device I / O performance. Large-scale data migrations usually take longer and may require longer monitoring intervals, while small-scale migrations can be monitored more frequently.

[0045] Based on system capacity and performance status, further optimize the monitoring interval. Assess the capacity of the current system's storage devices (such as hard drives or SSDs). Larger capacity devices typically support longer monitoring intervals, while smaller capacity devices require more intensive monitoring. Evaluate the current usage of CPU, memory, disk I / O, and network using pre-acquired system load data. If the system load is currently high, the monitoring interval should be appropriately shortened to promptly capture load fluctuation information; conversely, the monitoring interval can be appropriately extended when the load is low.

[0046] After considering the scale of the data migration and the system capacity, select an appropriate monitoring interval. For small-scale migrations (e.g., several GB to tens of GB of data): a monitoring interval of 1 to 5 seconds is recommended to capture subtle system changes. For medium-scale migrations (e.g., tens to hundreds of GB of data): a monitoring interval of 5 to 10 seconds is suggested. For large-scale migrations (e.g., TB-level data migration): a monitoring interval of 10 to 30 seconds can be selected, but it needs to be adjusted as needed based on performance fluctuations during the migration process.

[0047] Data transfer monitoring tools can be used to obtain read and write throughput data in real time for each monitoring time interval. Commonly used tools include iostat, iotop, and dstat. These tools can be used to monitor the read and write operations of storage devices and help analyze the performance of the storage system during data migration. Specifically, iostat can provide disk I / O transfer rate and utilization, displaying the number of read and write operations per second, the amount of data read and written, and the disk I / O wait time; iotop is used to display process-level I / O operations, helping to identify which processes are consuming a large amount of I / O resources.

[0048] Read / write throughput data refers to the amount of data transferred per second during data migration. This data reflects the efficiency of the storage device and is a key indicator for assessing the smoothness of the migration process. The specific data acquired includes:

[0049] Transfer rate: The amount of data read and written by a storage device per second, usually measured in MB / s. If read / write throughput remains high and stable over a long period, it indicates high system migration efficiency. Drastic fluctuations or a significant drop in read / write speed may indicate a resource bottleneck, such as disk I / O overload.

[0050] Average latency: This is the average time it takes for the storage device to respond to a request. A significant increase in latency indicates that the storage device may be overloaded, causing read and write requests to queue for processing and limiting migration efficiency.

[0051] Disk utilization: The I / O utilization of storage devices. If this value is close to 100%, it means that the storage device is almost running at full capacity and may be unable to handle more concurrent operations.

[0052] System performance monitoring tools can be used to obtain real-time data on resource contention and usage across different monitoring time intervals. Commonly used tools include vmstat, sar, and top. These tools can be used to monitor the usage of CPU, memory, disk I / O, and network resources, thereby identifying the contention for system resources by different tasks during data migration. Specifically, vmstat provides overall system performance data, including CPU utilization, memory and swap space usage, as well as disk and process status; sar can generate historical records and provide detailed analysis of the performance of various system components, such as I / O and CPU load; top primarily displays the current list of processes and their resource consumption, helping to identify high-consumption tasks.

[0053] Resource contention and usage data reflects the competition for critical resources (such as CPU, memory, disk I / O, and network) among different tasks within the system during data migration, especially when multiple concurrent tasks are running simultaneously, which may create performance bottlenecks. The acquired data includes:

[0054] CPU utilization: This assesses the CPU resource usage of migration tasks, especially the time allocated to user processes and system processes. Excessively high CPU utilization (e.g., close to 90% or higher) can prevent other system tasks from executing in a timely manner, impacting normal file access or other system functions.

[0055] Memory usage: This includes the current amount of available memory and the usage of swap space. When memory is insufficient, the system will use swap (i.e., virtual memory on the hard drive), leading to a decrease in overall system performance. If the migration task consumes too many memory resources, it may affect the normal operation of other applications.

[0056] Disk I / O contention: This reflects the competition among multiple tasks accessing storage devices simultaneously. Disk I / O contention can slow down task response, especially during large-scale data migrations, where frequent disk reads and writes can significantly increase response time.

[0057] Network bandwidth usage: For cross-network data migration (such as cloud storage or remote storage devices), network bandwidth usage is a critical factor. If data migration consumes a large amount of bandwidth, the network needs of other services may be affected, leading to increased network latency and slower response times.

[0058] S2: After preprocessing the resource contention and usage data within each monitoring time interval, feature extraction is performed to extract the I / O wait time deviation features and network bandwidth usage frequency abnormal features between different tasks.

[0059] Preprocessing resource contention and usage data across monitoring time intervals improves data quality, eliminates noise and outliers, and ensures the accuracy of subsequent analysis. The main preprocessing steps include data cleaning, smoothing, outlier handling, and standardization.

[0060] Remove incomplete or invalid data records to ensure data integrity and consistency. Check for missing data in each monitoring time interval (e.g., CPU utilization or I / O data not recorded in a certain time interval). If missing data is found, choose an appropriate imputation method (e.g., interpolation) or discard the records for that time interval. Ensure that all monitoring metrics (such as CPU utilization, memory usage, disk I / O, etc.) are correctly recorded in each time interval to avoid missing data.

[0061] Use moving averages or exponential smoothing to smooth monitoring data. For example, in data such as CPU utilization or I / O utilization, short-term sharp fluctuations may not reflect the true state of the system; smoothing can eliminate such short-term fluctuations. Performing a sliding window calculation on resource usage within each monitoring period results in smoothed values ​​that better reflect long-term trends, preventing anomalies at a single point in time from disrupting the overall analysis.

[0062] Outlier detection is performed for each monitoring metric (such as CPU utilization, I / O operation time, etc.). Statistical methods such as Z-score or IQR (interquartile range) can be used to identify anomalies in the data. Detected outliers can be deleted, replaced by interpolation, or marked and then have their weight reduced in subsequent analyses, depending on the situation. For example, if network bandwidth suddenly spikes within a certain time period, but other resource usage remains normal, this may be a temporary network failure requiring special handling.

[0063] Since various resource metrics (such as CPU utilization, memory usage, and I / O latency) may have different dimensions and ranges (e.g., CPU utilization is a percentage, while I / O latency is in milliseconds), it is necessary to standardize all monitoring metrics to the same range (e.g., 0 to 1) to facilitate weight calculation and comprehensive evaluation in subsequent analysis. Common standardization methods include min-max normalization and Z-score standardization; the specific choice depends on the distribution characteristics of the data.

[0064] Based on the extracted I / O latency deviation characteristics between different tasks, an I / O latency deviation index is generated. The method for obtaining the I / O latency deviation index is as follows:

[0065] Real-time acquisition of I / O wait time data for n tasks within a time period M. To eliminate the dimensional differences in I / O latency across tasks, the I / O latency data for each task is normalized to a proportional value, expressed as: ; It represents the proportion of the total latency to I / O for the i-th task. After normalization, it represents the latency for each task. This represents the relative proportion of its waiting time in the entire system, all The sum is 1. Based on the normalized wait time for each task. The contribution of each task to the overall system entropy is calculated using the following expression: ; It is the entropy contribution of the i-th task, and the entropy contribution of all tasks. Adding them together, we get the total entropy of the entire system, expressed as: H represents the total entropy of the system, reflecting the uncertainty in the distribution of I / O wait times for all tasks within the system. A higher total entropy indicates a more uniform distribution of I / O wait times; a lower total entropy indicates a less uniform distribution and larger wait time deviations. The standardized total entropy is expressed as: ; It is the standardized entropy value, ranging from 0 to 1. This is the maximum entropy value in the system, reached when the I / O latency of all tasks is perfectly uniformly distributed. It is based on the standardized entropy value. The I / O latency deviation index is calculated using the following expression: In the formula, This is the I / O latency deviation index.

[0066] A larger I / O wait time deviation index indicates a significant difference in I / O wait times among different tasks during data migration, suggesting uneven resource utilization. Some tasks may consume a large amount of I / O resources, while others are in a long waiting state. This indicates that resource allocation is unreasonable during data migration, and competition between tasks is too intense, potentially leading to bottlenecks and significant delays in the response time of some tasks, impacting overall performance. A larger index indicates a more severe system anomaly, meaning that resource scheduling strategies need to be adjusted to reduce wait time differences.

[0067] A smaller I / O latency deviation index indicates a more balanced I / O latency across tasks in the system, more reasonable resource allocation, and smoother I / O resource sharing among all tasks during data migration. In this case, the system load distribution is more even, resource contention between tasks is lower, and overall performance is more stable. A smaller deviation index indicates that system resource utilization is closer to the ideal state, with lower anomalies, and data migration can proceed efficiently and smoothly, without significant bottlenecks or performance degradation.

[0068] The network bandwidth usage frequency anomaly index is generated based on the extracted network bandwidth usage frequency anomaly characteristics. The method for obtaining the network bandwidth usage frequency anomaly index is as follows:

[0069] Collect network bandwidth usage data for the system over m different time periods, including bandwidth utilization rate and bandwidth occupancy rate; standardize the network bandwidth usage data and label it as Z; based on the standardized data, calculate the covariance matrix of the network bandwidth usage data. The covariance matrix is ​​used to describe the linear relationship between various features, and its expression is: In the formula, This is the bandwidth usage data for the i-th data point. Let X be the mean of all data points, Cov(X) be the covariance matrix of the bandwidth usage data, and T be the matrix transpose. Eigenvalue decomposition is performed on the covariance matrix to obtain eigenvalues ​​and eigenvectors. The eigenvalues ​​represent the variance of the data explained by each principal component, and the eigenvectors represent the directions of the principal components. The expressions are as follows: In the formula, It is the i-th eigenvector, representing the direction of the i-th principal component. This is the i-th eigenvalue, representing the variance explained by this principal component. Principal components are selected based on the magnitude of the eigenvalues, typically choosing the first few principal components with the largest explained variance. These principal components contain the most important information in the bandwidth usage data. The cumulative variance contribution rate is calculated to measure whether the selected principal components are sufficient to explain the abnormal fluctuations in bandwidth usage; the expression is: k is the number of principal components selected, and q is the total number of features. Generally, a cumulative variance contribution rate of 85%-95% is required, meaning these principal components can explain the vast majority of bandwidth usage fluctuations.

[0070] By selecting the principal components, a score is calculated for each data point along the principal component direction. This score represents the projection of bandwidth usage into the principal component space for each time period, reflecting the main characteristics of bandwidth usage. The expression is: In the formula, This is the score of the i-th principal component. Based on the principal component scores and their weights, the network bandwidth usage frequency anomaly index for each time period is calculated. A higher anomaly index indicates that the bandwidth usage frequency deviates more from the normal range within that time period. The expression is: In the formula, The weight of the i-th principal component is equal to the eigenvalue. The proportion of the total feature values, SC is the network bandwidth usage frequency anomaly index.

[0071] A higher network bandwidth usage frequency anomaly index indicates greater fluctuations in bandwidth usage frequency during data migration, suggesting an imbalance in bandwidth resource allocation and usage. Some tasks may be consuming excessive bandwidth, causing a significant decrease in the transmission speed of other tasks. This typically reflects inefficient system resource scheduling, with some tasks potentially affected by bandwidth bottlenecks, resulting in delays or interruptions in data transmission. Uneven bandwidth usage leads to system performance degradation, intensified competition between tasks, and a greater degree of overall resource usage anomalies.

[0072] The lower the network bandwidth usage frequency anomaly index, the more stable the bandwidth usage frequency is during the data migration operation. This indicates a relatively balanced allocation of bandwidth resources among tasks, with a lighter or less fluctuating network load. This suggests that the system's resource scheduling is more rational, bandwidth usage is as expected, and there is no excessive competition or resource waste. At this point, overall resource usage is relatively normal, the data migration process can proceed smoothly and efficiently, the degree of anomaly is low, and system performance and task transmission efficiency are optimized.

[0073] S3: After comprehensively analyzing the I / O wait time deviation characteristics and network bandwidth usage frequency abnormal characteristics extracted from different tasks, the weight assignment of resource contention usage data in each monitoring time interval is determined. The overall resource usage abnormality index is obtained by weighted averaging of the weight assignments of resource contention usage data in each monitoring time interval.

[0074] The waiting time deviation index and the network bandwidth usage frequency anomaly index are converted into a first feature vector. This first feature vector is used as the input to the machine learning model. The machine learning model uses the prediction of the weighted labels of resource contention usage data in each monitoring time interval as the prediction objective, and minimizes the sum of prediction errors of the weighted labels of resource contention usage data in all monitoring time intervals as the training objective. The machine learning model is trained until the sum of prediction errors converges, at which point the model training stops. The weighted labels of resource contention usage data in each monitoring time interval are determined based on the model output. The machine learning model is a multinomial regression model. The overall resource usage anomaly index is obtained by calculating the weighted average of the weighted labels of resource contention usage data in each monitoring time interval.

[0075] The method for obtaining the weights of resource contention and usage data within each monitoring time interval is as follows: Obtain the corresponding function expression from the training data of the first feature vector of the trained machine learning model. In the formula, It is the output function of the model. SC is the I / O latency deviation index, and SC is the network bandwidth usage frequency anomaly index. Assign weights to the resource contention and usage data within each monitoring time interval.

[0076] S4: After analyzing the fluctuation frequency of read and write throughput data within each monitoring time interval, determine the throughput fluctuation value of read and write throughput data within each monitoring time interval.

[0077] Determine the size Y of the sliding window, which is how many time intervals of data are included in each throughput fluctuation calculation. The choice of window size typically depends on the frequency of system monitoring and the timeliness of throughput changes. Small window: More sensitive to short-term fluctuations, suitable for rapidly changing systems (e.g., at the second level). Large window: Smooths out short-term fluctuations, suitable for long-term trend analysis (e.g., at the minute level). Example: Assuming each monitoring interval is 5 seconds, a sliding window size of 3 time intervals is chosen, meaning that the data from the most recent 15 seconds is used each time fluctuation is calculated.

[0078] Collect system read / write throughput data for each time interval, and record it as... , where x is the index of the time interval. Data may include the total number of disk read / write operations (e.g., bytes transferred per second, in MB / s). The average throughput is calculated within each sliding window. The length of the sliding window is d, and its expression is: ; Calculate the throughput change between adjacent sliding windows, i.e., calculate the absolute rate of change, expressed as: In the formula, This is the absolute rate of change; based on the calculated rate of change in throughput. This allows us to determine the degree of throughput fluctuation in the system over different time periods. Generally, larger fluctuations indicate unstable system throughput, potentially indicating bottlenecks or uneven load distribution; smaller fluctuations indicate stable system operation. If the absolute rate of change is greater than 0.1 (i.e., throughput change rate exceeds 10%), it is considered a large throughput fluctuation, requiring attention to potential performance issues. If the absolute rate of change is approximately 0 or very small, it indicates stable system throughput and relatively even load distribution. The expression for calculating throughput fluctuation is: In the formula, j is the number of sliding windows, and LM is the throughput fluctuation value.

[0079] S5: Based on the obtained overall resource usage anomaly index and throughput fluctuation value, analyze them using fuzzy logic and dynamically adjust the storage level of cold data and hot data.

[0080] The overall resources obtained are used as the input items of the fuzzy logic using the anomaly index QS and the throughput fluctuation value LM, and the storage hierarchy priority R of cold data and hot data is used as the output item of the fuzzy logic.

[0081] The QS (Overall Resource Usage Anomaly Index) is divided into three fuzzy sets:

[0082] Low (L): When system resource usage is low, the QS value is small (0 ≤ QS < 0.3).

[0083] Medium (M): System resource usage is moderate, and QS value is in the medium range (0.3 ≤ QS < 0.7).

[0084] High (H): When system resource usage is close to the limit, QS takes a larger value (0.7 ≤ QS ≤ 1.0).

[0085] The LM (throughput fluctuation value) is divided into three fuzzy sets:

[0086] Low (L): The throughput fluctuates less, and the read and write operations are relatively stable (0 ≤LM < 0.2).

[0087] Medium (M): The throughput fluctuates moderately, and the system read and write operations show some fluctuation (0.2 ≤LM < 0.5).

[0088] High (H): Throughput fluctuates significantly, and read / write operations are unstable (0.5 ≤ LM ≤ 1.0).

[0089] To adjust storage tier priorities, a set of fuzzy rules is defined to reflect the impact of different combinations of QS and LM on R (storage tier priority). Common examples of fuzzy rules are as follows:

[0090] Low QS, Low LM: If both QS and LM are low, it means that the system resources are sufficient and the read and write operations are stable. In this case, cold data can be migrated to low-speed storage devices first, and the output R is "low".

[0091] Low QS, Medium LM: If QS is low and LM is medium, system resources are still sufficient, but throughput fluctuates slightly. Cold data can continue to be migrated first, and the output R is "low".

[0092] Low QS, high LM: If QS is low but LM is high, it means that the system resources are sufficient, but the throughput fluctuates greatly. It is necessary to keep hot data in high-speed storage and postpone the migration of cold data. The output R is "medium".

[0093] Medium QS, Low LM: If QS is medium and LM is low, resource contention is moderate, but throughput is stable, and cold data is appropriately migrated, the output R is "medium".

[0094] Medium QS, Medium LM: If both QS and LM are medium, the system resources and throughput are in a medium state. Keep the current storage level unchanged, and the output R is "medium".

[0095] Medium QS, High LM: If QS is medium but LM is high, it indicates that the throughput fluctuates greatly. Hot data is given higher priority and cold data migration is postponed. The output R is "high".

[0096] High QS, low LM: If QS is high but LM is low, system resource contention is severe but throughput is stable, cold data migration is delayed, and the output R is "medium".

[0097] High QS, Medium LM: If QS is high and LM is medium, both resource and throughput issues arise. Prioritize protecting hot data, and output R is set to "High".

[0098] High QS, High LM: If both QS and LM are high, it indicates that the system resources are severely contested and the throughput fluctuates greatly. It is necessary to prioritize ensuring that hot data is stored in high-speed storage. The output R is "high".

[0099] Fuzzy inference methods (such as the Mamdani model) are used to process the input terms QS and LM, and the output R is calculated using preset fuzzy rules.

[0100] Assuming the current monitoring data is: QS = 0.8 (severe system resource contention), LM = 0.6 (large throughput fluctuations), according to the fuzzy rule: the rule corresponding to QS = high and LM = high is that the output R = high priority.

[0101] By using defuzzification methods (such as the centroid method), the fuzzy inference results are converted into precise numerical outputs R, representing the adjustment priority of the storage level.

[0102] Defuzzification calculation: Using the centroid method, the final output value R is calculated based on the membership degree output by the fuzzy rules. For example, after defuzzification, the output R = 0.85, indicating that hot data has a higher storage priority and the migration of cold data needs to be postponed.

[0103] Based on the defuzzified priority R, the system automatically adjusts the storage hierarchy of cold and hot data. Dynamic adjustment strategy:

[0104] A low R value (e.g., R < 0.3) indicates that the system resources are sufficient, the read / write throughput is stable, and the migration of cold data can be accelerated. The system will migrate cold data that has not been accessed for a long time from high-speed storage media (such as SSDs or high-performance HDDs) to low-speed storage media (such as HDDs or cloud storage) in the background, freeing up more high-speed storage space for hot data.

[0105] A medium R value (e.g., 0.3 ≤ R < 0.7) indicates that the system is under medium load, with fluctuating read / write throughput. In this state, the system maintains its current storage tier configuration and does not perform large-scale data migration. Migration of cold data may be delayed to ensure that current storage resources are not overburdened.

[0106] A high R-value (e.g., R≥0.7) indicates intense competition for system resources and significant throughput fluctuations. Hot data must be preferentially retained on high-speed storage media, while the migration of cold data should be postponed or suspended. The system will avoid migrating cold data to low-speed storage devices to ensure the high-frequency access performance of hot data.

[0107] Specific implementation:

[0108] Cold data migration: When R is in a low priority state, the system can migrate cold data to low-speed storage media in batches to free up more high-speed storage space.

[0109] Hot data residency: When R is in high priority, the system will prioritize ensuring that frequently accessed hot data resides in high-speed storage media and may postpone the migration of cold data if necessary.

[0110] Resource balancing adjustment: Under medium priority conditions, the system maintains the current storage tier and makes adjustments only when monitoring data indicates changes in resources or throughput.

[0111] In this application, by processing QS and LM using fuzzy logic and combining them with the storage hierarchy priority R generated after defuzzification, the system can dynamically adjust the storage hierarchy of cold and hot data. This dynamic adjustment method ensures that the storage space of hot data is prioritized for protection under high load or when system resources are scarce, while cold data is migrated as much as possible when system resources are sufficient, thereby optimizing the efficiency and performance of the entire storage system.

[0112] In this embodiment, during the data migration operation, the monitoring time interval is first determined based on the migration scale and system capacity, and read / write throughput and resource contention status data are acquired in real time using monitoring tools. Then, the resource contention data within each time interval is preprocessed to extract I / O wait time deviation characteristics and network bandwidth usage frequency anomaly characteristics. Next, these characteristics are comprehensively analyzed to determine the weights of the resource contention status data and a weighted average is calculated to obtain an overall resource usage anomaly index. Simultaneously, the fluctuation frequency of read / write throughput is analyzed to calculate the throughput fluctuation value. Finally, fuzzy logic is used, with the overall resource anomaly index and throughput fluctuation value as input, to dynamically adjust the storage levels of cold and hot data, ensuring efficient resource utilization and stable system operation.

[0113] Example 2: The electronic records management system described in this example includes a monitoring module, a feature extraction module, a comprehensive analysis module, a throughput fluctuation analysis module, and a fuzzy logic decision-making module.

[0114] Monitoring module: When performing data migration operations, it determines several monitoring time intervals based on the scale of data migration and system capacity, and obtains read and write throughput data in real time within each monitoring time interval through data transmission monitoring tools, and obtains resource contention and usage data in real time within each monitoring time interval through system performance monitoring tools.

[0115] Feature extraction module: After preprocessing the resource contention and usage data within each monitoring time interval, feature extraction is performed to extract the I / O wait time deviation features and network bandwidth usage frequency abnormal features between different tasks.

[0116] The comprehensive analysis module analyzes the I / O wait time deviation characteristics and network bandwidth usage frequency anomaly characteristics extracted from different tasks. It then determines the weight of the resource contention usage data in each monitoring time interval and calculates the overall resource usage anomaly index by weighted averaging of the weighted values ​​of the resource contention usage data in each monitoring time interval.

[0117] Throughput Fluctuation Analysis Module: After analyzing the fluctuation frequency changes of read and write throughput data within each monitoring time interval, the throughput fluctuation value of read and write throughput data within each monitoring time interval is determined.

[0118] Fuzzy logic decision module: Based on the obtained overall resource usage anomaly index and throughput fluctuation value, the module uses fuzzy logic to analyze the data and then dynamically adjusts the storage levels of cold and hot data.

[0119] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for managing electronic records, characterized in that: Includes the following steps: S1: When performing data migration operations, determine several monitoring time intervals based on the scale of data migration and system capacity, and obtain read and write throughput data in real time within each monitoring time interval through data transmission monitoring tools, and obtain resource contention and usage data in real time within each monitoring time interval through system performance monitoring tools. S2: After preprocessing the resource contention and usage data within each monitoring time interval, feature extraction is performed to extract the I / O wait time deviation features and network bandwidth usage frequency abnormal features between different tasks. Specifically, an I / O latency deviation index is generated based on the extracted I / O latency deviation characteristics between different tasks. The method for obtaining the I / O latency deviation index is as follows: Real-time acquisition of I / O wait time data for n tasks within a time period M. The task's I / O latency data is normalized to a proportional value, expressed as: ; It represents the proportion of the total latency to the I / O latency of the i-th task, based on the normalized latency of each task. The contribution of each task to the overall system entropy is calculated using the following expression: ; It is the entropy contribution of the i-th task, and the entropy contribution of all tasks. Adding them together, we get the total entropy of the entire system, expressed as: H is the total entropy of the system, the standardized total entropy, expressed as: ; It is the standardized entropy value. It is the maximum entropy value in the system, based on the standardized entropy value. The I / O latency deviation index is calculated using the following expression: In the formula, This refers to the I / O latency deviation index. S3: After comprehensively analyzing the I / O wait time deviation characteristics and network bandwidth usage frequency abnormal characteristics extracted from different tasks, determine the weight assignment of resource contention usage data in each monitoring time interval, and calculate the overall resource usage abnormality index by weighted average of the weight assignment of resource contention usage data in each monitoring time interval. S4: After analyzing the fluctuation frequency of read and write throughput data in each monitoring time interval, determine the throughput fluctuation value of read and write throughput data in each monitoring time interval. S5: Based on the obtained overall resource usage anomaly index and throughput fluctuation value, after analyzing them using fuzzy logic, dynamically adjust the storage levels of cold and hot data, including: The overall resources obtained are used as the input items of the fuzzy logic using the anomaly index QS and the throughput fluctuation value LM, and the storage hierarchy priority R of cold data and hot data is used as the output item of the fuzzy logic. Define a fuzzy set based on the overall resource usage anomaly index, throughput fluctuation value, and storage hierarchy priority of cold and hot data, and construct fuzzy rules. The fuzzy inference method is used to process the overall resource usage anomaly index QS and throughput fluctuation value LM of the input items, and the storage level priority R of cold data and hot data is calculated by the preset fuzzy rules. The fuzzy inference result is converted into a precise numerical output R by the defuzzification method, which represents the storage level priority of cold data and hot data. Based on the storage priority R of the defuzzified cold and hot data, they are divided into high priority, medium priority, and low priority. The storage level of cold and hot data is automatically adjusted according to the priority level of R.

2. The electronic record management method according to claim 1, characterized in that: In S2, a network bandwidth usage frequency anomaly index is generated based on the extracted network bandwidth usage frequency anomaly characteristics. The method for obtaining the network bandwidth usage frequency anomaly index is as follows: Collect network bandwidth usage data for the system over m different time periods, including bandwidth utilization rate and bandwidth occupancy rate; standardize the network bandwidth usage data and label it as Z; based on the standardized data, calculate the covariance matrix of the network bandwidth usage data, expressed as: In the formula, This is the bandwidth usage data for the i-th data point. Let X be the mean of all data points, Cov(X) be the covariance matrix of the bandwidth usage data, and T be the matrix transpose. Eigenvalue decomposition is performed on the covariance matrix to obtain eigenvalues ​​and eigenvectors. The eigenvalues ​​represent the variance of the data explained by each principal component, and the eigenvectors represent the directions of the principal components. The expression is: In the formula, It is the i-th eigenvector, representing the direction of the i-th principal component. The i-th eigenvalue is used to select principal components based on their magnitudes. The cumulative variance contribution rate is calculated using the following expression: ; k is the number of principal components selected, and q is the total number of features. Using the selected principal components, the score for each data point along the principal component direction is calculated, expressed as: In the formula, This is the score of the i-th principal component. Based on the principal component scores and their weights, the network bandwidth usage frequency anomaly index for each time period is calculated, expressed as: In the formula, The weight of the i-th principal component is equal to the eigenvalue. The proportion of the total feature values, SC is the network bandwidth usage frequency anomaly index.

3. The electronic record management method according to claim 2, characterized in that: In S3, the waiting time deviation index and the network bandwidth usage frequency anomaly index are converted into a first feature vector. The first feature vector is used as the input of the machine learning model. The machine learning model uses the prediction of the weight assignment labels of the resource contention usage data in each monitoring time interval as the prediction objective, and minimizes the sum of prediction errors of the weight assignment labels of the resource contention usage data in all monitoring time intervals as the training objective. The machine learning model is trained until the sum of prediction errors converges and the model training stops. The weight assignment of the resource contention usage data in each monitoring time interval is determined according to the model output. The machine learning model is a multinomial regression model. The overall resource usage anomaly index is obtained by weighted averaging the weight assignments of the resource contention usage data in each monitoring time interval.

4. The electronic record management method according to claim 3, characterized in that: In S4, after analyzing the fluctuation frequency of read / write throughput data within each monitoring time interval, the throughput fluctuation value of read / write throughput data within each monitoring time interval is determined, specifically: Determine the size Y of the sliding window, collect the system's read and write throughput data in each time interval, and denot it as... Where x is the index of the time interval, the average throughput is calculated within each sliding window. The length of the sliding window is d, and its expression is: ; Calculate the throughput change between adjacent sliding windows, i.e., calculate the absolute rate of change, expressed as: In the formula, This is the absolute rate of change; based on the calculated rate of change in throughput. The throughput fluctuation value is calculated using the following expression: In the formula, j is the number of sliding windows, and LM is the throughput fluctuation value.

5. An electronic records management system, used to implement the electronic records management method according to any one of claims 1-4, characterized in that: It includes a monitoring module, a feature extraction module, a comprehensive analysis module, a throughput fluctuation analysis module, and a fuzzy logic decision-making module; Monitoring module: When performing data migration operations, it determines several monitoring time intervals based on the scale of data migration and system capacity, and obtains read and write throughput data in real time within each monitoring time interval through data transmission monitoring tools, and obtains resource contention and usage data in real time within each monitoring time interval through system performance monitoring tools. Feature extraction module: After preprocessing the resource contention and usage data within each monitoring time interval, feature extraction is performed to extract the I / O wait time deviation features and network bandwidth usage frequency abnormal features between different tasks. The comprehensive analysis module analyzes the I / O wait time deviation characteristics and network bandwidth usage frequency anomaly characteristics extracted from different tasks. It then determines the weight of the resource contention usage data in each monitoring time interval and calculates the overall resource usage anomaly index by weighted averaging of the weighted values ​​of the resource contention usage data in each monitoring time interval. Throughput Fluctuation Analysis Module: After analyzing the fluctuation frequency changes of read and write throughput data within each monitoring time interval, the throughput fluctuation value of read and write throughput data within each monitoring time interval is determined. Fuzzy logic decision module: Based on the obtained overall resource usage anomaly index and throughput fluctuation value, the module uses fuzzy logic to analyze the data and then dynamically adjusts the storage levels of cold and hot data.

Images