Data storage management system and method based on big data

Abstract

The invention relates to the technical field of storage optimization, in particular to a data storage management system and method based on big data, and the system comprises an efficiency monitoring module, a medium evaluation module, a decision generation module, a strategy execution module and an effect feedback module. According to the method, input and output delay, a medium health attenuation curve and a bandwidth fluctuation map are collected, a dynamic feature space is constructed, the stability of the system is judged in combination with a Lyapunov index, early warning of performance abnormity is achieved, health indexes such as an erasing cycle, a seeking error rate and a charge retention rate are fused, and the access frequency and the load pressure are combined. Calculating a compensation coefficient, improving heterogeneous medium cooperation efficiency, pre-migration evaluation cost and topology adjustment income, improving resource configuration accuracy, dynamically adjusting hot and cold data distribution and node load, enhancing system scheduling flexibility, comparing throughput, medium wear rate and synchronous delay, and quantifying stability gain brought by configuration change; and a closed-loop process from identification to feedback is constructed.

Description

Technical Field

[0001] The present invention relates to the field of storage optimization technology, and in particular to a data storage management system and method based on big data. Background Art

[0002] The field of storage optimization encompasses storage resource management, data access efficiency improvement, and storage media performance tuning. Its core focus is on a systematic technical approach encompassing three dimensions: storage system architecture design, data distribution strategy optimization, and storage device performance evaluation. This area builds a dynamic storage resource allocation model, implements storage virtualization layer reconstruction, and establishes a data hot and cold tiering mechanism, forming a comprehensive technical system encompassing storage network topology optimization, storage device health monitoring, and storage capacity prediction and early warning.

[0003] The big data-based data storage management system and method addresses technical issues such as low cross-regional storage node coordination efficiency, insufficient hybrid storage media adaptability, and high data access path redundancy in scenarios with hundreds of petabytes of unstructured data. This technical solution optimizes the storage area network management protocol to achieve unified management of heterogeneous storage devices, employs a load balancing algorithm for intelligent block-level data distribution, applies a storage media performance optimization algorithm to adjust the IO scheduling strategy for hybrid SSD and HDD arrays, and combines storage resource profiling modeling technology to build a multi-dimensional storage performance evaluation system.

[0004] In cross-regional storage scenarios involving hundreds of petabytes of unstructured data, existing systems often rely on static allocation strategies and fixed-path access mechanisms. These systems fail to respond in real time to dynamic factors such as changes in input and output latency, storage media aging, and network bandwidth fluctuations, making it difficult to identify performance bottlenecks promptly. Existing storage resource management often relies on static profiling and periodic testing, failing to accurately match the health of mixed storage media. This can lead to persistent calls to degraded devices on heavily loaded nodes, significantly increasing access response times. Data migration strategies often rely on single-dimensional evaluation metrics, lacking a comprehensive assessment of overall system benefits. This can result in migration benefits insufficient to offset migration costs, leading to decreased resource utilization. Furthermore, the lack of a closed-loop performance evaluation mechanism after optimization execution prevents sufficient quantitative evidence to support future configuration adjustments, hindering the system's ability to continuously evolve. For example, certain cross-region synchronization operations, despite successful execution, fail to improve throughput due to bandwidth bottlenecks. Instead, they increase device wear, causing performance fluctuations and significantly reducing overall system stability. Summary of the Invention

[0005] The purpose of the present invention is to solve the shortcomings of the prior art and to propose a data storage management system and method based on big data.

[0006] In order to achieve the above objectives, the present invention adopts the following technical solutions: A data storage and management system based on big data includes: The performance monitoring module collects storage node input and output delay sequences, media health curves, and bandwidth fluctuation maps to construct a storage performance feature space. It calculates the Lyapunov exponent to monitor stability and generates a storage performance chaos warning signal when the exponent continuously exceeds a preset threshold. The medium evaluation module collects the erase / write cycle count, seek error rate, and charge retention rate based on the storage efficiency chaos warning signal, and calculates the medium performance degradation compensation coefficient based on the data access frequency distribution and the storage pool load pressure value, and outputs a hybrid storage medium adaptation decision table; The decision generation module evaluates the data migration cost, calculates the predicted value of the storage topology reconstruction benefit, and generates a data dynamic distribution strategy based on the hybrid storage medium adaptation decision table; The policy execution module performs cold and hot data media migration, storage node load rebalancing, and redundant copy topology optimization based on the data dynamic distribution strategy, and generates a storage topology optimization configuration; The effect feedback module monitors the cluster input and output throughput, media wear rate and cross-domain synchronization delay based on the storage topology optimization configuration, compares and analyzes the baseline values ​​before and after optimization to quantify the stability gain, and outputs the storage optimization performance evaluation results.

[0007] As a further solution of the present invention, the storage efficiency chaos warning signal includes input and output delay sequence, storage medium health attenuation curve, network bandwidth fluctuation map, Lyapunov exponent, and system stability threshold; the hybrid storage medium adaptation decision table includes erase and write cycle count, seek error rate, charge retention rate, data access frequency distribution, storage pool load pressure value, and medium performance degradation compensation coefficient; the data dynamic distribution strategy includes data migration cost, storage topology reconstruction benefit prediction value, and hybrid storage medium adaptation decision table; the storage topology optimization configuration includes cross-media migration of hot and cold data, storage node load balancing adjustment, and redundant copy topology reconstruction operation; the storage optimization efficiency evaluation results include input and output throughput, medium wear rate, and cross-domain synchronization delay.

[0008] As a further solution of the present invention, the performance monitoring module includes: The delay data acquisition submodule obtains the input and output request response delays of distributed storage nodes, the health change curve of the storage medium during its operation cycle, and the bandwidth fluctuation trajectory during network transmission. It extracts the time series fluctuation characteristic values ​​of each type of data, classifies and integrates the fluctuation characteristic values ​​according to a unified time standard, and constructs an indicator combination data sequence to generate a storage efficiency fluctuation parameter sequence. The feature space construction submodule extracts the index deviation trend, variation range, and fluctuation interval under different time segments based on the storage efficiency fluctuation parameter sequence, performs group statistics on the trend data based on time period division, and constructs a mapping relationship between features based on the variation density and direction difference of the grouped trend values ​​to generate the storage efficiency dynamic feature space mapping degree; The chaos threshold warning submodule calls the dimensional trend data in the storage efficiency dynamic feature space mapping, identifies continuous abnormal fluctuation sections according to the stability standard, counts the duration and frequency characteristics, and combines the density of the offset trend to generate a storage efficiency chaos warning signal.

[0009] As a further solution of the present invention, the medium evaluation module includes: The health parameter acquisition submodule obtains the storage efficiency chaos warning signal, collects three types of media health indicators, namely erase and write cycle count, seek error rate and charge retention rate, from distributed storage nodes, classifies and organizes them by storage medium type, establishes an indicator association mapping based on the node identifier, and generates a media health data set; The adaptation pressure analysis submodule, based on the medium health data group, calls the data access frequency and storage pool load pressure value, jointly divides the frequency distribution and pressure interval of the node data block, marks the access density under different pressure conditions, and generates a frequency-pressure coupling identification interval value; The fading compensation calculation submodule identifies the interval value according to the frequency-voltage coupling, extracts the medium health parameters corresponding to the interval, constructs the change trend according to the interval characteristics, evaluates the performance deviation degree of the indicator combination in the interval, and generates a hybrid storage medium adaptation decision table.

[0010] As a further solution of the present invention, the decision generation module includes: The migration cost evaluation submodule obtains the access frequency of the data block, the node input and output occupancy and the medium capacity status based on the hybrid storage medium adaptation decision table, calls the number of access paths, the migration distance between nodes and the bandwidth occupancy, calculates the migration efficiency, analyzes the resource consumption of the data block under the path and the path redundancy, and generates the migration cost interval value; The benefit prediction calculation submodule calls the target node's available capacity, write rate, and current load status based on the migration cost interval value, analyzes the node load change trend and access throughput change direction, establishes a difference sequence between benefits and costs, and generates a storage topology reconstruction benefit prediction value; The distribution strategy generation submodule calls the storage topology reconstruction benefit prediction value, obtains the current node data block distribution quantity and medium matching status, analyzes the distribution density change between nodes and the medium adaptation interval, determines the corresponding relationship between data distribution and benefit, and generates a data dynamic distribution strategy.

[0011] As a further solution of the present invention, the specific calculation formula for calculating the migration efficiency is: ; in, represents the comprehensive evaluation value of the migration efficiency from node z to x, represents the physical migration distance from source node z to target node x, represents the available bandwidth from node z to x, represents the current input and output load rate of source node z, represents the current input and output load rate of the target node x, ε represents the smoothing constant, represents the weight coefficient of the migration task in the mth time window, represents the duration of the mth time window, represents the path contention factor of the mth time window, γ represents the basic bandwidth reservation, and n represents the total number of statistical time windows.

[0012] As a further solution of the present invention, the policy execution module includes: The data migration submodule obtains the access frequency, access latency, and throughput information of hot and cold data on different storage media based on the dynamic data distribution strategy, determines the access characteristic deviation status based on the corresponding relationship between the access frequency and the latency threshold, calculates the load balancing evaluation value, and selects the migration path based on the throughput corresponding to the data and the available capacity of the node, and generates the migration path interval value; The load adjustment submodule extracts the load pressure, concurrent access volume and read / write parallelism of the target node based on the migratable path interval value, and selects a set of paths that meet the load balancing conditions based on the load difference and parallelism distribution between nodes to generate a balanced load distribution coefficient; The replica reconstruction submodule calls the node distribution, cross-domain synchronization frequency and replication delay information of the replica in the topology based on the balanced load distribution coefficient, determines the adjustable replica set according to the node density and synchronization frequency, determines the reconstruction order in combination with the replication delay sorting, and generates a storage topology optimization configuration.

[0013] As a further solution of the present invention, the specific calculation formula for calculating the load balancing evaluation value is: ; in, Represents the load balancing evaluation value, represents the absolute value of the load difference between node i and node j, Represents the arithmetic mean of the read and write parallelism of all nodes, represents the standard deviation of the load differences between nodes, N represents the total number of nodes in the cluster, and α represents the adjustment coefficient based on the system architecture. Represents the read and write parallelism of the kth node.

[0014] As a further solution of the present invention, the effect feedback module includes: The throughput monitoring submodule obtains the input and output throughput of the nodes in the storage topology optimization configuration, calls the baseline throughput data of the corresponding nodes before the configuration, compares the current throughput with the baseline state based on the data unit volume in the same time period, and summarizes the comparison results of all nodes to obtain the throughput balance value; The wear analysis submodule obtains the wear rate sampling results and operation cycle parameters of the storage node's corresponding medium based on the throughput balance value, builds a proportional relationship between the scheduling frequency and the cycle, integrates the parameters of the node medium status, and obtains the medium operation load rate; The stability evaluation submodule calls the medium operation load rate to obtain the synchronization delay and transmission flow information during the cross-domain data synchronization process, combines the node distribution and bandwidth occupancy of the storage area, extracts the communication status indicators under the synchronization path, and compares the synchronization delay and the bandwidth status between nodes in different partitions to obtain the storage optimization efficiency evaluation results.

[0015] A data storage management method based on big data, comprising the following steps: S1: Monitors the input and output delay sequences of storage nodes, collects media health decay curves and network bandwidth fluctuation maps, constructs a dynamic feature space, and performs stability analysis. When feature space parameters exceed system thresholds, a storage efficiency chaos warning signal is generated. S2: Based on the storage efficiency chaos warning signal, three types of media indicators, namely, erase / write cycle count, seek error rate, and charge retention rate, are collected. Correlation analysis is performed based on data access frequency distribution and storage pool load pressure values, and a hybrid storage media adaptation decision table is output. S3: Calling the hybrid storage medium adaptation decision table, evaluating data migration latency loss, storage pool life loss, and topology reconstruction energy consumption, performing multi-dimensional evaluation and calculation, and generating a data dynamic distribution strategy; S4: Execute the data dynamic distribution strategy, implement hot and cold data cross-media migration operations, adjust storage node load balancing parameters, reconstruct the redundant replica distribution topology, and generate a storage topology optimization configuration; S5: Monitor the throughput fluctuations, media wear rate changes, and cross-domain synchronization delay data after the implementation of the storage topology optimization configuration, analyze the difference between indicators before and after optimization, and output storage optimization performance evaluation results.

[0016] Compared with the prior art, the advantages and positive effects of the present invention are: In the present invention, a dynamic feature space is constructed by collecting input and output delays, medium health attenuation curves and bandwidth fluctuation maps, and the Lyapunov exponent is combined to judge the system stability, realize early warning of performance anomalies, integrate health indicators such as erase and write cycles, seek error rate, and charge retention rate, and combine access frequency and load pressure to calculate the compensation coefficient, improve the collaborative efficiency of heterogeneous media, evaluate the cost and topology adjustment benefits before migration, improve the accuracy of resource allocation, dynamically adjust the hot and cold data distribution and node load, enhance the flexibility of system scheduling, compare throughput, medium wear rate and synchronization delay, quantify the stability gain brought by configuration changes, and construct a closed-loop process from identification to feedback. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a system flow chart of the present invention; Figure 2 is a system block diagram of the present invention; Figure 3 The figure is a flow chart of the steps of the method of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0019] In the description of the present invention, it should be understood that the terms "length," "width," "up," "down," "front," "back," "left," "right," "vertical," "horizontal," "top," "bottom," "inside," "outside," and the like, indicating positions or relationships, are based on the positions or relationships shown in the accompanying drawings and are intended only to facilitate the description of the present invention and simplify the description. They do not indicate or imply that the devices or elements referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limiting the present invention. Furthermore, in the description of the present invention, "plurality" means two or more, unless otherwise expressly and specifically defined.

[0020] See also Figure 1 , a data storage and management system based on big data includes: The performance monitoring module collects input and output delay sequences of distributed storage nodes, storage media health decay curves, and network bandwidth fluctuation maps to construct a dynamic feature space for storage performance and calculate the Lyapunov exponent of dimensional parameters. When the exponent continuously exceeds the system stability threshold, a storage performance chaos warning signal is generated. The media assessment module collects three types of media health indicators: erase / write cycle count, seek error rate, and charge retention rate, based on storage efficiency chaos warning signals. It then combines data access frequency distribution with storage pool load pressure values ​​to calculate the media performance degradation compensation coefficient and output a hybrid storage media adaptation decision table. The decision generation module uses the hybrid storage media adaptation decision table to evaluate data migration costs in multiple dimensions, calculate the predicted value of storage topology reconstruction benefits, and generate a dynamic data distribution strategy. The policy execution module performs operations such as cross-media migration of hot and cold data, load balancing adjustment of storage nodes, and topology reconstruction of redundant replicas based on the data dynamic distribution strategy, thus generating an optimized storage topology configuration. The effect feedback module continuously monitors three core indicators: cluster input and output throughput, media wear rate, and cross-domain synchronization delay based on storage topology optimization configuration. It quantifies the stability gain by comparing and analyzing the baseline values ​​before and after optimization, and outputs the storage optimization performance evaluation results.

[0021] Storage efficiency chaos warning signals include input and output delay sequence, storage medium health decay curve, network bandwidth fluctuation map, Lyapunov exponent, and system stability threshold. The hybrid storage medium adaptation decision table includes erase and write cycle count, seek error rate, charge retention rate, data access frequency distribution, storage pool load pressure value, and medium performance degradation compensation coefficient. The data dynamic distribution strategy includes data migration cost, storage topology reconstruction benefit prediction value, and hybrid storage medium adaptation decision table. The storage topology optimization configuration includes cross-media migration of hot and cold data, storage node load balancing adjustment, and redundant copy topology reconstruction operations. The storage optimization efficiency evaluation results include input and output throughput, medium wear rate, and cross-domain synchronization delay.

[0022] See also Figure 2 , the performance monitoring module includes: The delay data acquisition submodule obtains the input and output request response delays of distributed storage nodes, the health change curve of the storage medium during its operation cycle, and the bandwidth fluctuation trajectory during network transmission. It extracts the time series fluctuation characteristic values ​​of each type of data, classifies and integrates all fluctuation characteristic values ​​according to a unified time standard, and constructs an indicator combination data sequence to generate a storage efficiency fluctuation parameter sequence. The delay data acquisition submodule records the request and response time of I / O operations from distributed storage nodes in real time. It is necessary to configure a timestamp recording mechanism on each node to mark the start and end time of each read and write operation, and calculate the operation delay by the difference. For example, if the average response time of multiple I / O operations in a certain period is 10ms, the node service processing capacity can be evaluated based on this. During the operation cycle, the status parameters of the storage medium are collected, such as temperature (°C), number of P / E cycles, bad block count, etc. These data are segmented and extracted according to a 24h cycle, and a standard change curve is formed through a normalization method to facilitate alignment with other features. In terms of network data, tools such as iperf are used to record bandwidth changes (in Mbps) at 1s intervals. For example, For example, if the bandwidth values ​​within a certain period of time are 960, 875, 780, 960, and 920, respectively, the fluctuation amplitude is 180 Mbps, and then the fluctuation characteristics such as standard deviation and maximum value difference are calculated; fluctuation characteristics are extracted from the three types of data respectively, such as the variance of I / O response time, the frequency of bandwidth changes, and the average slope of the health indicator, and are embedded as feature items in a unified time axis, for example, the record number is once per minute, forming a three-dimensional data set arranged by time; then, weight ratios are set, such as 50% for delay characteristics, 30% for bandwidth, and 20% for medium status, and a comprehensive performance indicator sequence is generated through linear combination, that is, one performance value corresponds to each minute, ultimately forming a storage performance fluctuation parameter sequence covering the entire time period, which is used for downstream analysis model calls.

[0023] The feature space construction submodule extracts the indicator deviation trend, variation range, and fluctuation interval under different time segments based on the storage efficiency fluctuation parameter sequence. It then groups and counts the trend data based on the time period. It then constructs a mapping relationship between features based on the variation density and direction differences of the grouped trend values ​​to generate the storage efficiency dynamic feature space mapping degree. The feature space construction submodule divides the aforementioned fluctuation parameter sequence into time segments, for example, every 30 minutes. The mean value (unit homologous data indicator) and the difference between the maximum and minimum values ​​within each segment are calculated to describe the fluctuation amplitude of the data in that segment. The direction and intensity of the change in the mean value between adjacent time segments are analyzed. If the mean value of the previous segment is 80 and the mean value of the next segment is 120, the deviation trend is upward and the deviation amplitude is 40. Time segments with consistent deviation trends are grouped together, such as all upward trend segments are grouped as positively skewed. The average change density of the data within each group is calculated. For example, if the average change value per minute exceeds 20, it is classified as a high-density group. Using the change density within the group as a reference, the relative difference between different groups is calculated, expressed as the ratio of the average density difference between each group to its mean. For example, if the average density of two groups is 50 and 80, respectively, the difference is 30, and the average is 65, the relative difference is approximately 46%. This method constructs an inter-group trend difference mapping matrix, and then outputs the storage efficiency dynamic feature space mapping degree, which is used to represent the spatial correlation and mapping strength between features in different time segments and serves as the basic input for judging the evolution of fluctuation structure.

[0024] The chaos threshold warning submodule uses the dimensional trend data in the storage efficiency dynamic feature space mapping to identify continuous abnormal fluctuation segments according to stability standards, calculates the duration and frequency characteristics, and combines the intensity of the deviation trend to generate a storage efficiency chaos warning signal. The chaos threshold warning submodule extracts dimensional trend data from the feature space mapping results and sets a stability threshold θ (dimensionless, representing the mapping difference rate). If the trend difference values ​​of several consecutive time slices are all greater than θ (for example, if θ is set to 0.2 and all differences within 10 consecutive 1-minute periods are greater than 0.3), the segment is judged to be unstable and marked as an abnormal segment. The duration T (in minutes) and the number of periodic occurrences f (in times / hour) of each segment are calculated. If T is greater than 5 and f is greater than 2, it is considered a persistent high-frequency abnormal signal. At the same time, the trend density D is calculated for each abnormal segment, expressed as the sum of the absolute values ​​of the trend changes divided by the duration. For example, if the trend change values ​​are 30, 45, and 20 units, the sum is 95. If the duration is 6 minutes, D is 15.8. If it is greater than the set density threshold η = 12, the trend segment is marked as a high-density deviation segment. When T, f, and D all meet the set criteria, the chaos warning mechanism is triggered, and the starting time point of the segment and the corresponding trend dimension number are recorded for subsequent use in locating the source of the abnormal state.

[0025] See also Figure 2 , the media assessment module includes: The health parameter acquisition submodule obtains storage efficiency chaos warning signals and collects three types of media health indicators: erase and write cycle count, seek error rate, and charge retention rate from distributed storage nodes. It classifies and organizes these indicators by storage medium type, establishes an indicator association map based on node identification, and generates a media health data set. The health parameter collection submodule connects to multiple nodes in the distributed storage architecture and, by invoking the underlying media interface of each node, extracts three metrics: erase / write cycle count, seek error rate, and charge retention. The erase / write cycle count uses the Flash media management chip to read the cumulative erase / write count of the current storage block. For example, for a 64GB MLC storage medium with a maximum allowable P / E cycle of 3000, a page in a node has been used 2650 times, and the count is recorded as 2650. The seek error rate is calculated using the I / O error statistics interface provided by the main control chip. A 24-hour collection window is set, during which the total number of accesses and errors is recorded. For example, if 14 errors occur and the total number of accesses is 5000, the calculated seek error rate is 0.28%. The charge retention rate is evaluated based on the charge stability of the floating gate memory cell. The current charge retention capability of the storage block is compared to the initial value using historical records and simulated aging test results. For example, a critical reference value for charge retention is set at 80%, and the actual detection value for a particular block is 84%. After collecting the above three types of indicators, they are categorized and organized by storage media type, such as creating independent parameter sets for SLC, MLC, and TLC types. Each parameter is bound to a node identifier, such as registering the node number in the form of a UUID. Data field combinations are created, including the media type, node identifier, and the three health indicator values. These combinations are aggregated to form a complete media health data set for subsequent analysis and call-out.

[0026] The adaptive pressure analysis submodule uses the data access frequency and storage pool load pressure values ​​based on the media health data group to jointly divide the frequency distribution and pressure interval of the node data blocks, mark the access intensity under different pressure conditions, and generate the frequency-pressure coupling identification interval value; Based on the generated media health data set, the adaptive stress analysis submodule first retrieves access frequency information for each data block from distributed nodes. Using a fixed time window (e.g., hourly), it collects the number of accesses for each block within that timeframe. For example, if a data block on a node was accessed 12, 14, 15, 10, and 13 times in the last five hours, the average access frequency is 12.8 times per hour. Simultaneously, the node's current load stress value is collected, defined as the ratio of currently active I / O requests to the maximum concurrency. For example, if the maximum concurrency is 1000 IOPS and the current value is 580 IOPS, the stress value is 58%. Access frequency and stress values ​​are mapped to partitions. Access frequency can be divided into four intervals: 0-5, 6-10, 11-15, and 16-20. Stress is divided into three intervals: 0%-30%, 31%-60%, and 61%-100%. This data block on the node falls into the combined interval corresponding to frequency interval 11-15 and stress interval 31%-60%. This process is executed item by item for all nodes and their data blocks, and the frequency-pressure interval of each block is marked. The node identifier, block number, frequency interval and pressure interval are combined to form the frequency-pressure coupling identification interval value to construct a complete mapping reference set.

[0027] The fading compensation calculation submodule identifies interval values ​​based on frequency-voltage coupling, extracts the corresponding medium health parameters, constructs change trends based on interval characteristics, evaluates the performance deviation of the indicator combination in the interval, and generates a hybrid storage medium adaptation decision table; The fading compensation calculation submodule uses the frequency-voltage coupling identification interval as its basis and selects the media health parameters of the data blocks contained in each interval as input. For example, for a node block number within a specific identification interval, the block health parameters include 2650 erase / write cycles, a seek error rate of 0.28%, and a charge retention rate of 84%. The module then collects historical changes in each parameter based on the interval characteristics, setting a unit time step, such as updating once per hour, and collecting data at multiple time points. For example, for erase / write cycles of 2630, 2640, and 2650, the seek error rates are 0.24%, 0.26%, and 0.28%, respectively, and the charge retention rates are 86%, 85%, and 84%, respectively. The rate of change is extracted by time difference. For example, if the erase / write cycle increases by 10 cycles per hour, the seek error rate increases by 0.02% per hour, and the charge retention rate decreases by 1% per hour. These change rates are then compared with corresponding thresholds. For example, if the maximum allowable erase / write cycle value is 3000, and the difference from the current value is 350 cycles, the change rate accounts for approximately 2.86% of the remaining capacity. Based on the importance of the rate of change, different parameters are weighted, such as a weight of 0.4 for the erase / write cycle, 0.3 for the seek error rate, and 0.3 for the charge retention rate. These parameters are then combined to calculate a score, for example, a composite score of approximately 0.025. The above process is repeated for all data blocks in the frequency / voltage interval, sorted by score, to form a data table containing the node number, block number, score, interval, and recommended allocation for subsequent hybrid storage media adaptation.

[0028] See also Figure 2 , the decision generation module includes: The migration cost evaluation submodule uses the hybrid storage medium adaptation decision table to obtain the access frequency of data blocks, node input and output occupancy, and media capacity status, and calls the number of access paths, inter-node migration distance, and bandwidth occupancy. It calculates migration efficiency, analyzes the resource consumption and path redundancy of data blocks along the path, and generates a migration cost interval value. The specific calculation formula for calculating migration efficiency is: ; in, represents the comprehensive evaluation value of the migration efficiency from node z to x, represents the physical migration distance from source node z to target node x (unit: kilometers), represents the available bandwidth from node z to x (unit: Gbps), Represents the current input and output load rate of source node z (in percentage form), represents the current input and output load rate of the target node x (in percentage form), ε represents the smoothing constant (with a value of 10^-5), represents the weight coefficient of the migration task in the mth time window, Represents the duration of the mth time window (unit: seconds), represents the path contention factor of the mth time window (range: 0.1-1.0), γ represents the basic bandwidth reservation (unit: Gbps), and n represents the total number of statistical time windows; Physical migration distance The source node z coordinate is obtained by GPS coordinate calculation (30.2672°N, 97.7431°W), and the target node x coordinate is obtained by Haversine formula calculation (40.7128°N, 74.0060°W). =2780 km; Available bandwidth Obtained through real-time network monitoring tools, the current link bandwidth is =25Gbps; Input and output load rate and Collected through the node resource monitoring system, =68%, =42%; The smoothing constant ε=0.00001 is used to avoid the denominator being zero. The total number of time windows n=6 is determined by the 15-minute time window statistical period set by the system; Migration task weight coefficient Determined by the task priority algorithm, urgent tasks α1=0.9, regular tasks α2=0.6, and batch tasks α3=0.3. The duration of the time window Extracted from the system log, Δt1=900 seconds, Δt2=900 seconds, Δt3=900 seconds; Path competition factor Obtained through the link quality detection protocol, β1=0.3, β2=0.5, β3=0.7; The basic bandwidth reservation γ = 5Gbps, set according to the network management policy; Compute the first component: ; Compute the second component: ; final =13630.7+298.4=13929.1. This value reflects the comprehensive evaluation value of the migration efficiency between nodes. The larger the value, the higher the migration cost. When the number exceeds the preset threshold of 15,000, the system will determine that the migration path needs to be optimized and adjusted.

[0029] The benefit prediction calculation submodule uses the migration cost interval value, the target node's available capacity, write rate, and current load status to analyze the node load change trend and access throughput change direction, establishes a difference sequence between benefits and costs, and generates a storage topology reconstruction benefit prediction value. The revenue prediction calculation submodule obtains the current available capacity of the target node based on the migration cost interval. For example, if the remaining space of a node is 400G, assuming that the average size of each data block is 10G, it can support up to 40 blocks of data. The write rate is given by the device parameters. For example, if the write rate of a node is 150M per second, the current load situation is recorded at the same time, including CPU utilization of 70% and I / O occupancy of 65%. The trend when the node load increases is predicted through the historical load change curve. For example, if historical data shows that the CPU utilization increases by about 5% for every 10% increase in load, then when 4 new blocks of data are added, the CPU utilization is expected to increase to about 90%. The current throughput is calculated based on the write rate and I / O idle ratio. For example, if the current throughput is 150 M / s x 35% idle ratio, the result is approximately 52.5 M / s. If it is predicted that the newly added data blocks will increase the I / O usage to 80%, the idle ratio will drop to 20% and the throughput will drop to 30 M / s. The direction of throughput change brought about by migration can be determined from this. The difference between benefit and cost is the throughput improvement divided by the cost. For example, if the throughput improvement is 10 M / s and the cost is 3.5, the difference is 2.86. This value exceeds the set threshold of 2.5 and is considered an acceptable migration condition. The difference values ​​calculated for multiple nodes are sorted to obtain a list of predicted benefit values, such as 2.86 for node A, 1.25 for node B, and 3.10 for node C. This list is used to guide the selection of target nodes for subsequent distribution strategies.

[0030] The distribution strategy generation submodule calls the storage topology reconstruction profit prediction value, obtains the current node data block distribution quantity and medium matching status, analyzes the distribution density changes between nodes and the medium adaptation interval, determines the corresponding relationship between data distribution and profit, and generates a dynamic data distribution strategy; The distribution strategy generation submodule obtains the number of data blocks currently distributed on each node and its media type matching status based on the aforementioned profit prediction values, such as 3.10 for node A, 2.86 for node B, and 1.25 for node C. For example, node A has 45 blocks of data, node B has 30 blocks, and node C has 20 blocks. Node A is mainly configured with SSD, node B is SAS, and node C is SATA. Combined with the media adaptation standard such as the recommended distribution density of SSD is 0.08 to 0.12 blocks per G, the current distribution density is calculated. For example, node A currently has 45 blocks / 500G, which is 0.09 blocks per G, which is within the adaptation range. Node C is 0.025 blocks per G, which is far below the adaptation range. The lower limit is obtained by dividing the current density by the median of the interval and then multiplying it by the predicted benefit value to obtain the adaptation index. For example, the adaptation index for node A is 0.09 / 0.1×3.10, which is approximately 2.79; for node B, it is 0.06 / 0.075×2.86, which is approximately 2.29; and for node C, it is 0.025 / 0.05×1.25, which is approximately 0.625. The priority migration direction is determined based on the size of the adaptation index. For example, the data blocks with low frequency access in node A are migrated to node C, and the data blocks with high frequency access in node C are migrated to node A. This realizes dynamic adjustment of data blocks. Priority control and strategy selection are performed based on the path migration cost data. Finally, a set of distribution strategies are formed for the scheduling module to call.

[0031] See also Figure 2 , the policy execution module includes: The data migration submodule, based on the data dynamic distribution strategy, obtains the access frequency, access latency, and throughput information of hot and cold data on different storage media. It determines the access characteristic deviation status based on the corresponding relationship between access frequency and latency threshold, calculates the load balancing evaluation value, and selects the migration path based on the corresponding data throughput and node available capacity information to generate the migration path interval value. The specific calculation formula for calculating the load balancing evaluation value is: ; in, Represents the load balancing evaluation value, represents the absolute value of the load difference between node i and node j, Represents the arithmetic mean of the read and write parallelism of all nodes, represents the standard deviation of the load differences between nodes, N represents the total number of nodes in the cluster, and α represents the adjustment coefficient based on the system architecture. Represents the read and write parallelism of the kth node; Parameter definition and data acquisition node load difference absolute value : The node monitoring system collects the target node's CPU usage (range 0%-100%), memory usage (range 0%-100%), and network bandwidth usage (range 0%-100%), and weights 0.4, 0.3, and 0.3 to obtain the comprehensive load value. and ,calculate =| - |; Measured nodes 1, 2, and 3 The arithmetic mean of read and write parallelism is 65%, 72%, and 58% respectively. : Extract the number of read and write operations per second (IOPS) of each node from the distributed storage system log, nodes 1, 2, and 3 They are 200, 250, and 300 respectively. =(200+250+300) / 3=250; Load difference standard deviation :Based on ΔL12=7%, ΔL13=7%, ΔL23=14%, the mean is (7+7+14) / 3=9.33, and the variance is [(7-9.33)²+(7-9.33)²+(14-9.33)²] / 3=10.89, = =3.3=3.3; Adjustment coefficient (α): According to the AWS ECS cluster architecture document, set α=150. This value increases logarithmically with the expansion of node scale, and the baseline value is taken when N=3; Total number of nodes (N): N=3 obtained from the cluster management interface; Molecular calculation: ; Denominator calculation: ; The numerator of the second fraction is: ; The denominator of the second fraction is: ; ; Substitute the complete formula into: ; The result shows that the load balancing evaluation value is negative, reflecting that the combined effect of the current inter-node load difference (standard deviation 3.3) and the read-write parallelism distribution (square root term 415.96) exceeds the control range of the adjustment coefficient α. It is necessary to screen the path set with a smaller absolute value of H (|H|≤0.02 is the threshold), and give priority to paths with H approaching zero when generating the balanced load distribution coefficient.

[0032] The load adjustment submodule extracts the load pressure, concurrent access volume, and read / write parallelism of the target node based on the migratable path interval value. Based on the load difference and parallelism distribution between nodes, it selects the path set that meets the load balancing conditions and generates a balanced load distribution coefficient. The load adjustment submodule continues to screen the node resource status according to the migration path interval value. Each target node needs to extract the current system load pressure, including CPU utilization, IO wait ratio and memory occupancy rate, and form a weighted load value after integration. For example, if the CPU usage of a node is 60%, IO wait is 20%, and memory usage is 70%, the weights are set to 5:3:2, and the integrated load is 60×0.5+20×0.3+70×0.2=47. If the integrated load of another node is 52, the load difference between the two nodes is 5. The difference threshold is set to 10%. If the difference is lower than this ratio, it is considered a balancing node. Then analyze the concurrent access volume and read and write parallelism. For example, a node accesses 200 nodes per second. The load is 40 times, the number of read and write threads is 60, and the normalization is set to 0.5 and 0.6 respectively. The sum of the two is 1.1 as the node processing capacity index. Compared with the required index of the data block, such as 1.0, the difference is 0.1, and the error threshold is set to 0.3. The node meets the conditions. The set of all nodes that meet the requirements of load difference and capacity error at the same time constitutes the set of paths that meet the load balancing requirements. Finally, for these paths, by constructing a balanced distribution coefficient, using the inverse of the load difference and capacity error and screening and sorting, the path with the best balance is selected. For example, if a path has a difference of 0.07 and a capacity error of 0.1, the balance coefficient is the inverse of the sum of the two, which is 6.25. If it is the maximum value, it is selected as the final migration path.

[0033] The replica reconstruction submodule uses the balanced load distribution coefficient, the node distribution of replicas in the topology, the cross-domain synchronization frequency, and the replication delay information to determine the set of adjustable replicas based on the node density and synchronization frequency. It then determines the reconstruction order based on the replication delay sorting and generates an optimized storage topology configuration. Based on the balanced path, the replica reconstruction submodule further analyzes the node distribution in the current replica topology. It calculates the node density by calculating the ratio of the number of replicas to the number of nodes in each network region. For example, if a region contains 10 nodes, including 4 replicas, the density is 0.4. If the upper limit density is set to 0.4, the region needs to be adjusted if it exceeds the limit. It then extracts cross-domain synchronization frequency and replica replication delay data. For example, if the synchronization frequency is 25 times per hour and the replication delay is 45 milliseconds, the delay benchmark can be calculated by the average delay of other replicas at the node. For example, if the average is 35 milliseconds, the replication delay deviation of the node is 10 milliseconds. If the deviation threshold is 5 milliseconds, the node has high synchronization frequency and delay deviation, which meets the reconstruction conditions. The replicas that meet the conditions are further sorted according to the order of replication delay, and the replicas with larger delays are adjusted first. The reconstruction path needs to migrate to the target node with a density below the upper limit, a high balancing coefficient, and a moderate synchronization frequency. For example, if a replica migrates to a node with a density of 0.3, a synchronization frequency of 15 times / h, and a replication delay of 30 milliseconds, all of which meet all conditions, thus forming a new replica distribution configuration.

[0034] See also Figure 2 , the effect feedback module includes: The throughput monitoring submodule obtains the input and output throughput of nodes in the storage topology optimization configuration, calls the baseline throughput data of the corresponding node before the configuration, compares the current throughput with the baseline state based on the data unit volume in the same time period, and summarizes the comparison results of all nodes to obtain the throughput balance value. After obtaining the input and output throughput of the node, the throughput monitoring submodule needs to call the benchmark throughput data before configuration for comparison. First, through the data collection mechanism in the system, the amount of write and read data is collected from each node interface every 5 minutes. For example, node A records 240MB of write and 180MB of read in one cycle, with a total throughput of 420MB, which is converted to an average throughput rate of 84MB / min per minute. Then, the historical benchmark data of the node before the optimized configuration is called, for example, the benchmark value is 60MB / min, and the difference between the current and the benchmark is calculated to be 24MB / min. The difference is then normalized. If the maximum allowable fluctuation range is set to plus or minus 4 0 MB / min, and the normalized offset value is 0.6. This step is performed on all nodes to form a set of normalized difference values. For example, in a topology with five nodes, the corresponding offset values ​​are 0.6, −0.2, 0.1, −0.4, and −0.1. The average deviation of these values ​​is calculated to measure the overall throughput balance of the system. The degree of balance can be determined according to the set level standard. For example, a fluctuation between 0 and 0.2 is considered highly balanced, between 0.2 and 0.4 is moderately balanced, and greater than 0.4 is poorly balanced. In the above example, the standard deviation of the calculated offset values ​​is approximately 0.34, corresponding to the moderately balanced range. The current throughput balance level of the topology is finally determined.

[0035] The wear analysis submodule obtains the wear rate sampling results and operation cycle parameters of the storage node's corresponding medium based on the throughput balance value. Combined with the node scheduling frequency, it constructs the proportional relationship between scheduling frequency and cycle time, integrates the parameters of the node medium status, and obtains the medium operation load rate. After obtaining the throughput balance, the wear analysis submodule needs to further sample the wear rate and operating cycle of the media used by each storage node. Based on this, its operating load is evaluated. The specific operation includes reading the write life indicator of the media. For example, according to the sampling results of a certain node, the remaining life count of the storage media used by this node is 480, the current cumulative usage cycle is 30 days, and the daily write volume is 250GB. The calculated average daily write ratio is 0.5%, that is, the daily wear level is approximately 0.005. Multiplying this value by the operating cycle yields a total wear ratio of 15% for the entire cycle. The ratio is further constructed based on the scheduling frequency. For example, if the node is scheduled 28 times in 30 days, the scheduling frequency ratio is 93%. Multiplying the total wear ratio by the scheduling frequency ratio yields a load ratio of approximately 14%. After other nodes execute the same operation process, a unified operating load ratio dataset is formed. This data can be used by lower-level modules to make performance judgments.

[0036] The stability assessment submodule uses the media load rate to obtain synchronization delay and transmission flow information during cross-domain data synchronization. It then combines the node distribution and bandwidth usage of the storage area to extract communication status indicators along the synchronization path. It then compares synchronization delay and inter-node bandwidth status by partition to obtain storage optimization performance evaluation results. After invoking the media load factor, the stability assessment submodule must simultaneously extract latency and traffic information for cross-domain data transmission. The process involves monitoring the latency records of inter-node synchronization paths. For example, if the latency from node A to node B is 120 milliseconds and the data volume is 2.5 GB, the submodule will also query the network bandwidth of the node's region. Assume that the bandwidth in node A's region is 10 Gbps with a 25% utilization rate, and the bandwidth in node B's region is 5 Gbps with a 50% utilization rate. If there is a relay node C in the path, the delays from node A to node B via node C are 40 milliseconds and 80 milliseconds, respectively, for a total of 120 milliseconds. The stability of bandwidth usage is determined by calculating the average and standard deviation of bandwidth fluctuations during each period of the bandwidth measurement cycle. The fluctuation range is then determined to determine whether it exceeds the set stability threshold. For example, a fluctuation coefficient below 20% is considered stable. The latency level and bandwidth fluctuation are combined, and the node distribution density is then regionalized and compared. This ultimately forms a state matching result for the synchronization path, resulting in a stability level assessment and storage optimization performance output.

[0037] See also Figure 3 , a data storage management method based on big data, comprising the following steps: S1: Monitors the input and output delay sequences of storage nodes, collects media health decay curves and network bandwidth fluctuation maps, constructs a dynamic feature space, and performs stability analysis. When feature space parameters exceed system thresholds, a storage efficiency chaos warning signal is generated. S2: Based on the storage efficiency chaos warning signal, it collects three types of media indicators: erase and write cycle count, seek error rate, and charge retention rate. It then performs a correlation analysis based on the data access frequency distribution and storage pool load pressure value, and outputs a hybrid storage media adaptation decision table. S3: Calls the hybrid storage media adaptation decision table to evaluate data migration latency, storage pool lifespan, and topology reconstruction energy consumption. It performs multi-dimensional evaluation and calculations to generate a dynamic data distribution strategy. S4: Executes the data dynamic distribution strategy, implements cross-media migration of hot and cold data, adjusts storage node load balancing parameters, reconstructs the redundant replica distribution topology, and generates an optimized storage topology configuration. S5: Monitors throughput fluctuations, media wear rate changes, and cross-domain synchronization delay data after the implementation of the storage topology optimization configuration, analyzes the difference between indicators before and after optimization, and outputs storage optimization performance evaluation results.

[0038] The above are merely preferred embodiments of the present invention and do not limit the present invention in any other form. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A data storage and management system based on big data, characterized by: The system comprises: The performance monitoring module collects storage node input and output delay sequences, media health curves, and bandwidth fluctuation maps to construct a storage performance feature space. It calculates the Lyapunov exponent to monitor stability and generates a storage performance chaos warning signal when the exponent continuously exceeds a preset threshold. The medium evaluation module collects the erase / write cycle count, seek error rate, and charge retention rate based on the storage efficiency chaos warning signal, and calculates the medium performance degradation compensation coefficient based on the data access frequency distribution and the storage pool load pressure value, and outputs a hybrid storage medium adaptation decision table; The decision generation module evaluates the data migration cost, calculates the predicted value of the storage topology reconstruction benefit, and generates a data dynamic distribution strategy based on the hybrid storage medium adaptation decision table; The policy execution module performs cold and hot data media migration, storage node load rebalancing, and redundant copy topology optimization based on the data dynamic distribution strategy, and generates a storage topology optimization configuration; The effect feedback module monitors the cluster input and output throughput, media wear rate and cross-domain synchronization delay based on the storage topology optimization configuration, compares and analyzes the baseline values ​​before and after optimization to quantify the stability gain, and outputs the storage optimization performance evaluation results.

2. The data storage and management system based on big data according to claim 1, characterized in that: The storage efficiency chaos warning signal includes input and output delay sequence, storage medium health decay curve, network bandwidth fluctuation map, Lyapunov exponent, and system stability threshold. The hybrid storage medium adaptation decision table includes erase and write cycle count, seek error rate, charge retention rate, data access frequency distribution, storage pool load pressure value, and medium performance degradation compensation coefficient. The data dynamic distribution strategy includes data migration cost, storage topology reconstruction benefit prediction value, and hybrid storage medium adaptation decision table. The storage topology optimization configuration includes cross-media migration of hot and cold data, storage node load balancing adjustment, and redundant copy topology reconstruction operation. The storage optimization efficiency evaluation results include input and output throughput, medium wear rate, and cross-domain synchronization delay.

3. The data storage and management system based on big data according to claim 1, characterized in that: The performance monitoring module includes: The delay data acquisition submodule obtains the input and output request response delays of distributed storage nodes, the health change curve of the storage medium during its operation cycle, and the bandwidth fluctuation trajectory during network transmission. It extracts the time series fluctuation characteristic values ​​of each type of data, classifies and integrates the fluctuation characteristic values ​​according to a unified time standard, and constructs an indicator combination data sequence to generate a storage efficiency fluctuation parameter sequence. The feature space construction submodule extracts the index deviation trend, variation range, and fluctuation interval under different time segments based on the storage efficiency fluctuation parameter sequence, performs group statistics on the trend data based on time period division, and constructs a mapping relationship between features based on the variation density and direction difference of the grouped trend values ​​to generate the storage efficiency dynamic feature space mapping degree; The chaos threshold warning submodule calls the dimensional trend data in the storage efficiency dynamic feature space mapping, identifies continuous abnormal fluctuation sections according to the stability standard, counts the duration and frequency characteristics, and combines the density of the offset trend to generate a storage efficiency chaos warning signal.

4. The data storage and management system based on big data according to claim 3, characterized in that: The medium evaluation module includes: The health parameter acquisition submodule obtains the storage efficiency chaos warning signal, collects three types of media health indicators, namely erase and write cycle count, seek error rate and charge retention rate, from distributed storage nodes, classifies and organizes them by storage medium type, establishes an indicator association mapping based on the node identifier, and generates a media health data set; The adaptation pressure analysis submodule, based on the medium health data group, calls the data access frequency and storage pool load pressure value, jointly divides the frequency distribution and pressure interval of the node data block, marks the access density under different pressure conditions, and generates a frequency-pressure coupling identification interval value; The fading compensation calculation submodule identifies the interval value according to the frequency-voltage coupling, extracts the medium health parameters corresponding to the interval, constructs the change trend according to the interval characteristics, evaluates the performance deviation degree of the indicator combination in the interval, and generates a hybrid storage medium adaptation decision table.

5. The data storage and management system based on big data according to claim 4, characterized in that: The decision making module includes: The migration cost evaluation submodule obtains the access frequency of the data block, the node input and output occupancy and the medium capacity status based on the hybrid storage medium adaptation decision table, calls the number of access paths, the migration distance between nodes and the bandwidth occupancy, calculates the migration efficiency, analyzes the resource consumption of the data block under the path and the path redundancy, and generates the migration cost interval value; The benefit prediction calculation submodule calls the target node's available capacity, write rate, and current load status based on the migration cost interval value, analyzes the node load change trend and access throughput change direction, establishes a difference sequence between benefits and costs, and generates a storage topology reconstruction benefit prediction value; The distribution strategy generation submodule calls the storage topology reconstruction benefit prediction value, obtains the current node data block distribution quantity and medium matching status, analyzes the distribution density change between nodes and the medium adaptation interval, determines the corresponding relationship between data distribution and benefit, and generates a data dynamic distribution strategy.

6. The data storage and management system based on big data according to claim 5, characterized in that: The specific calculation formula for calculating the migration efficiency is: ； in, represents the comprehensive evaluation value of the migration efficiency from node z to x, represents the physical migration distance from source node z to target node x, represents the available bandwidth from node z to x, represents the current input and output load rate of source node z, represents the current input and output load rate of the target node x, ε represents the smoothing constant, represents the weight coefficient of the migration task in the mth time window, represents the duration of the mth time window, represents the path contention factor of the mth time window, γ represents the basic bandwidth reservation, and n represents the total number of statistical time windows.

7. The data storage and management system based on big data according to claim 5, characterized in that: The policy execution module includes: The data migration submodule obtains the access frequency, access latency, and throughput information of hot and cold data on different storage media based on the dynamic data distribution strategy, determines the access characteristic deviation status based on the corresponding relationship between the access frequency and the latency threshold, calculates the load balancing evaluation value, and selects the migration path based on the throughput corresponding to the data and the available capacity of the node, and generates the migration path interval value; The load adjustment submodule extracts the load pressure, concurrent access volume and read / write parallelism of the target node based on the migratable path interval value, and selects a set of paths that meet the load balancing conditions based on the load difference and parallelism distribution between nodes to generate a balanced load distribution coefficient; The replica reconstruction submodule calls the node distribution, cross-domain synchronization frequency and replication delay information of the replica in the topology based on the balanced load distribution coefficient, determines the adjustable replica set according to the node density and synchronization frequency, determines the reconstruction order in combination with the replication delay sorting, and generates a storage topology optimization configuration.

8. The data storage and management system based on big data according to claim 7, characterized in that: The specific calculation formula for calculating the load balancing evaluation value is: ； in, Represents the load balancing evaluation value, represents the absolute value of the load difference between node i and node j, Represents the arithmetic mean of the read and write parallelism of all nodes, represents the standard deviation of the load differences between nodes, N represents the total number of nodes in the cluster, and α represents the adjustment coefficient based on the system architecture. Represents the read and write parallelism of the kth node.

9. The data storage and management system based on big data according to claim 7, characterized in that: The effect feedback module includes: The throughput monitoring submodule obtains the input and output throughput of the nodes in the storage topology optimization configuration, calls the baseline throughput data of the corresponding nodes before the configuration, compares the current throughput with the baseline state based on the data unit volume in the same time period, and summarizes the comparison results of all nodes to obtain the throughput balance value; The wear analysis submodule obtains the wear rate sampling results and operation cycle parameters of the storage node's corresponding medium based on the throughput balance value, builds a proportional relationship between the scheduling frequency and the cycle, integrates the parameters of the node medium status, and obtains the medium operation load rate; The stability evaluation submodule calls the medium operation load rate to obtain the synchronization delay and transmission flow information during the cross-domain data synchronization process, combines the node distribution and bandwidth occupancy of the storage area, extracts the communication status indicators under the synchronization path, and compares the synchronization delay and the bandwidth status between nodes in different partitions to obtain the storage optimization efficiency evaluation results.

10. A data storage management method based on big data, characterized in that: The data storage management system based on big data according to any one of claims 1 to 9 comprises the following steps: S1: Monitors the input and output delay sequences of storage nodes, collects media health decay curves and network bandwidth fluctuation maps, constructs a dynamic feature space, and performs stability analysis. When feature space parameters exceed system thresholds, a storage efficiency chaos warning signal is generated. S2: Based on the storage efficiency chaos warning signal, three types of media indicators, namely, erase / write cycle count, seek error rate, and charge retention rate, are collected. Correlation analysis is performed based on data access frequency distribution and storage pool load pressure values, and a hybrid storage media adaptation decision table is output. S3: Calling the hybrid storage medium adaptation decision table, evaluating data migration latency loss, storage pool life loss, and topology reconstruction energy consumption, performing multi-dimensional evaluation and calculation, and generating a data dynamic distribution strategy; S4: Execute the data dynamic distribution strategy, implement hot and cold data cross-media migration operations, adjust storage node load balancing parameters, reconstruct the redundant replica distribution topology, and generate a storage topology optimization configuration; S5: Monitor the throughput fluctuations, media wear rate changes, and cross-domain synchronization delay data after the implementation of the storage topology optimization configuration, analyze the difference between indicators before and after optimization, and output storage optimization performance evaluation results.

Images