A big data-based data storage management system and method

Abstract

The application relates to the technical field of storage optimization, in particular to a data storage management system and method based on big data, which comprises an efficiency monitoring module, a medium evaluation module, a decision generation module, a strategy execution module and an effect feedback module.In the application, a dynamic characteristic space is constructed by collecting input and output delays, medium health attenuation curves and bandwidth fluctuation atlases, system stability is judged by combining Lyapunov indexes, performance abnormality early warning is realized, health indexes such as erasing and writing cycles, track error rates and charge retention rates are fused, compensation coefficients are calculated by combining access frequencies and load pressures, heterogeneous medium collaborative efficiency is improved, migration cost and topological adjustment benefits are evaluated before migration, resource allocation accuracy is improved, hot and cold data distribution and node load are dynamically adjusted, system scheduling flexibility is enhanced, stability gains brought by configuration changes are quantified by comparing throughput, medium wear rates and synchronization delays, and a closed-loop process from identification to feedback is constructed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of storage optimization, in particular to a data storage management system and method based on big data. BACKGROUND

[0002] The technical field of storage optimization includes storage resource management, data access efficiency improvement, and storage medium performance optimization. The core content of this field focuses on three dimensions: storage system architecture design, data distribution strategy optimization, and storage device performance evaluation. This field builds a dynamic storage resource allocation model, implements storage virtualization layer reconstruction, establishes a data hot and cold classification mechanism, and forms a complete technical system covering storage network topology optimization, storage device health monitoring, and storage capacity prediction and early warning.

[0003] Among them, the data storage management system and method based on big data refers to the solution to the technical issues such as low coordination efficiency of cross-regional storage nodes, insufficient adaptability of mixed storage media, and high redundancy of data access paths in the scenario of non-structured data storage of hundreds of PB. This technical solution realizes the unified management of heterogeneous storage devices through storage area network management protocol optimization, completes intelligent distribution of data blocks using load balancing algorithm, adjusts the IO scheduling strategy of SSD and HDD mixed array using storage medium performance optimization algorithm, and constructs a multi-dimensional storage performance evaluation system combined with storage resource portrait modeling technology.

[0004] In the cross-regional storage scenario of hundreds of PB of non-structured data, the existing system usually relies on static allocation strategy and fixed path access mechanism, and cannot respond to dynamic factors such as input / output delay changes, storage medium aging state, and network bandwidth fluctuations in real time, resulting in performance bottlenecks that are difficult to be identified in time. The existing storage resource management relies on static portrait and periodic detection, and cannot accurately match the health difference of mixed storage media, which may cause performance degradation devices on high-load nodes to be continuously called, resulting in a significant increase in access response time. The data migration strategy is based on a single-dimensional evaluation index, lacks comprehensive judgment of the overall benefit of the system, and may cause the migration benefit to be insufficient to offset the migration cost, resulting in a decrease in resource utilization. In addition, there is a lack of closed-loop performance evaluation mechanism after optimization, which cannot provide sufficient quantitative basis to support the next round of configuration adjustment, affecting the continuous evolution ability of the system. For example, although some cross-domain synchronization operations are completed, the bandwidth bottleneck cannot improve the throughput, but increases the device wear rate, causing performance to fluctuate repeatedly, and significantly reducing the overall stability of the system. SUMMARY

[0005] The purpose of the present application is to solve the shortcomings in the prior art and to provide a data storage management system and method based on big data.

[0006] To achieve the above object, the present application adopts the following technical scheme: a data storage management system based on big data comprises:

[0007] The performance monitoring module collects input and output delay sequence, medium health degree curve and bandwidth fluctuation atlas of the storage node, constructs a storage performance characteristic space, calculates Lyapunov index to monitor stability, and generates a storage performance chaos early warning signal when the index continuously exceeds a preset threshold value;

[0008] The medium evaluation module, based on the storage performance chaos early warning signal, collects the number of erase-write cycles, the track error rate and the charge retention rate, combines the data access frequency distribution and the storage pool load pressure value, calculates a medium performance degradation compensation coefficient, and outputs a hybrid storage medium adaptation decision table;

[0009] The decision generation module, based on the hybrid storage medium adaptation decision table, evaluates the data migration cost, calculates a storage topology reconstruction revenue prediction value, and generates a data dynamic distribution strategy;

[0010] The strategy execution module, based on the data dynamic distribution strategy, performs cold and hot data medium migration, storage node load rebalancing and redundant copy topology optimization, and generates a storage topology optimization configuration;

[0011] The effect feedback module, according to the storage topology optimization configuration, monitors the cluster input and output throughput, the medium wear rate and the cross-domain synchronization delay, compares and analyzes the stability gain of the benchmark value before and after optimization, and outputs a storage optimization performance evaluation result.

[0012] As a further scheme of the present application, the storage performance chaos early warning signal includes input and output delay sequence, storage medium health degree attenuation curve, network bandwidth fluctuation atlas, Lyapunov index and system stability threshold value, the hybrid storage medium adaptation decision table includes erase-write cycle count, track error rate, charge retention rate, data access frequency distribution, storage pool load pressure value and medium performance degradation compensation coefficient, the data dynamic distribution strategy includes data migration cost, storage topology reconstruction revenue prediction value and hybrid storage medium adaptation decision table, the storage topology optimization configuration includes cold and hot data cross-medium migration, storage node load balancing adjustment and redundant copy topology reconstruction operation, and the storage optimization performance evaluation result includes input and output throughput, medium wear rate and cross-domain synchronization delay.

[0013] As a further scheme of the present application, the performance monitoring module comprises:

[0014] The delay data collection submodule obtains the input-output request response delay of the distributed storage node, the health change curve in the running period of the storage medium, and the bandwidth fluctuation trajectory in the network transmission process, respectively extracts the time sequence fluctuation characteristic value of the data, classifies and integrates the fluctuation characteristic value according to a unified time standard, constructs an index combination data sequence, and generates a storage efficiency fluctuation parameter sequence;

[0015] The feature space construction submodule extracts the index offset trend, change range and fluctuation interval under the difference time segment according to the storage efficiency fluctuation parameter sequence, groups and counts the trend data based on time period division, constructs the mapping relationship between the features according to the change density and direction difference of the grouped trend value, and generates a storage efficiency dynamic feature space mapping degree.

[0016] The chaotic threshold early warning submodule calls the dimension trend data in the storage efficiency dynamic feature space mapping degree, identifies the continuous abnormal fluctuation paragraph according to the stability standard, counts the duration and frequency characteristics, and generates a storage efficiency chaotic early warning signal combined with the offset trend density.

[0017] As a further scheme of the application, the medium evaluation module comprises:

[0018] The health parameter collection submodule obtains the storage efficiency chaotic early warning signal, collects three types of medium health indicators of the erase-write cycle count, the seek error rate and the charge retention rate in the distributed storage node, classifies and organizes the indicators according to the storage medium type, establishes an index association mapping combined with the node identifier, and generates a medium health degree data group.

[0019] The adaptive pressure analysis submodule calls the data access frequency and storage pool load pressure value based on the medium health degree data group, jointly divides the frequency distribution and pressure interval of the node data block, marks the access intensity under the different pressure conditions, and generates a frequency-pressure coupling identification interval value.

[0020] The degradation compensation calculation submodule extracts the medium health parameters corresponding to the interval according to the frequency-pressure coupling identification interval value, constructs the change trend according to the interval characteristics, evaluates the performance offset degree of the index combination in the interval, and generates a mixed storage medium adaptive decision table.

[0021] As a further scheme of the application, the decision generation module comprises:

[0022] The migration cost evaluation submodule obtains the access frequency of the data block, the node input-output occupation condition and the medium capacity state based on the mixed storage medium adaptive decision table, calls the access path number, the inter-node migration distance and the bandwidth occupation condition, calculates the migration efficiency, analyzes the resource consumption and path redundancy degree of the data block under the path, and generates a migration cost interval value.

[0023] The benefit prediction calculation sub-module calls the available capacity of the target node, the write rate and the current load state according to the migration cost interval value, analyzes the node load change trend and the access throughput change direction, establishes the difference sequence between the benefit and the cost, and generates the storage topology reconstruction benefit prediction value;

[0024] The distribution strategy generation sub-module calls the storage topology reconstruction benefit prediction value, obtains the current node data block distribution quantity and the medium matching state, analyzes the node distribution density change and the medium adaptation interval, judges the corresponding relationship between the data distribution and the benefit, and generates the data dynamic distribution strategy.

[0025] As a further scheme of the application, the specific calculation formula of the migration efficiency is:

[0026] ;

[0027] Among them, represents the comprehensive evaluation value of the migration efficiency from the node z to the node x, represents the physical migration distance from the source node z to the target node x, represents the available bandwidth from the node z to the node x, represents the current input / output load rate of the source node z, represents the current input / output load rate of the target node x, and ε represents a smoothing constant, represents the migration task weight coefficient of the mth time window, represents the duration of the mth time window, represents the path competition factor of the mth time window, γ represents a basic bandwidth reservation, and n represents the total number of statistical time windows.

[0028] As a further scheme of the application, the strategy execution module comprises:

[0029] The data migration sub-module obtains the access frequency, access delay and throughput information of hot and cold data on different storage media based on the data dynamic distribution strategy, judges the access characteristic deviation state based on the corresponding relationship between the access frequency and the delay threshold, calculates the load balancing evaluation value, filters the migration path in combination with the throughput corresponding to the data and the available capacity information of the node, and generates the migratable path interval value;

[0030] The load adjustment sub-module extracts the load pressure, concurrent access quantity and read-write parallelism of the target node according to the migratable path interval value, filters the path set satisfying the load balancing condition according to the load difference and parallelism distribution between nodes, and generates the balanced load distribution coefficient;

[0031] The copy reconstruction submodule calls node distribution of the copy in the topology, cross-domain synchronization frequency and replication delay information based on the balanced load distribution coefficient, determines the reconstruction order according to node density and synchronization frequency, and generates a storage topology optimization configuration.

[0032] As a further scheme of the present application, the specific calculation formula of the calculation load balancing evaluation value is:

[0033] ;

[0034] Among them, represents the load balancing evaluation value, represents the absolute value of the load difference between node i and node j, represents the arithmetic mean of the read-write parallelism of all nodes, represents the standard deviation of the load difference between nodes, N represents the total number of nodes in the cluster, and a represents an adjustment coefficient based on the system architecture, represents the read-write parallelism of the kth node.

[0035] As a further scheme of the present application, the effect feedback module comprises:

[0036] The throughput monitoring submodule obtains the input and output throughput of the nodes in the storage topology optimization configuration, calls the baseline throughput data of the corresponding nodes before configuration, compares the current throughput with the baseline state according to the data unit quantity in the same time period, and obtains the throughput balancing degree value by collecting and integrating the comparison results of all nodes;

[0037] The wear analysis submodule obtains the wear rate sampling result and the running cycle parameter of the medium corresponding to the storage node according to the throughput balancing degree value, constructs the proportional relationship between the scheduling frequency and the cycle in combination with the node scheduling frequency, integrates the parameters of the node medium state, and obtains the medium running load rate;

[0038] The stability evaluation submodule calls the medium running load rate, obtains the synchronization delay and transmission flow information in the cross-domain data synchronization process, extracts the communication state index of the synchronization path in combination with the node distribution and bandwidth occupation of the storage area, and performs partition comparison on the synchronization delay and the bandwidth state between nodes to obtain the storage optimization efficiency evaluation result.

[0039] A data storage management method based on big data comprises the following steps:

[0040] S1: Monitor the storage node input and output delay sequence, collect the medium health degree attenuation curve and network bandwidth fluctuation atlas, construct a dynamic characteristic space and perform stability analysis, and generate a storage efficiency chaos warning signal when the characteristic space parameter exceeds the system threshold.

[0041] S2: Based on the storage performance chaos early warning signal, collect three types of medium indexes of erase-write cycle count, seek error rate and charge retention rate, and perform correlation analysis on the data access frequency distribution and storage pool load pressure value, and output a mixed storage medium adaptation decision table;

[0042] S3: Call the mixed storage medium adaptation decision table, evaluate data migration delay loss, storage pool life loss and topology reconstruction energy consumption, perform multi-dimensional evaluation and operation, and generate a data dynamic distribution strategy;

[0043] S4: Perform the data dynamic distribution strategy, implement cold and hot data cross-medium migration operation, adjust the storage node load balancing parameter, reconstruct the redundant copy distribution topology, and generate a storage topology optimization configuration;

[0044] S5: Monitor the throughput fluctuation, medium wear rate change and cross-domain synchronization delay data after the implementation of the storage topology optimization configuration, perform pre- and post-optimization index difference analysis, and output a storage optimization performance evaluation result.

[0045] Compared with the prior art, the advantages and positive effects of the present application are:

[0046] In the present application, by collecting input-output delay, medium health attenuation curve and bandwidth fluctuation atlas, a dynamic characteristic space is constructed, the system stability is judged in combination with Lyapunov index, performance abnormality early warning is realized, health indexes such as erase-write cycle, seek error rate and charge retention rate are fused, access frequency and load pressure are combined, compensation coefficient is calculated, heterogeneous medium cooperation efficiency is improved, migration cost and topology adjustment benefit are evaluated before migration, resource configuration precision is improved, hot and cold data distribution and node load are dynamically adjusted, system scheduling flexibility is enhanced, throughput, medium wear rate and synchronization delay are compared, stability gain brought by configuration change is quantified, and a closed-loop process from identification to feedback is constructed. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 The system flowchart of the present application;

[0048] Figure 2 The system block diagram of the present application;

[0049] Figure 3 The method step flowchart of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application, and are not used to limit the present application.

[0051] In the description of the present application, it should be understood that the terms "length", "width", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, in the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly and specifically limited.

[0052] Please refer to Figure 1 A data storage management system based on big data comprises:

[0053] The performance monitoring module collects the input-output delay sequence of the distributed storage node, the storage medium health degree attenuation curve, and the network bandwidth fluctuation spectrum, constructs a storage performance dynamic characteristic space, calculates the Lyapunov index of the dimension parameter, and generates a storage performance chaos early warning signal when the index continuously exceeds the system stability threshold;

[0054] The medium evaluation module collects three types of medium health indicators, including the erase-write cycle count, the seek error rate, and the charge retention rate, based on the storage performance chaos early warning signal, combines the data access frequency distribution and the storage pool load pressure value, calculates the medium performance degradation compensation coefficient, and outputs a hybrid storage medium adaptation decision table;

[0055] The decision generation module generates a data dynamic distribution strategy based on the hybrid storage medium adaptation decision table, multi-dimensionally evaluates the data migration cost, calculates the storage topology reconstruction benefit prediction value, and generates a data dynamic distribution strategy;

[0056] The strategy execution module performs cold and hot data cross-medium migration, storage node load balancing adjustment, and redundant copy topology reconstruction operation based on the data dynamic distribution strategy, and generates a storage topology optimization configuration;

[0057] The effect feedback module continuously monitors three core indicators, including the cluster input-output throughput, the medium wear rate, and the cross-domain synchronization delay, according to the storage topology optimization configuration, quantifies the stability gain by comparing and analyzing the benchmark values before and after optimization, and outputs the storage optimization performance evaluation result.

[0058] The storage performance chaos early warning signal includes input-output delay sequence, storage medium health degree attenuation curve, network bandwidth fluctuation atlas, Lyapunov exponent, system stability threshold, the mixed storage medium adaptation decision table includes erase-write cycle count, seek error rate, charge retention rate, data access frequency distribution, storage pool load pressure value, medium performance degradation compensation coefficient, the data dynamic distribution strategy includes data migration cost, storage topology reconstruction benefit prediction value, mixed storage medium adaptation decision table, the storage topology optimization configuration includes cold-hot data cross-medium migration, storage node load balancing adjustment, redundant copy topology reconstruction operation, and the storage optimization performance evaluation result includes input-output throughput, medium wear rate, cross-domain synchronization delay.

[0059] Please refer to Figure 2 The performance monitoring module includes:

[0060] The delay data acquisition submodule obtains the input-output request response delay of the distributed storage node, the health change curve in the running period of the storage medium, and the bandwidth fluctuation trajectory in the network transmission process, respectively extracts the time sequence fluctuation characteristic value of the data, classifies and integrates all the fluctuation characteristic values according to the unified time standard, constructs the index combination data sequence, and generates the storage performance fluctuation parameter sequence;

[0061] The delay data acquisition submodule records the request and response time of I / O operation in real time from the distributed storage node. The timestamp recording mechanism needs to be configured in each node, and the start and end time of each read-write operation is marked respectively. The delay of the operation is calculated by the difference value. For example, the average response time of multiple I / O operations in a period is 10ms, which can be used to evaluate the node service processing capacity. In the running period, the state parameters of the storage medium are collected, such as temperature (°C), P / E cycle number, bad block count, etc. These data are segmented according to 24h period, and the standard change curve is formed by the normalization method, which is convenient for alignment processing with other characteristics. In terms of network data, tools such as iperf are used to record the bandwidth change (unit: Mbps) at 1s intervals. For example, the bandwidth values in a period of time are 960, 875, 780, 960, and 920 respectively, and the fluctuation range is 180Mbps. Then the standard deviation, maximum difference and other fluctuation characteristics are calculated. The fluctuation characteristics of the three types of data are extracted respectively, such as the variance of I / O response time, the frequency of bandwidth change, and the average slope of health indicators, which are embedded into the unified time axis, for example, the record number is recorded once a minute, forming a three-dimensional data set arranged by time. Thereafter, a weight ratio is set, for example, the delay characteristic is 50%, the bandwidth is 30%, and the medium state is 20%. The comprehensive performance index sequence is generated by linear combination, that is, each minute corresponds to a performance value, and finally a storage performance fluctuation parameter sequence covering the entire time period is formed. The sequence is used for downstream analysis model calling.

[0062] The feature space construction submodule extracts the index offset trend, change range and fluctuation interval under the difference time segment according to the storage performance fluctuation parameter sequence, performs grouped statistics on the trend data based on time period division, constructs the mapping relationship between the features according to the change density and direction difference of the grouped trend values, and generates a storage performance dynamic feature space mapping degree;

[0063] The feature space construction submodule divides the foregoing fluctuation parameter sequence by time, for example, every 30 min as a paragraph, respectively statistics the average value (unit homologous data index), maximum value and minimum value difference in each segment, for describing the fluctuation amplitude of the segment data; analyzes the change direction and intensity of the average value between adjacent time periods, if the average value of the previous segment is 80 and that of the next segment is 120, the offset trend is upward and the offset amplitude is 40; the time periods with consistent offset trend are grouped, for example, all upward trend segments are grouped into a positive bias group, the average change density of the internal data of each group is calculated, for example, if the average change value per minute exceeds 20, it is judged as a high density group; then, taking the change density in the group as a reference, the relative difference degree between different groups is calculated, which is represented by the ratio of the average density difference of each group to the average value, for example, if the average density of two groups is 50 and 80 respectively, the difference is 30, and the average is 65, the relative difference degree is about 46%; in this way, a trend difference mapping matrix between groups is constructed, and then a storage performance dynamic feature space mapping degree is output, which is used to represent the spatial correlation and mapping strength between different time period features, as a basic input for judging the evolution of fluctuation structure.

[0064] The chaotic threshold early warning submodule calls the trend data in the storage performance dynamic feature space mapping degree, identifies continuous abnormal fluctuation paragraphs according to the stability standard, statistics the duration and frequency characteristics, and generates a storage performance chaotic early warning signal combined with the offset trend density;

[0065] The chaotic threshold early warning submodule extracts the dimension trend data in the feature space mapping result, sets a stability threshold θ (dimensionless, representing the mapping difference rate), if the trend difference values of continuous time slices are all greater than θ, for example, θ is set to 0.2, and all the difference values in the continuous 10 1 min periods are all greater than 0.3, it is judged as an unstable segment, which is marked as an abnormal segment; the duration T (unit: min) of each segment and the periodic occurrence frequency f (unit: times / h) are calculated, if T is greater than 5 and f is greater than 2, it is regarded as a continuous high-frequency abnormal signal; at the same time, the trend density D of each abnormal segment is calculated, which is represented in the form of the sum of the absolute values of the trend changes divided by the duration, for example, the trend change values are 30, 45 and 20 units, the sum is 95, and if the duration is 6 min, D is 15.8, if it is higher than the set density threshold η = 12, the trend of this segment is marked as a high-density offset segment; when T, f and D all meet the set standard, the chaotic early warning mechanism is triggered, and the starting time point and the corresponding trend dimension number of the segment are recorded, which are used for positioning the source of the abnormal state subsequently.

[0066] Referring to Figure 2 The medium evaluation module comprises:

[0067] The health parameter acquisition submodule acquires the storage performance chaos early warning signal, acquires three types of medium health indicators of the erase-write cycle count, the seek error rate and the charge retention rate in the distributed storage node, classifies and organizes them according to the storage medium type, establishes an index association mapping in combination with the node identifier, and generates a medium health degree data set;

[0068] The health parameter acquisition submodule accesses multiple nodes in the distributed storage architecture, and extracts the three types of indicators such as the erase-write cycle count, the seek error rate and the charge retention rate item by item by calling the underlying medium interface of each node. The erase-write cycle count is read by the management chip of the Flash medium to obtain the cumulative erase-write times of the current storage block. For example, a certain MLC type storage medium has a capacity of 64G, and the maximum allowed P / E times is set to 3000. A certain block page area in the node has been used 2650 times, and the count value is recorded as 2650. The seek error rate is calculated according to the I / O error statistical interface provided by the main control chip. A 24-hour collection window is set, and the total access times and the number of error occurrences in the time period are recorded. For example, 14 errors occur, and the total access is 5000 times. The calculated seek error rate is 0.28%. The charge retention rate is evaluated based on the charge stability of the floating gate storage unit. By comparing the current charge retention capability of the storage block with the initial value through historical records and simulation aging test results, for example, the charge retention rate is set to decrease to 80% as the critical reference value, and the actual detection of a certain block is 84%. After the collection of the above three types of indicators is completed, they are classified and organized according to the storage medium type, for example, SLC, MLC and TLC types are established independently. Each parameter is bound to the node identifier, for example, the node number is registered in the form of UUID, and the data field combination includes the medium type, the node identifier, and the values of the three types of health indicators. By aggregating these combinations, a complete medium health degree data set is formed, which is used for subsequent analysis and calling.

[0069] The adaptive pressure analysis submodule is based on the medium health degree data set, calls the data access frequency and the storage pool load pressure value, jointly divides the frequency distribution and the pressure interval of the node data block, marks the access intensity under the different pressure conditions, and generates a frequency-pressure coupling identification interval value;

[0070] The adaptive pressure analysis sub-module, based on the generated medium health data set, first calls the access frequency information of each data block from the distributed nodes, collects the access times of each block in the time period by setting a fixed time window, for example, in hours. For example, the access of a certain data block in the node in the last five hours is 12 times, 14 times, 15 times, 10 times and 13 times, and the average access frequency is 12.8 times per hour. The current load pressure value of the node is collected synchronously, which is defined as the proportion of the current active I / O request number to the maximum concurrent capacity. For example, the maximum concurrent capacity is 1000 IOPS, and the current value is 580 IOPS, so the pressure value is 58%. The access frequency and the pressure value are mapped, the access frequency can be divided into 0-5, 6-10, 11-15, 16-20 four intervals, and the pressure is divided into 0%-30%, 31%-60%, 61%-100% three intervals. The data block in the node is classified into the combined interval corresponding to the frequency interval 11-15 and the pressure interval 31%-60%. The process is performed for all nodes and their data blocks, and each block is marked with the frequency and pressure interval. The node identifier, block number, frequency interval and pressure interval are combined to form a frequency and pressure coupling identification interval value, and a complete mapping reference set is constructed.

[0071] The degradation compensation calculation submodule extracts the medium health parameters corresponding to the interval according to the frequency and pressure coupling identification interval value, constructs the change trend according to the interval characteristics, evaluates the performance deviation degree of the index combination in the interval, and generates a mixed storage medium adaptive decision table.

[0072] The degradation compensation calculation sub-module is based on the frequency-voltage coupling identification interval, and selects the medium health parameters of the data block contained in each interval as the input basis. Taking the node Block number in a certain identification interval as an example, the block health parameters include 2650 times of write-erase cycles, 0.28% of seek error rate, and 84% of charge retention rate. Further, according to the interval characteristics, the historical change values of each parameter are collected, and a unit time step is set, such as updating every hour, and the change at multiple time points is collected, for example, the write-erase cycles are 2630, 2640 and 2650 times, the seek error rates are 0.24%, 0.26% and 0.28%, and the charge retention rates are 86%, 85% and 84%. The parameter change speed is extracted through the time difference value, the write-erase cycle increases by 10 times per hour, the seek error rate increases by 0.02% per hour, and the charge retention rate decreases by 1% per hour. Compare these change speeds with the corresponding threshold values, for example, the maximum allowed value of the write-erase cycle is 3000 times, which is 350 times different from the current value, and the change speed accounts for about 2.86% of the remaining capacity. According to the importance of the change rate, different parameters are assigned weights, such as the write-erase cycle weight is 0.4, the seek error rate is 0.3, and the charge retention rate is 0.3. The scores are calculated by combining the change speeds, for example, the comprehensive score is about 0.025. Repeat the above process for the data blocks in all frequency-voltage intervals, sort according to the scores, and form a data table containing node number, block number, score value, interval and allocation suggestion, which is used for subsequent mixed storage medium adaptation processing.

[0073] Please refer to Figure 2 , the decision generation module includes:

[0074] The migration cost evaluation sub-module obtains the access frequency of the data block, the node input-output occupancy and the medium capacity state based on the mixed storage medium adaptation decision table, calls the number of access paths, the migration distance between nodes and the bandwidth occupancy, calculates the migration efficiency, analyzes the resource consumption and path redundancy of the data block under the path, and generates the migration cost interval value;

[0075] The specific calculation formula of the migration efficiency is:

[0076] ;

[0077] Among them, represents the comprehensive evaluation value of the migration efficiency from node z to x, represents the physical migration distance from the source node z to the target node x (unit: kilometers), represents the available bandwidth from node z to x (unit: Gbps), represents the current input-output load rate of the source node z (in percentage form), and Current input / output load rate of target node x (in percentage), ε represents a smoothing constant (10^-5), Migration task weight coefficient of the mth time window, Duration of the mth time window (in seconds), Path competition factor of the mth time window (0.1-1.0), γ represents the basic bandwidth reservation (in Gbps), n represents the total number of statistical time windows;

[0078] Physical migration distance Calculated by GPS coordinates, source node z coordinates (30.2672°N, 97.7431°W), target node x coordinates (40.7128°N, 74.0060°W), calculated using the Haversine formula = 2780 kilometers;

[0079] Available bandwidth Obtained through real-time network monitoring tools, the current link bandwidth is = 25 Gbps;

[0080] Input / output load rate And Collected by node resource monitoring system, = 68%, = 42%;

[0081] The smoothing constant ε = 0.00001 is used to avoid zero denominator. The total number of time windows n = 6 is determined by the system setting of 15-minute time window statistical period;

[0082] Migration task weight coefficient Determined by the task priority algorithm, urgent task α1 = 0.9, regular task α2 = 0.6, batch task α3 = 0.3. The duration of the time window Extracted from system logs, Δt1 = 900 seconds, Δt2 = 900 seconds, Δt3 = 900 seconds;

[0083] Path competition factor Obtained through link quality detection protocol, β1 = 0.3, β2 = 0.5, β3 = 0.7;

[0084] The basic bandwidth reservation γ = 5 Gbps is set according to the network management strategy;

[0085] Calculate the first component:

[0086] ;

[0087] Calculate the second component:

[0088] ;

[0089] Finally = 13630.7 + 298.4 = 13929.1. The value reflects the comprehensive evaluation value of the migration efficiency between nodes, and the larger the value, the higher the migration cost. When exceeds the preset threshold 15000, the system will determine that the migration path needs to be optimized and adjusted.

[0090] The benefit prediction calculation submodule calls the available capacity, write rate and current load state of the target node according to the migration cost interval value, analyzes the node load change trend and access throughput change direction, establishes the difference sequence between benefit and cost, and generates the storage topology reconstruction benefit prediction value;

[0091] The benefit prediction calculation submodule obtains the current available capacity of the target node based on the migration cost interval, such as a node with a remaining space of 400G, assuming an average data block size of 10G, which can support up to 40 data blocks, and the write rate is given by the device parameters, such as a node with a write rate of 150M per second, while recording the current load, including CPU usage of 70% and I / O occupancy of 65%. Through the historical load change curve, the trend of the node load growth is predicted, such as historical data showing that the load increases by 10% and the CPU usage increases by about 5%. Under the condition of adding 4 data blocks, the CPU usage is expected to increase to about 90%. The current throughput is calculated according to the write rate and I / O idle ratio, such as the current throughput of 150M per second x 35% idle ratio, which is about 52.5M per second. If it is predicted that the new data block will cause the I / O occupancy to rise to 80%, the idle ratio will drop to 20% and the throughput will drop to 30M per second. The direction of the throughput change caused by migration is thus determined, and the difference between the benefit and the cost is the throughput improvement divided by the cost. For example, the throughput improvement is 10M per second and the cost is 3.5, so the difference is 2.86. If the value exceeds the set threshold of 2.5, it is considered to be an acceptable migration condition. The difference values calculated for multiple nodes are sorted to obtain a benefit prediction value list, such as node A being 2.86, node B being 1.25, and node C being 3.10. This list is used to guide the target node selection of the subsequent distribution strategy.

[0092] The distribution strategy generation submodule calls the storage topology reconstruction benefit prediction value, obtains the current node data block distribution quantity and medium matching state, analyzes the node distribution density change and medium adaptation interval, judges the corresponding relationship between data distribution and benefit, and generates a data dynamic distribution strategy.

[0093] The distribution strategy generation submodule obtains the number of data blocks currently distributed on each node and the matching state of the medium type according to the aforementioned profit prediction values, such as 3.10 for node A, 2.86 for node B, and 1.25 for node C. For example, node A stores 45 data blocks, node B stores 30 data blocks, and node C stores 20 data blocks. Node A is mainly configured with SSD, node B is configured with SAS, and node C is configured with SATA. In combination with the medium adaptation standard, such as the recommended distribution density of 0.08 to 0.12 blocks per G for SSD, the current distribution density is calculated, such as 0.09 blocks per G for node A, which is within the adaptation range, and 0.025 blocks per G for node C, which is far below the lower limit of the adaptation range. According to the current density divided by the interval median multiplied by the profit prediction value, the adaptation index is obtained, such as 0.09 / 0.1 x 3.10 for node A, which is about 2.79, 0.06 / 0.075 x 2.86 for node B, which is about 2.29, and 0.025 / 0.05 x 1.25 for node C, which is about 0.625. According to the size of the adaptation index, the priority migration direction is determined, such as migrating the low-frequency access data blocks in node A to node C and migrating the high-frequency access data blocks in node C to node A, to achieve dynamic adjustment of data blocks. In combination with the path migration cost data, priority control and strategy selection are performed, and finally a set of distribution strategies are formed for the scheduling module to call.

[0094] Please refer to Figure 2 The strategy execution module includes:

[0095] The data migration submodule obtains the access frequency, access delay, and throughput information of hot and cold data on different storage media based on the dynamic data distribution strategy, judges the access characteristic deviation state based on the corresponding relationship between the access frequency and the delay threshold, calculates the load balancing evaluation value, selects the migration path based on the throughput of the data and the available capacity information of the node, and generates the migratable path interval value.

[0096] The specific calculation formula of the load balancing evaluation value is:

[0097] ;

[0098] Among them, represents the load balancing evaluation value, represents the absolute value of the load difference between node i and node j, represents the arithmetic mean of the read-write parallelism of all nodes, represents the standard deviation of the load difference between nodes, N represents the total number of nodes in the cluster, and a represents the adjustment coefficient based on the system architecture, represents the read-write parallelism of the kth node.

[0099] Parameter definition and data acquisition : The CPU usage rate (0%-100%) of the target node, the memory occupancy rate (0%-100%) and the network bandwidth occupancy rate (0%-100%) are collected by the node monitoring system, and the comprehensive load value is obtained by weighted summation with weights of 0.4, 0.3 and 0.3 and , the calculation =| - |;

[0100] The actual measured node 1, 2, 3 are 65%, 72% and 58% respectively. The arithmetic mean of read-write parallelism : The read-write operation number (IOPS) of each node per second is extracted from the distributed storage system log, and the read-write operation number (IOPS) of node 1, 2, 3 are 200, 250 and 300 respectively, and the calculation =(200+250+300) / 3=250;

[0101] Load difference standard deviation : Based on ΔL12=7%, ΔL13=7%, ΔL23=14%, the mean (7+7+14) / 3=9.33 is calculated, and the variance [(7-9.33)²+(7-9.33)²+(14-9.33)²] / 3=10.89, = =3.3=3.3;

[0102] Adjustment coefficient (α): According to the AWS ECS cluster architecture document, set α=150, which increases logarithmically with the expansion of node scale, and take the benchmark value when N=3;

[0103] Total number of nodes (N): Get N=3 from the cluster management interface;

[0104] Numerator calculation:

[0105] ;

[0106] Denominator calculation:

[0107] ;

[0108] Second fraction numerator:

[0109] ;

[0110] Second fraction denominator:

[0111] ;

[0112] ;

[0113] Complete formula substitution:

[0114] ;

[0115] The result shows that the load balancing evaluation value is negative, reflecting the comprehensive effect of the current node load difference (standard deviation 3.3) and read-write parallelism distribution (square root term 415.96) exceeding the adjustment coefficient a control range, and the path set with smaller absolute value of H (|H|≤0.02 as the threshold value) needs to be selected to generate the balanced load distribution coefficient when the path with H close to zero is preferred.

[0116] The load adjustment submodule extracts the load pressure, concurrent access volume and read-write parallelism of the target node according to the migratable path interval value, and screens the path set that meets the load balancing condition according to the load difference and parallelism distribution between nodes to generate the balanced load distribution coefficient;

[0117] The load adjustment submodule continues to screen the node resource status according to the migratable path interval value, wherein the current system load pressure of each target node needs to be extracted, including CPU utilization, IO waiting ratio and memory occupancy ratio, which are combined to form a weighted load value, for example, the CPU usage of a node is 60%, the IO waiting is 20%, and the memory usage is 70%, the weight is set as 5:3:2, and the combined load is 60x0.5+20x0.3+70x0.2=47, if the comprehensive load of another node is 52, the load difference between the two nodes is 5, and the difference value threshold is set to 10%, if the difference is lower than the proportion, it is considered as a balanced node; Then analyze the concurrent access volume and read-write parallelism, for example, the access volume of a node is 40 times per second, and the read-write thread number is 60, which are normalized to 0.5 and 0.6 respectively, and the sum of the two is taken as the node processing capacity index, which is 1.1, compared with the required index of the data block, such as 1.0, the difference is 0.1, and the error threshold is set to 0.3, the node meets the condition, and all the nodes that meet the load difference and capacity error requirements form the path set that meets the load balancing, finally, for these paths, the balanced distribution coefficient is constructed, the reciprocal of the load difference and the capacity error is used for screening and sorting, and the path with the optimal balancing degree is selected, for example, a path with a difference of 0.07 and a capacity error of 0.1, the balanced coefficient is the reciprocal of the sum of the two, which is 6.25, if it is the maximum value, it is selected as the final migration path.

[0118] The replica reconstruction submodule calls the node distribution, cross-domain synchronization frequency and replication delay information of the replica in the topology based on the balanced load distribution coefficient, judges the adjustable replica set according to the node density and synchronization frequency, determines the reconstruction order in combination with the replication delay sorting, and generates the storage topology optimization configuration;

[0119] The copy reconstruction submodule further analyzes the node distribution in the current copy topology structure on the basis of the balanced path, calculates the ratio of the number of copies in each network region to the number of nodes in the region as the node density, for example, a region contains 10 nodes, of which 4 are copies, then the density is 0.4, if the upper limit density is set to 0.4, the region needs to be adjusted; then the cross-domain synchronization frequency and copy replication delay data are extracted, the synchronization frequency is, for example, 25 times per hour, the replication delay is, for example, 45 milliseconds, the delay benchmark can be calculated by the average delay of other copies of the node, for example, the average is 35 milliseconds, then the replication delay deviation of the node is 10 milliseconds, the deviation threshold is set to 5 milliseconds, the node synchronization frequency and delay deviation are both high, which meets the reconstruction condition; further, according to the high and low order of the replication delay, the copies that meet the condition are sorted, and the copies with larger delay are adjusted preferentially, the reconstruction path needs to be migrated to the target node with density lower than the upper limit, higher balanced coefficient and moderate synchronization frequency, for example, a copy is migrated to a node, the node density is 0.3, the synchronization frequency is 15 times / h, and the replication delay is 30 milliseconds, which meets all the conditions, that is, a new copy distribution configuration is formed.

[0120] Please refer to Figure 2 , the effect feedback module comprises:

[0121] The throughput monitoring submodule obtains the input and output throughput of the nodes in the storage topology optimization configuration, calls the benchmark throughput data of the corresponding nodes before configuration, compares the current throughput with the benchmark state according to the data unit quantity in the same time period, combines and collects the comparison results of all nodes to obtain the throughput balance degree value;

[0122] The throughput monitoring submodule needs to call the benchmark throughput data before configuration for comparison after obtaining the node input and output throughput. First, through the data collection mechanism in the system, the write and read data volume is collected from each node interface every 5 minutes. For example, node A records a write of 240 MB and a read of 180 MB in a cycle, and the total throughput is 420 MB, which is converted into an average throughput rate of 84 MB / min. Then, the historical benchmark data before optimization configuration of the node is retrieved, for example, the benchmark value is 60 MB / min. The difference between the current and the benchmark is 24 MB / min. Then, the difference is normalized. If the maximum allowed fluctuation range is set to plus or minus 40 MB / min, the normalized offset value is 0.6. This step is performed on all nodes to form a set of normalized differences. For example, in a certain topology structure, there are 5 nodes, and the corresponding offset values are 0.6, -0.2, 0.1, -0.4, and -0.1. The average deviation of these values is calculated to measure the overall throughput balance of the system. The balance level can be distinguished according to the set level standard. For example, the fluctuation degree between 0 and 0.2 is considered as high balance, the fluctuation degree between 0.2 and 0.4 is considered as moderate balance, and the fluctuation degree greater than 0.4 is considered as low balance. In the above example, the standard deviation of the offset value is about 0.34, which corresponds to a moderate balance interval. Finally, the current throughput balance level of the topology is obtained.

[0123] The wear analysis submodule obtains the wear rate sampling results and the running cycle parameters of the storage node corresponding medium according to the throughput balance degree value, integrates the parameters of the node medium state by constructing the proportional relationship between the scheduling frequency and the cycle, and obtains the medium running load rate;

[0124] After obtaining the throughput balance degree, the wear analysis submodule needs to further sample the wear rate and running cycle of the medium used by each storage node. Based on this, the running load is evaluated. The specific operation includes reading the write life index of the medium. For example, according to the sampling results of a certain node, the remaining life count of the storage medium used by the node is 480, the current cumulative use cycle is 30 days, and the write volume is 250 GB per day. The average daily write proportion is calculated to be 0.5%, i.e. the daily wear degree is about 0.005. Multiply this value by the running cycle to get the total wear proportion in the overall cycle, which is about 15%. Further, the proportion is constructed by combining the scheduling frequency, for example, the node is scheduled 28 times in 30 days, and the scheduling frequency proportion is 93%. Multiply the total wear proportion by the scheduling frequency ratio to get the load rate of about 14%. After the same operation process is performed on other nodes, a unified running load rate data set is formed. This data can be used for performance judgment by the lower module.

[0125] The stability evaluation submodule calls the medium running load rate, obtains synchronization time delay and transmission flow information in the cross-domain data synchronization process, combines node distribution and bandwidth occupation of the storage area, extracts communication state indicators on the synchronization path, and performs partition comparison on the synchronization time delay and bandwidth state between nodes to obtain a storage optimization efficiency evaluation result.

[0126] After calling the medium running load rate, the stability evaluation submodule needs to synchronously extract the time delay and flow information during cross-domain data transmission. The operation process includes monitoring the delay record of the synchronization path between nodes, such as the delay from node A to node B being 120 milliseconds and the transmission data being 2.5 GB. At the same time, the network bandwidth of the region where the nodes are located is queried. It is assumed that the bandwidth of the region where node A is located is 10 Gbps, and the occupation rate is 25%. The bandwidth of the region where node B is located is 5 Gbps, and the occupation rate is 50%. If there is a relay node C in the path, the relay path segment delays are 40 milliseconds and 80 milliseconds, respectively, and the total delay is still 120 milliseconds. The stability of bandwidth usage is calculated by averaging the bandwidth fluctuations in each period during the bandwidth measurement period and then determining whether the fluctuation range exceeds the set stability threshold. For example, if the fluctuation coefficient is less than 20%, it can be determined to be stable. By comprehensively considering the delay level and bandwidth fluctuation, the distribution density of the nodes is divided into regions for comparison. Finally, the state matching result of the synchronization path is formed, the stability level is evaluated, and the storage optimization efficiency output is generated.

[0127] Please refer to Figure 3 A data storage management method based on big data includes the following steps:

[0128] S1: Monitor the input-output delay sequence of the storage node, collect the medium health degree attenuation curve and the network bandwidth fluctuation atlas, construct a dynamic characteristic space and perform stability analysis, and generate a storage efficiency chaos warning signal when the characteristic space parameter exceeds the system threshold;

[0129] S2: Based on the storage efficiency chaos warning signal, collect three types of medium indicators, including the erase-write cycle count, the seek error rate, and the charge retention rate, perform correlation analysis on the data access frequency distribution and the storage pool load pressure value, and output a hybrid storage medium adaptation decision table;

[0130] S3: Call the hybrid storage medium adaptation decision table, evaluate the data migration time delay loss, the storage pool life loss, and the topology reconstruction energy consumption, perform multi-dimensional evaluation and operation, and generate a data dynamic distribution strategy;

[0131] S4: Execute the data dynamic distribution strategy, implement the cold and hot data cross-medium migration operation, adjust the storage node load balancing parameters, reconstruct the redundant copy distribution topology, and generate a storage topology optimization configuration;

[0132] S5: Monitor the throughput fluctuation, media wear rate change and cross-domain synchronization delay data after the storage topology optimization configuration is implemented, analyze the index difference before and after optimization, and output the storage optimization performance evaluation result.

[0133] The above is only a preferred embodiment of the present application, and does not limit the present application in other forms. Any person skilled in the art can use the disclosed technical content to make changes or modifications as equivalent embodiments applied to other fields, but any simple modification, equivalent change and modification made according to the technical essence of the present application to the above embodiments without departing from the technical solution content of the present application still belongs to the protection scope of the technical solution of the present application.

Claims

1. A data storage management system based on big data, characterized in that: The system includes: The performance monitoring module collects the input and output delay sequence of storage nodes, media health curve, and bandwidth fluctuation spectrum to construct a storage performance feature space, calculates the Lyapunov index to monitor stability, and generates a storage performance chaos early warning signal when the index continuously exceeds the preset threshold. Based on the storage performance chaos warning signal, the media evaluation module collects erase / write cycle counts, seek error rate and charge retention rate, and calculates the media performance degradation compensation coefficient by combining the data access frequency distribution and storage pool load pressure value, and outputs a hybrid storage media adaptation decision table. The decision generation module evaluates the data migration cost, calculates the predicted value of storage topology reconstruction benefits, and generates a dynamic data distribution strategy based on the hybrid storage medium adaptation decision table. Based on the aforementioned dynamic data distribution strategy, the strategy execution module performs cold and hot data media migration, storage node load rebalancing, and redundant replica topology optimization, generating a storage topology optimization configuration. The performance feedback module optimizes the configuration based on the storage topology, monitors the cluster's input / output throughput, media wear rate, and cross-domain synchronization latency, compares and analyzes the baseline values ​​before and after optimization to quantify the stability gain, and outputs the storage optimization performance evaluation results.

2. The data storage management system based on big data according to claim 1, characterized in that: The storage performance chaos early warning signal includes input / output delay sequence, storage medium health decay curve, network bandwidth fluctuation spectrum, Lyapunov exponent, and system stability threshold. The hybrid storage medium adaptation decision table includes erase / write cycle count, seek error rate, charge retention rate, data access frequency distribution, storage pool load pressure value, and medium performance degradation compensation coefficient. The data dynamic distribution strategy includes data migration cost, storage topology reconstruction benefit prediction value, and hybrid storage medium adaptation decision table. The storage topology optimization configuration includes cold and hot data cross-media migration, storage node load balancing adjustment, and redundant replica topology reconstruction operation. The storage optimization performance evaluation results include input / output throughput, medium wear rate, and cross-domain synchronization latency.

3. The data storage management system based on big data according to claim 1, characterized in that: The performance monitoring module includes: The latency data acquisition submodule acquires the input / output request response latency of distributed storage nodes, the health change curve of storage media during the operating cycle, and the bandwidth fluctuation trajectory during network transmission. It extracts the time-series fluctuation feature values ​​of the data types, classifies and integrates the fluctuation feature values ​​according to a unified time standard, and constructs a data sequence of indicator combinations to generate a storage performance fluctuation parameter sequence. The feature space construction submodule extracts the index offset trend, change range and fluctuation interval under the difference time segment based on the storage performance fluctuation parameter sequence, groups and statistically analyzes the trend data based on the time segment division, and constructs the mapping relationship between features based on the change density and direction difference of the group trend values ​​to generate the dynamic feature space mapping degree of storage performance. The chaos threshold early warning submodule calls the dimensional trend data in the dynamic feature space mapping degree of the storage performance, identifies continuous abnormal fluctuation segments according to the stability standard, counts the duration and frequency characteristics, and generates a storage performance chaos early warning signal by combining the density of the offset trend.

4. The data storage management system based on big data according to claim 3, characterized in that: The media evaluation module includes: The health parameter acquisition submodule acquires the storage performance chaos warning signal, collects three types of media health indicators in the distributed storage nodes: erase / write cycle count, seek error rate, and charge retention rate, classifies and organizes them according to storage media type, establishes an indicator association mapping with node identifier, and generates media health data group. The adaptation pressure analysis submodule, based on the media health data group, calls the data access frequency and storage pool load pressure value to jointly divide the frequency distribution and pressure range of node data blocks, marks the access density under different pressure conditions, and generates frequency-pressure coupling identification range value. The degradation compensation calculation submodule extracts the media health parameters corresponding to the interval based on the frequency-voltage coupling identification interval value, constructs the change trend according to the interval characteristics, evaluates the performance deviation of the index combination in the interval, and generates a hybrid storage media adaptation decision table.

5. The data storage management system based on big data according to claim 4, characterized in that: The decision generation module includes: The migration cost assessment submodule, based on the hybrid storage medium adaptation decision table, obtains the access frequency of data blocks, node input / output occupancy and medium capacity status, calls the number of access paths, migration distance between nodes and bandwidth occupancy, calculates migration efficiency, analyzes the resource consumption and path redundancy of data blocks under the path, and generates migration cost range values. The revenue prediction calculation submodule, based on the migration cost range, calls the target node's available capacity, write rate, and current load status to analyze the node load change trend and access throughput change direction, establishes a difference sequence between revenue and cost, and generates a storage topology reconstruction revenue prediction value. The distribution strategy generation submodule calls the storage topology reconstruction revenue prediction value, obtains the current node data block distribution quantity and media matching status, analyzes the distribution density change and media adaptation range between nodes, determines the correspondence between data distribution and revenue, and generates a dynamic data distribution strategy.

6. The data storage management system based on big data according to claim 5, characterized in that: The specific formula for calculating migration efficiency is as follows: ； in, This represents the comprehensive evaluation value of the migration efficiency from node z to x. This represents the physical migration distance from the source node z to the target node x. This represents the available bandwidth from node z to x. This represents the current input / output load rate of the source node z. ε represents the current input / output load rate of the target node x, and ε represents the smoothing constant. This represents the migration task weight coefficient for the m-th time window. This represents the duration of the m-th time window. γ represents the path contention factor for the m-th time window, γ represents the base bandwidth reservation, and n represents the total number of statistical time windows.

7. The data storage management system based on big data according to claim 5, characterized in that: The strategy execution module includes: The data migration submodule, based on the dynamic data distribution strategy, obtains the access frequency, access latency, and throughput information of cold and hot data on different storage media. Based on the correspondence between access frequency and latency threshold, it judges the deviation state of access characteristics, calculates the load balancing evaluation value, and combines the throughput and available capacity information of the data to filter migration paths and generate a range of migration paths. The load adjustment submodule extracts the load pressure, concurrent access volume and read / write parallelism of the target node based on the range of the migrated paths. Based on the load differences and parallelism distribution between nodes, it filters the set of paths that meet the load balancing conditions and generates a balanced load distribution coefficient. The replica reconstruction submodule, based on the balanced load distribution coefficient, calls the node distribution, cross-domain synchronization frequency and replication latency information of the replicas in the topology, determines the set of replicas that can be adjusted based on the node density and synchronization frequency, determines the reconstruction order based on the replication latency sorting, and generates an optimized storage topology configuration.

8. The data storage management system based on big data according to claim 7, characterized in that: The specific formula for calculating the load balancing evaluation value is as follows: ； in, This represents the load balancing evaluation value. This represents the absolute value of the load difference between node i and node j. The arithmetic mean of the read / write parallelism of all nodes. The standard deviation represents the load difference between nodes, N represents the total number of nodes in the cluster, and α represents an adjustment coefficient based on the system architecture. This represents the read / write parallelism of the k-th node.

9. The data storage management system based on big data according to claim 7, characterized in that: The effect feedback module includes: The throughput monitoring submodule obtains the input and output throughput of the nodes in the storage topology optimization configuration, calls the baseline throughput data of the corresponding nodes before configuration, compares the current throughput with the baseline state based on the data unit volume in the same time period, and collects and summarizes the comparison results of all nodes to obtain the throughput balance value. The wear analysis submodule obtains the wear rate sampling results and operating cycle parameters of the medium corresponding to the storage node based on the throughput balance value. Combined with the node scheduling frequency, it constructs the ratio relationship between scheduling frequency and cycle period, integrates the parameters of the node medium status, and obtains the medium operating load rate. The stability assessment submodule calls the operating load rate of the medium to obtain the synchronization latency and transmission traffic information during the cross-domain data synchronization process. Combined with the node distribution and bandwidth occupancy of the storage area, it extracts the communication status indicators under the synchronization path, compares the synchronization latency and the bandwidth status between nodes in different partitions, and obtains the storage optimization performance assessment results.

10. A data storage management method based on big data, characterized in that, The data storage management system based on big data as described in any one of claims 1-9 is executed, comprising the following steps: S1: Monitor the input and output delay sequence of storage nodes, collect the media health decay curve and network bandwidth fluctuation spectrum, construct a dynamic feature space and perform stability analysis. When the feature space parameters exceed the system threshold, generate a storage performance chaos early warning signal. S2: Based on the storage performance chaos warning signal, collect three types of media indicators: erase / write cycle count, seek error rate, and charge retention rate. Combine the data access frequency distribution with the storage pool load pressure value to perform correlation analysis and output a hybrid storage media adaptation decision table. S3: Call the hybrid storage medium adaptation decision table to evaluate data migration latency loss, storage pool lifetime loss and topology reconstruction energy consumption, perform multi-dimensional evaluation and calculation, and generate a dynamic data distribution strategy. S4: Execute the dynamic data distribution strategy, carry out cross-media migration of hot and cold data, adjust the storage node load balancing parameters, reconstruct the redundant replica distribution topology, and generate an optimized storage topology configuration. S5: Monitor the throughput fluctuation, media wear rate change and cross-domain synchronization delay data after the implementation of the storage topology optimization configuration, perform difference analysis of indicators before and after optimization, and output storage optimization performance evaluation results.

Images