Data all-flash storage optimization method and system based on cloud computing

Abstract

The present invention provides a data all-flash storage optimization method and system based on cloud computing. The method collects a multi-dimensional performance data set of each node of an all-flash storage cluster in real time, including a storage space fragmentation rate, an input and output request queue depth, and a network link bandwidth utilization rate, receives a user terminal storage operation request stream, parses a set of identifiers of data blocks to be stored and an operation mode label, traverses a historical access record library based on the identifiers of the data blocks to be stored, extracts historical access time series features and the proximity of the physical storage locations of associated data blocks, combines the multi-dimensional performance data with the historical access time series features, constructs a dynamic access heat prediction model within a preset time period, and finally generates a cross-node shard storage topology map and a cache preloading strategy matrix based on the dynamic access heat prediction model and the proximity of the physical storage locations, thereby optimizing the all-flash storage and improving the data storage performance in a cloud computing environment.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud computing, and in particular to a data all-flash storage optimization method and system based on cloud computing. Background Art

[0002] With the rapid development of cloud computing technology, data volumes are experiencing explosive growth, placing higher demands on data storage performance, reliability, and efficiency. All-flash storage, with its high-speed data read and write performance, is becoming an increasingly important data storage option in cloud computing environments. However, in practical applications, all-flash storage systems face numerous challenges, and existing technical solutions struggle to meet these increasingly complex demands.

[0003] In the current field of cloud computing data storage, traditional approaches to optimizing all-flash storage clusters typically simply allocate storage locations according to fixed rules when processing storage operation requests from user terminals. These methods lack in-depth analysis of the characteristics of the data blocks in the request and their historical access history. For example, these methods fail to consider the historical access timing characteristics of the data blocks to be stored, resulting in a lack of scientific data storage distribution and an inability to rationally arrange data based on the actual access frequency and patterns, which in turn affects data read and write efficiency. Furthermore, existing technologies rarely consider the physical storage location proximity of associated data blocks, resulting in a more dispersed distribution of data within the storage cluster, increasing seek time and network transmission overhead during data access. Summary of the Invention

[0004] In view of the above-mentioned problems, in combination with the first aspect of the present invention, an embodiment of the present invention provides a data all-flash storage optimization method based on cloud computing, the method comprising:

[0005] Real-time collection of multi-dimensional performance data sets of each node in the all-flash storage cluster, including the node storage space fragmentation rate, input and output request queue depth and network link bandwidth utilization;

[0006] Receiving a storage operation request stream sent by a user terminal, parsing a set of identifiers of data blocks to be stored and corresponding operation mode tags in the storage operation request stream;

[0007] Traversing a historical access record library based on the set of identifiers of the data blocks to be stored, extracting historical access time sequence features of the data blocks to be stored and physical storage location proximity of the associated data blocks;

[0008] Constructing a dynamic access popularity prediction model for the data block to be stored within a preset time period based on the multi-dimensional performance data set and the historical access time series characteristics;

[0009] Based on the dynamic access heat prediction model and the physical storage location proximity, a cross-node shard storage topology map and a cache preloading strategy matrix are generated.

[0010] For example, traversing the historical access record library based on the set of identifiers of the data blocks to be stored to extract the historical access time sequence characteristics of the data blocks to be stored and the physical storage location proximity of the associated data blocks may include:

[0011] According to each data block identifier in the set of data block identifiers to be stored, matching a corresponding historical access record in the historical access record library, the historical access record including an access timestamp set and an associated data block identifier subset;

[0012] Extracting an access timestamp sequence from the historical access records, and calculating a periodic access interval distribution parameter and a timestamp set of a sudden access event based on the access timestamp sequence;

[0013] Filtering out a set of associated data block identifiers whose access times with the data block to be stored exceed a collaboration threshold within a preset time window from the associated data block identifier subset;

[0014] Obtaining a physical storage coordinate set of the associated data block identifier set in the all-flash storage cluster, where the physical storage coordinate set includes a storage node identifier and a logical unit address;

[0015] Based on the topological connection relationship between the storage node identifiers and the distance difference between the logical unit addresses in the physical storage coordinate set, a physical storage location proximity matrix of the associated data blocks is generated.

[0016] On the other hand, an embodiment of the present invention also provides a cloud computing-based data all-flash storage optimization system, including a processor and a machine-readable storage medium, the machine-readable storage medium is connected to the processor, the machine-readable storage medium is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the machine-readable storage medium to implement the above method.

[0017] Based on the above aspects, after the embodiment of the present application collects the multi-dimensional performance data set of each node in the all-flash storage cluster in real time, it receives the storage operation request stream of the user terminal and parses the set of identifiers of the data blocks to be stored and the operation mode labels therein, and at the same time traverses the historical access record library based on the set of identifiers of the data blocks to be stored, extracts the historical access timing characteristics and the physical storage location proximity of the associated data blocks, and deeply mines the historical access characteristics of the data blocks themselves and their spatial relationship with the surrounding data blocks. Furthermore, based on the multi-dimensional performance data set and the historical access timing characteristics, a dynamic access heat prediction model of the data blocks to be stored within a preset time period is constructed. The dynamic access heat prediction model integrates real-time performance data and historical access rules, and can dynamically and accurately predict the access heat of the data blocks in the future. It can better adapt to the variability and complexity of data access in the cloud computing environment, and effectively improves the ability to predict data access trends. Finally, based on the dynamic access heat prediction model and the physical storage location proximity, a cross-node shard storage topology map and a cache preloading strategy matrix are generated, realizing the optimized allocation and efficient utilization of storage resources. The cross-node sharded storage topology considers the relationship between data block access popularity and physical storage location, rationally planning data storage distribution across different nodes. This effectively balances the load pressure on each node, reduces storage space fragmentation, and improves the read and write performance of the entire storage cluster. Furthermore, the cache preloading strategy matrix pre-loads frequently accessed data blocks into the cache based on predicted access popularity, significantly reducing data access wait times and significantly improving system responsiveness and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 The present invention provides a method for optimizing data storage based on cloud computing.

[0019] Figure 2 Schematic diagram of exemplary hardware and software components of a cloud computing-based all-flash data storage optimization system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The present invention will be described in detail below with reference to the accompanying drawings. Figure 1 FIG1 is a flow chart of a method for optimizing data all-flash storage based on cloud computing provided by an embodiment of the present invention. The method for optimizing data all-flash storage based on cloud computing is introduced in detail below.

[0021] Step S110 , collecting a multi-dimensional performance data set of each node in the all-flash storage cluster in real time, wherein the multi-dimensional performance data set includes a node storage space fragmentation rate, an input / output request queue depth, and a network link bandwidth utilization rate.

[0022] Specifically, an enterprise data center can deploy an all-flash storage cluster to store massive amounts of business data, such as the company's financial data, customer information, sales records, etc. The all-flash storage cluster consists of multiple nodes, each of which is responsible for storing and processing a portion of the data.

[0023] In this scenario, the node storage space fragmentation rate is collected as follows: As an enterprise continuously stores, deletes, and modifies data, the storage space within the node gradually becomes fragmented. For example, when the finance department frequently updates financial report data, many small blocks of free space may form in the storage space, rather than continuous large blocks of free space. The node storage space fragmentation rate is calculated by calculating the ratio of these fragmented spaces.

[0024] Regarding the depth of the I / O request queue, let's assume that a company's sales department collects and reports sales data at the end of each month. During this period, a large number of read and write requests are sent to the all-flash storage cluster. Each node receives these requests and places them into the I / O request queue in the order they arrive. The number of requests in this queue is monitored in real time, and this number represents the I / O request queue depth. For example, during peak sales data processing periods, the I / O request queue depth of a node may reach hundreds or even thousands of requests.

[0025] The collection of network link bandwidth utilization is related to data transmission between nodes. Specifically, different departments within an enterprise may share data. When the customer service department retrieves customer information from the storage cluster to provide services, data needs to be transmitted between nodes via network links. During this process, the ratio of the actual amount of data transmitted per unit time to the total link bandwidth can be monitored. This is the network link bandwidth utilization. For example, if the total bandwidth of a network link is 1000 Mbps and the actual data transmission volume at a given moment is 500 Mbps, the network link bandwidth utilization at that time is 50%.

[0026] Step S120: receiving a storage operation request stream sent by a user terminal, and parsing a set of identifiers of data blocks to be stored and corresponding operation mode tags in the storage operation request stream.

[0027] Continuing with the enterprise data center example, user terminals can be computers, servers, and other devices used by employees in various departments within the enterprise. Suppose the enterprise's marketing department wants to store a new market research report in an all-flash storage cluster. The marketing employee uses specialized software (user terminals) to send a stream of storage operation requests to the storage cluster.

[0028] The storage operation request stream contains a set of identifiers for the data blocks to be stored. For example, a market research report might be stored in multiple data blocks, each with a unique identifier. The request stream also carries a corresponding operation mode tag, which might be "write," indicating a storage (write) operation. Upon receiving the storage operation request stream, the system parses it to accurately extract the identifiers of the data blocks to be stored and the "write" operation mode tag, enabling subsequent processing based on this information.

[0029] Step S130 , traversing a historical access record library based on the set of identifiers of the data blocks to be stored, extracting historical access time sequence features of the data blocks to be stored and physical storage location proximity of the associated data blocks.

[0030] Still based on the enterprise data center scenario, for the storage operation of the market research report, for example, each data block identifier in the set of data block identifiers to be stored can be matched against the historical access record library. Assume that the historical access record library records all past accesses to the enterprise's data.

[0031] To extract historical access time series features, take a data block as an example. If past access records for this data block show frequent accesses between 9:00 AM and 10:00 AM every weekday, this constitutes an access timestamp sequence. Based on this access timestamp sequence, we can calculate the periodic access interval distribution parameter. For example, if we find an average peak in accesses every seven days, this is the periodic access interval distribution parameter. Furthermore, if there is a sudden surge in accesses during a special event (such as a company's new product launch), we can record the timestamp collection of this sudden access event.

[0032] To determine the physical storage location proximity of associated data blocks, a set of associated data block identifiers is selected from the associated data block identifier subset whose co-access counts with the data block to be stored exceed a co-access threshold (assuming 10 times) within a preset time window (e.g., the past month). For example, consider market analysis data blocks related to a market research report, which were co-accessed 15 times within the past month. The physical storage coordinates of these associated data block identifiers in the all-flash storage cluster are then obtained. Assume that the market research report data block is stored at logical unit address 100 on node A, while the associated market analysis data blocks are stored at logical unit addresses 105 on node A and 200 on node B. Based on the topological connectivity between these storage node identifiers (nodes A and B are connected via a high-speed network) and the distance differences between the logical unit addresses (logical unit addresses 100 and 105 are closer, but farther from 200), a physical storage location proximity matrix for the associated data blocks is generated.

[0033] Step S140 : constructing a dynamic access popularity prediction model of the data block to be stored within a preset time period based on the multi-dimensional performance data set and the historical access time series characteristics.

[0034] Taking the above scenario as an example, we can extract periodic access peak intervals (such as 9:00 to 10:00 a.m. every weekday), burst access event timestamps (such as access peaks during new product releases), and access interval distribution statistics (such as an average of one access peak every seven days) from historical access time series features.

[0035] Suppose a market research report has a "write" operation mode tag, but some of the data contained in it may be frequently read later to adjust marketing strategies. The read and write operation ratio corresponding to the operation mode tag is identified. When the read operation ratio exceeds a preset threshold (assuming 60%), the hot data prediction flag is activated. For example, analysis reveals that the market share analysis section in a market research report has an 80% chance of being read later, exceeding the preset 60% threshold, so the hot data prediction flag is activated.

[0036] Combine the periodic access peak intervals and the timestamps of sudden access events to generate a time-dimensional access probability density function. For example, if the access probability between 9:00 AM and 10:00 AM is 0.3, and the access probability corresponding to a sudden access event during a new product launch is 0.2, then a time-dimensional access probability density function can be constructed based on this data.

[0037] Based on the access interval distribution statistics and hotspot data prediction markers, the weight parameters of the time-dimension access probability density function are modified. Assuming that the access interval distribution statistics indicate that the access probability gradually decreases between two access peaks, the weight parameters of the time-dimension access probability density function are adjusted accordingly.

[0038] The corrected time-dimensional access probability density function is coupled with the network link bandwidth utilization in the multidimensional performance data set for calculation. For example, a bandwidth fluctuation dataset of network link bandwidth utilization within multiple historical time windows is extracted from the multidimensional performance data set. Assume that the peak bandwidth utilization interval is 80% between 9:00 AM and 10:00 AM on weekdays, and the average transmission rate is 500 Mbps. The bandwidth utilization peak interval is aligned with the time axis of the time-dimensional access probability density function to generate a time-synchronized bandwidth utilization distribution sequence. Based on the average transmission rate, the bandwidth weight factor for each time unit of the time-dimensional access probability density function is calculated. The access probability value of the time-dimensional access probability density function in the corresponding time unit is weighted and corrected according to the bandwidth weight factor to generate a bandwidth-aware access probability density distribution function. The bandwidth-aware access probability density function is normalized in the time dimension. Based on the normalized maximum probability value and the slope change of the probability distribution curve, the popularity level parameter of the dynamic access popularity prediction model is generated. This popularity level parameter can be used to represent the access popularity of the market research report data block within a preset future time period.

[0039] Step S150: Based on the dynamic access heat prediction model and the physical storage location proximity, a cross-node shard storage topology map and a cache preloading strategy matrix are generated.

[0040] Specifically, for the storage of market research report data blocks, the shard redundancy threshold and minimum number of replicas for the data blocks to be stored are determined based on the popularity level parameter output by the dynamic access popularity prediction model. Assuming a high popularity level, the shard redundancy threshold is determined to be 3 (indicating that the data block can be divided into a maximum of three shards for storage), and the minimum number of replicas is determined to be 2 (each shard has at least two replicas).

[0041] The storage space fragmentation rate and input / output request queue depth of all nodes in the all-flash storage cluster are traversed to select a subset of candidate nodes that meet the shard capacity constraints. For example, if node A has a low storage space fragmentation rate, an acceptable input / output request queue depth, and sufficient space to store the shards of the market research report data block, then node A may be selected into the candidate node subset.

[0042] Based on physical storage location proximity, a storage location relevance score is calculated for each node in the candidate node subset. Assume that node A is closer to the storage location of the associated data block and therefore has a higher storage location relevance score. Based on the storage location relevance scores and the network link bandwidth utilization from the multi-dimensional performance data set, a storage path weight table between nodes is constructed. For example, if the network link bandwidth utilization from node A to node B is high and the storage location relevance score is high, then the storage path weight between them is high.

[0043] Based on the storage path weight table and the minimum number of replicas, a shard storage topology diagram with redundant path cross-connections is generated. For example, based on the inter-node storage path weights in the storage path weight table, a set of candidate storage paths with weights exceeding a preset weight threshold (assuming it is 0.6) is selected. Based on the minimum number of replicas, the number of shard replicas for the market research report data block is determined to be two, and an initial storage path is assigned to each shard replica. Assuming the first shard replica is initially assigned a storage path from Node A to Node B, the storage path weights in the candidate storage path set are traversed. The path from Node A to Node B has the highest weight, so it is selected as the default storage path for the shard replica. The inter-node connectivity of this default storage path is checked. If only one path from Node A to Node B is found, posing a single point of failure risk, a backup storage path with the next highest weight (assuming the path from Node A to Node C has the next highest weight) is selected from the candidate storage path set as a cross-redundant path. The cross-redundant path is bidirectionally connected to the default storage path to generate a shard storage topology diagram with redundant path cross-connections. Verify whether the path redundancy of each shard copy in the shard storage topology diagram meets the redundancy constraint corresponding to the minimum number of copies. If not, reselect the backup storage path and update the cross-connection relationship of the shard storage topology diagram.

[0044] Based on the dynamic access popularity prediction model and physical storage location proximity, a cache preloading strategy matrix is ​​generated. Based on physical storage location proximity, a target data block sequence with a spatial correlation to the market research report data block is identified, such as the market analysis data block mentioned earlier. Based on the dynamic access popularity prediction model, the concurrent access probability distribution of the target data block sequence within a preset time period is predicted. Assume that the market analysis data block has a 60% probability of being accessed simultaneously with the market research report data block within the next week. Based on the concurrent access probability distribution and the node storage space fragmentation rate in the multi-dimensional performance data set, a preloading priority coefficient is calculated for each target data block. If the node storage space fragmentation rate is low, the preloading priority coefficient may be higher. Based on the preloading priority coefficient, differentiated cache retention periods and compression level parameters are assigned to the target data block sequence. For example, for the market analysis data block with a high preloading priority coefficient, the current cache space occupancy and historical cache replacement frequency of the edge nodes in the all-flash storage cluster are obtained. When the concurrent access probability distribution exceeds a first preset threshold (assuming 50%), a fixed retention period (e.g., 3 days) is assigned to the corresponding data block, and the cache space is locked. When the concurrent access probability distribution falls below a second preset threshold (assuming 30%), the retention period decay rate is dynamically adjusted based on the historical cache replacement frequency. Low-priority data blocks are subjected to variable compression rates based on the compression level parameters, and metadata verification information for the compressed data blocks is recorded.

[0045] Based on the above steps, after the embodiment of the present application collects the multi-dimensional performance data set of each node in the all-flash storage cluster in real time, it receives the storage operation request stream of the user terminal and parses the set of identifiers of the data blocks to be stored and the operation mode labels therein, and at the same time traverses the historical access record library based on the set of identifiers of the data blocks to be stored, extracts the historical access timing characteristics and the physical storage location proximity of the associated data blocks, and deeply mines the historical access characteristics of the data blocks themselves and their spatial relationship with the surrounding data blocks. Furthermore, based on the multi-dimensional performance data set and the historical access timing characteristics, a dynamic access heat prediction model of the data blocks to be stored within a preset time period is constructed. The dynamic access heat prediction model integrates real-time performance data and historical access rules, and can dynamically and accurately predict the access heat of the data blocks in the future. It can better adapt to the variability and complexity of data access in the cloud computing environment, and effectively improves the ability to predict data access trends. Finally, based on the dynamic access heat prediction model and the physical storage location proximity, a cross-node shard storage topology map and a cache preloading strategy matrix are generated to achieve optimized allocation and efficient utilization of storage resources. The cross-node sharded storage topology considers the relationship between data block access popularity and physical storage location, rationally planning data storage distribution across different nodes. This effectively balances the load pressure on each node, reduces storage space fragmentation, and improves the read and write performance of the entire storage cluster. Furthermore, the cache preloading strategy matrix pre-loads frequently accessed data blocks into the cache based on predicted access popularity, significantly reducing data access wait times and significantly improving system responsiveness and user experience.

[0046] For example, in one possible implementation, step S130 includes:

[0047] Step S131 : matching a corresponding historical access record in the historical access record library according to each data block identifier in the set of data block identifiers to be stored, wherein the historical access record includes an access timestamp set and an associated data block identifier subset.

[0048] In the enterprise data center scenario, in the process related to the storage operation of the market research report data block, first, according to the identifier of each data block in the market research report, the corresponding historical access record is matched in the historical access record library. Since the historical access record library records the past access of all enterprise data in detail, it is possible to accurately find the records corresponding to the market research report data block. These historical access records contain an access timestamp set and an associated data block identifier subset. For example, for a specific data block in the market research report, the access timestamp set in its historical access record shows the specific time of each access in the past, which may include a series of time points since the company launched a large-scale marketing campaign, such as 10:15 on March 1, 2023, 14:30 on March 5, 2023, etc. The associated data block identifier subset contains other data block identifiers that may be involved when accessing the data block.

[0049] Step S132: extracting the access timestamp sequence in the historical access records, and calculating the periodic access interval distribution parameter and the timestamp set of the sudden access event based on the access timestamp sequence.

[0050] For example, the access timestamp sequence for a specific data block in the aforementioned market research report is, in chronological order, 10:15 AM on March 1, 2023, 2:30 PM on March 5, 2023, and so on. A periodic access interval distribution parameter is calculated based on this access timestamp sequence. The calculation process is to first count the time intervals between adjacent access timestamps. For example, the time interval from 10:15 AM on March 1, 2023, to 2:30 PM on March 5, 2023 is 4 days, 4 hours, and 15 minutes. This calculation is performed for all adjacent access timestamps. These time intervals are then analyzed, and the frequency of each time interval is counted. If the 4-day, 4-hour, and 15-minute interval occurs three times, which is the most frequent, then this interval is used as an important periodic access interval reference. By counting and analyzing all time intervals in this way, a periodic access interval distribution parameter is obtained, which can reflect the periodic pattern of access to the data block in different time periods. At the same time, in this process, the timestamp set of sudden access events can also be determined. For example, when a company conducts an annual financial audit, it needs to refer to a large amount of data in the market research report, which leads to a large number of concentrated accesses between April 1, 2023 and April 5, 2023. The timestamps during this period constitute the timestamp set of sudden access events.

[0051] Step S133 : Filtering out a set of associated data block identifiers whose access times with the data block to be stored exceed a collaboration threshold within a preset time window from the associated data block identifier subset.

[0052] Assume the preset time window is the past six months, and the collaboration threshold is set to 15. Check the number of co-accesses corresponding to each identifier in the associated data block identifier subset. For example, if a market analysis data block has been co-accessed 20 times with a market research report data block in the past six months, exceeding the collaboration threshold of 15, then filter it out. These filtered data block identifiers constitute the associated data block identifier set.

[0053] Step S134: obtaining a physical storage coordinate set of the associated data block identifier set in the all-flash storage cluster, wherein the physical storage coordinate set includes a storage node identifier and a logical unit address.

[0054] For example, for the selected market analysis data block, its storage node identifier in the all-flash storage cluster is node A, and its logical unit address is 105. There may be other related data blocks stored in node B, with a logical unit address of 200, etc. These storage node identifiers and logical unit addresses constitute a physical storage coordinate set.

[0055] Step S135 : generating a physical storage location proximity matrix of the associated data blocks based on the topological connection relationship between the storage node identifiers and the distance differences between the logical unit addresses in the physical storage coordinate set.

[0056] For example, with respect to the topological connection relationship between storage node identifiers, if Node A and Node B are connected via a high-speed network and the network latency is low, then the topological connection relationship between them is relatively close. With respect to the distance difference between logical unit addresses, for example, the distance difference between the logical unit address 105 of Node A and another logical unit address 110 of Node A is relatively small, while the distance difference between the logical unit address 200 of Node B is relatively large. By comprehensively considering these factors, the relationship between each pair of associated data blocks is quantitatively evaluated, thereby constructing a physical storage location proximity matrix for the associated data blocks. This physical storage location proximity matrix can accurately reflect the proximity relationship between each associated data block in the physical storage location, providing an important basis for subsequent data storage and management strategies.

[0057] In a possible implementation, step S140 includes:

[0058] Step S141 , extracting periodic access peak intervals, burst access event timestamps, and access interval distribution statistics from the historical access time series features.

[0059] Specifically, regarding the periodic access peak interval, a review of historical access records revealed that the data block received the most visits between 9:00 AM and 10:00 AM every weekday. This time period is the periodic access peak interval. Regarding the timestamps of sudden access events, for example, during a company's quarterly sales strategy adjustments, the market research report data block experienced a large number of sudden accesses between June 15, 2023, and June 20, 2023. These time points are the timestamps of sudden access events. For the access interval distribution statistics, the time interval between each access is counted, such as the interval between the first and second visits being 3 days, and the interval between the second and third visits being 5 days, etc. By performing statistical analysis on a large amount of such interval data, the access interval distribution statistics are obtained.

[0060] Step S142 , identifying the read-write operation ratio corresponding to the operation mode tag, and activating a hotspot data prediction mark when the proportion of read operations in the read-write operation ratio exceeds a preset threshold.

[0061] Specifically, a market research report data block is stored with an operation mode tag. Suppose analysis reveals that 70% of the operations are read operations. A threshold of 60% is set. Since the read operation ratio exceeds the threshold, the hot data prediction tag is activated.

[0062] Step S143 : generating a time dimension access probability density function by combining the periodic access peak interval and the burst access event timestamp.

[0063] To illustrate, let's use a simple calculation process. Assume that the initial access probability during the periodic access peak (9:00 AM to 10:00 AM on weekdays) is set to 0.3. The access probability during the period from June 15, 2023, to June 20, 2023, corresponding to the timestamps of sudden access events, is set to 0.2. For other time periods, corresponding probabilities are set based on factors such as historical access frequency, thereby constructing a preliminary time-based access probability density function.

[0064] Step S144 , based on the access interval distribution statistics and the hotspot data prediction mark, modify the weight parameter of the time dimension access probability density function.

[0065] Suppose the access interval distribution statistics indicate that the access probability gradually decreases between two access peaks. For example, after a periodic access peak, the access probability decreases by 0.05 every day. Since the hot data prediction flag is activated, indicating that this data block may be hot data, in this case, the weight parameters need to be adjusted to slow the decay of the access probability after the peak. For example, the original decay rate of 0.05 per day can be adjusted to 0.03 per day, thereby correcting the weight parameters of the time dimension access probability density function.

[0066] Step S145, coupling the corrected time dimension access probability density function with the network link bandwidth utilization in the multi-dimensional performance data set to generate the dynamic access heat prediction model, and the dynamic access heat prediction model outputs the heat level parameter of the data block to be stored.

[0067] For example, step S145 includes:

[0068] Step S1451: extracting a bandwidth fluctuation data set of the network link bandwidth utilization within multiple historical time windows from the multi-dimensional performance data set, wherein the bandwidth fluctuation data set includes a bandwidth utilization peak interval and an average transmission rate.

[0069] Step S1452 : Align and map the bandwidth utilization peak interval in the bandwidth fluctuation data set with the time axis of the time dimension access probability density function to generate a time-synchronized bandwidth utilization distribution sequence.

[0070] Step S1453 : Calculate the bandwidth weight factor of the time dimension access probability density function in each time unit based on the average transmission rate in the bandwidth utilization distribution sequence.

[0071] Step S1454 : performing weighted correction on the access probability value of the time dimension access probability density function in the corresponding time unit according to the bandwidth weight factor to generate a bandwidth-aware access probability density distribution function.

[0072] Step S1455: normalize the bandwidth-aware access probability density distribution function in the time dimension, and generate the heat level parameter of the dynamic access heat prediction model based on the maximum probability value of the normalized bandwidth-aware access probability density distribution function and the slope change of the probability distribution curve.

[0073] For example, consider the past 10 working days as a historical time window and calculate the network link bandwidth utilization for each working day. Peak bandwidth utilization may occur between 10:00 AM and 11:00 AM, reaching 80% and an average transmission rate of 500 Mbps. Align the bandwidth utilization peak intervals in the bandwidth fluctuation dataset with the time axis of the time-dimension access probability density function to generate a time-synchronized bandwidth utilization distribution sequence. Assume the time unit of the time-dimension access probability density function is hourly. The access probability between 9:00 AM and 10:00 AM is 0.3, and the bandwidth utilization peak is 80% between 10:00 AM and 11:00 AM. The bandwidth weight factor for each time unit in the time-dimension access probability density function is calculated as follows: First, determine the degree to which the relationship between the average transmission rate and peak bandwidth utilization affects the access probability. A higher average transmission rate and a higher peak bandwidth utilization indicate high data transmission demand and sufficient network resource utilization during that time period. Therefore, the bandwidth weight factor for that time unit should be larger. For example, using a simple calculation, if the average transmission rate is 500 Mbps and the peak bandwidth utilization is 80%, and a base weight is set to 1, then the bandwidth weight factor calculated based on these two values ​​might be 1.2 (this is just an example calculation; actual calculations will use more complex logic). Based on this bandwidth weight factor, the access probability value of the time dimension access probability density function in the corresponding time unit is weighted and corrected. For example, the original access probability from 9:00 to 10:00 AM is 0.3, but after weighted correction, it becomes 0.3 × 1.2 = 0.36, thus generating a bandwidth-aware access probability density distribution function.

[0074] Furthermore, normalization involves adjusting the access probability values ​​of all time units to a specific range, such as between 0 and 1. The calculation process involves first finding the maximum probability value in the bandwidth-aware access probability density distribution function, assuming it is 0.4. The access probability value for each time unit is then divided by this maximum probability value. For example, if the original access probability of a time unit is 0.2, after normalization, it becomes 0.2 ÷ 0.4 = 0.5. The popularity level parameter of the dynamic access popularity prediction model is generated based on the maximum probability value of the normalized bandwidth-aware access probability density distribution function and the slope of the probability distribution curve. If the maximum probability value is close to 1 and the slope of the probability distribution curve varies significantly, this indicates that the data block has high access popularity in certain time periods and that popularity fluctuates significantly. Therefore, the popularity level parameter may be set to a higher level, such as level 3 (assuming the popularity levels are 1 to 5). If the maximum probability value is small and the slope changes gradually, the popularity level parameter may be set to a lower level, such as level 1. This heat level parameter can accurately reflect the access popularity of the market research report data block within a preset time period, providing an important basis for subsequent data storage management strategies.

[0075] In a possible implementation, after step S143, the method further includes:

[0076] Step S210 : dividing a plurality of data access areas according to the node topology of the all-flash storage cluster, and configuring an independent access frequency monitor for each area.

[0077] The node topology of an all-flash storage cluster can be a complex network consisting of multiple storage nodes. For example, nodes storing corporate financial data can be divided into one zone, nodes storing sales data into another zone, and nodes storing data blocks for market research reports into a specific zone. Each of these zones can be equipped with an independent access frequency monitor, which accurately measures access activity in each zone.

[0078] Step S220 : collecting the actual number of visits distribution data of each area within the historical time window through the visit frequency monitor.

[0079] For example, let's set the historical time window for the past month as the area containing the market research report data block. Within that month, the access frequency monitor records the actual number of visits to that area on a daily basis or within a specific time period. For example, in the first few days of the month, as various company departments are just formulating their work plans, the number of visits to the market research report data block is low, perhaps only 10 per day. By the middle of the month, when the marketing plan is being adjusted, the number of visits increases to 30 per day. Near the end of the month, when the monthly summary is being conducted, the number of visits changes again, perhaps reaching 20 per day. This provides the actual distribution of visits to that area within that historical time window.

[0080] Step S230 , performing fitting verification on the actual access frequency distribution data and the time dimension access probability density function, and adjusting the curve shape of the probability density function.

[0081] To illustrate, let's use a simple calculation process. Suppose the probability density function (PDF) for visits in the time dimension sets a visit probability of 0.1 at the beginning of the month, but the actual visit count data shows fewer visits at the beginning of the month. Calculate the difference between the actual number of visits and the number of visits predicted by the PDF. For example, if the PDF predicts 15 visits per day at the beginning of the month, but there are only 10 visits per day, the difference is 5. Perform this difference calculation for each time period within the entire historical time window. If the difference is large for a particular time period, it indicates that the PDF prediction for that time period is inaccurate. For example, during the mid-month marketing plan adjustment period, the PDF predicted a visit probability of 25 visits, but the actual number was 30, a difference of 5. Based on these differences, adjust the PDF curve. If the actual number of visits in a time period exceeds the predicted number, appropriately increase the probability of visits for that time period, adjusting the curve upward; otherwise, adjust downward.

[0082] Step S240: Based on the adjusted probability density function, the heat level parameter calculation rules of the dynamic access heat prediction model are updated.

[0083] Before adjusting the probability density function, the heat level parameter calculation rules may be based on certain characteristic values ​​of the original probability density function, such as the maximum probability value and the slope of the probability distribution curve. For example, the original rule is that when the maximum probability value is greater than 0.3 and the slope is greater than a certain value, the heat level parameter is level 3. After adjusting the probability density function, these characteristic values ​​have changed. New characteristic values ​​are re-determined based on the adjusted probability density function. For example, if the maximum probability value changes to 0.35, the slope also changes. The heat level parameter calculation rules are updated based on these new characteristic values. For example, the new rule may become: when the maximum probability value is greater than 0.35 and the slope meets the new conditions, the heat level parameter is level 3. This enables the dynamic access heat prediction model to more accurately reflect the access heat of market research report data blocks under different circumstances, thereby providing a more accurate basis for operations such as data storage, management, and pre-fetching.

[0084] In a possible implementation, step S150 includes:

[0085] Step S151: Determine the shard redundancy threshold and the minimum number of copies of the data block to be stored according to the heat level parameter output by the dynamic access heat prediction model.

[0086] Assume that the market research report data block has a high popularity level parameter, indicating that it is likely to be accessed frequently in the future. Based on this, the shard redundancy threshold is set to 3, meaning that the data block can be stored in a maximum of three shards to improve data availability and reliability. At the same time, the minimum number of replicas is set to 2, meaning that each shard must have at least two replicas. This ensures data accessibility in the event of node failures, data corruption, and other situations.

[0087] Step S152 , traversing the storage space fragmentation rates and input / output request queue depths of all nodes in the all-flash storage cluster, and screening a subset of candidate nodes that meet the shard capacity constraint.

[0088] For example, an all-flash storage cluster contains multiple nodes, including Node A, Node B, and Node C. Node A has a storage space fragmentation rate of 20% and an input / output request queue depth of 50 requests at the current moment. Node B has a storage space fragmentation rate of 30% and an input / output request queue depth of 80 requests. Assuming the shard capacity constraint is that the storage space fragmentation rate does not exceed 30% and the input / output request queue depth does not exceed 100 requests, both Node A and Node B meet the shard capacity constraint and are therefore selected into the candidate node subset. However, if Node C has a storage space fragmentation rate of 40% or an input / output request queue depth of 150 requests, which does not meet the shard capacity constraint, it will not be selected into the candidate node subset.

[0089] Step S153 : calculating a storage location association score of each node in the candidate node subset based on the physical storage location proximity.

[0090] Taking the aforementioned association between the market analysis data block and the market research report data block as an example, assuming the market analysis data block is stored on node A, and a data block associated with the market research report data block is stored at a logical unit address adjacent to node A, then the storage location association score between node A and the market research report data block is high. If another node, B, is farther away from the storage location of the data block associated with the market research report data block, its storage location association score is relatively low. By comprehensively considering these factors, an accurate storage location association score is calculated for each node in the candidate node subset.

[0091] Step S154 : constructing a storage path weight table between nodes based on the storage location association score and the network link bandwidth utilization in the multi-dimensional performance data set.

[0092] For example, if the storage location association score between nodes A and B is high and the network link bandwidth utilization between them is 70%, this indicates that data transmission between the two nodes is highly efficient. Based on these factors, the storage path between nodes A and B is assigned a higher weight. If the storage location association score between nodes A and C is low and the network link bandwidth utilization is 50%, then the storage path weight between them is relatively low. By performing this evaluation on the relationships between all nodes in the candidate node subset, a complete storage path weight table between nodes is constructed.

[0093] Step S155: Based on the storage path weight table and the minimum number of replicas, a shard storage topology diagram including redundant path cross connections is generated.

[0094] For example, step S155 includes:

[0095] Step S1551 : Filter a set of candidate storage paths whose storage path weight values ​​exceed a preset weight threshold according to the inter-node storage path weight values ​​in the storage path weight table.

[0096] Step S1552: Determine the number of shard copies of the data block to be stored based on the minimum number of copies, and allocate an initial storage path to each shard copy.

[0097] Step S1553: traverse the storage path weight values ​​in the candidate storage path set, and select the primary storage path with the highest weight value as the default storage path for the shard copy.

[0098] Step S1554 : detecting the inter-node connection status of the default storage path; if a single node connection path exists, selecting a backup storage path with the second highest weight value from the candidate storage path set as a cross-redundant path.

[0099] Step S1555 : bidirectionally connect the cross-redundant path and the default storage path to generate a shard storage topology diagram including a cross-connection of redundant paths.

[0100] Assuming the preset weight threshold is 0.6, and in the storage path weight table, the storage path weight from node A to node B is 0.7, and the storage path weight from node B to node C is 0.8, then the storage paths from node A to node B and from node B to node C are selected into the candidate storage path set. Based on the minimum number of replicas, the number of shard replicas for the data block to be stored is determined to be two, and an initial storage path is assigned to each shard replica. For example, the first shard replica is assigned the storage path from node A to node B. The storage path weights in the candidate storage path set are then traversed, and the path from node A to node B is found to have the highest weight, so it is selected as the default storage path for the shard replica. The inter-node connectivity of this default storage path is checked. If only one path from node A to node B is found, there is a single point of failure risk. In this case, the backup storage path with the next highest weight (assuming the path from node A to node C has the next highest weight) is selected from the candidate storage path set as a cross-redundant path. The cross-redundant path is bidirectionally connected to the default storage path to generate a shard storage topology map containing cross-connections of redundant paths. For example, in this shard storage topology diagram, the first shard copy from node A to node B is the primary storage path, and from node A to node C is the cross redundant path. The two are connected in both directions to ensure that when node A or node B fails, the data can still be accessed through node C.

[0101] Step S1556: Verify whether the path redundancy of each shard copy in the shard storage topology diagram meets the redundancy constraint condition corresponding to the minimum number of copies. If not, reselect the backup storage path and update the cross-connection relationship of the shard storage topology diagram.

[0102] In detail, since the minimum number of replicas is 2, for each shard replica, its path redundancy should be at least 1, that is, in addition to the primary storage path, there should be at least one backup storage path. Check the path status of each shard replica in the shard storage topology. If it is found that the path redundancy of a shard replica does not meet the requirements, for example, there is only a primary storage path but no backup storage path, then reselect the backup storage path and update the cross-connection relationship of the shard storage topology. Assuming that a shard replica originally only has a primary storage path from node A to node B, which does not meet the redundancy constraint, reselect the path from node A to node C from the candidate storage path set as the backup storage path, and update the shard storage topology to establish the correct cross-connection relationship between node A to node B and node A to node C, thereby ensuring the reliability and availability of the entire shard storage topology to meet the requirements of the enterprise data center for market research report data block storage and improve data storage security and access efficiency.

[0103] In one possible implementation, the method further includes:

[0104] Step S310: Detect whether there is a single point failure risk path in the shard storage topology diagram.

[0105] For example, for a shard copy of a market research report data block, its primary storage path is from node A to node B. If this is the only storage path, with no other backup paths connected to it, then it is a single point of failure. If either node A or node B fails, such as a power supply problem on node A or damage to the storage medium on node B, the shard copy will be inaccessible, affecting the availability of the entire data block.

[0106] Step S320: If there is a single point failure risk path, then based on the real-time performance data of the candidate node subset, a spare storage node is dynamically inserted to form a ring redundant path.

[0107] Assume that the candidate node subset includes Node A, Node B, Node C, and so on. Review the real-time performance data of these nodes. For example, Node C's real-time storage space fragmentation rate is low, the input and output request queue depth is within a reasonable range, and the network link bandwidth utilization is high, indicating good performance. At this point, Node C is dynamically inserted into the path with the risk of single point failure. The previous path from Node A to Node B is expanded into a ring-shaped redundant path from Node A to Node C and then to Node B. This way, even if Node A or Node B fails, data can still be transmitted through Node C, improving data reliability. During this process, it is necessary to carefully consider the connection relationships between nodes and the direction of data flow to ensure the rationality and effectiveness of the ring-shaped redundant path.

[0108] Step S330: Optimize the replica synchronization priority in the shard storage topology diagram according to the inter-node delay data of the ring redundant path.

[0109] For example, by measuring the latency data between each node in a ring redundant path, for example, the latency from node A to node C is 5 milliseconds, the latency from node C to node B is 3 milliseconds, and the latency from node B to node A is 4 milliseconds. The priority of replica synchronization is determined based on this latency data. If a shard replica is updated on node A, since the latency from node C to node B is relatively small, the update can be synchronized to node C first, then from node C to node B, and finally from node B back to node A to ensure data consistency and timeliness. This process requires precise calculation and weighing of the impact of latency on data synchronization of different paths to ensure efficient data synchronization while improving data reliability.

[0110] Step S340 , associate and map the optimized shard storage topology map with the cache preloading strategy matrix to generate a joint storage strategy configuration file.

[0111] The cache preloading policy matrix contains cache policy information for data blocks related to the market research report data block, such as preloading priority coefficient, cache retention period, and compression level parameters. The optimized shard storage topology is associated and mapped with these cache policy information. For example, for a shard copy stored on node A in the shard storage topology, if its corresponding associated data block has a higher preloading priority coefficient in the cache preloading policy matrix, then the node A where the shard copy is located will be marked in the joint storage policy configuration file as requiring a higher priority for cache preloading operations. Through this association mapping, a comprehensive joint storage policy configuration file is generated, which will guide the entire data storage system on how to store, cache, and synchronize data for the market research report data block and its related data blocks.

[0112] In one possible implementation, the method further includes:

[0113] Step S410: Create a storage lifecycle tracking log for the data block to be stored, so as to record the creation timestamp, migration event and cache status change history of the shard copy through the storage lifecycle tracking log.

[0114] Specifically, when a shard copy of a market research report data block is created at a certain moment, the storage lifecycle tracking log will record the creation timestamp, for example, 10:15 on July 1, 2023. During the storage of the data block, if a shard copy undergoes a migration event due to performance issues on a certain node or the need for load balancing, such as migrating from node A to node C, then the migration event will be recorded in detail in the log, including information such as the starting node and target node of the migration, and the time of the migration. At the same time, the change history of the cache status will also be recorded. If the cache of a shard copy changes from the initial uncached state to the cached state, or if the cache retention period changes, this information will be recorded.

[0115] Step S420: Analyze the abnormal event sequence in the storage lifecycle tracking log to identify potential causes of storage performance bottlenecks.

[0116] For example, storage lifecycle tracking logs revealed frequent migration events for shard copies of a market research report data block during a certain period, accompanied by unstable cache status. Analyzing this sequence of abnormal events might reveal that the high storage fragmentation rate of a node (such as Node A) reduced the storage and access efficiency of data blocks, leading to frequent migration events and unstable cache status. Alternatively, it could be that network link bandwidth utilization suddenly decreased at certain moments, impacting data transmission and caching, potentially causing a storage performance bottleneck.

[0117] Step S430: Adjust the parameter update frequency of the dynamic access popularity prediction model and the calculation logic of the shard redundancy threshold according to the potential cause.

[0118] If it is found that the problem is caused by the excessively high fragmentation rate of the node's storage space, the parameter update frequency of the dynamic access heat prediction model can be adjusted. For example, the original parameter update frequency was once a day. Since changes in the storage space fragmentation rate may have a greater impact on data access heat, the parameter update frequency is increased to once an hour to more promptly reflect the actual access situation of the data. For the calculation logic of the shard redundancy threshold, if it is found that the unstable network link bandwidth utilization has a greater impact on data availability, then when calculating the shard redundancy threshold, the consideration weight of the network link bandwidth utilization will be increased. For example, the shard redundancy threshold was previously determined to be 3 based only on the heat level parameter. Now, due to the influence of network factors, the shard redundancy threshold may be adjusted to 4 based on the fluctuation of the network link bandwidth utilization to improve data reliability.

[0119] Step S440: Synchronize the adjusted parameter update frequency and calculation logic to all management nodes of the all-flash storage cluster in real time.

[0120] Every management node in the all-flash storage cluster needs to obtain this adjusted information to ensure that the entire storage system stores and manages data according to the new policy. For example, management nodes A and B need to receive and update these parameters. This allows them to make decisions about storing market research report data blocks and other data blocks based on the parameter update frequency and shard redundancy threshold calculation logic of the new dynamic access popularity prediction model, ensuring the efficiency, reliability, and data availability of the entire storage system.

[0121] In a possible implementation, step S150 further includes:

[0122] Step S156 : identifying a target data block sequence that is spatially associated with the data block to be stored according to the physical storage location proximity.

[0123] Specifically, in an all-flash storage cluster, market research report data blocks are stored at specific nodes and logical unit addresses. By analyzing physical storage location proximity, for example, by examining the relationship between storage node identifiers and logical unit addresses, other data blocks stored adjacent to the market research report data block are identified. Assuming the market research report data block is stored at logical unit address 100 on node A, data blocks with similar logical unit addresses (e.g., 101-105) on the same node may be auxiliary data blocks related to the market research report, such as raw data collection records and preliminary analysis results from the market research process. These data blocks constitute a target data block sequence that is spatially associated with the market research report data block.

[0124] Step S157 : Based on the dynamic access popularity prediction model, predict the concurrent access probability distribution of the target data block sequence within the preset time period.

[0125] The dynamic access popularity prediction model outputs information such as popularity level parameters for the market research report data block, and based on this, predicts the concurrent access probability distribution of the target data block sequence. For example, based on the high access popularity of the market research report data block from 9 to 10 am on weekdays, it is inferred that the original data collection record data block spatially associated with it also has a high concurrent access probability during this time period, which may reach 0.4. For the preliminary analysis result data block, its concurrent access probability may be slightly lower, at 0.3. For other time periods, the concurrent access probability of each time period is calculated based on the access popularity trend of the market research report data block and the degree of association between each target data block and the market research report data block, thereby obtaining the concurrent access probability distribution of the target data block sequence within the preset time period.

[0126] Step S158 , calculating the preloading priority coefficient of each target data block according to the concurrent access probability distribution and the node storage space fragmentation rate in the multi-dimensional performance data set.

[0127] For example, let's take the raw data collection record data block, whose concurrent access probability is 0.4. Assume that the storage space fragmentation rate of node A is 20%. The calculation process is as follows: If the storage space fragmentation rate is low, the node has more space for caching operations, which has a positive impact on the preloading priority. A base score can be set, such as 40 points for a concurrent access probability of 0.4. This score can then be adjusted based on the storage space fragmentation rate. Since the storage space fragmentation rate is 20%, which is relatively low, a bonus score of 10 points is given. Therefore, the preloading priority coefficient for the raw data collection record data block is 50 points. For the preliminary analysis result data block, whose concurrent access probability is 0.3 and the storage space fragmentation rate of its node is 30%, which is relatively high and has a negative impact on the preloading priority. Using the same calculation method, its base score is 30 points. Due to the high fragmentation rate, 5 points may be subtracted, resulting in a preloading priority coefficient of 25 points. This calculation method is used to calculate the preloading priority coefficient for each target data block.

[0128] Step S159: Based on the preloading priority coefficient, differentiated cache retention periods and compression level parameters are allocated to the target data block sequence, and the cache retention periods and compression level parameters are integrated according to the node dimension to generate a multi-layer structure of the cache preloading strategy matrix.

[0129] An all-flash storage cluster has multiple nodes. For each node, the cache retention period and compression level parameters for the target data blocks stored on that node are integrated. For example, on node A, the cache retention period for the original data acquisition record data block is 3 hours, with a low compression level. Preliminary analysis results in a data block with a cache retention period dynamically adjusted (assuming it's currently 1 hour) and a high compression level. This information is organized along the node dimension to form a multi-layered cache preloading strategy matrix. This cache preloading strategy matrix clearly guides the cache preloading operations for target data block sequences on different nodes of the all-flash storage cluster, including when to cache, how long to cache, and what compression ratio to use, thereby improving data access efficiency and storage resource utilization.

[0130] Wherein, step S159 includes:

[0131] Step S1591: Obtain the current cache space occupancy and historical cache replacement frequency of the edge nodes in the all-flash storage cluster.

[0132] Step S1592: When the concurrent access probability distribution is higher than a first preset threshold, a fixed retention period is allocated to the corresponding data block and the cache space is locked.

[0133] Step S1593: When the concurrent access probability distribution is lower than a second preset threshold, dynamically adjust the retention period decay rate based on the historical cache replacement frequency.

[0134] Step S1594: Variable compression rate processing is performed on the low-priority data block according to the compression level parameter, and metadata verification information of the compressed data block is recorded.

[0135] Assume that the current cache occupancy rate of an edge node is 60%, and the historical cache replacement frequency is every two hours. For raw data acquisition record data blocks with a high preload priority coefficient, when their concurrent access probability distribution exceeds a first preset threshold (assuming 0.35), a fixed retention period is assigned to the corresponding data block and the cache space is locked. For example, a fixed cache retention period of three hours is assigned. During this three-hour period, even if cache space is limited, the data block will not be replaced. For preliminary analysis result data blocks, whose concurrent access probability distribution is below a second preset threshold (assuming 0.25), the retention period decay rate is dynamically adjusted based on the historical cache replacement frequency. Since the historical cache replacement frequency is every two hours, when cache space is limited, its cache retention period decays faster, for example, by 10 minutes every 30 minutes, to free up cache space for higher-priority data blocks that need it more quickly. Low-priority data blocks are compressed at a variable rate based on the compression level parameter. Low-priority data blocks such as preliminary analysis results are compressed at a higher rate based on their compression level parameter (assuming a higher compression level). During the compression process, metadata verification information of the compressed data block, such as the checksum, is recorded to verify data integrity when the data block is subsequently accessed.

[0136] In one possible implementation, the method further includes:

[0137] Step S510: deploying a dynamic load balancing controller in the all-flash storage cluster to collect performance data fluctuation trends of each node in real time.

[0138] For example, the dynamic load balancing controller continuously monitors the performance of each node storing the market research report data block and its related data blocks. It records performance data changes over time for nodes such as Node A and Node B. For Node A, for example, it monitors changes in its storage space fragmentation rate over time, such as from 15% in the morning to 18% at noon and then to 20% in the afternoon. It also monitors fluctuations in the depth of the input and output request queues. For example, at the start of the morning business, the queue depth may be 30 requests. As the marketing department frequently accesses the market research report data block and its related data blocks, the queue depth increases to 80 requests during the morning business peak, then gradually decreases to 50 requests in the afternoon. By recording this data at different time points, performance fluctuation trends of each node can be analyzed.

[0139] Step S520: identifying potential overloaded nodes based on the performance data fluctuation trend, and triggering a data block replica migration early warning mechanism.

[0140] For example, consider observing a rapid increase in the input and output request queue depth of Node A, from 30 requests to 80 requests over the past hour. Calculate the growth rate (increase in requests divided by the time interval) and assume this rate exceeds the pre-set queue depth warning threshold (e.g., 50 requests per hour). Simultaneously, the storage space fragmentation rate of Node A also experiences a significant gradient, rising from 15% to 20%, reaching the fragmentation threshold (assuming a range of 15% to 20%). At this point, Node A is identified as a potentially overloaded node, triggering the data block replica migration warning mechanism.

[0141] Step S530 : Based on the redundant path cross-connection relationship in the shard storage topology graph, a migration target node is selected and an optimal delay parameter of the migration path is calculated.

[0142] In the sharded storage topology, node A has redundant cross-connections with other nodes (such as node B and node C). Check the real-time performance data for these nodes, including storage space fragmentation, input / output request queue depth, and network link bandwidth utilization. Assuming node B has a low storage space fragmentation rate, a small input / output request queue depth, and high network link bandwidth utilization, node B is selected as the migration target. Calculate the optimal latency parameters for the migration path from node A to node B. Consider each link between node A and node B, such as intermediate nodes and network devices. Measure the latency of each link. For example, if the latency from node A to an intermediate node is 3 milliseconds and the latency from the intermediate node to node B is 2 milliseconds, the total latency is 5 milliseconds. Also consider the impact of network congestion and data transmission rates that may occur during the migration process. By comprehensively analyzing this data, determine the optimal latency parameters for the migration path from node A to node B.

[0143] Step S540: During the migration operation, the access availability of the original data block is maintained and the joint storage policy configuration file is synchronously updated.

[0144] When starting to migrate the shard copy of the market research report data block on node A to node B, it is necessary to ensure that during the migration process, the marketing department or other departments that need to access the data block can still access the data normally. This may require the use of some technical means, such as temporary copies of data, multi-path access to data, etc. At the same time, during the migration process, the joint storage policy configuration file is synchronously updated. The joint storage policy configuration file contains information about the shard storage topology map, cache preloading strategy matrix, etc. of the market research report data block. Since the storage location of the data block has changed, the relevant information needs to be updated in the joint storage policy configuration file. For example, the original storage information about the data block on node A is modified to the storage information on node B, including updating the storage path, cache strategy and other related information of the shard copy on node B, to ensure that the strategy of the entire storage system matches the actual storage situation of the data.

[0145] Wherein, step S520 includes:

[0146] Step S521 : monitoring the growth rate of the input / output request queue depth and the change gradient of the storage space fragmentation rate of the potential overloaded node.

[0147] Step S522 : When the growth rate exceeds the queue depth warning line and the change gradient reaches a fragmentation threshold, a migration task queue is generated.

[0148] Step S523 : sorting the migration execution order and allocating migration bandwidth resources according to the data block priority tags in the migration task queue.

[0149] For example, as described above, for node A, a detailed record of a significant increase in its input / output request queue depth over a short period of time, as well as a significant increase in storage space fragmentation, is kept. When the rate of increase exceeds the queue depth warning line and the gradient of change reaches the fragmentation threshold, a migration task queue is generated. Assume that the market research report data block has multiple shard replicas on node A. These shard replicas are tagged with different data block priority tags based on factors such as their importance. When generating the migration task queue, these priority tags are used to sort the data blocks. For example, shard replicas related to the core data of the market research report are tagged with high priority, those related to auxiliary data are tagged with medium priority, and other less relevant data block replicas are tagged with low priority. Based on this priority order, high-priority data block replicas are placed at the front of the migration task queue, followed by medium priority, and finally low priority. Simultaneously, migration bandwidth resources are allocated to the data block replicas in the migration task queue based on the current network bandwidth resources of the all-flash storage cluster. If the total bandwidth is 1000 Mbps, based on factors such as the priority and size of the data blocks, 500 Mbps of bandwidth may be allocated to high-priority data block copies, 300 Mbps to medium-priority copies, and 200 Mbps to low-priority copies.

[0150] Step S524: After the migration is completed, verify the data consistency of the target node and update the node status identifier of the shard storage topology map.

[0151] After the data block copy is completely migrated from node A to node B, the consistency of the data on node B must be verified. This may include comparing the checksum of the data block to check whether the size, content, etc. of the data block are consistent with those on node A. If the data is consistent, it means that the migration is successful. Then, the node status flag of the shard storage topology map is updated, and the status flag of the data block copy on node A is changed to migrated, and the status flag on node B is changed to received and stored. Through such operations, the shard storage topology map can accurately reflect the storage location and status of the data block copy, providing an accurate basis for subsequent storage management operations.

[0152] In one possible implementation, the method further includes:

[0153] Step S610: Implement a data integrity verification loop in the all-flash storage cluster, periodically scan the checksum information of the shard replicas, and when a checksum anomaly is detected, initiate a replica repair request based on the redundant path cross-connection relationship in the shard storage topology diagram.

[0154] In this embodiment, taking a market research report data block as an example, the all-flash storage cluster scans the checksum information of the shard copies of the market research report data block stored on each node at predetermined intervals, such as every hour or every day. A checksum is a value representing the content characteristics of a data block, calculated using a specific algorithm. For each shard copy of the market research report data block, its checksum is recalculated and then compared with the previously stored checksum information.

[0155] Suppose, during a scan, it is discovered that the checksum of a shard copy of a market research report data block stored on Node A does not match the previously stored value. This indicates that the data of the shard copy may be damaged or incorrect. In this case, you can refer to the redundant path cross-connection relationship in the shard storage topology diagram. For example, the shard storage topology diagram shows that the shard copy has a primary storage path on Node A and is connected to Node B via a redundant path. Node B also stores a redundant copy of the shard copy. Based on this, you can initiate a replica repair request to Node B, requesting Node B to provide the correct shard copy data to repair the damaged copy on Node A.

[0156] Step S620: recompress the repaired data block and refresh the cache status according to the compression level parameter in the cache preloading strategy matrix, update the metadata mapping relationship in the joint storage strategy configuration file, and feed back the repair result to the user terminal.

[0157] In the cache preloading strategy matrix, each data block has a corresponding compression level parameter. Assume that the compression level parameter of the shard copy of the market research report data block is a medium compression level. After the repair is completed, the repaired data block can be re-compressed according to the medium compression level. For the compression process, the duplicate data in the data block can be processed according to the compression algorithm to reduce the storage space occupied by the data block. At the same time, since the content of the data block has changed (after repair and recompression), its cache status needs to be refreshed. If the data block was previously in a cached state in the cache and had a certain cache retention period and cache policy, then these cache-related states need to be updated according to the new situation. For example, it may be necessary to recalculate the cache retention period or adjust the cache priority.

[0158] The federated storage policy configuration file contains various information regarding the storage of market research report data blocks. The metadata mappings record the associations between data blocks and various information, such as storage location, caching policy, and compression level. As data blocks undergo repair, recompression, and cache status refresh, this associated information changes. For example, a data block's location on a storage node may change due to the repair operation, or a change in compression level may alter its relationship in the cache preload policy. Therefore, the metadata mappings in the federated storage policy configuration file need to be updated to ensure that the information in the file matches the actual storage and management of the data blocks. Finally, the repair results can be fed back to the user terminal. Users of the market research report data blocks, such as employees in the company's marketing department, can receive repair result notifications through their terminals. If the repair is successful, the notification may display "The shard replica of the market research report data block on node A has been successfully repaired. Data integrity has been restored and it can be used normally." If the repair fails, the notification may display "The shard replica of the market research report data block on node A has failed to be repaired. Please contact the administrator for further investigation." This allows relevant personnel in the company to promptly understand the status of the data blocks and take further action if necessary.

[0159] Figure 2 A schematic diagram illustrates exemplary hardware and software components of a cloud computing-based all-flash data storage optimization system 100 that can implement the concepts of the present application, as provided in some embodiments of the present application. For example, a processor 120 can be used in the cloud computing-based all-flash data storage optimization system 100 to perform the functions described in the present application.

[0160] The cloud computing-based all-flash data storage optimization system 100 can be a general-purpose server or a special-purpose server, both of which can be used to implement the cloud computing-based all-flash data storage optimization method of this application. Although only one server is shown in this application, for convenience, the functions described in this application can be implemented in a distributed manner on multiple similar platforms to balance the processing load.

[0161] For example, the cloud computing-based data all-flash storage optimization system 100 may include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and storage media 140 in different forms, such as a disk, ROM, or RAM, or any combination thereof. Exemplarily, the cloud computing-based data all-flash storage optimization system 100 may also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The method of the present application can be implemented according to these program instructions. The cloud computing-based data all-flash storage optimization system 100 also includes an input / output (I / O) interface 150 between the computer and other input / output devices.

[0162] For ease of explanation, only one processor is described in the cloud computing-based data all-flash storage optimization system 100. However, it should be noted that the cloud computing-based data all-flash storage optimization system 100 in the present application may also include multiple processors, so the steps performed by one processor described in the present application may also be performed jointly or individually by multiple processors. For example, if the processor of the cloud computing-based data all-flash storage optimization system 100 executes step A and step B, it should be understood that step A and step B may also be performed jointly by two different processors or individually in one processor. For example, the first processor executes step A, the second processor executes step B, or the first processor and the second processor execute steps A and B together.

[0163] In addition, an embodiment of the present invention further provides a readable storage medium having computer-executable instructions preset therein. When a processor executes the computer-executable instructions, the above-mentioned cloud computing-based data all-flash storage optimization method is implemented.

[0164] It should be noted that in order to simplify the description of the present invention and thus help understand one or more embodiments of the invention, in the foregoing description of the embodiments of the present invention, multiple features are sometimes combined into one embodiment, figure or description thereof.

Claims

1. A data all-flash storage optimization method based on cloud computing, characterized in that: The method comprises: Real-time collection of multi-dimensional performance data sets of each node in the all-flash storage cluster, including the node storage space fragmentation rate, input and output request queue depth and network link bandwidth utilization; Receiving a storage operation request stream sent by a user terminal, parsing a set of identifiers of data blocks to be stored and corresponding operation mode tags in the storage operation request stream; Traversing a historical access record library based on the set of identifiers of the data blocks to be stored, extracting historical access time sequence features of the data blocks to be stored and physical storage location proximity of the associated data blocks; Constructing a dynamic access popularity prediction model for the data block to be stored within a preset time period based on the multi-dimensional performance data set and the historical access time series characteristics; Generate a cross-node shard storage topology map and a cache preloading strategy matrix based on the dynamic access heat prediction model and the physical storage location proximity; Generating a cache preloading strategy matrix based on the dynamic access popularity prediction model and the physical storage location proximity includes: identifying a target data block sequence spatially associated with the data block to be stored according to the physical storage location proximity; Based on the dynamic access popularity prediction model, predicting the concurrent access probability distribution of the target data block sequence within the preset time period; Calculating a preloading priority coefficient for each target data block based on the concurrent access probability distribution and the node storage space fragmentation rate in the multidimensional performance data set; Allocating differentiated cache retention period and compression level parameters to the target data block sequence based on the preloading priority coefficient; Integrating the cache retention period and compression level parameters according to the node dimension to generate a multi-layer structure of the cache preloading strategy matrix; The step of allocating differentiated cache retention periods and compression level parameters to the target data block sequence based on the preloading priority coefficient includes: Obtaining the current cache space occupancy rate and historical cache replacement frequency of the edge nodes in the all-flash storage cluster; When the concurrent access probability distribution is higher than a first preset threshold, allocating a fixed retention period for the corresponding data block and locking the cache space; When the concurrent access probability distribution is lower than a second preset threshold, dynamically adjusting the retention period decay rate based on the historical cache replacement frequency; Variable compression rate processing is performed on the low priority data block according to the compression level parameter, and metadata verification information of the compressed data block is recorded.

2. The method for optimizing data all-flash storage based on cloud computing according to claim 1, characterized in that: The step of constructing a dynamic access popularity prediction model for the data block to be stored within a preset time period based on the multi-dimensional performance data set and the historical access time series characteristics includes: Extracting periodic access peak intervals, burst access event timestamps, and access interval distribution statistics from the historical access time series features; Identifying a read-write operation ratio corresponding to the operation mode tag, and activating a hotspot data prediction mark when a proportion of read operations in the read-write operation ratio exceeds a preset threshold; Combining the periodic access peak interval and the burst access event timestamp, generating a time dimension access probability density function; Based on the access interval distribution statistics and the hotspot data prediction mark, modifying the weight parameter of the time dimension access probability density function; The corrected time dimension access probability density function is coupled with the network link bandwidth utilization in the multi-dimensional performance data set to generate the dynamic access heat prediction model, and the dynamic access heat prediction model outputs the heat level parameter of the data block to be stored.

3. The method for optimizing data all-flash storage based on cloud computing according to claim 2, characterized in that: After generating a time dimension access probability density function by combining the periodic access peak interval and the burst access event timestamp, the method further includes: Divide the data access area into multiple areas according to the node topology of the all-flash storage cluster, and configure an independent access frequency monitor for each area; Collecting the actual number of visits to each area within a historical time window through the visit frequency monitor; Fitting and verifying the actual access frequency distribution data with the time dimension access probability density function, and adjusting the curve shape of the probability density function; Based on the adjusted probability density function, the heat level parameter calculation rules of the dynamic access heat prediction model are updated.

4. The method for optimizing data all-flash storage based on cloud computing according to claim 1, characterized in that: Based on the dynamic access popularity prediction model and the physical storage location proximity, a cross-node shard storage topology diagram is generated, including: Determining a shard redundancy threshold and a minimum number of copies of the data block to be stored according to the heat level parameter output by the dynamic access heat prediction model; Traversing the storage space fragmentation rates and input and output request queue depths of all nodes in the all-flash storage cluster, and screening a subset of candidate nodes that meet the shard capacity constraints; Calculating a storage location relevance score for each node in the candidate node subset based on the physical storage location proximity; Constructing a storage path weight table between nodes based on the storage location association score and the network link bandwidth utilization in the multi-dimensional performance data set; Based on the storage path weight table and the minimum number of replicas, a shard storage topology diagram including redundant path cross connections is generated.

5. The method for optimizing data all-flash storage based on cloud computing according to claim 4, characterized in that: The method further comprises: Detect whether there is a single point failure risk path in the shard storage topology diagram; If there is a single point failure risk path, dynamically insert a backup storage node based on the real-time performance data of the candidate node subset to form a ring redundant path; Optimizing the replica synchronization priority in the shard storage topology diagram according to the inter-node delay data of the ring redundant path; The optimized shard storage topology map is associated and mapped with the cache preloading strategy matrix to generate a joint storage strategy configuration file.

6. The method for optimizing data all-flash storage based on cloud computing according to claim 4, characterized in that: The method further comprises: Creating a storage lifecycle tracking log for the data block to be stored, so as to record the creation timestamp, migration events and cache status change history of the shard copy through the storage lifecycle tracking log; Analyzing abnormal event sequences in the storage lifecycle tracking log to identify potential causes of storage performance bottlenecks; Adjusting the parameter update frequency of the dynamic access popularity prediction model and the calculation logic of the shard redundancy threshold according to the potential cause; The adjusted parameter update frequency and calculation logic are synchronized to all management nodes of the all-flash storage cluster in real time.

7. The method for optimizing data all-flash storage based on cloud computing according to claim 5, characterized in that: The method further comprises: Deploy a dynamic load balancing controller in the all-flash storage cluster to collect performance data fluctuation trends of each node in real time; Identify potential overloaded nodes based on the performance data fluctuation trend and trigger a data block replica migration early warning mechanism; Based on the redundant path cross-connection relationship in the shard storage topology graph, selecting a migration target node and calculating an optimal delay parameter of the migration path; During the migration operation, maintaining the access availability of the original data block and synchronously updating the joint storage policy configuration file; The triggering of the data block replica migration early warning mechanism includes: Monitoring the growth rate of the input and output request queue depth and the change gradient of the storage space fragmentation rate of the potential overloaded node; When the growth rate exceeds the queue depth warning line and the change gradient reaches the fragmentation threshold, a migration task queue is generated; sorting the migration execution order and allocating migration bandwidth resources according to the data block priority tags in the migration task queue; After the migration is completed, the data consistency of the target node is verified and the node status identifier of the shard storage topology map is updated.

8. The method for optimizing data all-flash storage based on cloud computing according to claim 5, characterized in that: The method further comprises: Implementing a data integrity verification cycle in the all-flash storage cluster, periodically scanning checksum information of shard replicas, and initiating a replica repair request based on redundant path cross-connection relationships in the shard storage topology graph when a checksum anomaly is detected; According to the compression level parameter in the cache preloading strategy matrix, recompression and cache status refresh operations are performed on the repaired data block, and the metadata mapping relationship in the joint storage strategy configuration file is updated and the repair result is fed back to the user terminal.

9. A data all-flash storage optimization system based on cloud computing, characterized in that: The cloud computing-based data all-flash storage optimization system includes a processor and a memory, the memory is connected to the processor, the memory is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the memory to implement the cloud computing-based data all-flash storage optimization method described in any one of claims 1 to 8.

Images