Dynamic Optimization Methods for Distributed Storage and Intelligent Caching for High-Concurrency Access

Abstract

This invention discloses a dynamic optimization method for distributed storage and intelligent caching for high-concurrency access, comprising: constructing a node basic index table for a distributed system; establishing a cache time window of length T seconds; updating the node basic index table to a node advanced index table; filtering and sorting high-frequency data blocks based on the window access heat of N data blocks to generate a high-frequency data block sequence; filtering data blocks to be migrated and generating a data block migration sequence; performing data block identifier updates on the node advanced index table to generate an updated node advanced index table; and entering the next cache optimization cycle based on the updated node advanced index table. This invention can reduce invalid replication of high-frequency data blocks on low-load nodes, enabling cache replicas to be preferentially extended to low-load nodes, and allowing subsequent cache time windows to continue load calculation and cache optimization based on the updated cache replica distribution.

Description

Technical Field

[0001] This invention relates to the field of cache optimization, specifically to a dynamic optimization method for distributed storage and intelligent caching for high-concurrency access. Background Technology

[0002] As file data, object data, business record data, and log data are continuously written to a distributed system, the data to be stored is typically divided into multiple data blocks and distributed across multiple storage nodes according to a certain placement strategy. To reduce frequent reads from the underlying persistent storage, storage nodes are usually configured with local cache units to store copies of frequently accessed data blocks. When clients or business servers access certain data blocks intensively within a short period of time, the corresponding data blocks will be accessed frequently in some local cache units. If these cache copies are concentrated in a few storage nodes, it can easily cause the CPU, memory, and network bandwidth of the corresponding storage nodes to be continuously under high load.

[0003] Existing cache optimization methods often adjust cache content based on access frequency, recent access time, or fixed eviction rules. They typically only focus on whether a data block is frequently accessed, lacking synchronous judgment on "which local cache units the data block is currently cached in" and "the current load status of the corresponding storage node." Therefore, within a certain cache time window, even if a data block exhibits high-frequency access, it's difficult to further determine whether it resides on a high-load storage node. This can easily lead to unnecessary replication of high-frequency data blocks on low-load nodes, or failure to promptly replicate high-frequency data blocks from high-load nodes to low-load nodes, impacting cache replica distribution and window access load balancing in distributed systems under high-concurrency access scenarios. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a dynamic optimization method for distributed storage and intelligent caching for high-concurrency access, which solves the technical problems mentioned in the background by introducing synchronization judgment of high-frequency data blocks.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A dynamic optimization method for distributed storage and intelligent caching for high-concurrency access includes the following steps: S1. Construct the basic node index table for the distributed system; The node basic index table uses the node identifiers of M storage nodes as index items to record the write correspondence between the data block identifiers and node identifiers of N data blocks; each storage node is configured with a unique local cache unit, and the local cache unit is assigned a local cache identifier. S2. Establish a cache time window with a length of T seconds along the access time axis of the distributed system; S3. Based on the access requests within the cache time window, update the node basic index table to the node advanced index table; S4. Filter and sort high-frequency data blocks based on window access popularity of N data blocks to generate a high-frequency data block sequence; S5. Based on the high-frequency data block sequence and the node advancement index table, filter the data blocks to be migrated and generate the data block migration sequence; S6. Based on the data block migration sequence and the window access load in the node advanced index table, perform data block identifier update on the node advanced index table to generate an updated node advanced index table. S7. Recalculate the window access load of M storage nodes based on the updated node advanced index table, and enter the next cache optimization cycle based on the next cache time window.

[0006] In some specific embodiments, the construction of a node-based index table for the distributed system includes: S1-1, Receive the write request for the data to be stored; S1-2. In response to the write request, the data to be stored is divided into N data blocks; wherein each data block is assigned a data block identifier, and N is a positive integer that changes dynamically with the write or delete operation; S1-3. Based on the preset placement strategy, write N data blocks and their data block identifiers into M storage nodes of the distributed system to construct the node basic index table of the distributed system.

[0007] In some specific embodiments, based on access requests within the cache time window, the node basic index table is updated to the node advanced index table, including: S3-1. Respond to access requests within the cache time window, and calculate the window access heat of N data blocks and the window access load of M storage nodes respectively; where the window access heat is associated with the data block identifier and the window access load is associated with the node identifier. S3-2. Read the cache directories of M local cache units within the cache time window to obtain the cached data block identifiers corresponding to each local cache identifier; wherein, each cached data block identifier inherits the window access popularity of its corresponding data block identifier; S3-3. Write the window access load, cached data block identifier, and window access popularity inherited by the cached data block identifier into the node basic index table, and update the node advanced index table.

[0008] In some specific embodiments, calculating the window access heat of N data blocks includes: S3-1-A1, Read the P access requests that fall within the cache time window; where P is a non-negative integer; S3-1-A2: Divide the P access requests into Q data block access events according to the data block identifiers they carry; where each data block access event includes an access timestamp and a data block identifier, and Q is a non-negative integer; S3-1-A3: Among Q data block access events, anchor the data block access event with the same data block identifier; S3-1-A4. Divide the data block access events with the same data block identifier into the same data block access event group until K data block access event groups are obtained; where K is the number of data blocks accessed within the cache time window, and K≤N; S3-1-A5, Select the event groups to be counted one by one along the K data blocks; S3-1-A6, Count the number of data block access events in the event group to be counted, and extract the access timestamp of each data block access event in the event group to be counted; S3-1-A7. Calculate the time interval between each access timestamp and the end time of the cache time window; S3-1-A8. Based on the number of data block access events and the time interval, perform time decay weighting on the event group to be counted to obtain the event group popularity of the event group to be counted. S3-1-A9, Define the event group popularity as the window access popularity of the data block identifier corresponding to the event group to be counted; S3-1-A10, Traverse the K data block access event groups, and repeatedly execute S3-1-A5 to S3-1-A9 until the window access heat of the K accessed data blocks is obtained; S3-1-A11. Mark the data blocks that do not appear in the access event group of K data blocks out of N data blocks as unaccessed data blocks, and set the window access heat of each unaccessed data block to 0; S3-1-A12. Combine the window access heat of K accessed data blocks and the window access heat of NK unaccessed data blocks to obtain the window access heat of N data blocks under the cache time window.

[0009] In some specific embodiments, the window access load of M storage nodes is calculated, including: S3-1-B1: Obtain C load parameters for each of the M storage nodes within the cache time window; wherein, the C load parameters include at least CPU utilization, memory usage, and network bandwidth utilization. S3-1-B2. Perform dimensionless processing on the C load parameters to obtain the characteristics of the C load parameters; S3-1-B3: Perform a weighted summation on the C load parameter features to obtain the window access load of M storage nodes; where each node identifier is associated with the corresponding window access load.

[0010] In some specific embodiments, the window access load, cached data block identifiers, and window access popularity inherited from the cached data block identifiers are written into the node basic index table, and the node advanced index table is updated and generated, including: S3-3-1. Synchronously anchor the M node identifiers in the node base index table, as well as each local cache identifier and the corresponding cached data block identifier in the cache directory; S3-3-2. Based on the correspondence between node identifiers and local cache identifiers, match the cached data block identifiers corresponding to each local cache identifier to the corresponding node identifier; S3-3-3. Write the window access load corresponding to each node identifier, the matched cached data block identifier, and the window access heat corresponding to the cached data block identifier into the node basic index table to generate the node advanced index table.

[0011] In some specific embodiments, high-frequency data blocks are filtered and sorted based on the window access popularity of N data blocks to generate a high-frequency data block sequence, including: S4-1. Select the current data block to be judged one by one along the data block identifiers of N data blocks; S4-2. Compare the window access popularity of the current data block to be judged with the popularity threshold; if the window access popularity of the current data block to be judged is greater than or equal to the popularity threshold, then mark the current data block to be judged as a high-frequency data block. S4-3: Traverse N data blocks and repeat S4-1 to S4-2 until N1 high-frequency data blocks are marked; S4-4. Sort the N1 high-frequency data blocks in descending order according to their window access frequency to generate a high-frequency data block sequence.

[0012] In some specific embodiments, based on the high-frequency data block sequence and the node progression index table, data blocks to be migrated are selected and a data block migration sequence is generated, including: S5-1. Select the current high-frequency data block one by one along the ordered direction of the high-frequency data block sequence; S5-2. Based on the data block identifier of the current high-frequency data block, query the J local cache identifiers of the current high-frequency data block in the node advanced index table; S5-3, anchor J local cache identifiers to J node identifiers one-to-one, and extract the window access load corresponding to the J node identifiers; S5-4. Select the highest window access load from the window access loads corresponding to the J node identifiers, and determine the node identifier corresponding to the highest window access load as the highest load node identifier. S5-5. If the highest window access load is greater than the load threshold, mark the current high-frequency data block as a data block to be migrated, and record the data block identifier, the highest load node identifier, and the window access heat of the current high-frequency data block. S5-6. Traverse the N1 high-frequency data blocks in the high-frequency data block sequence and repeat S5-1 to S5-5 until N2 data blocks to be migrated are obtained. S5-7. Sort the N2 data blocks to be migrated in descending order of their window access popularity to generate a data block migration sequence; wherein, each data block to be migrated in the data block migration sequence is associated with its data block identifier, the identifier of the highest load node and its window access popularity.

[0013] In some specific embodiments, the node advancement index table is updated with data block identifiers based on the data block migration sequence and the window access load in the node advancement index table to generate an updated node advancement index table, including: S6-1. Select the current data block to be migrated one by one along the ordered direction of the data block migration sequence; S6-2. Based on the window access load corresponding to the M node identifiers in the node advancement index table, select candidate node identifiers one by one in the direction of window access load from low to high. S6-3. If the candidate node identifier is the same as the highest load node identifier associated with the current migrated data block, or if the local cache unit corresponding to the candidate node identifier has already cached the current migrated data block, then continue to select the next candidate node identifier according to the direction of window access load from low to high. S6-4. If the candidate node identifier is different from the highest load node identifier associated with the current migrated data block, and the local cache unit corresponding to the candidate node identifier does not cache the current migrated data block, then the candidate node identifier is determined as the target node identifier. S6-5. Copy the current migrated data block to the local cache unit corresponding to the target node identifier, and simultaneously write the data block identifier of the current migrated data block to the local cache identifier corresponding to the target node identifier in the node advanced index table; S6-6. Traverse the N2 data blocks to be migrated in the data block migration sequence, repeatedly execute the write of the current data block to be migrated, until the cache copy of the N2 data blocks to be migrated is completed, and obtain the updated node advanced index table.

[0014] This invention provides a dynamic optimization method for distributed storage and intelligent caching for high-concurrency access, which has the following beneficial effects: This invention constructs a basic node index table with the node identifier of the storage node as the index item, fixing the basic correspondence between storage nodes, data block identifiers, and local cache identifiers. Within each cache time window, the window access popularity of data blocks, the window access load of storage nodes, and the cached data block identifiers in the local cache unit are written into the basic node index table to generate an advanced node index table. Thus, the advanced node index table can simultaneously represent data block access popularity, cache replica location, and window access load status, enabling subsequent cache optimization to move from relying solely on access frequency to incorporating the actual load status of the node where the data block resides for migration decisions.

[0015] Furthermore, this invention uses a node advancement index table to select data blocks to be migrated from high-frequency data blocks located on high-load nodes, and generates a data block migration sequence according to window access frequency. During cache replication, target nodes are preferentially identified from storage nodes with lower loads that do not cache the data block. The data block to be migrated is replicated to the local cache unit corresponding to the target node, and the node advancement index table is updated. This reduces invalid replication of high-frequency data blocks on low-load nodes, allows cache replicas to be preferentially extended to low-load nodes, and enables subsequent cache time windows to continue load calculation and cache optimization based on the updated cache replica distribution. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the dynamic optimization method for distributed storage and intelligent caching for high-concurrency access according to the present invention. Figure 2 This is a schematic diagram illustrating the process of generating the node advanced index table described in this invention; Figure 3 This is a schematic diagram illustrating the update process of the node advanced index table described in this invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example 1: Please refer to Figure 1 This invention provides a dynamic optimization method for distributed storage and intelligent caching for high-concurrency access, comprising the following steps: S1. Construct the basic node index table for the distributed system; The node basic index table uses the node identifiers of M storage nodes as index items to record the write correspondence between the data block identifiers and node identifiers of N data blocks; each storage node is configured with a unique local cache unit, and the local cache unit is assigned a local cache identifier. S2. Establish a cache time window with a length of T seconds along the access time axis of the distributed system; Specifically, the access timeline of the distributed system is formed according to the arrival time sequence of access requests, and is used to limit the statistical range of access requests, data block access events, and window access popularity.

[0019] S3. Based on the access requests within the cache time window, update the node basic index table to the node advanced index table; S4. Filter and sort high-frequency data blocks based on window access popularity of N data blocks to generate a high-frequency data block sequence; S5. Based on the high-frequency data block sequence and the node advancement index table, filter the data blocks to be migrated and generate the data block migration sequence; S6. Based on the data block migration sequence and the window access load in the node advanced index table, perform data block identifier update on the node advanced index table to generate an updated node advanced index table. S7. Recalculate the window access load of M storage nodes based on the updated node advanced index table, and enter the next cache optimization cycle based on the next cache time window.

[0020] Specifically, the next cache optimization cycle corresponds to the cache time window obtained by sliding up and down the access time axis once; within the next cache optimization cycle, the access heat of the data block window under the corresponding cache time window is recalculated, and cache optimization continues to be performed based on the recalculated window access load.

[0021] In this embodiment, the window access heat of data blocks and the window access load of storage nodes are obtained synchronously through cache time windows. The cached data block identifier, window access heat, and window access load are written together into the node basic index table, so that the node advanced index table reflects the data block access activity, the location of cache replicas, and the window access load status. Based on this, data blocks to be migrated are selected from high-frequency data blocks and copied to the local cache unit corresponding to the target node. This allows the updated node advanced index table to record the distribution status of data blocks after cache copying and provides an updated window access load basis for cache optimization in the next cache time window.

[0022] Example 2: See Figures 2 to 3 This embodiment further discloses the specific execution steps of each step based on embodiment 1.

[0023] The specific execution steps of step S1 include: S1-1, Receive the write request for the data to be stored; Specifically, the data to be stored includes file data, object data, business record data, or log data to be written into the distributed system.

[0024] S1-2. In response to the write request, the data to be stored is divided into N data blocks; wherein each data block is assigned a data block identifier, and N is a positive integer that changes dynamically with the write or delete operation; Specifically, the division of N data blocks is based on preset block division rules; in this embodiment, the block division rules include fixed-size blocks, blocks based on data object boundaries, or blocks based on access granularity.

[0025] Data block identifiers can be generated based on at least one of the data source identifier, block sequence number, and data block hash value of the data to be stored.

[0026] S1-3. Based on the preset placement strategy, write N data blocks and their data block identifiers into M storage nodes of the distributed system to construct the node basic index table of the distributed system. The distributed system includes multiple storage nodes for performing persistent data storage and cache access responses. Specifically, the placement strategy is preferably a placement strategy based on the available storage capacity of the nodes and the evenness of data block distribution, and its steps include: ① Obtain the available storage capacity of M storage nodes; ② Sort the M storage nodes in descending order according to their available storage capacity to obtain the sequence of available node capacity; ③ Select the data blocks to be placed one by one in the order of the data block identifiers of the N data blocks; ④ Select a storage node from the available capacity sequence whose available storage capacity meets the storage requirements of the data block to be placed; ⑤ Write the data block identifier of the data block to be placed under the node identifier of the selected storage node, and update the available storage capacity of the selected storage node after writing; Repeat the node selection and writing process for the data blocks to be placed until all N data blocks are allocated to their corresponding storage nodes.

[0027] It should be noted that the node base index table is used to record the distribution of persistent data and the correspondence between local cache units of each storage node, but does not include window access load and cached data block status within the cache time window.

[0028] In this embodiment, the specific execution steps of step S3 include: S3-1. Respond to access requests within the cache time window, and calculate the window access heat of N data blocks and the window access load of M storage nodes respectively; where the window access heat is associated with the data block identifier and the window access load is associated with the node identifier. S3-2. Read the cache directories of M local cache units within the cache time window to obtain the cached data block identifiers corresponding to each local cache identifier; wherein, each cached data block identifier inherits the window access popularity of its corresponding data block identifier; It should be noted that the cache directory is a cache record maintained by the local cache unit, which at least records the identifier of the data block currently stored by the local cache unit; when a local cache unit does not cache a data block, its corresponding cached data block identifier is empty.

[0029] S3-3. Write the window access load, cached data block identifier, and window access popularity inherited by the cached data block identifier into the node basic index table, and update the node advanced index table.

[0030] In this embodiment, the specific execution steps for calculating the window access heat of N data blocks in step S3-1 include: S3-1-A1, Read the P access requests that fall within the cache time window; where P is a non-negative integer; The access request is a data read request initiated by the client or business server for one or more data blocks. The access request carries at least an access timestamp and the identifier of the data block to be accessed.

[0031] S3-1-A2: Divide the P access requests into Q data block access events according to the data block identifiers they carry; where each data block access event includes an access timestamp and a data block identifier, and Q is a non-negative integer; Among them, a data block access event is used to represent an access to a data block under an access timestamp; when an access request carries multiple data block identifiers, the access request is split into multiple data block access events, and multiple data block access events have the same access timestamp and different data block identifiers.

[0032] S3-1-A3: Among Q data block access events, anchor the data block access event with the same data block identifier; S3-1-A4. Divide the data block access events with the same data block identifier into the same data block access event group until K data block access event groups are obtained; where K is the number of data blocks accessed within the cache time window, and K≤N; Specifically, a data block access event group corresponds to all access events of a data block within a cache time window, and each data block access event in the same data block access event group has the same data block identifier.

[0033] S3-1-A5, Select the event groups to be counted one by one along the K data blocks; S3-1-A6, Count the number of data block access events in the event group to be counted, and extract the access timestamp of each data block access event in the event group to be counted; S3-1-A7. Calculate the time interval between each access timestamp and the end time of the cache time window; S3-1-A8. Based on the number of data block access events and the time interval, perform time decay weighting on the event group to be counted to obtain the event group popularity of the event group to be counted. S3-1-A9, Define the event group popularity as the window access popularity of the data block identifier corresponding to the event group to be counted; The formula for calculating the window access popularity is: The formula for calculating the window access popularity is: ; in, The table represents the window access popularity of the data block identifier corresponding to the k-th data block access event group. This indicates the number of data block access events in the k-th data block access event group. Indicates the end time of the cache time window. This represents the access timestamp of the q-th data block access event within the k-th data block access event group. This represents the time decay coefficient.

[0034] Specifically, window access heat is calculated by counting the number of times the same data block is accessed within the cache time window, combined with the time interval between each access occurrence and the end time of the cache time window. An exponential function with a base of the natural constant is used to weight each access event by time decay. The smaller the time interval, the closer the corresponding exponential decay value is to 1, and the greater the contribution of that access event to the window access heat; the larger the time interval, the closer the corresponding exponential decay value is to 0, and the smaller the contribution of that access event to the window access heat. The exponential decay values ​​of all access events corresponding to the same data block are summed to obtain the window access heat of that data block within the cache time window.

[0035] S3-1-A10, Traverse the K data block access event groups, and repeatedly execute S3-1-A5 to S3-1-A9 until the window access heat of the K accessed data blocks is obtained; S3-1-A11. Mark the data blocks that do not appear in the access event group of K data blocks out of N data blocks as unaccessed data blocks, and set the window access heat of each unaccessed data block to 0; S3-1-A12. Combine the window access heat of K accessed data blocks and the window access heat of NK unaccessed data blocks to obtain the window access heat of N data blocks under the cache time window.

[0036] In this embodiment, by splitting access requests within the cache time window into data block access events and classifying them into data block access event groups according to data block identifiers, each accessed data block corresponds to a window access heat; for data blocks that do not appear in the cache time window, their window access heat is set to 0, thereby forming a complete access heat distribution of N data blocks under the same cache time window.

[0037] In this embodiment, the specific execution steps for calculating the window access load of the M storage nodes in step S3-1 further include: S3-1-B1: Obtain C load parameters for each of the M storage nodes within the cache time window; wherein, the C load parameters include at least CPU utilization, memory usage, and network bandwidth utilization. S3-1-B2. Perform dimensionless processing on the C load parameters to obtain the characteristics of the C load parameters; S3-1-B3: Perform a weighted summation on the C load parameter features to obtain the window access load of M storage nodes; where each node identifier is associated with the corresponding window access load.

[0038] Specifically, the weighting coefficients of load parameter characteristics can be preset according to the sensitivity of access response latency to each load parameter.

[0039] In this embodiment, by making the CPU utilization, memory usage, and network bandwidth utilization dimensionless and weighted summation, the window access load of the M storage nodes has a uniform numerical scale. After each window access load is associated with a node identifier, it can be used to determine whether the storage node caching high-frequency data blocks is in a high-load state.

[0040] In this embodiment, the specific execution steps of step S3-3 include: S3-3-1. Synchronously anchor the M node identifiers in the node base index table, as well as each local cache identifier and the corresponding cached data block identifier in the cache directory; S3-3-2. Based on the correspondence between node identifiers and local cache identifiers, match the cached data block identifiers corresponding to each local cache identifier to the corresponding node identifier; S3-3-3: Write the window access load corresponding to each node identifier, the matched cached data block identifier, and the window access heat corresponding to the cached data block identifier into the node basic index table to generate the node advanced index table. The window access heat corresponding to the cached data block identifier is determined by the window access heat of N data blocks under the current cache time window; if the cached data block is not accessed within the current cache time window, its window access heat is 0.

[0041] In this embodiment, the specific execution steps of step S4 include: S4-1. Select the current data block to be judged one by one along the data block identifiers of N data blocks; S4-2. Compare the window access popularity of the current data block to be judged with the popularity threshold; if the window access popularity of the current data block to be judged is greater than or equal to the popularity threshold, then mark the current data block to be judged as a high-frequency data block. In this embodiment, the heat threshold is used to filter highly active data blocks from the access heat distribution under the current cache time window; data blocks that do not reach the heat threshold are not included in the migration determination of the cache time window.

[0042] S4-3: Traverse N data blocks and repeat S4-1 to S4-2 until N1 high-frequency data blocks are marked; S4-4. Sort the N1 high-frequency data blocks in descending order according to their window access frequency to generate a high-frequency data block sequence.

[0043] In this embodiment, N1 high-frequency data blocks are selected from N data blocks by using a heat threshold, and the N1 high-frequency data blocks are sorted in descending order according to the window access heat, so that the migration determination only applies to the data blocks that are actively accessed within the current cache time window, and the data blocks with higher window access heat are processed first.

[0044] In this embodiment, the specific execution steps of step S5 include: S5-1. Select the current high-frequency data block one by one along the ordered direction of the high-frequency data block sequence; S5-2. Based on the data block identifier of the current high-frequency data block, query the J local cache identifiers of the current high-frequency data block in the node advanced index table; S5-3, anchor J local cache identifiers to J node identifiers one-to-one, and extract the window access load corresponding to the J node identifiers; S5-4. Select the highest window access load from the window access loads corresponding to the J node identifiers, and determine the node identifier corresponding to the highest window access load as the highest load node identifier. S5-5. If the highest window access load is greater than the load threshold, mark the current high-frequency data block as a data block to be migrated, and record the data block identifier, the highest load node identifier, and the window access heat of the current high-frequency data block. In this embodiment, when the window access load is normalized to the [0,1] interval, the load threshold ranges from 0.75 to 0.85. This range can be used to filter storage nodes that are already under high access pressure or resource consumption, avoiding unnecessary cache copying of high-frequency data blocks on low-load nodes.

[0045] S5-6. Traverse the N1 high-frequency data blocks in the high-frequency data block sequence and repeat S5-1 to S5-5 until N2 data blocks to be migrated are obtained. S5-7. Sort the N2 data blocks to be migrated in descending order of their window access popularity to generate a data block migration sequence; wherein, each data block to be migrated in the data block migration sequence is associated with its data block identifier, the identifier of the highest load node and its window access popularity. Specifically, the highest load node identifier is the node with the highest window access load among the multiple storage nodes caching the data block to be migrated; window access heat is used to determine the cache replication priority of the data block to be migrated.

[0046] In this embodiment, the local cache identifier corresponding to the current high-frequency data block is queried in the node advanced index table, and the highest load node identifier is selected from the corresponding node identifiers, so that each data block to be migrated is associated with its current high load cache node; then the data blocks to be migrated are sorted in descending order according to the window access popularity, so that cache replication is performed preferentially on the high-frequency data blocks on the high load nodes.

[0047] In this embodiment, the specific execution steps of step S6 include: S6-1. Select the current data block to be migrated one by one along the ordered direction of the data block migration sequence; S6-2. Based on the window access load corresponding to the M node identifiers in the node advancement index table, select candidate node identifiers one by one in the direction of window access load from low to high. S6-3. If the candidate node identifier is the same as the highest load node identifier associated with the current migrated data block, or if the local cache unit corresponding to the candidate node identifier has already cached the current migrated data block, then continue to select the next candidate node identifier according to the direction of window access load from low to high. Among them, the candidate node identifier is the same as the highest load node identifier, indicating that the candidate node is the high load cache node corresponding to the current migrated data block; the local cache unit corresponding to the candidate node has cached the current migrated data block, indicating that the candidate node already has a cache copy of the current migrated data block.

[0048] S6-4. If the candidate node identifier is different from the highest load node identifier associated with the current migrated data block, and the local cache unit corresponding to the candidate node identifier does not cache the current migrated data block, then the candidate node identifier is determined as the target node identifier. During the selection of the target node identifier, the target node identifier is determined first from the storage nodes with lower window access load.

[0049] S6-5. Copy the current migrated data block to the local cache unit corresponding to the target node identifier, and simultaneously write the data block identifier of the current migrated data block to the local cache identifier corresponding to the target node identifier in the node advanced index table; Specifically, the replication is a cache copy replication, which does not change the persistent storage location of the current migrated data block in the distributed system; the node advanced index table is used to record that the local cache unit corresponding to the target node identifier has added a cache copy of the current migrated data block.

[0050] S6-6. Traverse the N2 data blocks to be migrated in the data block migration sequence, repeatedly execute the write of the current data block to be migrated, until the cache copy of the N2 data blocks to be migrated is completed, and obtain the updated node advanced index table. During the cache replication process, each time a current migration data block is replicated, the corresponding target node identifier, local cache identifier, and newly added cache data block identifier are synchronously added to the node advanced index table.

[0051] In this embodiment, candidate node identifiers are selected according to the window access load from low to high, and the node identifier with the highest load and the node that has cached the current migration data block are excluded. This allows the current migration data block to be copied to a local cache unit with a lower load and which has not yet saved a copy of the data block. After the copy is completed, the identifier of the data block with the newly cached copy is written into the node advanced index table, so that the updated node advanced index table records the distribution status of the cached data blocks after the copy.

[0052] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A dynamic optimization method for distributed storage and intelligent caching for high-concurrency access, characterized in that, include: S1. Construct the basic node index table for the distributed system; The node basic index table uses the node identifiers of M storage nodes as index items to record the write correspondence between the data block identifiers and node identifiers of N data blocks; each storage node is configured with a unique local cache unit, and the local cache unit is assigned a local cache identifier. S2. Establish a cache time window with a length of T seconds along the access time axis of the distributed system; S3. Based on the access requests within the cache time window, update the node basic index table to the node advanced index table; S4. Filter and sort high-frequency data blocks based on window access popularity of N data blocks to generate a high-frequency data block sequence; S5. Based on the high-frequency data block sequence and the node advancement index table, filter the data blocks to be migrated and generate a data block migration sequence; S6. Based on the data block migration sequence and the window access load in the node advanced index table, perform data block identifier update on the node advanced index table to generate an updated node advanced index table. S7. Recalculate the window access load of M storage nodes based on the updated node advanced index table, and enter the next cache optimization cycle based on the next cache time window.

2. The distributed storage and intelligent caching dynamic optimization method for high-concurrency access as described in claim 1, characterized in that, Construct the node base index table for the distributed system, including: S1-1, Receive the write request for the data to be stored; S1-2. In response to the write request, the data to be stored is divided into N data blocks; wherein each data block is assigned a data block identifier, and N is a positive integer that changes dynamically with the write or delete operation; S1-3. Based on the preset placement strategy, write N data blocks and their data block identifiers into M storage nodes of the distributed system to construct the node basic index table of the distributed system.

3. The distributed storage and intelligent caching dynamic optimization method for high-concurrency access as described in claim 1, characterized in that, Based on the access requests within the cache time window, the node basic index table is updated to the node advanced index table, including: S3-1. Respond to access requests within the cache time window, and calculate the window access heat of N data blocks and the window access load of M storage nodes respectively; where the window access heat is associated with the data block identifier and the window access load is associated with the node identifier. S3-2. Read the cache directories of M local cache units within the cache time window to obtain the cached data block identifiers corresponding to each local cache identifier; wherein, each cached data block identifier inherits the window access popularity of its corresponding data block identifier; S3-3. Write the window access load, cached data block identifier, and window access popularity inherited by the cached data block identifier into the node basic index table, and update the node advanced index table.

4. The distributed storage and intelligent caching dynamic optimization method for high-concurrency access according to claim 3, characterized in that, Calculate the window access frequency of N data blocks, including: S3-1-A1, Read the P access requests that fall within the cache time window; where P is a non-negative integer; S3-1-A2: Divide the P access requests into Q data block access events according to the data block identifiers they carry; where each data block access event includes an access timestamp and a data block identifier, and Q is a non-negative integer; S3-1-A3: Among Q data block access events, anchor the data block access event with the same data block identifier; S3-1-A4. Divide the data block access events with the same data block identifier into the same data block access event group until K data block access event groups are obtained; where K is the number of data blocks accessed within the cache time window, and K≤N; S3-1-A5, Select the event groups to be counted one by one along the K data blocks; S3-1-A6, Count the number of data block access events in the event group to be counted, and extract the access timestamp of each data block access event in the event group to be counted; S3-1-A7. Calculate the time interval between each access timestamp and the end time of the cache time window; S3-1-A8. Based on the number of data block access events and the time interval, perform time decay weighting on the event group to be counted to obtain the event group popularity of the event group to be counted. S3-1-A9, Define the event group popularity as the window access popularity of the data block identifier corresponding to the event group to be counted; S3-1-A10, Traverse the K data block access event groups, and repeatedly execute S3-1-A5 to S3-1-A9 until the window access popularity of the K accessed data blocks is obtained; S3-1-A11. Mark the data blocks that do not appear in the access event groups of K data blocks out of N data blocks as unaccessed data blocks, and set the window access heat of each unaccessed data block to 0; S3-1-A12. Combine the window access heat of K accessed data blocks and the window access heat of NK unaccessed data blocks to obtain the window access heat of N data blocks under the cache time window.

5. The distributed storage and intelligent caching dynamic optimization method for high-concurrency access according to claim 3, characterized in that, Calculate the window access load for M storage nodes; include: S3-1-B1. Obtain the C load parameters for each of the M storage nodes within the cache time window; wherein, the C load parameters include at least CPU utilization, memory usage, and network bandwidth utilization. S3-1-B2. Perform dimensionless processing on the C load parameters to obtain the characteristics of the C load parameters; S3-1-B3: Perform a weighted summation on the C load parameter features to obtain the window access load of M storage nodes; where each node identifier is associated with the corresponding window access load.

6. The distributed storage and intelligent caching dynamic optimization method for high-concurrency access according to claim 3, characterized in that, Write the window access load, cached data block identifiers, and window access popularity inherited from the cached data block identifiers into the node's basic index table, and update the node's advanced index table, including: S3-3-1. Synchronously anchor the M node identifiers in the node base index table, as well as each local cache identifier and the corresponding cached data block identifier in the cache directory; S3-3-2. Based on the correspondence between node identifiers and local cache identifiers, match the cached data block identifiers corresponding to each local cache identifier to the corresponding node identifier; S3-3-3. Write the window access load corresponding to each node identifier, the matched cached data block identifier, and the window access heat corresponding to the cached data block identifier into the node basic index table to generate the node advanced index table.

7. The distributed storage and intelligent caching dynamic optimization method for high-concurrency access according to claim 1, characterized in that, Based on the window access popularity of N data blocks, high-frequency data blocks are filtered and sorted to generate a high-frequency data block sequence, including: S4-1. Select the current data block to be judged one by one along the data block identifiers of N data blocks; S4-2. Compare the window access popularity of the current data block to be judged with the popularity threshold; if the window access popularity of the current data block to be judged is greater than or equal to the popularity threshold, then mark the current data block to be judged as a high-frequency data block. S4-3: Traverse N data blocks and repeat S4-1 to S4-2 until N1 high-frequency data blocks are marked; S4-4. Sort the N1 high-frequency data blocks in descending order according to their window access frequency to generate a high-frequency data block sequence.

8. The distributed storage and intelligent caching dynamic optimization method for high-concurrency access according to claim 7, characterized in that, Based on the high-frequency data block sequence and the node advancement index table, the data blocks to be migrated are selected and a data block migration sequence is generated, including: S5-1. Select the current high-frequency data block one by one along the ordered direction of the high-frequency data block sequence; S5-2. Based on the data block identifier of the current high-frequency data block, query the J local cache identifiers of the current high-frequency data block in the node advanced index table; S5-3, anchor J local cache identifiers to J node identifiers one-to-one, and extract the window access load corresponding to the J node identifiers; S5-4. Select the highest window access load from the window access loads corresponding to the J node identifiers, and determine the node identifier corresponding to the highest window access load as the highest load node identifier. S5-5. If the highest window access load is greater than the load threshold, mark the current high-frequency data block as a data block to be migrated, and record the data block identifier, the highest load node identifier, and the window access heat of the current high-frequency data block. S5-6. Traverse the N1 high-frequency data blocks in the high-frequency data block sequence and repeat S5-1 to S5-5 until N2 data blocks to be migrated are obtained. S5-7. Arrange the N2 data blocks to be migrated in descending order according to their window access popularity to generate a data block migration sequence; wherein, each data block to be migrated in the data block migration sequence is associated with its data block identifier, the identifier of the highest load node and its window access popularity.

9. The distributed storage and intelligent caching dynamic optimization method for high-concurrency access according to claim 8, characterized in that, Based on the data block migration sequence and the window access load in the node advancement index table, the node advancement index table is updated with data block identifiers to generate an updated node advancement index table, including: S6-1. Select the current data block to be migrated one by one along the ordered direction of the data block migration sequence; S6-2. Based on the window access load corresponding to the M node identifiers in the node advancement index table, select candidate node identifiers one by one in order of window access load from low to high. S6-3. If the candidate node identifier is the same as the highest load node identifier associated with the current migrated data block, or if the local cache unit corresponding to the candidate node identifier has already cached the current migrated data block, then continue to select the next candidate node identifier according to the direction of window access load from low to high. S6-4. If the candidate node identifier is different from the highest load node identifier associated with the current migrated data block, and the local cache unit corresponding to the candidate node identifier does not cache the current migrated data block, then the candidate node identifier is determined as the target node identifier. S6-5. Copy the current migrated data block to the local cache unit corresponding to the target node identifier, and simultaneously write the data block identifier of the current migrated data block to the local cache identifier corresponding to the target node identifier in the node advanced index table; S6-6. Traverse the N2 data blocks to be migrated in the data block migration sequence, repeatedly execute the write of the current data block to be migrated, until the cache copy of the N2 data blocks to be migrated is completed, and obtain the updated node advanced index table.

Images