Distributed data storage dynamic optimization method based on edge computing

Abstract

The application relates to the technical field of edge computing storage, and discloses a distributed data storage dynamic optimization method based on edge computing. The method acquires data storage state data and data access request data of distributed edge nodes, and constructs a data resource distribution model according to the data; the method combines preset optimization conditions with the model to determine a system resource early warning state, enters a dynamic optimization mode when the early warning is triggered, and generates a data migration priority queue, and executes a data migration operation according to the queue; the method updates the data resource distribution model according to a migration result, determines a system resource exit state based on the updated model, exits the dynamic optimization mode when the exit state is triggered, and transmits the optimized storage state data to a central management system. The method dynamically adjusts data distribution by real-time sensing of node states and access demands, adapts to edge node heterogeneity and data access dynamics, and improves distributed data storage efficiency.

Description

Technical Field

[0001] This invention relates to the field of edge computing storage technology, specifically to a dynamic optimization method for distributed data storage based on edge computing. Background Technology

[0002] With the rapid development of technologies such as the Internet of Things (IoT) and 5G communication, the amount of data generated by terminal devices is growing exponentially. Traditional centralized data storage architectures face problems such as high data transmission latency, high network bandwidth pressure, and uneven distribution of computing resources. To address these challenges, edge computing has emerged as a technology that pushes computing and storage capabilities down to network edge nodes. By processing and storing data at edge nodes close to the data generation source, it can effectively reduce data transmission distance and improve response speed.

[0003] In edge computing environments, distributed data storage systems consist of numerous geographically dispersed edge nodes. These nodes are typically heterogeneous, with significant differences in hardware configuration, storage capacity, and network connectivity. Furthermore, data access to edge nodes is highly dynamic, with access frequency and data type requirements potentially changing drastically over time. For example, in intelligent transportation scenarios, traffic data access volume at intersection edge nodes is far higher during morning rush hour than at other times; in the Industrial Internet of Things (IIoT), sudden equipment failures generate large amounts of real-time monitoring data, leading to a surge in storage and access pressure on the corresponding edge nodes. Existing distributed data storage methods are mostly based on static strategies, which are insufficiently adapted to the heterogeneity of edge nodes and the dynamic nature of data access. Some methods use fixed data allocation rules, which cannot adjust data distribution according to changes in node storage status, easily leading to storage overflow on some nodes while other nodes are idle. Other methods, while introducing dynamic adjustment mechanisms, lack precise awareness of data access patterns, resulting in delayed data migration decisions. This not only increases network transmission overhead but may also affect service response speed due to untimely migration of critical data. Furthermore, most methods lack effective resource early warning mechanisms, making it difficult to quickly trigger optimization processes when the system experiences resource stress, further exacerbating the problems of low data storage efficiency and degraded service quality. Summary of the Invention

[0004] The purpose of this invention is to provide a dynamic optimization method for distributed data storage based on edge computing, so as to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, this invention provides a dynamic optimization method for distributed data storage based on edge computing, the method comprising: Obtain data storage status data and data access request data from distributed edge nodes; A data resource distribution model is constructed based on the data storage status data and data access request data; The system resource early warning status is determined based on preset optimization conditions and the data resource distribution model. When the system resource warning state is triggered, the system enters dynamic optimization mode and generates a data migration priority queue. In the dynamic optimization mode, data migration operations are performed based on the data migration priority queue; Update the data resource distribution model based on the results of the data migration operation; The system resource exit status is determined based on the updated data resource distribution model; When the system resource exit state is triggered, the dynamic optimization mode is exited. The optimized data storage status is transmitted to the central management system via the edge computing network.

[0006] Preferably, the acquisition of data storage status data and data access request data of distributed edge nodes specifically includes: Collect storage capacity utilization data, data access latency data, and data block size distribution data for each edge node; Based on the adaptive data sharding algorithm, request features are extracted from the data access request data to obtain data access frequency features and data type features. The storage capacity utilization data, data access latency data, data block size distribution data, data access frequency characteristics, and data type characteristics are integrated to generate a data storage status feature set.

[0007] Preferably, the step of constructing a data resource distribution model based on the data storage status data and data access request data specifically includes: The data storage state feature set is subjected to multi-scale spatial analysis, and a storage feature tensor is constructed through gridding processing; Based on the resource consumption classifier, the storage feature tensor is used to identify resource types, and a data resource distribution model including storage capacity distribution, access latency distribution and data type distribution is generated. Data block location information is obtained based on the data access request data, and the storage characteristics of each data block are mapped to the data resource distribution model based on the data block location information.

[0008] Preferably, the step of determining the system resource early warning status based on preset optimization conditions and the data resource distribution model specifically includes: The preset optimization conditions include system resource load thresholds and storage performance thresholds; The load assessment value and performance assessment value of each edge node are calculated based on the data resource distribution model; When the load assessment value of any edge node reaches the system resource load threshold or the performance assessment value is lower than the storage performance threshold, a system resource warning status is issued. The system resource warning status includes high load warning status and low performance warning status.

[0009] Preferably, when the system resource warning state is triggered, entering the dynamic optimization mode and generating a data migration priority queue specifically involves: After entering dynamic optimization mode, all new data access requests enter a waiting sequence; Priority is evaluated for each data block based on the data resource distribution model, and priority labels are generated. Based on historical resource consumption data, resource type assessment is performed on each data block to generate resource type labels; Construct a data migration priority queue based on priority labels and resource type labels.

[0010] Preferably, in the dynamic optimization mode, performing the data migration operation based on the data migration priority queue specifically involves: Obtain real-time idle rate data of system resources; Select the data block with the highest priority label from the data migration priority queue; For data blocks with the same priority label, select the data block with the resource type label that matches the resource type with the highest idle rate based on the real-time idle rate data; Perform the migration operation of the selected data block to the target edge node.

[0011] Preferably, updating the data resource distribution model based on the data migration operation results specifically involves: Collect data on changes in storage capacity, access latency, and data block location after the migration operation; Perform feature reconstruction on data showing changes in storage capacity, access latency, and data block location, and update the storage feature tensor. The data resource distribution model is regenerated based on the updated storage feature tensor.

[0012] Preferably, determining the system resource exit status based on the updated data resource distribution model specifically involves: The preset exit conditions include system resource stability thresholds and storage performance recovery thresholds; The load stability value and performance recovery value of each edge node are calculated based on the updated data resource distribution model; When the load stability value of all edge nodes remains below the system resource stability threshold and the performance recovery value is higher than the storage performance recovery threshold, the system resource exit status is issued. The system resource exit status includes load stabilization exit status and performance recovery exit status.

[0013] Preferably, when the system resource exit state is triggered, the dynamic optimization mode is exited, specifically as follows: After exiting dynamic optimization mode, new data access requests directly enter the normal processing sequence; Data access request processing is performed based on the normal processing sequence; Stop generating and maintaining the data migration priority queue.

[0014] Preferably, the step of transmitting the optimized data storage status data to the central management system via an edge computing network specifically involves: Based on a hierarchical data transmission protocol, the optimized data storage status data is modulated to generate a data storage transmission signal; Data storage transmission signals are sent to the central management system via edge computing networks; The central management system decodes the data storage and transmission signals to restore the optimized data storage status data.

[0015] Compared with the prior art, the beneficial effects of the present invention are: By acquiring real-time data storage status and data access request data from distributed edge nodes, comprehensive and timely foundational information is provided for subsequent model building and optimization decisions. This precise grasp of node status and access needs allows the system to clearly perceive the heterogeneity of edge nodes and the dynamic nature of data access, avoiding decision-making biases caused by insufficient information. Based on the acquired data, a data resource distribution model is constructed, integrating the scattered node states and access data into a systematic model representation, thus intuitively presenting the storage resource distribution and data access characteristics of edge nodes. With this model, the system can more clearly identify imbalances in resource allocation, providing quantifiable analytical basis for subsequent optimization operations and helping to grasp the overall state of data storage from a holistic perspective. By combining preset optimization conditions with a data resource distribution model, the system's resource early warning status is determined, enabling the system to issue timely warnings before resource strain or imbalance occurs. This proactive detection mechanism breaks away from the traditional approach of passively addressing resource problems only after they erupt, providing the system with a window of opportunity for proactive optimization and helping to prevent service interruptions or performance degradation caused by the accumulation of resource issues. When an alert is triggered, the system enters dynamic optimization mode and generates a data migration priority queue. This queue allows for the rational planning of the data migration order based on factors such as data importance, access frequency, and node resource status. This prioritized migration strategy ensures that critical data is allocated preferentially, avoiding resource waste and inefficiency caused by disordered migration, and making data migration operations more targeted and effective. In dynamic optimization mode, data migration operations are performed based on priority queues, which can adjust the distribution of data among edge nodes in an orderly manner according to the plan. By migrating data from resource-constrained nodes to resource-abundant nodes, balanced utilization of storage resources is achieved, reducing the situation where some nodes are overloaded while others are idle, and improving the storage resource utilization of the entire edge computing system. The data resource distribution model is updated based on the data migration results, enabling the model to reflect changes in the system state in real time after the data migration. The updated model provides the latest basis for determining the exit status of system resources, ensuring the accuracy of exit decisions and allowing the system to exit dynamic optimization mode in a timely manner after achieving the expected optimization results, avoiding unnecessary resource consumption. The optimized data storage status data is transmitted to the central management system, enabling the linkage between edge node optimization results and central management. The central management system can then perform global coordination and long-term analysis based on this data, providing references for the overall planning, node deployment, and resource expansion of the edge computing network, thus promoting the coordinated development of the entire edge computing system. Attached Figure Description

[0016] Figure 1 This is a schematic diagram illustrating the working principle of the dynamic optimization method for distributed data storage based on edge computing described in this invention. Figure 2 A flowchart for obtaining data storage status data and data access request data; Figure 3 A flowchart for constructing a data resource distribution model; Figure 4 A flowchart for generating a data migration priority queue; Figure 5 A flowchart for updating the data resource distribution model. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Please see Figure 1 This invention provides a dynamic optimization method for distributed data storage based on edge computing, the method comprising: A dynamic data resource distribution model is constructed by monitoring the storage status and access requests of distributed edge nodes in real time. The system continuously evaluates the resource usage of each node and automatically triggers an optimization mechanism when a resource warning state is detected. In dynamic optimization mode, the system generates a data migration priority queue according to a preset algorithm and executes intelligent data migration operations. After each migration, the resource distribution model is updated until the system resource status returns to normal, at which point the system exits optimization mode. Finally, the optimized storage status data is transmitted to the central management system through the edge network, achieving dynamic balancing of global storage resources. The entire process employs a closed-loop control mechanism to ensure that the distributed storage system is always in an optimal operating state.

[0019] Example 1: See Figure 2 Data storage status monitoring of distributed edge nodes is achieved through a lightweight acquisition module deployed at the node's operating system kernel layer. This module employs a dual-channel acquisition mechanism: the primary channel periodically polls system resources, collecting storage capacity utilization data every 5 seconds; the secondary channel monitors network stack events in real time, immediately capturing latency data when a data access request occurs. During storage capacity utilization data acquisition, the ratio of used space to total capacity of disk partitions is obtained by calling the file system interface, while simultaneously recording the usage status of block device caches. Data access latency measurement is performed at the transport layer of the network protocol stack, recording the time difference from receiving an access command to returning an acknowledgment signal. This process utilizes a high-precision timer to achieve millisecond-level timestamps.

[0020] Data block size distribution is collected using metadata scanning technology, which accesses the storage engine's index structure to statistically analyze the distribution of data block numbers across different size ranges. The system sets three key threshold ranges: 256KB, 1MB, and 4MB, recording the number and proportion of data blocks falling within each range. Each scan generates three sets of values: fragmented file counts in the 0-256KB range, standard block counts in the 256KB-1MB range, and large file block counts exceeding 1MB. These data form a dynamic profile of spatial distribution.

[0021] The adaptive data sharding algorithm employs a two-layer processing architecture. In the request receiving layer, the traffic parser splits the original request into three parts: a protocol header, an opcode, and a data address. The opcode is used to identify the read / write type, and the data address is used to locate the target block. The feature extraction layer includes a parallel frequency analysis unit and a type identification unit. The frequency analysis unit maintains a circular buffer recording the access count of each data block within the last 15 minutes. Each time a new request arrives, the corresponding count is updated, and the count value is converted into an exponentially smoothed weighted value to generate a dynamic frequency coefficient. The type identification unit establishes a multi-level classification tree: first, it distinguishes application types such as HTTP, FTP, and MQTT based on the protocol header; second, it parses content type identifiers to distinguish basic types such as text, images, and videos; finally, it detects data patterns using a load feature identification tool to complete the final classification of structured / unstructured data.

[0022] Data storage status feature integration is achieved using a distributed message queue. Each node's acquisition agent encapsulates the processed raw data into structured messages, containing the following in a fixed format: 1) Node identifier (64-bit hash value); 2) Acquisition timestamp (ISO8601 format); 3) Five-dimensional feature vector (storage capacity ratio, average latency, fragmentation degree, access frequency, and type encoding). The integration engine subscribes to the message queue and performs time window alignment to ensure data comparability between different nodes. The fragmentation degree in the feature vector is calculated using the Shannon entropy formula to determine the dispersion of distributions across different intervals. The type encoding uses a three-bit bitmap to represent data attribute combinations. The final unified feature set is stored in a columnar database on a node-by-node basis, with each row containing complete five-dimensional features and their metadata.

[0023] The edge network transmission process utilizes a data compression channel. Inter-node communication employs the DEFLATE algorithm to compress feature data, with a compression block size set to 32KB. The transport layer establishes two parallel channels: a high-priority channel transmits real-time access delay data, and a low-priority channel transmits periodic capacity distribution data. Sequence checksums are added to data distribution packets, and the receiving end uses CRC-32 verification to ensure data integrity. In the event of a transmission interruption, the system automatically switches to buffered forwarding mode, retransmitting lost data packets once the network recovers. The feature integration process employs zero-copy technology, directly mapping received cluster data into memory for processing, avoiding redundant data copying operations.

[0024] During the generation of the stored state feature set, the anomaly handling mechanism is continuously running. When a node goes offline during the acquisition phase, the associated agent triggers a timeout retry mechanism; after three failed retries, the node's features are marked as missing. The data verification module checks the validity of feature values, initiating a second acquisition verification for feature values ​​exceeding a set threshold. The feature value smoothing module suppresses short-term fluctuations by processing noisy data using a first-order lag filter method, maintaining the stability and continuity of the feature data. All anomaly handling logs are recorded during the final feature set output for reference during the model building phase.

[0025] Data access frequency characteristics are analyzed using a dynamic weighted statistical method. The frequency analysis unit is configured with three time windows: a 15-minute short window for detecting sudden access bursts, a 6-hour medium window for identifying business access patterns, and a 24-hour long window for analyzing cyclical patterns. Frequency values ​​are calculated independently for each window and then weighted and merged. The window weights are adaptively adjusted based on the access volume fluctuation coefficient. Data type feature encoding uses a tree structure: the root node distinguishes between real-time and non-real-time data, branch nodes distinguish between transactional and analytical uses, and leaf nodes indicate the specific encoding scheme. Each access request generates a 6-byte type identifier code, where the first byte indicates the privacy level, the second byte indicates the compression status, and the following four bytes encode the specific format characteristics.

[0026] The resource utilization control module of the acquisition system implements a dynamic degradation strategy. When the node's CPU utilization exceeds 60%, the acquisition cycle is automatically extended to 10 seconds; when memory usage exceeds the limit, the data block distribution scanning function is shut down; and when network bandwidth is strained, low-priority data transmission is paused. The device support module monitors the health status of various storage media, adding write / erase count monitoring for SSDs and recording bad sector information for mechanical hard drives. All these additional features are added as hidden dimensions to the feature vector, participating in the calculation only during the resource evaluation phase, maintaining the universality of the basic feature structure.

[0027] Example 2: See Figure 3 The data resource distribution model is constructed using a spatiotemporal tensor computation method, transforming discrete edge node features into a continuous spatial distribution representation. The construction of the stored feature tensor begins with spatiotemporal alignment preprocessing. The system deploys a network time protocol service to synchronize the clocks of each node, eliminating time discrepancies in cross-node data acquisition. The aligned feature data is input into a gridded processor, which uses a dynamic resolution algorithm to divide the three-dimensional space: the node dimension is clustered according to physical location, the time dimension uses 5-minute units as the basic unit, and the feature dimension maintains the original five-dimensional structure. The grid cell size is automatically adjusted according to node density, using fine-grained partitioning in densely populated areas and merging sparse areas into macroscopic units. Each grid cell records the statistics of all node feature vectors within the cell, including the mean vector, variance matrix, and sample size.

[0028] The resource consumption classifier is implemented based on a deep neural network architecture. The input layer receives a standardized feature tensor with dimensions of grid number × time slice number × feature number. Hidden layers use 3D convolutional kernels to extract spatiotemporal features. The first convolutional layer uses a 5×3×5 kernel size, the second reduces it to 3×2×3, and the third further compresses it to 2×1×2. Each convolutional operation is followed by batch normalization and LeakyReLU activation to prevent gradient vanishing. The output layer uses spatial pyramid pooling to transform the variable-length input into a fixed-dimensional output, ultimately outputting a resource type probability distribution. The classifier is trained using a weighted cross-entropy loss function, with weight coefficients dynamically adjusted based on the type sample size.

[0029] The data block location mapping establishes a mechanism for translating logical addresses to physical grids. The system maintains a distributed hash table to record the mapping relationship between data block identifiers and storage nodes, and the hash function uses a consistent hashing algorithm to reduce the impact of migration. When an access request arrives, the location resolver performs the following calculations to determine the target grid coordinates:

[0030] in: Represents the target grid coordinate vector. For the node position hash function, It is the logical address of the data block. Represents the feature space projection function. To access the feature vector of the request, This represents a vector concatenation operation. This formula ensures that the same data block may be mapped to different feature space locations under different access modes.

[0031] System resource early warning judgment implements multi-level threshold monitoring. Load assessment values ​​are calculated based on six dimensions: CPU utilization. Memory usage Storage I / O throughput Network bandwidth utilization Thread pool activity and request queue length Each dimensional indicator is normalized to the [0,1] interval through a non-linear transformation, and then the overall load score is synthesized according to dynamic weights. Weight coefficients are automatically configured based on node type: computationally intensive nodes are boosted. and Weighting, storage-intensive nodes emphasize and Strengthening network-dense nodes and Considerations.

[0032] The performance evaluation system is based on quantitative standards derived from Service Quality Level Agreements (SLAs). Key performance metrics include the 95th percentile of request response time. Transaction count successfully processed per second Data persistence delay and cache hit rate The evaluation value calculation incorporates a decay factor to process historical data, with the current period indicator accounting for 70% of the weight, and the indicators from the previous three periods accounting for 15%, 10%, and 5%, respectively. Performance status is divided into five levels: Excellent (>80), Good (60-80), Average (40-60), Poor (20-40), and Inferior (<20). A lag interval is set at the level boundaries to prevent status oscillation.

[0033] The early warning triggering mechanism employs composite logic judgment. A basic early warning is activated when the load score exceeds 0.85 or the performance level drops to "medium," while an advanced early warning requires the load score to exceed 0.95 or the performance level to fall to "poor." A 5-minute observation window is set for status persistence; if the metric falls below the threshold during this period, the early warning is canceled. The early warning information includes three parts: a list of triggering nodes, details of the exceeding metrics, and suggested handling strategies, broadcast to relevant nodes via a publish-subscribe model. The early warning state machine implements four state transitions: Normal → Basic Early Warning → Advanced Early Warning → Recovery in Progress → Normal, with each transition generating an audit log recording the complete context.

[0034] Model updates and maintenance employ an incremental approach. Local updates to the stored feature tensors target changed grid cells, recalculating the cell's statistics and updating the corresponding tensor slices. Global reconstruction is triggered when more than 30% of grid cells have changed. The reconstruction process uses a parallel MapReduce framework: during the mapping phase, each node calculates its local grid statistics, and during the reduction phase, these are merged to generate the global tensor. Online learning of the resource classifier utilizes mini-batch gradient descent, updating model parameters every 100 new samples, with the learning rate dynamically adjusted based on classification accuracy.

[0035] The data block location index implements a multi-level caching strategy. Hot data mappings are cached in an in-memory hash table with an LRU eviction mechanism; warm data is stored in a local RockDB key-value database; and cold data is backed up to a distributed file system. Index updates use copy-on-write technology to ensure query consistency during migration. The location resolver implements short-circuit query optimization, directly returning cached results for consecutively accessed identical data blocks, skipping the complete calculation process.

[0036] Real-time load assessment is implemented using a stream processing framework. Raw metrics collected from each node are sent to a Kafka message queue, and Flink jobs consume the data and perform window aggregation. The sliding window size is set to 1 minute, with a sliding step of 10 seconds, balancing real-time performance and computational overhead. The performance assessment module adopts a Lambda architecture: the speed layer processes real-time streaming data, the batch processing layer recalculates all historical metrics daily, and the service layer merges the results from both parts to provide a unified query interface.

[0037] The distribution of early warning information adopts a tiered push strategy. Basic warnings only notify the local scheduler, while advanced warnings trigger the global event bus. Push content is encoded using Protocol Buffers, containing binary-formatted metric snapshots and text-formatted analysis suggestions. The early warning processor implements a chain-of-responsibility pattern, supporting dynamic insertion of filters and processors for easy business logic expansion. A silent period mechanism is set for state recovery detection; recovery is only confirmed after three consecutive detection cycles of metric compliance, avoiding frequent state switching.

[0038] Example 3: See Figure 4 The activation of the dynamic optimization mode triggers the start of the request buffering mechanism, which employs a dual-queue structure. The main queue implements a weighted fair queuing algorithm, where each newly arriving access request receives an initial weight value based on the service level field in its protocol header, ranging from 1 (normal business) to 8 (critical tasks). The secondary queue implements an emergency channel mechanism, directly allowing control requests carrying special markers to pass. The queue manager maintains a dynamic weight adjustment table, automatically increasing the weight of long-waiting requests based on request processing latency, with the adjustment magnitude following a logarithmic growth curve to prevent weight explosion. The buffer capacity is set with an adaptive threshold, not exceeding 30% of the node's available memory. When the capacity reaches the warning line, a selective discarding strategy is activated, prioritizing the discarding of low-weight and non-persistent query requests.

[0039] The data block priority evaluation module executes a multi-stage decision-making process. The first stage involves scoring basic attributes. Access frequency scores are mapped to a 0-10 range using an inverse hyperbolic tangent function. Data freshness scores are calculated based on a linear decay of the last modified timestamp. Dependency is determined by analyzing co-occurrence patterns in access logs. The second stage implements business rule corrections, loading a predefined business priority matrix and applying weighted adjustments to specific data types. The final priority label generation formula is as follows:

[0040] in: Indicates the final priority score. Represents the dynamic weight of the i-th evaluation dimension. It is the original score of the i-th dimension. This is a counter for the current migration depth of the data block. The formula suppresses the priority of frequently migrating data blocks through the logarithmic term in the denominator, thus avoiding system oscillations. The score is discretized into five levels of labels, and the transition boundary is dynamically adjusted based on the current system load.

[0041] Resource type assessment employs a hybrid clustering approach. In the offline phase, a K-means++ model is trained using three months of historical data to determine prototype vectors for three resource consumption patterns. In the online phase, streaming clustering is implemented, calculating the cosine similarity between the feature vector and the prototype vector for each data block. If the maximum similarity exceeds 0.7, the data is directly classified; otherwise, an incremental clustering process is initiated. Resource type labels are accompanied by confidence scores for risk control during migration decisions. Intermediate data generated during the assessment process is written to a circular buffer for subsequent model optimization.

[0042] Real-time idle rate monitoring implements fine-grained resource sharding statistics. Computing resources are divided into vCPU thread slots, memory resources are managed in 2MB pages, and storage I / O bandwidth is measured in 16KB units. The monitor maintains a three-dimensional resource matrix, recording the utilization percentage of each shard over the past 15 seconds. Idle rate calculation excludes system-reserved resources, and median filtering is used to handle instantaneous fluctuations. Matrix data is persisted every 5 seconds to form a resource utilization time series for trend prediction.

[0043] The migration selector implements a multi-constraint satisfaction algorithm. The core selection logic comprises three decision layers: the primary layer filters data blocks whose priority labels reach the current processing level; the intermediate layer selects instances from these candidate data blocks whose resource type labels best match the idle resources, considering the complementarity of resource types in the matching degree calculation; the final layer applies a location optimization strategy, prioritizing data blocks with the shortest migration path that do not cross racks. The decision tree generated during the selection process is recorded in the audit log, containing the reasons for the elimination of all excluded candidates.

[0044] Target node selection employs a multi-objective optimization approach. Candidate node evaluation considers six key factors: physical distance weight, network topology hop count, remaining storage capacity percentage, expected load balancing, data locality preservation rate, and migration cost estimation. Each factor is analyzed using Pareto front analysis to determine the optimal solution set. A random perturbation factor is incorporated into the final decision to prevent swarm intelligence from getting trapped in local optima. The selection results include a backoff flag, automatically triggering alternative solutions when resource conflicts occur during the actual migration process.

[0045] The migration execution engine implements a segmented transmission protocol. Large data blocks are split into multiple 2MB transmission segments, each independently checked and encrypted. Transmission control employs an improved BBR algorithm, dynamically adjusting the window size and transmission rate based on network conditions. The engine maintains a migration state machine, recording the status of each data block across four stages: preparation, transmission, verification, and commit. The exception handling mechanism includes three recovery strategies: automatic retry, breakpoint resumption, and failure rollback, employing differentiated responses for different types of error codes.

[0046] The concurrency control module implements distributed transaction management. Before the migration operation begins, exclusive locks are acquired on the source and target nodes, with lock granularity controlled at the data block level. Lock management employs a lease mechanism with a default lease period of 30 seconds, renewable three times. The transaction coordinator implements a two-phase commit protocol: the preparation phase verifies the target node's capacity and permissions, and the commit phase atomically updates the position index. All participating nodes maintain operation logs, using the Raft algorithm to ensure log consistency and guarantee state synchronization after fault recovery.

[0047] A dynamic weight adjuster continuously optimizes evaluation parameters. During system runtime, it collects actual migration performance data, including latency reduction, throughput improvement, and resource utilization changes. The adjuster employs a reinforcement learning framework, using weight coefficients as the action space and system performance metrics as reward signals, progressively optimizing the decision model through policy gradient methods. After each weight update, a snapshot of the old version is retained; if the new weights cause a performance degradation exceeding 5%, it automatically reverts to the previous stable version. Training data generated during the adjustment process is stored in an experience replay buffer, supporting offline batch learning.

[0048] Example 4: See Figure 5 During the model update process after data migration, the system initiates a three-dimensional monitoring mechanism. The capacity change monitor compares the storage usage of each node before and after the migration, recording the differences to form a capacity change matrix. For example, when data block B-7F3 is migrated from node E-05 to E-11, the recorded capacity changes for the two nodes are -1.8GB and +1.8GB, respectively. The latency change analyzer sends probe request packets to measure the access latency of all migrated data blocks at their new locations, forming a latency change comparison table. The location tracker updates the global index records and simultaneously records information about intermediate nodes along the migration path.

[0049] The feature reconstruction engine employs an incremental update strategy to handle changed data. Updates to stored feature tensors follow two modes: when the capacity change of a single node is less than 15%, a local grid refresh operation is performed, recalculating only the statistics of the grid cell containing that node; when more than four nodes migrate in the cluster (taking a 10-node cluster as an example), a full model recalculation is triggered. Data smoothing incorporates a time decay factor, with new measurements accounting for 70% of the weight and historical values ​​accounting for 30%, to prevent model oscillations caused by sudden changes in metrics.

[0050] Table 1: Data records after a migration operation.

[0051] E-05 B-7F3 186.4 184.6 -12.3 Removed E-11 B-7F3 92.8 94.6 +5.2 Loaded E-09 D-A22 154.2 152.0 -8.7 Removed E-13 D-A22 87.3 89.5 +3.1 Loaded E-02 F-9B1 203.7 203.7 - transit node The resource distribution model reconstruction process involves four steps: First, the updated storage feature tensor is loaded, maintaining its dimensional structure. Then, feature vector normalization is performed, mapping the values ​​of each dimension to the [0,1] interval. Next, forward propagation calculations are performed using a resource allocation classifier. Finally, the output includes a heatmap of the new storage capacity, an updated access latency distribution matrix, and an adjusted data type distribution model. This process employs a pipelined approach, where each node independently completes its local model update, which is then integrated into a global view by the coordinator.

[0052] The system exit determination employs a progressive verification mechanism. Load stability values ​​are calculated based on datasets from ten consecutive monitoring periods, using analysis of variance to assess load fluctuations at each node. For example, within a 10-minute observation window, the load value sequence for node E-11 is [68, 72, 69, 71, 70, 73, 69, 70, 71, 70], and its variance of 4.2, below the threshold of 5.0, is considered stable. Performance recovery value verification uses a three-level check: during the initial test, E-11's access latency is 34ms (recovery threshold 45ms), and the throughput reaches 142 operations per second (threshold 120 operations per second); the subsequent two tests yielded results of 31ms / 148ops and 33ms / 145ops respectively, both meeting the continuous improvement requirements.

[0053] The exit condition check enforces strict security rules. All nodes must simultaneously meet two conditions: the load stability value must be below 20 units for three consecutive periods, and the performance recovery value must be above 80 points for three consecutive periods. The regional coordinator periodically broadcasts status query requests to collect the real-time load index and performance score of each node. When the load fluctuation coefficient of node E-09 is detected to be 22 (threshold is 20), the system delays the exit decision until the node's load falls back to the stable range.

[0054] The state transition process includes a two-phase buffer period. The preparatory phase maintains the optimized mode but pauses new migration tasks, during which the persistence of migration effects is verified. Before formal exit, an eventual consistency check is performed: the data location index validator scans the global mapping table; the configuration synchronizer compares the local configurations of each node with the central record; and the performance baseline generator establishes a new monitoring baseline. Once all checks pass, the system marks the resource distribution model as a stable version.

[0055] The anomaly handling mechanism remains active throughout the update process. When the latency measurement value of node E-11 abnormally jumps to 128ms, the diagnostic tool automatically initiates a three-stage troubleshooting process: first, it retests key indicators to rule out false readings; second, it checks the network link status; and finally, it analyzes the integrity of the target data block. After confirming that it is a temporary network congestion, the system replaces the abnormal measurement value with a historically reasonable value. If a configuration inconsistency is detected (such as node E-09 recording a missing data block D-A22), the repairer copies metadata from the nearest neighbor node to achieve rapid recovery.

[0056] The resource distribution model version control system maintains the change history. Each model update generates a timestamped version record, retaining a complete snapshot of the past six versions. Data persistence uses a differential storage model, saving only the changes relative to the previous version. The version rollback mechanism can restore the system to the previous stable state within 45 seconds when performance degradation is detected. Audit logs record detailed version switching information, including the trigger reason, a list of change nodes, and duration statistics.

[0057] The performance monitoring baseline is dynamically adjusted during the exit process. After successfully exiting dynamic optimization mode, the system establishes new monitoring thresholds based on the measured values ​​after migration. This baseline includes three key parameters: the average latency baseline is the median of three measurements taken during the exit verification period (e.g., 34ms); the throughput baseline is the minimum safe value (e.g., 135 ops); and the capacity warning threshold is set to 85% of the node's maximum capacity. These baseline parameters serve as the evaluation criteria for subsequent operational status until they are reset when the optimization mode is activated again.

[0058] The coordinator communication protocol enhances fault tolerance during the exit phase. State switching commands are confirmed via a three-way handshake: the master coordinator sends a pre-exit command; the node returns its current load value; and the coordinator verifies the data and sends an acknowledgment signal. Each node is equipped with a timeout retransmission mechanism, automatically attempting transmission via an alternative path in case of communication failure. Final state synchronization employs an atomic broadcast protocol to ensure all nodes switch operating modes simultaneously, preventing inconsistent resource access states.

[0059] Example 5: After exiting dynamic optimization mode, the system switches to standard operation mode. The priority queue manager closes the buffer channel entry, and newly arriving access requests are directly routed to the target node. The request parser analyzes the message header to extract key parameters, including protocol version, opcode, data block identifier, and security token. The authentication unit verifies the digital signature of the security token and rejects unauthorized access attempts. Each request is assigned an independent thread for processing, and the thread pool is divided into priority groups according to request type: high-priority threads are assigned to real-time control instructions, and low-priority threads are used for batch data processing. The execution engine establishes a four-level processing pipeline: the first level loads data block metadata, the second level checks data integrity, the third level performs core read and write operations, and the final level generates response messages. The response time monitor records the time consumed in each stage, and timeout requests trigger an automatic retry mechanism.

[0060] The data migration priority queue generator enters a dormant state, and historical evaluation data is compressed and stored. A snapshot of the last active queue is serialized and stored in a distributed key-value database, with a retention period spanning the three most recent optimization cycles. The snapshot includes a complete data block priority label mapping table and resource type classification records. A data aging strategy implements periodic cleanup, automatically removing historical data older than thirty days, but retaining statistical summary information. The queue management interface remains responsive, accepting manual query commands to return historical decision logs.

[0061] Data transmission employs a layered modulation scheme. The control plane encapsulation module processes node topology information, converting dynamic identifiers into fixed-area codes. Each node is assigned a unique location code, and resource summary information is represented by bit fields: the first 3 bits identify the CPU type, the next 4 bits indicate the memory level, and the last 12 bits record the storage capacity level. The data plane processing module uses differential encoding technology; metrics that have changed by more than 15% compared to the last transmission are fully recorded, while other metrics only transmit the incremental change. Metric encapsulation uses a custom binary protocol, and the field structure includes a version number, timestamp, node identifier, metric type flag, and payload.

[0062] Edge network transmission employs dual-path redundancy. The primary channel uses UDP to transmit control plane data, with forward error correction codes added to combat network packet loss. The backup channel transmits detailed metric data via TCP, with each packet appended with a cyclic redundancy check (CRC) code. The path selector monitors network quality in real time, automatically switching to the backup channel when the primary channel's packet loss rate exceeds 5%. The transmission window size is dynamically adjusted based on round-trip delay, with an initial value of 32KB and an upper limit not exceeding 80% of the node's available network buffer.

[0063] The central management system receiver establishes a multi-layer decoding pipeline. The signal preprocessor performs frame synchronization and clock recovery, correcting timing offsets during transmission. The verification unit verifies data integrity, discards frames that fail verification, and triggers a retransmission mechanism. The control plane decoder parses the location code mapping table to restore the node physical topology. The resource summary decoder parses bit fields to reconstruct the resource configuration view of each node. The data plane decoder processes the differential encoded stream and applies a reverse conversion algorithm to restore the complete index set. Fields that fail to decode trigger an error isolation procedure, and the relevant data is transferred to a buffer to be repaired.

[0064] The data reorganization engine performs three-dimensional data fusion. Spatially, it integrates data reported by each node to build a cluster-level storage status view. Temporally, it aligns the time-series data of each node, with a maximum tolerable time difference of 500 milliseconds. Metrics-wise, it associates different performance parameters to establish a correlation between capacity utilization and access latency. The fused dataset is stored in a time-series database, with the index key containing the node identifier, data type, and collection time point. Storage employs a sharding strategy, dividing the data into shards based on time ranges. Data from the most recent three hours is stored in an in-memory database, while historical data is archived in columnar storage.

[0065] A heartbeat feedback mechanism is implemented for transmission link maintenance. Edge nodes send a beacon frame every 30 seconds, containing the current transmission sequence number and resource digest. The central system responds with an acknowledgment frame, carrying the range of most recently successfully received sequence numbers. A packet loss detector analyzes sequence number continuity and initiates a link diagnostic procedure when two consecutive heartbeats fail to respond. The diagnostic procedure consists of three phases: network layer connectivity testing, transport layer session state checking, and application layer protocol verification. During fault recovery, the system prioritizes attempting to rebuild the existing connection; after three failures, a new session establishment process is initiated.

[0066] The configuration synchronization system implements a version control strategy. The central management system maintains the primary and secondary configuration copies, while nodes store read-only caches locally. At the end of each optimization cycle, a configuration update package is distributed, appending a version marker and timestamp. Nodes apply the update and return a confirmation signal; unconfirmed nodes are added to the retransmission list. Version conflict resolution follows the last-write-first principle, and configuration change records are fully audited. A rollback mechanism allows restoration to any valid version from the past 24 hours; rollback operations require triple security authentication.

[0067] The status monitoring console implements a multi-level display strategy. The basic view displays a heatmap of the overall cluster storage utilization, while secondary views present detailed trend charts of individual node metrics. The alert interface highlights abnormal parameters, and anomaly detection is based on a dynamic baseline algorithm: the base threshold is set to the three-sigma range of the historical average, and the baseline value is automatically adjusted hourly. The data drill-down function supports accessing individual data block records by progressively descending from the cluster view. All displayed data is dynamically sampled and rendered to ensure the visualization performance of large-scale datasets. Detailed logs of user operation events are recorded, including view switching, time range selection, and anomaly confirmation actions.

[0068] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0069] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A dynamic optimization method for distributed data storage based on edge computing, characterized in that, Includes the following steps: Obtain data storage status data and data access request data from distributed edge nodes; A data resource distribution model is constructed based on the data storage status data and data access request data; The system resource early warning status is determined based on preset optimization conditions and the data resource distribution model. When the system resource warning state is triggered, the system enters dynamic optimization mode and generates a data migration priority queue, specifically as follows: After entering dynamic optimization mode, all new data access requests enter a waiting sequence; Priority is evaluated for each data block based on the data resource distribution model, and priority labels are generated. Based on historical resource consumption data, resource type assessment is performed on each data block to generate resource type labels; Construct a data migration priority queue based on priority labels and resource type labels; In the dynamic optimization mode, data migration operations are performed based on the data migration priority queue, specifically as follows: Obtain real-time idle rate data of system resources; Select the data block with the highest priority label from the data migration priority queue; For data blocks with the same priority label, select the data block with the resource type label that matches the resource type with the highest idle rate based on the real-time idle rate data; Perform the migration operation on the selected data block to the target edge node; Update the data resource distribution model based on the results of the data migration operation; The system resource exit status is determined based on the updated data resource distribution model; When the system resource exit state is triggered, the dynamic optimization mode is exited. The optimized data storage status is transmitted to the central management system via the edge computing network.

2. The dynamic optimization method for distributed data storage based on edge computing according to claim 1, characterized in that, The specific steps for obtaining the data storage status data and data access request data of the distributed edge nodes are as follows: Collect storage capacity utilization data, data access latency data, and data block size distribution data for each edge node; Based on the adaptive data sharding algorithm, request features are extracted from the data access request data to obtain data access frequency features and data type features. The storage capacity utilization data, data access latency data, data block size distribution data, data access frequency characteristics, and data type characteristics are integrated to generate a data storage status feature set.

3. The dynamic optimization method for distributed data storage based on edge computing according to claim 2, characterized in that, The step of constructing a data resource distribution model based on the data storage status data and data access request data specifically involves: The data storage state feature set is subjected to multi-scale spatial analysis, and a storage feature tensor is constructed through gridding processing; Based on the resource consumption classifier, the storage feature tensor is used to identify resource types, and a data resource distribution model including storage capacity distribution, access latency distribution and data type distribution is generated. Data block location information is obtained based on the data access request data, and the storage characteristics of each data block are mapped to the data resource distribution model based on the data block location information.

4. The dynamic optimization method for distributed data storage based on edge computing according to claim 3, characterized in that, The determination of the system resource early warning status based on preset optimization conditions and the data resource distribution model specifically includes: The preset optimization conditions include system resource load thresholds and storage performance thresholds; The load assessment value and performance assessment value of each edge node are calculated based on the data resource distribution model; When the load assessment value of any edge node reaches the system resource load threshold or the performance assessment value is lower than the storage performance threshold, a system resource warning status is issued. The system resource warning status includes high load warning status and low performance warning status.

5. The dynamic optimization method for distributed data storage based on edge computing according to claim 4, characterized in that, The step of updating the data resource distribution model based on the data migration operation results specifically involves: Collect data on changes in storage capacity, access latency, and data block location after the migration operation; Perform feature reconstruction on data showing changes in storage capacity, access latency, and data block location, and update the storage feature tensor. The data resource distribution model is regenerated based on the updated storage feature tensor.

6. The dynamic optimization method for distributed data storage based on edge computing according to claim 5, characterized in that, The determination of system resource exit status based on the updated data resource distribution model specifically involves: The preset exit conditions include system resource stability thresholds and storage performance recovery thresholds; The load stability value and performance recovery value of each edge node are calculated based on the updated data resource distribution model; When the load stability value of all edge nodes remains below the system resource stability threshold and the performance recovery value is higher than the storage performance recovery threshold, the system resource exit status is issued. The system resource exit status includes load stabilization exit status and performance recovery exit status.

7. The dynamic optimization method for distributed data storage based on edge computing according to claim 6, characterized in that, The step of exiting dynamic optimization mode when the system resource exit state is triggered is specifically as follows: After exiting dynamic optimization mode, new data access requests directly enter the normal processing sequence; Data access request processing is performed based on the normal processing sequence; Stop generating and maintaining the data migration priority queue.

8. The dynamic optimization method for distributed data storage based on edge computing according to claim 7, characterized in that, The process of transmitting the optimized data storage status data to the central management system via the edge computing network specifically involves: The optimized data storage status data is modulated based on a hierarchical data transmission protocol to generate a data storage transmission signal. Data storage transmission signals are sent to the central management system via edge computing networks; The central management system decodes the data storage and transmission signals to restore the optimized data storage status data.

Images