Distributed data storage dynamic optimization method based on edge computing

Abstract

The invention relates to the technical field of edge computing storage, and discloses a distributed data storage dynamic optimization method based on edge computing. The method comprises the following steps: acquiring data storage state data and data access request data of distributed edge nodes, and constructing a data resource distribution model according to the data storage state data and the data access request data; determining a system resource early warning state in combination with a preset optimization condition and the model, entering a dynamic optimization mode and generating a data migration priority queue when early warning is triggered, and executing data migration operation according to the queue; and updating the data resource distribution model according to a migration result, determining a system resource quit state based on the updated model, quitting the dynamic optimization mode during triggering, and transmitting the optimized storage state data to a central management system. According to the method, the node state and the access requirement are sensed in real time, the data distribution is dynamically adjusted, the edge node isomerism and the data access dynamics are adapted, and the distributed data storage efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of edge computing storage, in particular to a distributed data storage dynamic optimization method based on edge computing. BACKGROUND

[0002] With the rapid development of technologies such as Internet of Things and 5G communication, the amount of data generated by terminal devices is growing exponentially, and traditional centralized data storage architecture is facing problems such as high data transmission delay, large network bandwidth pressure, and uneven distribution of computing resources. To address these challenges, edge computing, as a technology that sinks computing and storage capabilities to network edge nodes, has emerged. By processing and storing data at edge nodes close to the data source, it can effectively reduce data transmission distance and improve response speed.

[0003] In the edge computing environment, the distributed data storage system is composed of a large number of geographically dispersed edge nodes, which are usually heterogeneous in terms of hardware configuration, storage capacity, and network connection status. At the same time, data access of edge nodes is highly dynamic, with access frequency and data type requirements varying dramatically at different times. For example, in the intelligent transportation scenario, the access volume of traffic data at intersection edge nodes during the morning rush hour is much higher than at other times; in the industrial Internet of Things, a large amount of real-time monitoring data is generated when a device fails, causing a sudden increase in storage and access pressure on the corresponding edge node. Existing distributed data storage methods are mostly based on static strategies, which are not well adapted to the heterogeneity of edge nodes and the dynamics of data access. Some methods use fixed data allocation rules, which cannot adjust data distribution according to changes in node storage state, leading to storage overflow in some nodes and idle resources in others. Other methods introduce dynamic adjustment mechanisms, but lack precise perception of data access patterns, resulting in lag in data migration decisions, which not only increases network transmission overhead but also may affect service response speed due to delayed migration of critical data. In addition, most methods do not establish effective resource warning mechanisms, making it difficult to quickly trigger optimization processes when the system is resource-constrained, further exacerbating the problems of low data storage efficiency and declining service quality. SUMMARY

[0004] The present application aims to provide a distributed data storage dynamic optimization method based on edge computing to solve the problems raised in the background.

[0005] To achieve the above-mentioned purpose, the present application provides a distributed data storage dynamic optimization method based on edge computing, which comprises: obtaining data storage state data and data access request data of distributed edge nodes; construct a data resource distribution model according to the data storage state data and the data access request data; determine a system resource warning state based on a preset optimization condition and the data resource distribution model; When the system resource warning state is triggered, enter a dynamic optimization mode, and generate a data migration priority queue; In the dynamic optimization mode, perform a data migration operation based on the data migration priority queue; update the data resource distribution model according to a data migration operation result; determine a system resource exit state based on the updated data resource distribution model; When the system resource exit state is triggered, exit the dynamic optimization mode; transmit the optimized data storage state data to a central management system through an edge computing network.

[0006] Preferably, the data storage state data and the data access request data of the distributed edge nodes are obtained as follows: collect storage capacity utilization rate data, data access delay data, and data block size distribution data of each edge node; extract request features from the data access request data based on an adaptive data sharding algorithm to obtain data access frequency features and data type features; integrate the storage capacity utilization rate data, the data access delay data, the data block size distribution data, the data access frequency features, and the data type features to generate a data storage state feature set.

[0007] Preferably, the data resource distribution model is constructed according to the data storage state data and the data access request data as follows: perform multi-scale spatial analysis on the data storage state feature set, and construct a storage feature tensor through gridding processing; perform resource type identification on the storage feature tensor based on a resource consumption classifier to generate a data resource distribution model containing storage capacity distribution, access delay distribution, and data type distribution; obtain data block location information according to the data access request data, and map the storage features of each data block to the data resource distribution model based on the data block location information.

[0008] Preferably, the system resource warning state is determined based on a preset optimization condition and the data resource distribution model as follows: The preset optimization condition includes a system resource load threshold and a storage performance threshold; calculate a load evaluation value and a performance evaluation value of each edge node according to the data resource distribution model; When the load evaluation value of any edge node reaches the system resource load threshold or the performance evaluation value is lower than the storage performance threshold, a system resource warning state is issued; The system resource warning state includes a high-load warning state and a low-performance warning state.

[0009] Preferably, when the system resource warning state is triggered, a dynamic optimization mode is entered, and a data migration priority queue is generated, specifically: After entering the dynamic optimization mode, all new data access requests enter a waiting sequence; Based on the data resource distribution model, a priority evaluation is performed on each data block to generate a priority label; Based on historical resource consumption data, a resource type evaluation is performed on each data block to generate a resource type label; According to the priority label and the resource type label, a data migration priority queue is constructed.

[0010] Preferably, in the dynamic optimization mode, a data migration operation is performed based on the data migration priority queue, specifically: Real-time idle rate data of system resources are obtained; From the data migration priority queue, a data block with the highest priority label is selected preferentially; For data blocks with the same priority label, according to the real-time idle rate data, a data block with a resource type label matching the highest idle rate resource type is selected; The selected data block is migrated to a target edge node.

[0011] Preferably, the data resource distribution model is updated according to the data migration operation result, specifically: Storage capacity change data, access delay change data, and data block position change data after the migration operation are collected; The storage capacity change data, access delay change data, and data block position change data are feature reconstructed, and the storage feature tensor is updated; Based on the updated storage feature tensor, the data resource distribution model is regenerated.

[0012] Preferably, the system resource exit state is determined based on the updated data resource distribution model, specifically: The preset exit conditions include a system resource stability threshold and a storage performance recovery threshold; According to the updated data resource distribution model, the load stability value and the performance recovery value of each edge node are calculated; When the load stability value of all edge nodes continuously falls below the system resource stability threshold and the performance recovery value is higher than the storage performance recovery threshold, a system resource exit state is issued; The system resource exit state comprises a load stable exit state and a performance recovery exit state.

[0013] Preferably, when the system resource exit state is triggered, the dynamic optimization mode is exited, in particular: After exiting the dynamic optimization mode, the new data access request directly enters a normal processing sequence. The data access request processing is performed based on the normal processing sequence. The generation and maintenance of the data migration priority queue are stopped.

[0014] Preferably, the optimized data storage state data is transmitted to the central management system through an edge computing network, in particular: The optimized data storage state data is modulated based on a hierarchical data transmission protocol to generate a data storage transmission signal. The data storage transmission signal is sent to the central management system through the edge computing network. The data storage transmission signal is decoded in the central management system to restore the optimized data storage state data.

[0015] Compared with the prior art, the present application has the following advantages: By obtaining the data storage state data and the data access request data of the distributed edge nodes in real time, comprehensive and timely basic information is provided for subsequent model construction and optimization decision-making. This precise grasp of node state and access demand enables the system to clearly perceive the heterogeneity of edge nodes and the dynamics of data access, avoiding decision-making bias caused by insufficient information. Based on the obtained data, a data resource distribution model is constructed, and the scattered node state and access data are integrated into a systematic model expression, so that the storage resource distribution and data access characteristics of the edge nodes are intuitively presented. With the help of the model, the system can more clearly identify the imbalance in resource allocation, providing a quantifiable analysis basis for subsequent optimization operations, which helps to grasp the global status of data storage from the overall level. The combination of the preset optimization condition and the data resource distribution model determines the system resource warning state, so that the system can timely issue a warning before the resource is about to be tense or unbalanced. This mechanism of early perception breaks the traditional passive processing mode after the resource problem occurs, and helps to prevent service interruption or performance degradation caused by resource accumulation. When the early warning state is triggered, the dynamic optimization mode is entered and the data migration priority queue is generated, and the order of data migration can be reasonably planned according to the importance of data, access frequency, node resource status and other factors. This priority-based migration strategy ensures that critical data can be reasonably allocated in priority, avoiding resource waste and inefficiency caused by disordered migration, making the data migration operation more targeted and effective. In the dynamic optimization mode, the data migration operation is performed based on the priority queue, which can adjust the distribution of data among edge nodes in an orderly manner according to the planning. By migrating data from nodes with insufficient resources to nodes with sufficient resources, balanced utilization of storage resources is achieved, reducing the situation of overloading some nodes while other nodes are idle, and improving the storage resource utilization of the entire edge computing system. The data resource distribution model is updated according to the data migration result, so that the model can reflect the system state change after data migration in real time. The updated model provides the latest basis for the judgment of system resource exit state, ensuring the accuracy of the exit decision, so that the system can exit the dynamic optimization mode in time after optimization achieves the expected effect, avoiding unnecessary resource consumption. The optimized data storage state data is transmitted to the central management system, realizing the linkage of edge node optimization results and central management. The central management system can conduct global overall planning and long-term analysis based on these data, providing reference for the overall planning, node deployment, resource expansion and other aspects of the edge computing network, promoting the coordinated development of the entire edge computing system. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 The working principle diagram of the distributed data storage dynamic optimization method based on edge computing is described. Figure 2 The flowchart for obtaining data storage state data and data access request data is described. Figure 3 The flowchart for constructing a data resource distribution model is described. Figure 4 The flowchart for generating a data migration priority queue is described. Figure 5 The flowchart for updating a data resource distribution model is described. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0018] Please refer toFigure 1 The application provides a distributed data storage dynamic optimization method based on edge computing, which comprises the following steps: By monitoring the storage state and access request of the distributed edge node in real time, a dynamic data resource distribution model is constructed. The system continuously evaluates the resource usage of each node, and automatically triggers the optimization mechanism when a resource warning state is detected. In the dynamic optimization mode, the system generates a data migration priority queue according to the preset algorithm, and performs intelligent data migration operation. The resource distribution model is updated after each migration is completed, and the optimization mode is exited until the system resource state returns to normal. Finally, the optimized storage state data is transmitted to the central management system through the edge network, realizing the dynamic balance of global storage resources. The whole process adopts a closed-loop control mechanism to ensure that the distributed storage system is always in the optimal operating state.

[0019] Embodiment 1: see Figure 2 The data storage state monitoring of the distributed edge node is realized by deploying a lightweight collection module in the kernel layer of the node operating system. The module adopts a dual-channel collection mechanism: the main channel periodically polls system resources, and collects storage capacity usage data every 5 seconds; the secondary channel listens to network stack events in real time, and immediately captures delay data when a data access request occurs. When collecting the storage capacity usage data, the used space and total capacity ratio of the disk partition are obtained by calling the file system interface, and the usage state of the block device cache is recorded. The data access delay measurement cuts in from the transmission layer of the network protocol stack, and records the time difference from receiving the access instruction to returning the confirmation signal. This process uses a high-precision timer to realize millisecond-level timestamp marking.

[0020] The data block size distribution collection adopts a metadata scanning technology, accesses the index structure of the storage engine, and counts the number distribution of data blocks in different size intervals. The system sets three key threshold intervals of 256KB, 1MB and 4MB, and records the number and proportion of data blocks falling within each interval. Each scan produces three sets of values: the number of fragmented files in the 0-256KB interval, the number of standard blocks in the 256KB-1MB interval, and the number of large file blocks exceeding 1MB. These data form a dynamic profile of spatial distribution.

[0021] The adaptive data slicing algorithm adopts a two-layer processing architecture. In the request receiving layer, the traffic parser splits the original request into three parts: protocol header, operation code, and data address. The operation code is used to identify the read-write type, and the data address is used to locate the target block position. The feature extraction layer includes parallel frequency analysis units and type identification units. The frequency analysis unit maintains a ring buffer to record the access frequency of each data block within the last 15 minutes. Each time a new request arrives, the corresponding count is updated, and the count value is converted into an exponential smoothing weighted value to generate a dynamic frequency coefficient. The type identification unit establishes a multi-level classification tree: first, it distinguishes between HTTP, FTP, MQTT, and other application types based on the protocol header; second, it distinguishes between text, image, and video basic types based on the content type identifier; finally, it detects data patterns through load feature recognition tools to complete the final classification of structured / unstructured data.

[0022] The data storage state feature integration is implemented using a distributed message queue. The node collection agent encapsulates the processed raw data into a structured message, which includes: 1) node identifier (64-bit hash value); 2) collection timestamp (ISO8601 format); 3) five-dimensional feature vector (storage capacity ratio, average delay, fragmentation degree, access frequency, type code). The integration engine subscribes to the message queue and performs time window alignment operations to ensure the comparability of data between different nodes. The fragmentation degree in the feature vector is calculated by the Shannon entropy formula, and the type code is represented by a three-bit bitmap. The final unified feature set is stored in a columnar database with each row containing the complete five-dimensional features and their metadata.

[0023] The edge network transmission implementation process enables data compression channels. DEFLATE algorithm is used to compress feature data between nodes, with a compression block size of 32KB. The transmission layer establishes two parallel channels: the high-priority channel transmits real-time access delay data, and the low-priority channel transmits periodic capacity distribution data. Data distribution packets add sequence check codes, and the receiving end ensures data integrity through CRC-32 verification. When the transmission is interrupted, it automatically switches to a cache forwarding mode, and retransmits the lost data packets after the network is restored. The feature integration process uses zero-copy technology to directly map received cluster data to memory processing, avoiding redundant data copy operations.

[0024] During the feature set generation process, the abnormal data processing mechanism is in a continuous running state. When the node is offline during the collection stage, the association agent triggers the timeout retry mechanism, and after 3 retries, the node feature is marked as missing. The data verification module detects the legality of the feature value, and the feature value exceeding the set threshold range starts secondary collection verification. The feature value smoothing module suppresses short-term fluctuation interference, processes noise data through a first-order lag filter method, and maintains the stability and continuity of the feature data. The final feature set output records all abnormal processing logs for reference by the model construction stage.

[0025] The data access frequency feature adopts a dynamic weighted statistical method. The frequency analysis unit configures three time windows: a 15-minute short window for detecting sudden access, a 6-hour medium window for identifying business access patterns, and a 24-hour long window for analyzing periodic rules. The frequency values of each window are independently calculated and then weighted and fused, and the window weights are adaptively adjusted according to the access volume fluctuation coefficient. The data type feature encoding uses a tree structure representation method: the root node distinguishes between real-time / non-real-time data, the branch node distinguishes between transactional / analytical purposes, and the leaf node marks the specific encoding scheme. Each access request generates a 6-byte type identification code, of which the first bit represents the privacy level, the second bit identifies the compression state, and the last four bits encode the specific format features.

[0026] The collection system resource occupation control module implements a dynamic degradation strategy. When the node CPU usage exceeds 60%, the collection period is automatically extended to 10 seconds; when the memory occupation exceeds the limit, the data block distribution scanning function is closed; when the network bandwidth is tight, low-priority data transmission is suspended. The device support module monitors the health status of various storage media, and for SSD media, it adds write count monitoring, and for mechanical disks, it records bad sector information. All these additional features are added as hidden dimensions to the feature vector, only participating in calculation during the resource evaluation stage, maintaining the universality of the basic feature structure.

[0027] Embodiment 2: refer to Figure 3 The construction process of the data resource distribution model adopts a spatiotemporal tensor calculation method to convert discrete edge node features into continuous spatial distribution representation. The construction of the storage feature tensor starts from spatiotemporal alignment preprocessing, and the system deploys a network time protocol service to synchronize node clocks, eliminating the time deviation of cross-node data collection. The aligned feature data is input into the gridding processor, which uses a dynamic resolution algorithm to divide the three-dimensional space: the node dimension is clustered according to physical location, the time dimension is based on 5-minute units, and the feature dimension maintains the original five-dimensional structure. The grid cell size is automatically adjusted according to the node density, with fine-grained division in dense areas and macro cells in sparse areas. Each grid cell records the statistics of all node feature vectors within the cell, including the mean vector, variance matrix, and sample size.

[0028] The implementation of the resource consumption classifier is based on a deep neural network architecture. The input layer receives the normalized feature tensor with dimensions of grid number x time slice number x feature number. The hidden layers use three-dimensional convolution kernels to extract spatio-temporal features. The first convolution layer uses a kernel size of 5x3x5, the second convolution layer is reduced to 3x2x3, and the third convolution layer is further compressed to 2x1x2. After each layer of convolution operation, batch normalization and LeakyReLU activation function are applied to prevent gradient vanishing. The output layer converts the variable-length input to fixed dimensions through spatial pyramid pooling, and finally outputs the resource type probability distribution. The classifier training uses a weighted cross-entropy loss function, and the weight coefficient is dynamically adjusted according to the type sample size.

[0029] The data block position mapping establishes a conversion mechanism from logical address to physical grid. The system maintains a distributed hash table to record the mapping relationship between data block identifiers and storage nodes, and the hash function uses a consistent hashing algorithm to reduce the impact of migration. When an access request arrives, the position resolver performs the following calculation to determine the target grid coordinates:

[0030] wherein: represents the target grid coordinate vector, is the node position hash function, is the data block logical address, represents the feature space projection function, is the access request feature vector, represents the vector concatenation operation. This formula ensures that the same data block may be mapped to different feature space positions under different access patterns.

[0031] The system resource early warning judgment implements multi-level threshold monitoring. The calculation of the load evaluation value integrates six-dimensional indicators: CPU utilization , memory occupancy rate , storage I / O throughput , network bandwidth usage , thread pool activity and request queue length . After normalizing each dimension indicator to the [0, 1] interval through nonlinear transformation, the overall load score is synthesized according to the dynamic weight. The weight coefficient is automatically configured according to the node type: the weight of and is increased for computation-intensive nodes, the weight of and is focused on for storage-intensive nodes, and the weight of and is strengthened for network-intensive nodes.

[0032] The performance evaluation system is established based on the quantitative criteria of service level agreement (SLA). The basic performance indicators include the 95th percentile value of request response time , the number of successfully processed transactions per second , data persistence delay , and cache hit rate . The evaluation value calculation introduces a decay factor to handle historical data, with the current period indicators accounting for 70% of the weight, and the previous three period indicators accounting for 15%, 10%, and 5%, respectively. The performance state is divided into five levels: excellent (> 80), good (60-80), medium (40-60), poor (20-40), and bad (< 20). The grade boundary is set with a lag interval to prevent state oscillation.

[0033] The early warning trigger mechanism uses a composite logic judgment. The primary warning is activated when the load score exceeds 0.85 or the performance level drops to "medium". The advanced warning requires the load score to break through 0.95 or the performance level to drop to "poor". The state persistence judgment sets a 5-minute observation window, during which if the indicators fall below the threshold, the warning is canceled. The warning information includes three parts: the trigger node list, the over-standard indicator details, and the recommended processing strategy, which are broadcast to related nodes through the publish-subscribe mode. The warning state machine realizes four state transitions: normal → primary warning → advanced warning → recovery in progress → normal, and generates an audit log record to record the complete context for each transition.

[0034] The model update maintenance adopts an incremental processing method. The local update of the feature tensor is targeted at the grid cells that have changed, and the statistics of the cells are recalculated and the corresponding tensor slices are updated. The global reconstruction operation is triggered when more than 30% of the grid cells have changed, and the reconstruction process uses a parallel MapReduce framework: the mapping stage calculates the local grid statistics for each node, and the reduction stage combines to generate a global tensor. The online learning of the resource classifier uses small batch gradient descent, updating the model parameters every 100 new samples, and the learning rate is dynamically adjusted according to the classification accuracy.

[0035] The data block position index implements a multi-level cache strategy. The hot mapping relationship is cached in the memory hash table, equipped with an LRU eviction mechanism; the warm data is stored in the local RockDB key-value database; the cold data is backed up to the distributed file system. The index update uses the copy-on-write technology to ensure the query consistency during the migration process. The position resolver realizes the short-circuit query optimization, and directly returns the cache result for the same data block accessed continuously, skipping the complete calculation process.

[0036] The real-time calculation of load evaluation is implemented by a stream processing framework. The raw metrics collected by each node are sent to a Kafka message queue, and the Flink job consumes the data and performs window aggregation. The size of the sliding window is set to 1 minute, and the sliding step is 10 seconds, balancing real-time performance and computational overhead. The performance evaluation module uses the Lambda architecture, with the speed layer processing real-time stream data, the batch processing layer recalculating historical metrics in full every day, and the service layer combining the two parts to provide a unified query interface.

[0037] The distribution of early warning information uses a hierarchical push strategy. The primary warning only notifies the local scheduler, and the advanced warning triggers the global event bus. The push content is encoded using ProtocolBuffers, including binary format metrics snapshots and text format analysis suggestions. The warning processor implements the chain of responsibility pattern, supporting dynamic insertion of filters and processors, and facilitating business logic expansion. The state recovery detection sets a silent period mechanism, and the index must meet the standard for three consecutive detection periods to confirm recovery, avoiding frequent state switching.

[0038] Example 3: refer to Figure 4 The activation of the dynamic optimization mode triggers the start of the request buffering mechanism, which uses a double queue structure design. The main queue implements the weighted fair queuing algorithm, and each newly arrived access request obtains an initial weight value according to the service level field in its protocol header, with the weight range from 1 (ordinary business) to 8 (critical task). The auxiliary queue implements the emergency channel mechanism, and the control request carrying a special marker is directly released. The queue manager maintains a dynamic weight adjustment table, automatically increasing the weight of long-waiting requests according to the request processing delay, and the adjustment amplitude follows a logarithmic growth curve to prevent weight explosion. The capacity of the buffer is set to an adaptive threshold, not exceeding 30% of the available memory of the node. When the capacity reaches the warning line, the selective discard strategy is started, preferentially discarding low-weight and non-persistent query requests.

[0039] The data block priority evaluation module runs a multi-stage decision-making process. In the first stage, basic attribute scoring is performed. The access frequency score uses the inverse hyperbolic tangent function to map the original frequency value to the range of 0-10 points, the data freshness score is calculated based on the linear decay of the last modification timestamp, and the association dependency is determined by analyzing the co-occurrence patterns in the access log. In the second stage, business rule correction is implemented, and a pre-defined business priority matrix is loaded to apply a weight bonus to specific types of data combinations. The final priority label generation formula is as follows:

[0040] Where: P represents the final priority score, w represents the dynamic weight of the i-th evaluation dimension, s represents the original score of the i-th dimension, is the current migration depth counter for the data chunk. This formula suppresses the priority of frequently migrated data chunks through the logarithmic term in the denominator, avoiding system oscillation. The score result is discretized into a five-level label, and the conversion boundary is dynamically adjusted according to the current system load.

[0041] The resource type evaluation adopts a hybrid clustering method. In the offline stage, a K-means++ model is trained using historical data from the past three months to determine the prototype vectors of three resource consumption patterns. In the online stage, streaming clustering is implemented, and the cosine similarity between the feature vector of each data chunk and the prototype vector is calculated. When the maximum similarity exceeds 0.7, it is directly classified, otherwise the incremental clustering process is started. The resource type label is accompanied by a confidence score, which is used for risk control in the migration decision. The intermediate data generated in the evaluation process is written to a ring buffer for subsequent model optimization.

[0042] Real-time idle rate monitoring implements fine-grained resource slice statistics. The resources are divided into vCPU thread slots, memory resources are managed by 2MB pages, and storage I / O bandwidth is measured in units of 16KB. The monitor maintains a three-dimensional resource matrix, recording the utilization percentage of each slice in the past 15 seconds. The idle rate calculation excludes system reserved resources, and median filtering is used to handle transient fluctuations. The matrix data is persisted every 5 seconds to form a resource utilization time series for trend prediction.

[0043] The migration selector implements a multi-condition constraint satisfaction algorithm. The core selection logic includes three decision layers: the first layer filters data chunks whose priority labels reach the current processing level; the middle layer selects the instance with the most matching resource type label from these candidate data chunks, taking into account the complementarity of resource types; the final layer applies a position optimization strategy, preferring data chunks with the shortest migration path and not crossing racks. The decision tree generated in the selection process is recorded in the audit log, including the elimination reasons for all excluded candidates.

[0044] The target node selection adopts a multi-objective optimization method. The candidate node evaluation considers six key factors: physical distance weight, network topology hop count, remaining storage capacity percentage, expected load balancing degree, data locality retention rate, and migration cost estimation. Each factor is analyzed through the Pareto frontier to determine the optimal solution set, and a random disturbance factor is added to the final decision to prevent the swarm intelligence from falling into local optimization. The selection result is accompanied by a rollback flag, which automatically triggers the alternative solution when resource conflicts occur during actual migration.

[0045] The migration execution engine implements a segmented transmission protocol. Large-size data blocks are split into multiple 2MB transmission segments, each of which is independently checksummed and encrypted. Transmission control uses an improved BBR algorithm to dynamically adjust window size and transmission rate based on network conditions. The engine maintains a migration state machine that records the status of each data block in the preparation, transmission, verification, and submission stages. The exception handling mechanism includes three recovery strategies: automatic retry, breakpoint resume, and failure rollback, which take different measures for different types of error codes.

[0046] The concurrency control module implements distributed transaction management. Exclusive locks are acquired for the source and target nodes before the migration operation begins, with a lock granularity controlled at the data block level. The lock management uses a lease mechanism with a default lease period of 30 seconds and a maximum of three renewals. The transaction coordinator implements a two-phase commit protocol, verifying target node capacity and permissions in the preparation phase and atomically updating the location index in the submission phase. All participating nodes maintain operation logs, use the Raft algorithm to ensure log consistency, and ensure state synchronization after failure recovery.

[0047] The dynamic weight adjuster continuously optimizes evaluation parameters. The system collects actual migration effect data during operation, including delay reduction amplitude, throughput improvement ratio, and resource utilization change. The adjuster uses a reinforcement learning framework, with weight coefficients as the action space and system performance indicators as reward signals, to gradually optimize the decision model through policy gradient methods. After each weight update, the old version is retained as a snapshot, and when the new weight causes a performance drop of more than 5%, it automatically reverts to the previous stable version. The training data generated during the adjustment process is stored in an experience replay buffer, supporting offline batch learning.

[0048] Embodiment 4: see Figure 5 During the model update process after completing the data migration operation, the system starts a three-dimensional monitoring mechanism. The capacity change monitor compares the storage usage of each node before and after migration, records the difference to form a capacity change matrix. For example, when data block B-7F3 is migrated from node E-05 to E-11, the capacity changes of the two nodes are recorded as -1.8GB and +1.8GB, respectively. The delay change analyzer sends probe request packets to measure the access delay of all involved migrated data blocks in the new location, forming a delay change comparison table. The location tracker updates the global index record and records the transit node information in the migration path.

[0049] The feature reconstruction engine adopts an incremental update strategy to handle change data. The update of the stored feature tensor is divided into two modes: when the capacity of a single node changes by less than 15%, a local grid refresh operation is performed, and only the statistics of the grid cell where the node is located are recalculated; when more than 4 nodes in the cluster are migrated (taking a 10-node cluster as an example), full-model recalculation is triggered. The data smoothing process introduces a time decay factor, with new measurement values accounting for 70% of the weight and historical values accounting for 30% of the weight, to avoid model oscillation caused by index mutations.

[0050] Table 1: Data records after a migration operation.

[0051] Node ID Data Block ID Original Capacity (GB) New Capacity (GB) Delay Change (ms) Location Status E-05 B-7F3 186.4 184.6 -12.3 Removed E-11 B-7F3 92.8 94.6 +5.2 Loaded E-09 D-A22 154.2 152.0 -8.7 Removed E-13 D-A22 87.3 89.5 +3.1 Loaded E-02 F-9B1 203.7 203.7 - Transit Node The resource distribution model reconstruction performs a four-step process: first, load the updated stored feature tensor, whose dimension structure remains unchanged; then perform feature vector normalization to map the values of each dimension to the [0, 1] interval; then perform forward propagation calculation through the resource allocation classifier; finally, output the new storage capacity heat map, updated access delay distribution matrix, and adjusted data type distribution model. This process uses pipelining, and each node independently completes local model updates, which are then integrated into a global view by the coordinator.

[0052] The system exit determination uses a gradual verification mechanism. The load stability value calculation is based on a set of ten consecutive monitoring periods, and the variance analysis method is used to evaluate the load fluctuations of each node. For example, within a 10-minute observation window, the load value sequence of node E-11 is [68, 72, 69, 71, 70, 73, 69, 70, 71, 70], and its variance value of 4.2 is lower than the threshold of 5.0, which is considered stable. The performance recovery value verification uses a three-level check: the first detection of E-11's access delay is 34ms (with a recovery threshold of 45ms), and the throughput reaches 142 operations per second (with a threshold of 120 operations). The results of the subsequent two detections are 31ms / 148ops and 33ms / 145ops, respectively, both of which meet the continuous improvement requirement.

[0053] The exit condition check implements strict safety rules. All nodes must simultaneously satisfy two conditions: the load stability value is continuously lower than 20 units for three consecutive periods, and the performance recovery value is continuously higher than 80 points for three consecutive periods. The regional coordinator regularly broadcasts state query requests, collecting real-time load indices and performance scores from each node. When the load fluctuation coefficient of node E-09 is detected to be 22 (threshold is 20), the system delays the exit decision until the node's load falls back to the stable interval.

[0054] The state transition process contains a two-stage buffer period. The preparation stage maintains the optimization mode but suspends new migration tasks, during which the persistence of migration effects is verified. Before formal exit, a final consistency check is performed: the data location index verifier scans the global mapping table; the configuration synchronizer compares the local configuration of each node with the central record; and the performance baseline generator establishes a new monitoring baseline. After all checks pass, the system marks the resource distribution model as a stable version.

[0055] The abnormality handling mechanism remains active throughout the update process. When the delay measurement value of node E-11 abnormally jumps to 128 ms, the diagnostic automatically initiates a three-stage investigation: first, re-measure critical indicators to rule out measurement errors; second, check network link status; and finally, analyze target data block integrity. After confirming that it is temporary network congestion, the system replaces the abnormal measurement value with a historically reasonable value. If configuration inconsistencies are detected (e.g., node E-09 local record missing data block D-A22), the repairer copies the metadata copy from the nearest neighbor node to achieve rapid recovery.

[0056] The resource distribution model version control system maintains a change history. Each model update generates a timestamped version record, retaining complete snapshots of the past six versions. Differential storage mode is used for data persistence, saving only the amount of change relative to the previous version. The version rollback mechanism can restore the system to the previous stable state within 45 seconds when performance degradation is detected. The audit log details version switching information, including the trigger reason, the list of changed nodes, and the duration statistics.

[0057] The performance monitoring baseline is dynamically adjusted in the exit process. After successfully exiting the dynamic optimization mode, the system establishes a new monitoring threshold based on the post-migration measurement value. This baseline includes three key parameters: the average delay baseline value takes the middle value of the three measurements during the exit verification period (e.g., 34 ms); the throughput baseline value takes the minimum safe value (e.g., 135 ops); and the capacity warning threshold is set to 85% of the maximum capacity of the node. These baseline parameters serve as evaluation criteria for subsequent running states until the next optimization mode activation, when they are reset.

[0058] The coordinator communication protocol enhances fault tolerance mechanisms in the exit phase. The state switching command is confirmed through a three-way handshake: the primary coordinator sends a preliminary exit instruction; the node returns the current load value; and the coordinator verifies and sends a confirmation signal. Each node is equipped with a timeout retransmission mechanism, which automatically attempts to transmit through the backup path when communication fails. The final state synchronization uses an atomic broadcast protocol to ensure that all nodes switch operation modes simultaneously, avoiding inconsistent resource access states.

[0059] Example 5: After exiting the dynamic optimization mode, the system switches to the standard operating state. The priority queue manager closes the buffer channel entrance, and newly arrived access requests are directly routed to the target node. The request parser analyzes the message header to extract key parameters, including the protocol version, operation code, data block identifier, and security token. The authentication unit verifies the digital signature of the security token and rejects unauthorized access attempts. Each request is assigned an independent thread for processing, and the thread pool is divided into priority groups based on request type: real-time control instructions are assigned high-priority threads, and batch data processing uses low-priority threads. The execution engine establishes a four-stage processing pipeline: the first stage loads data block metadata, the second stage checks data integrity, the third stage performs read and write operations, and the fourth stage generates response messages. The response time monitor records the time spent in each stage, and the automatic retry mechanism is triggered for timeout requests.

[0060] The data migration priority queue generator enters a dormant state, and historical evaluation data is compressed and stored. The snapshot of the last active queue is serialized and stored in a distributed key-value database, with a time span of the last three optimization periods. The snapshot includes a complete data block priority tag mapping table and resource type classification record. The data aging strategy implements a timed cleanup, automatically removing historical data older than thirty days, but retaining statistical summary information. The queue management interface remains in a responsive state and can receive manual query instructions to return historical decision logs.

[0061] The data transmission uses a hierarchical modulation scheme. The control plane encapsulation module processes node topology information and converts dynamic identifiers into fixed area codes. Each node is assigned a unique location code, and resource summary information is represented using bit fields: the first 3 bits identify the CPU type, the next 4 bits represent memory hierarchy, and the last 12 bits record storage capacity levels. The data plane processing module uses differential encoding technology: for indicators with a change greater than 15% compared to the last transmission, full records are executed, and for the remaining indicators, only the change increment is transmitted. The indicator encapsulation uses a custom binary protocol, and the field structure includes version number, timestamp, node identifier, indicator type flag, and payload.

[0062] The edge network transmission implements dual-path redundancy. The primary channel uses UDP protocol to transmit control plane data, adding forward error correction codes to resist network packet loss. The backup channel transmits detailed indicator data through TCP protocol, and each transmission packet is attached with a cyclic redundancy check code. The path selector monitors network quality in real time and automatically switches to the backup channel when the packet loss rate of the primary channel exceeds 5%. The transmission window size is dynamically adjusted based on the round-trip delay, with a starting value of 32KB and an upper limit not exceeding 80% of the node's available network buffer.

[0063] The central management system receives the end of the establishment of a multi-layer decoding pipeline. The signal preprocessor completes frame synchronization and clock recovery, and corrects the timing offset in the transmission process. The check unit verifies the data integrity, discards the frames that fail the check and triggers the retransmission mechanism. The control plane decoder parses the location code mapping table and recovers the node physical topology structure. The resource summary decoder parses the bit field and reconstructs the resource configuration view of each node. The data plane decoder processes the difference encoding stream and applies the reverse conversion algorithm to restore the complete index set. The fields that fail to decode trigger the error isolation program, and the related data is transferred to the repair buffer.

[0064] The data reorganization engine performs three-dimensional data fusion. The spatial dimension integrates the data reported by each node to construct a cluster-level storage state view. The time dimension aligns the time series data of each node, with a maximum tolerance time difference of 500 milliseconds. The index dimension associates different performance parameters to establish the correspondence between capacity utilization and access delay. The fused data set is stored in the time series database, and the index key includes the node identifier, data type and collection time point. The storage implements a sharding strategy, divides data shards by time range, and stores the last three hours of data in the in-memory database and archives the historical data to the columnar storage.

[0065] The transmission link maintenance implements a heartbeat feedback mechanism. The edge node sends a beacon frame every 30 seconds, containing the current transmission sequence number and resource summary. The central system responds with an acknowledgment frame, carrying the latest successfully received sequence number range. The packet loss detector analyzes the sequence number continuity and starts the link diagnosis program when it finds that two consecutive heartbeats have not responded. The diagnosis program has three stages: network layer connectivity test, transmission layer session state check, and application layer protocol verification. The fault recovery process prioritizes the reconstruction of existing connections, and after three failures, it starts the new session establishment process.

[0066] The configuration synchronization system implements a version control strategy. The central management system maintains the configuration master copy, and the node locally saves the read-only cache. After each optimization period ends, a configuration update package is issued, and the update package is attached with a version mark and a timestamp. The node returns an acknowledgment signal after applying the update, and unconfirmed nodes are included in the retransmission list. The version conflict resolution adopts the last write priority principle, and the configuration change record completes the audit trail. The rollback mechanism allows recovery to any valid version within the last 24 hours, and the rollback operation requires three security authentications.

[0067] The state monitoring console implements a multi-level display strategy. The base view presents a cluster-wide storage utilization heat map, and the secondary view presents a single node detailed metric trend chart. The alert interface highlights abnormal parameters, and the abnormality detection is based on a dynamic baseline algorithm: the base threshold is set to a three-sigma range of the historical mean, and the baseline value is automatically adjusted every hour. The data drill-down function supports step-by-step exploration from the cluster view to the single data block access record, and all the displayed data implements dynamic sampling rendering to ensure the visualization performance of large-scale data sets. User operation events are recorded in detailed logs, including view switching, time range selection, and abnormality confirmation actions.

[0068] It should be noted that the relational terms herein, such as first and second, and the like, are used solely to distinguish one from another entity or action without necessarily requiring or implying any actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0069] While embodiments of the application have been shown and described, it is to be understood that the embodiments described are only by way of example and that changes can be made to the embodiments without departing from the spirit and the scope of the application, which is defined in the appended claims and their equivalents.

Claims

1. A dynamic optimization method for distributed data storage based on edge computing, characterized in that, Includes the following steps: Obtain data storage status data and data access request data from distributed edge nodes; A data resource distribution model is constructed based on the data storage status data and data access request data; The system resource early warning status is determined based on preset optimization conditions and the data resource distribution model. When the system resource warning state is triggered, the system enters dynamic optimization mode and generates a data migration priority queue. In the dynamic optimization mode, data migration operations are performed based on the data migration priority queue; Update the data resource distribution model based on the results of the data migration operation; The system resource exit status is determined based on the updated data resource distribution model; When the system resource exit state is triggered, the dynamic optimization mode is exited. The optimized data storage status is transmitted to the central management system via the edge computing network.

2. The dynamic optimization method for distributed data storage based on edge computing according to claim 1, characterized in that, The specific steps for obtaining the data storage status data and data access request data of the distributed edge nodes are as follows: Collect storage capacity utilization data, data access latency data, and data block size distribution data for each edge node; Based on the adaptive data sharding algorithm, request features are extracted from the data access request data to obtain data access frequency features and data type features. The storage capacity utilization data, data access latency data, data block size distribution data, data access frequency characteristics, and data type characteristics are integrated to generate a data storage status feature set.

3. The dynamic optimization method for distributed data storage based on edge computing according to claim 2, characterized in that, The step of constructing a data resource distribution model based on the data storage status data and data access request data specifically involves: The data storage state feature set is subjected to multi-scale spatial analysis, and a storage feature tensor is constructed through gridding processing; Based on the resource consumption classifier, the storage feature tensor is used to identify resource types, and a data resource distribution model including storage capacity distribution, access latency distribution and data type distribution is generated. Data block location information is obtained based on the data access request data, and the storage characteristics of each data block are mapped to the data resource distribution model based on the data block location information.

4. The dynamic optimization method for distributed data storage based on edge computing according to claim 3, characterized in that, The determination of the system resource early warning status based on preset optimization conditions and the data resource distribution model specifically includes: The preset optimization conditions include system resource load thresholds and storage performance thresholds; The load assessment value and performance assessment value of each edge node are calculated based on the data resource distribution model; When the load assessment value of any edge node reaches the system resource load threshold or the performance assessment value is lower than the storage performance threshold, a system resource warning status is issued. The system resource warning status includes high load warning status and low performance warning status.

5. The dynamic optimization method for distributed data storage based on edge computing according to claim 4, characterized in that, When the system resource warning state is triggered, the system enters dynamic optimization mode and generates a data migration priority queue, specifically as follows: After entering dynamic optimization mode, all new data access requests enter a waiting sequence; Priority is evaluated for each data block based on the data resource distribution model, and priority labels are generated. Based on historical resource consumption data, resource type assessment is performed on each data block to generate resource type labels; Construct a data migration priority queue based on priority labels and resource type labels.

6. The dynamic optimization method for distributed data storage based on edge computing according to claim 5, characterized in that, In the dynamic optimization mode, the data migration operation is performed based on the data migration priority queue, specifically as follows: Obtain real-time idle rate data of system resources; Select the data block with the highest priority label from the data migration priority queue; For data blocks with the same priority label, select the data block with the resource type label that matches the resource type with the highest idle rate based on the real-time idle rate data; Perform the migration operation of the selected data block to the target edge node.

7. The dynamic optimization method for distributed data storage based on edge computing according to claim 6, characterized in that, The step of updating the data resource distribution model based on the data migration operation results specifically involves: Collect data on changes in storage capacity, access latency, and data block location after the migration operation; Perform feature reconstruction on data showing changes in storage capacity, access latency, and data block location, and update the storage feature tensor. The data resource distribution model is regenerated based on the updated storage feature tensor.

8. The dynamic optimization method for distributed data storage based on edge computing according to claim 7, characterized in that, The determination of system resource exit status based on the updated data resource distribution model specifically involves: The preset exit conditions include system resource stability thresholds and storage performance recovery thresholds; The load stability value and performance recovery value of each edge node are calculated based on the updated data resource distribution model; When the load stability value of all edge nodes remains below the system resource stability threshold and the performance recovery value is higher than the storage performance recovery threshold, the system resource exit status is issued. The system resource exit status includes load stabilization exit status and performance recovery exit status.

9. The method for dynamic optimization of distributed data storage based on edge computing according to claim 8, characterized in that, The step of exiting dynamic optimization mode when the system resource exit state is triggered is specifically as follows: After exiting dynamic optimization mode, new data access requests directly enter the normal processing sequence; Data access request processing is performed based on the normal processing sequence; Stop generating and maintaining the data migration priority queue.

10. The dynamic optimization method for distributed data storage based on edge computing according to claim 9, characterized in that, The process of transmitting the optimized data storage status data to the central management system via the edge computing network specifically involves: The optimized data storage status data is modulated based on a hierarchical data transmission protocol to generate a data storage transmission signal. Data storage transmission signals are sent to the central management system via edge computing networks; The central management system decodes the data storage and transmission signals to restore the optimized data storage status data.

Images