Data hierarchical storage method for unmanned aerial vehicle distributed cluster

Abstract

The invention relates to the technical field of distributed storage, and discloses a data hierarchical storage method for an unmanned aerial vehicle distributed cluster, which comprises the following steps: S1, initializing system storage parameters, including capacity of unmanned aerial vehicle storage nodes, used storage capacity and priority data distribution; s2, collecting real-time data in an unmanned aerial vehicle cluster, wherein the real-time data comprises the storage utilization rate of nodes, the priority of task data and inter-node communication conditions; s3, calculating the storage distribution balance of the current system based on the collected data; and S4, constructing an optimization function taking the storage balance and the data migration cost as targets. The balance of system storage resources is quantified by introducing a distribution entropy theory, and storage distribution is dynamically adjusted in combination with a reinforcement learning algorithm. Compared with a traditional static distribution mode, the method can adapt to task requirements and changes of storage node states in real time, the utilization rate of storage resources is effectively increased in a complex dynamic environment, and efficient storage and access of task data are ensured.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage technology, and in particular to a data hierarchical storage method for a distributed cluster of unmanned aerial vehicles. Background Art

[0002] In the application of drone distributed clusters, such as disaster monitoring, logistics distribution and smart city management, efficient storage and rapid response of massive data are important requirements for improving system performance. As distributed storage nodes, drones have the characteristics of dynamic, decentralized and limited storage resources. These characteristics make how to efficiently use storage resources under dynamic task requirements and complex network environments an important research direction.

[0003] Traditional distributed storage methods mainly include static distributed storage, centralized storage optimization, and task-independent balanced distribution strategies. The static distributed storage method allocates storage resources through predefined strategies, but it lacks adaptability when faced with real-time changing task requirements, which easily leads to uneven distribution of storage resources. Centralized storage optimization achieves global optimization by uniformly scheduling storage resources, but in a distributed environment, the high communication cost and potential single point failure problems of centralized control affect the reliability of this method. On the other hand, the task-independent balanced distribution strategy attempts to improve the storage utilization of the system, but it does not consider the priority of task data enough, which easily causes response delays for high-priority tasks.

[0004] The data storage of distributed drone clusters faces diverse task requirements. For example, the requirements for storage resources for real-time tasks and historical tasks are significantly different. High-priority tasks such as real-time video data storage usually require higher response speed and reliability, while low-priority tasks such as historical data storage can be delayed appropriately. In addition, the communication conditions between drone nodes, such as bandwidth and latency, will fluctuate with the task execution location and environmental changes, which further increases the complexity of data migration and storage adjustment. On this basis, how to efficiently allocate and dynamically adjust limited storage resources is particularly important. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a data hierarchical storage method for a distributed cluster of drones to solve the problem of dynamic allocation and optimization of storage resources in a distributed cluster of drones.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: A data hierarchical storage method for a distributed cluster of drones, comprising the following steps: S1. Initialize system storage parameters, including the capacity of drone storage nodes, used storage, and priority data distribution; S2. Collect real-time data in the drone cluster, including node storage utilization, task data priority, and inter-node communication conditions; S3. Calculate the storage distribution balance of the current system based on the collected data; S4. Construct an optimization function with storage balance and data migration cost as the goal; S5. Generate storage adjustment strategy through reinforcement learning algorithm; S6. Perform data migration according to the storage adjustment strategy to optimize storage distribution; S7. Update the storage status and continue to optimize according to the updated status to form a closed-loop process.

[0007] Preferably, the step of initializing system storage parameters includes: S11. Determine the total storage capacity and currently used storage capacity of each storage node; S12. Divide the task data into multiple categories according to priority, and assign a weight to each priority to indicate the degree of influence of different priority data on the system.

[0008] Preferably, the step of collecting real-time data from the drone cluster includes: S21. Obtain the storage utilization of each storage node; S22. Monitor the distribution ratio of task data of different priorities stored in each node; S23. Monitor the communication conditions between nodes, including communication delay and transmission cost.

[0009] Preferably, the calculation of the storage distribution balance is obtained by weighting the proportion of data with different priorities in each node, so as to quantify the distribution state of the current storage resources and reflect the degree of balance of the system.

[0010] Preferably, the optimization function is based on storage distribution balance and takes into account the cost of data migration, wherein the data migration cost is represented by a weighted result of the amount of migrated data and the communication cost between nodes.

[0011] Preferably, the reinforcement learning algorithm is a deep Q-learning algorithm, which generates an optimal storage adjustment strategy by learning and iteratively optimizing the current storage state, adjustment actions and reward values ​​of system feedback.

[0012] Preferably, the reward value is calculated based on storage distribution balance and migration cost, the positive impact of storage distribution balance on the reward value increases as the system's balance level increases, and the negative impact of migration cost on the reward value increases as the migration cost increases.

[0013] Preferably, the data migration comprises the following steps: S61. Determine the amount of data to be migrated and its target node; S62. Compression is used for large data files to reduce transmission overhead; S63. Use the breakpoint resume mechanism to ensure the integrity of the data migration process.

[0014] Preferably, the updating of the storage status includes: S71. Update the storage utilization of the storage node and the distribution status of the task data; S72. Recalculate the storage distribution balance of the system and use the new state as input for the next round of optimization.

[0015] Preferably, the method is optimized through a closed-loop process, which includes collecting the real-time status of the system, evaluating storage balance, optimizing storage adjustment strategies, performing data migration and updating storage status, so as to gradually improve the system's storage resource utilization efficiency and task priority matching capabilities.

[0016] The present invention provides a data hierarchical storage method for a distributed cluster of drones. It has the following beneficial effects: 1. The present invention introduces the distribution entropy theory to quantify the balance of system storage resources, and dynamically adjusts the storage distribution in combination with the reinforcement learning algorithm. Compared with the traditional static distribution method, the present invention can adapt to the changes in task requirements and storage node status in real time, effectively improve the utilization of storage resources in a complex dynamic environment, and ensure efficient storage and access of task data.

[0017] 2. The present invention introduces priority weights to give priority to the data storage needs of high-priority tasks during the optimization process, ensuring that critical task data is preferentially allocated to low-latency, high-reliability storage nodes. This mechanism significantly improves the system's responsiveness to emergency and high-priority tasks and enhances the reliability of overall task execution.

[0018] 3. The present invention minimizes the cost of data migration while maintaining storage balance by optimizing the objective function. Specifically, the amount of migrated data and the transmission path are optimized, reducing the occupation of communication bandwidth and transmission time. In addition, through compression technology and breakpoint resume mechanism, the communication cost and energy consumption of large data volume transmission are further reduced.

[0019] 4. The present invention builds a closed-loop optimization process based on reinforcement learning, which can autonomously learn and adjust storage strategies, and quickly adapt to new system environments when task requirements, network status, or node storage loads change. Compared with traditional methods that rely on manual intervention, the present invention significantly improves the intelligence level and operating efficiency of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1The figure is a schematic flow chart of the steps of the hierarchical storage method of the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the specification of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] Please refer to the attached Figure 1 The embodiment of the present invention provides a data hierarchical storage method for a distributed cluster of drones, comprising the following steps: Initialize system storage parameters, including the capacity of drone storage nodes, used storage, and priority data distribution; In this embodiment, initializing system storage parameters includes the following: Initialization of node storage capacity: As an implementation method, the system first determines the number of storage nodes n in the drone distributed cluster and allocates storage capacity to each storage node ,in Indicates the maximum available storage capacity of the i-th node, in GB or TB.

[0023] Specifically, the node storage capacity It can be determined according to the hardware configuration, for example, a node equipped with a high-performance storage device can be allocated a larger storage capacity. In some embodiments, the node capacity can also be dynamically adjusted according to task requirements, for example, a regional node with a higher task load can be assigned more storage resources.

[0024] It should be noted that the storage capacity initialization of the drone node should take into account the overall requirements of the task and the storage balance, for example, sufficient configuration margin should be provided for subsequent distribution balance optimization.

[0025] Calculation of storage utilization: In one possible implementation, the initialization step also requires obtaining the used storage of each node. . use and , calculate the storage utilization of each node : , in, represents the current used storage of node i, Indicates the storage capacity of the node; For example, if a node , used storage , then its storage utilization is This storage utilization can be used to measure the current load of the node and provide initial parameters for the distribution entropy calculation.

[0026] It can be understood that the calculation of storage utilization can not only reflect the storage status of the node, but also provide a quantitative basis for the initialization of priority data distribution.

[0027] Classification and weight setting of task data priority: In the present invention, in order to realize hierarchical storage, the task data needs to be divided into m categories according to priority, and a priority weight is set for each category of data. , where k is the priority category index.

[0028] Specifically, the priority weight The following constraints are met: , As an option, high real-time task data can be assigned a higher weight. For example, for a disaster monitoring task, the weight of real-time sensor data can be set to , and the weight of video archive data can be set to .

[0029] It should be noted that the setting of priority weights should be flexibly adjusted according to the requirements of task importance and access frequency, so as to ensure that high-priority data is given priority in storage distribution.

[0030] Initialization of priority data distribution: In this embodiment, the storage distribution ratio of the priority data is initialized for each node. ,in Indicates the proportion of the k-th type of data stored in node i, satisfying: , Where m represents the number of priority categories of task data.

[0031] Specifically, in a possible implementation, the access statistics of historical task data can be used to initialize For example, if the storage utilization of node i is , where real-time data and historical data occupy 60% and 40% of the storage respectively, the priority data distribution is: , .

[0032] Exemplarily, a node has the following storage configuration: Total storage capacity ; Used storage ; Storage Utilization ; Real-time data weight , historical data weight ; Priority data distribution , .

[0033] It can be understood that this initialization process can provide accurate initial conditions for the subsequent optimization strategy generation.

[0034] Technical description of the extended implementation: As a possible extension, the initialization process can also be globally configured in combination with the task collaboration requirements between nodes. For example, if the communication capability of some nodes is weak or the task load is high, their initial storage utilization can be appropriately reduced. , thereby reducing the pressure of subsequent data migration.

[0035] It should be noted that the initialization parameter configuration in the present invention is flexible, but should always meet the overall requirements of task priority matching and storage balance, and provide an efficient initial state for subsequent optimization algorithms.

[0036] S2. Collect real-time data in the drone cluster, including node storage utilization, task data priority, and inter-node communication conditions; In this embodiment, collecting real-time data from the drone cluster includes the following: Storage status collection: In one possible implementation, in order to understand the load of each storage node in real time, the system needs to regularly collect the storage utilization of each node. , the value range is [0,1], and its calculation formula is: , in, Indicates the current used storage capacity of node i, in GB or MB. Indicates the storage capacity of the node in GB or MB.

[0037] Specifically, the acquisition frequency can be adjusted dynamically according to task requirements. For example, for high-load task scenarios, the acquisition frequency can be increased to obtain more timely storage status.

[0038] It should be noted that storage utilization Real-time updates are essential for monitoring the dynamic distribution of storage resources. For example, if the storage capacity of a node is 100 GB and the current used storage capacity is 50 GB, the storage utilization is 0.5, indicating that half of the storage resources of the node are used.

[0039] Collection of priority data distribution: As an option, the system also needs to collect the distribution ratio of task data of different priorities stored in each node . It represents the storage ratio of the k-th priority data on node i, satisfying the following formula: , Specifically, the system classifies and counts the storage content of the nodes to obtain the storage ratio of each type of task data.

[0040] In some embodiments, the collection of priority data distribution can be optimized in combination with task history records. For example, for data with high access frequency, its distribution information can be collected first to improve the accuracy of the storage strategy.

[0041] It should be noted that the collection of priority data distribution not only helps to quantify the task data distribution of each node, but also provides necessary support for the storage guarantee of high-priority tasks.

[0042] Communication conditions collection: In the present invention, in order to optimize the data migration path and minimize the migration cost, it is necessary to collect the communication conditions between nodes in real time, including communication delay and transmission cost. .

[0043] As an implementation method, the transmission cost It can be calculated by measuring the data transmission delay, bandwidth occupancy and other parameters between node i and node j. Specifically, the transmission cost can be defined as: , For example, if the amount of data transmitted at a time is 10 MB, the bandwidth is 5 MBps, and the network delay is 10 ms, the transmission cost is 2.1 s.

[0044] It should be noted that real-time collection of communication conditions is crucial to building an accurate migration cost model. When communication conditions change rapidly, the system can increase the collection frequency to capture dynamic network status.

[0045] Update of task priority weights: As an implementation method, in order to adapt to dynamic task requirements, the system also needs to regularly update the priority weight of task data during data collection. , where k is the priority category index, satisfying the following constraints: , in, : The priority weight of the k-th task data.

[0046] m: The number of priority categories of task data.

[0047] Specifically, the priority weight can be set according to the real-time and importance of the task. For example, for a disaster monitoring task, the weight of real-time sensor data can be set to 0.8, while the weight of historical record data can be set to 0.2.

[0048] It should be noted that the dynamic update of task priority weights can effectively improve the system's responsiveness to key tasks.

[0049] Technical description of the extended implementation: In a possible extended implementation, real-time data collection can also be combined with an edge computing module to pre-process the collected data locally at the node to reduce communication overhead. For example, priority data distribution can be compressed in real time, and only statistical information of high-priority data can be transmitted.

[0050] It is understandable that by combining the real-time collection of storage status, priority distribution, communication conditions and task weights, the present invention can fully grasp the operating status of the drone cluster and provide a reliable basis for the calculation of distribution entropy and the generation of optimization strategies. This implementation method enables those skilled in the art to easily reproduce the technology based on the disclosed content.

[0051] S3. Calculate the storage distribution balance of the current system based on the collected data; In this embodiment, the step of calculating storage distribution balance based on the collected data includes the following contents: Definition and basic calculation of distribution entropy: In a possible implementation, the distribution entropy is used to quantify the balance of storage resources in the distributed cluster of drones. and priority data distribution Perform weighted calculation to obtain the distribution entropy value H of the current system.

[0052] Specifically, the distribution entropy is defined as follows: , in: n represents the total number of storage nodes; m represents the number of priority categories of task data; represents the priority weight of the k-th task data, and represents the importance weight of task k in the overall task data, satisfying ; Indicates the proportion of the k-th type of data stored in the i-th node, satisfying ; express The natural logarithm of quantifies the contribution of each storage distribution to the system distribution entropy.

[0053] It should be noted that the larger the value of the distribution entropy H is, the more balanced the storage distribution is; the smaller the value of the distribution entropy is, the more balanced the storage resource allocation is and the more optimization is needed.

[0054] The process of calculating distribution entropy: In one possible implementation, the system first calculates the storage utilization of each node. and priority data distribution The preprocessed data is used as the input for calculating the distribution entropy.

[0055] For example, if the system contains n=3 nodes, the task data is divided into m=2 categories, and the storage status of the node is: Node 1: , ; Node 2: , ; Node 3: , ; The priority weights of task data are: , , At this time, the calculation of distribution entropy is: ; It can be understood that by calculating the distribution entropy, the system can quantify the storage allocation status of each node and provide a basis for subsequent optimization.

[0056] Threshold judgment of distribution entropy: As an option, the present invention can set a distribution entropy equilibrium threshold , used to determine whether the system needs storage adjustment. Specifically: If the current distribution entropy , it means that the system's storage distribution is well balanced and the current state can be maintained; If the current distribution entropy , it means that the system has uneven storage resource allocation and needs to start the optimization process.

[0057] It should be noted that the setting of the distribution entropy threshold should be dynamically adjusted according to task requirements and system characteristics to balance storage balance and migration cost.

[0058] Technical description of extended calculation: In a possible extended implementation, the distribution entropy calculation can be improved by combining the storage load correlation between nodes. For example, for nodes with collaborative tasks, the task correlation weight between nodes can be introduced. , and modify the distribution entropy formula: ; in: H: Improved distribution entropy, used to quantify the balance of storage distribution, combined with the task correlation between nodes.

[0059] n: The total number of storage nodes.

[0060] m: The number of priority categories of task data.

[0061] : The priority weight of the k-th task data.

[0062] : The storage ratio of the kth priority data on node i.

[0063] : Natural logarithm.

[0064] : Balancing factor used to control the impact of correlation on distribution entropy.

[0065] : The task correlation between node i and node j.

[0066] : The storage ratio of the kth priority data on node j.

[0067] It is understandable that this improved calculation method can better reflect the storage distribution status of nodes with collaborative tasks and further improve the optimization effect.

[0068] S4. Construct an optimization function with storage balance and data migration cost as the goal; In this embodiment, the steps of constructing an optimization function with storage balance and data migration cost as the objectives include the following: The optimization goal of distribution balance is: In a possible implementation, storage distribution balance is represented by distribution entropy H, which is used to quantify the storage resource utilization of the system. The distribution entropy formula is: ; in: m represents the total number of priority categories; n represents the total number of storage nodes; is the priority weight of the k-th category data, satisfying ; Indicates the proportion of the k-th type of data stored in the i-th node, satisfying .

[0069] It should be noted that the larger the distribution entropy H value is, the more balanced the storage distribution of the system is. During the optimization process, the goal is to maximize the distribution entropy to ensure balanced resource utilization of the system.

[0070] Specifically, the distribution entropy optimization part can be achieved by dynamically adjusting the storage distribution of nodes. For example, for high-priority task data, it can be distributed and stored in multiple nodes first, thereby improving the distribution entropy value.

[0071] Optimization goal of data migration cost: In the present invention, the data migration cost ( ) The amount of data migrated between nodes and transmission cost The formula for migration cost is: ; in: Indicates the amount of data migrated from node i to node j, in MB or GB; is the transmission cost between node i and node j, taking into account the data volume, bandwidth and network delay.

[0072] In one possible implementation, the transmission cost It can be expressed as: ; in: : The transmission cost between node i and node j.

[0073] Data volume: The total amount of data transferred from node i to node j, in MB or GB.

[0074] Bandwidth: The available network bandwidth between node i and node j, in MBps.

[0075] Network delay: The communication delay between node i and node j, in seconds.

[0076] For example, if the bandwidth between nodes i and j is 10MBps, the amount of data to be migrated is 50MB, and the network delay is 0.1s, then .

[0077] It can be understood that the optimization goal of the migration cost is to minimize the total cost, thereby reducing system energy consumption and communication overhead.

[0078] Comprehensive construction of optimization functions: As an option, the optimization function constructs the following objective function by integrating the quantitative model of distribution entropy and data migration cost: ; in: :The total value of the optimization objective function, which reflects the comprehensive index of storage balance and data migration cost. The optimization goal is to make minimize.

[0079] : The priority weight of the k-th task data, the same distribution entropy formula.

[0080] : Priority data distribution of node i, with the same distribution entropy formula.

[0081] : Weight factor, used to balance the weight between storage balance and migration cost.

[0082] : The amount of data migrated from node i to node j, in GB or MB.

[0083] : The migration cost between node i and node j is the same as the data migration cost formula.

[0084] The first term is the distribution entropy, which represents the optimization goal of storage distribution balance; The second item is the migration cost, which represents the cost incurred during the data migration process; It is a balancing factor used to control the weight of balance and migration cost.

[0085] It should be noted that The value of can be adjusted dynamically according to system characteristics. For example, in a high-load task scenario, you can appropriately increase The value of is used to prioritize reducing the migration cost.

[0086] Specifically, the goal of the optimization function is to minimize That is, on the basis of ensuring storage balance, minimize the cost of data migration.

[0087] The solution process of the optimization function: In a possible implementation, the optimization function is solved by the following steps: First, based on the collected storage distribution data and transmission conditions , calculate the distribution entropy H and migration cost .

[0088] Then, combined with the reinforcement learning algorithm, the gradient descent method is used to dynamically adjust the amount of migration data and node distribution , update the result of the optimization function.

[0089] For example, if the distribution entropy of a node is low and the migration cost is high, the node is preferentially migrated by reducing the storage load of the node. To improve the overall optimization effect.

[0090] Extended technical description: As a possible extension, the optimization function can also combine the task importance weight of the node , further adjust the optimization goal: ; in, : The expanded optimization objective function.

[0091] : The task importance weight of node i.

[0092] : The original optimization function objective value.

[0093] : Energy consumption balance factor, used to weigh the impact of the migration process on network energy consumption.

[0094] Network energy consumption: The energy consumption of network resources between nodes during the migration process.

[0095] It is understandable that by introducing task weights and energy consumption factors, the optimization function can more comprehensively adapt to complex task scenarios and further improve the system's performance and resource utilization efficiency. S5. Generate storage adjustment strategy through reinforcement learning algorithm; In this embodiment, the steps of generating a storage adjustment strategy by using a reinforcement learning algorithm include the following: Definition of state space: In one possible implementation, the storage state S of the distributed cluster of drones is defined as the storage utilization of each node , Priority data distribution And network communication conditions . Specifically, the state space can be expressed as: ; in: Indicates the proportion of the k-th type of data at node i; represents the communication conditions between nodes i and j, including transmission delay and bandwidth.

[0096] It should be noted that the definition of the state space can fully characterize the storage resource distribution and communication conditions of the current system, and provide comprehensive input for generating adjustment strategies.

[0097] Definition of action space: The action space A is defined as the storage adjustment operation, including the amount of data migration from node i to node j Specifically: ; The following constraints need to be met: ; in: is the used storage capacity of node i; is the total storage capacity of node j.

[0098] As an option, the action space can be reduced according to the priority weight of the task data, that is, only the migration operation is performed on high-priority data, thereby reducing the computational overhead of the system.

[0099] Construction of reward function: The reward function in this invention Taking into account the storage distribution balance and data migration cost, it is defined as: ; It is the distribution entropy, which quantifies the balance of the current storage distribution; is the migration cost, which represents the communication overhead when executing action a; and are the weight factors of distribution entropy and migration cost respectively.

[0100] In one possible implementation, the weight of the reward function can be adjusted dynamically. For example, when the system communication resources are limited, the weight of the reward function can be increased. weights to prioritize reducing the migration cost.

[0101] Implementation of reinforcement learning algorithm: In this embodiment, the deep Q learning (DQN) algorithm is used to generate the storage adjustment strategy. The core of the DQN algorithm is to use the Q value function Evaluate the value of executing action A under the current state S. The specific formula is: ; in: : Q-value function, used to represent the long-term cumulative value of performing action a in state s.

[0102] : The state of the system at time t.

[0103] : The action of the system at time t.

[0104] : In state Next action Rewards received; is a discount factor used to balance short-term and long-term rewards.

[0105] For example, if the current system status ,action This results in improved storage distribution balance and lower migration costs. The value of is larger.

[0106] Initialization: Randomly initialize the parameters of the Q network and experience pool.

[0107] Experience sampling: randomly extract state transition samples from the experience pool .

[0108] Q value update: ;

[0109] in, : Represents the value evaluation of executing action a in state s.

[0110] : Learning rate, used to control the step size of Q value update.

[0111] : Reward value, which represents the immediate reward obtained by executing action a in state s.

[0112] : Discount factor, used to balance the weights of short-term rewards and long-term rewards, with a value range of [0,1].

[0113] : Indicates the next state The maximum possible Q value reflects the strategy of maximizing future benefits.

[0114] Action selection: According to The maximum value of the optimal action is selected .

[0115] It should be noted that, through the above optimization process, the system can generate the optimal storage adjustment strategy in a dynamic environment.

[0116] Generation of storage adjustment policies: Based on the results of reinforcement learning, the storage adjustment strategy is generated in the form of a migration matrix, where each element Represents the amount of data migration from node i to node j.

[0117] In one possible implementation, the generated migration matrix will preferentially allocate high-priority task data to low-latency nodes, thereby improving the system's responsiveness to critical tasks.

[0118] For example, if the storage load of a node is high and the communication condition is poor, the migration strategy may preferentially transfer some high-priority data to a neighboring node with a lower load.

[0119] Extended technical description: As a possible extended implementation, the reinforcement learning algorithm of the present invention can also be combined with multi-agent reinforcement learning to further improve the overall performance of the system through collaborative decision-making while each node independently performs optimization tasks.

[0120] It is understandable that this extended implementation method can effectively cope with the challenges of fast node status changes and high task complexity in distributed clusters, and provide stronger adaptability for large-scale distributed systems.

[0121] S6. Perform data migration according to the storage adjustment strategy to optimize storage distribution; In this embodiment, the specific steps of performing data migration according to the storage adjustment policy include the following: Determining the amount of data to be migrated: In one possible implementation, the amount of data to be migrated The migration matrix in the storage adjustment strategy is used for determination. Specifically, each element of the migration matrix M represents the amount of data migrated from node i to node j, which must meet the following constraints: ; in: is the amount of data migrated from node i to node j; is the used storage capacity of node i; is the total storage capacity of node j; is the current used storage of node j; n is the total number of nodes.

[0122] As an option, the amount of data to be migrated can be determined by giving priority to the storage requirements of high-priority task data. For example, if the high-priority data of node i occupies a high proportion, the storage pressure can be relieved by migrating the high-priority data first.

[0123] Migration matching of priority data: In one possible implementation, the priority of data migration is assigned by the task priority weight Specifically, data with high weight priority will be migrated to the target node first, while data with low weight priority can be migrated later or stored in place.

[0124] For example, if the priority weight of a task is , and the weight of the other task , the data of high-priority tasks will be preferentially distributed to the target nodes with lower storage load and better communication conditions.

[0125] It should be noted that by matching the migration of priority data, the system can maximize the storage needs of critical tasks within limited storage resources.

[0126] Optimization of data transfer process: During the execution of data migration, transmission efficiency and integrity are the focus of this step.

[0127] As an implementation method, the system uses data compression technology for large data files to reduce transmission overhead. The metadata of the data is recorded during the compression process to facilitate the target node to decompress and verify after receiving it.

[0128] Specifically, the transmission cost It can be expressed as: ; For example, if the bandwidth between node i and node j is 100MBps, the amount of data to be migrated is 50MB, and the network delay is 0.1s, then the transmission cost is .

[0129] As an option, the system also introduces a breakpoint resume mechanism to cope with possible interruptions or failures during the transmission process. The breakpoint resume mechanism records the migration progress and ensures that data continues to be transmitted after the network is restored, thus avoiding data loss caused by transmission failure.

[0130] Status update after migration is complete: In the present invention, after the data migration is completed, the storage status of the source node and the target node needs to be updated. Specifically, the update of the storage status includes: Update the storage utilization of a node : ; in, : Storage utilization of node i.

[0131] : The current amount of used storage of node i.

[0132] : The amount of data migrated from node i to node j.

[0133] : The total storage capacity of node i.

[0134] n: the total number of nodes.

[0135] Update the storage distribution of priority data : , : Indicates the storage ratio of the k-th priority task data stored in node i, and the value range is [0,1].

[0136] Priority k data volume: indicates the amount of data belonging to task priority k stored in node i, in GB or MB.

[0137] Total node storage capacity: equivalent to , that is, the total storage capacity of node i, in GB or MB.

[0138] It should be noted that the accuracy of status updates directly affects the optimization effect of subsequent storage distribution.

[0139] Extended technical description: As a possible extension, the data migration step of the present invention can also be combined with an artificial intelligence-based prediction model to dynamically evaluate the migration cost. For example, by learning from historical transmission conditions, the system can predict the transmission bandwidth and network delay in advance, thereby more accurately adjusting the migration path and target node.

[0140] It is understandable that this expansion implementation method can further improve the efficiency of data migration, while reducing the occupation of network resources, and provide guarantee for the long-term operation of the distributed storage system. S7. Update the storage status and continue to optimize according to the updated status to form a closed-loop process.

[0141] In this embodiment, the specific contents of updating the storage state and forming a closed-loop process include the following steps: Update of storage status: In a possible implementation, the update of the storage status includes dynamic update of storage utilization and priority data distribution.

[0142] First, update the storage utilization of each node , and its calculation formula is: , in: is the current used storage capacity of node i; is the storage capacity of node i.

[0143] As an option, when a node completes data migration, its storage utilization You need to subtract the amount of data that is migrated out, or add the amount of data that is migrated in. For example, if the initial used storage capacity of a node is 40GB and the total storage capacity is 100GB, after migrating 10GB of data, its storage utilization will be updated to .

[0144] Secondly, update the storage distribution of priority data , and its calculation formula is: , It should be noted that the distribution of priority data directly affects the task responsiveness of the system. It will be used as input for subsequent distribution entropy calculation.

[0145] Re-evaluate storage balance: After the storage status update is completed, the present invention quantifies the current storage balance of the system by recalculating the distribution entropy H. The calculation formula of the distribution entropy is: ; in: m represents the total number of priority categories; n represents the total number of storage nodes; is the weight of the task priority; Represents the priority data distribution of node i.

[0146] Specifically, the update of the distribution entropy H is used to determine whether the system needs further optimization. For example, if H is less than the set equilibrium threshold , the system needs to enter the next round of optimization process; otherwise, the existing storage status can be maintained.

[0147] For example, if the storage status of three nodes in the system is updated, the priority data distribution is: Node 1: , ; Node 2: , ; Node 3: , ; Combined priority weights , , calculate the distribution entropy H and compare it with the threshold Compare to decide the next step.

[0148] Design to form a closed-loop process: As a possible implementation, the closed-loop process of the present invention includes the following core steps: Data collection: Based on the updated storage status, re-collect the node storage utilization , Priority data distribution And communication conditions .

[0149] Optimization target reconstruction: Based on the latest storage status, reconstruct the optimization function with storage balance and migration cost as the target: ;

[0150] in, :The total value of the optimization objective function, which reflects the comprehensive index of storage balance and data migration cost. The optimization goal is to make minimize.

[0151] : Weight factor, used to balance the weight between storage balance and migration cost.

[0152] Strategy generation and execution: Combined with reinforcement learning algorithms, a new round of storage adjustment strategies is generated and data migration is performed.

[0153] It should be noted that this closed-loop process design ensures that the system can adaptively adjust the storage distribution in a dynamic environment and effectively respond to task changes and network fluctuations.

[0154] Extended technical description: In a possible extended implementation, the closed-loop process can also introduce a time factor to periodically evaluate storage status and system performance. For example, the system can set an optimization interval T, and trigger a status update and the next round of optimization after each time period. This design is suitable for scenarios where the task load changes less frequently.

[0155] In addition, the prediction model can be combined to predict the storage demand of the next cycle, thereby optimizing the adjustment efficiency of the system. For example, when it is predicted that the storage demand of high-priority tasks may increase, the storage space of the corresponding nodes can be released in advance to improve the task response speed.

[0156] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A data hierarchical storage method for a distributed cluster of drones, characterized in that: The following steps are involved: S1. Initialize system storage parameters, including the capacity of drone storage nodes, used storage, and priority data distribution; S2. Collect real-time data in the drone cluster, including node storage utilization, task data priority, and inter-node communication conditions; S3. Calculate the storage distribution balance of the current system based on the collected data; S4. Construct an optimization function with storage balance and data migration cost as the goal; S5. Generate storage adjustment strategy through reinforcement learning algorithm; S6. Perform data migration according to the storage adjustment strategy to optimize storage distribution; S7. Update the storage status and continue to optimize according to the updated status to form a closed-loop process.

2. The data hierarchical storage method of a distributed cluster of unmanned aerial vehicles according to claim 1 is characterized in that: The step of initializing system storage parameters comprises: S11. Determine the total storage capacity and currently used storage capacity of each storage node; S12. Divide the task data into multiple categories according to priority, and assign a weight to each priority to indicate the degree of influence of different priority data on the system.

3. The data hierarchical storage method of a distributed cluster of drones according to claim 1 is characterized in that: The step of collecting real-time data in the drone cluster includes: S21. Obtain the storage utilization of each storage node; S22. Monitor the distribution ratio of task data of different priorities stored in each node; S23. Monitor the communication conditions between nodes, including communication delay and transmission cost.

4. The data hierarchical storage method of a distributed cluster of unmanned aerial vehicles according to claim 1 is characterized in that: The calculation of the storage distribution balance is obtained by weighting the proportion of data with different priorities in each node, so as to quantify the distribution state of the current storage resources and reflect the degree of balance of the system.

5. The data hierarchical storage method of a distributed cluster of drones according to claim 1 is characterized in that: The optimization function is based on storage distribution balance and takes into account the cost of data migration. The data migration cost is represented by a weighted result of the amount of migrated data and the communication cost between nodes.

6. The data hierarchical storage method of a distributed cluster of drones according to claim 1 is characterized in that: The reinforcement learning algorithm is a deep Q-learning algorithm, which generates an optimal storage adjustment strategy by learning and iteratively optimizing the current storage state, adjustment actions, and reward values ​​of system feedback.

7. The data hierarchical storage method of a distributed cluster of drones according to claim 1 is characterized in that: The reward value is calculated based on the storage distribution balance and migration cost. The positive impact of storage distribution balance on the reward value increases with the improvement of the system's balance level, and the negative impact of migration cost on the reward value increases with the increase of migration cost.

8. The data hierarchical storage method of a distributed cluster of drones according to claim 1 is characterized in that: The data migration includes the following steps: S61. Determine the amount of data to be migrated and its target node; S62. Compression is used for large data files to reduce transmission overhead; S63. Use the breakpoint resume mechanism to ensure the integrity of the data migration process.

9. The data hierarchical storage method of a distributed cluster of drones according to claim 1 is characterized in that: The updating of the storage status includes: S71. Update the storage utilization of the storage node and the distribution status of the task data; S72. Recalculate the storage distribution balance of the system and use the new state as input for the next round of optimization.

10. The data hierarchical storage method of a distributed cluster of drones according to claim 1, characterized in that: The method achieves optimization through a closed-loop process, which includes collecting real-time system status, evaluating storage balance, optimizing storage adjustment strategies, performing data migration, and updating storage status, so as to gradually improve the system's storage resource utilization efficiency and task priority matching capabilities.

Images