Data layout method for cloud-edge-end collaborative environment

Abstract

This paper proposes a data layout method for a cloud-edge-device collaborative environment, and studies and designs a data placement strategy tailored to the characteristics of the cloud-edge-device environment and application requirements. The overall approach of the present invention is to analyze the node, network, and data characteristics of the cloud-edge-device environment, extract and calculate the key factors affecting data distribution, and construct a node-data model for subsequent analysis and use; establish an overall optimization objective with constraints to provide mathematical tools for data layout tasks; based on the node-data model, achieve rapid initial data layout, reduce the overhead of large-scale optimization calculations, and alleviate the storage pressure on nodes connected to the terminals collecting data; and perform data rearrangement and layout optimization based on the actual user needs.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a data layout method for a cloud-edge-end collaborative environment. Background Art

[0002] Advances in computing technology have opened up new application possibilities for a wider range of physical and virtual industries. Cloud computing integrates computing and storage capabilities in the form of resource pools, providing on-demand computing and storage resources for many current applications. However, because cloud centers are often geographically distant from users, long-distance network transmission prevents cloud services from responding to user needs in a timely manner, resulting in access delays. This conflict has given rise to edge computing. Compared to cloud centers, edge centers are more compact and flexible, closer to devices and users, and more responsive. Furthermore, with the widespread deployment of various types of devices (such as cameras and smart sensors) in scenarios such as smart industry and smart transportation, and the proliferation of cloud-edge centers, this cloud-edge-device collaborative computing model is becoming a promising option for developing high-performance, intelligent applications. Currently, cloud computing and edge computing are playing an increasingly important role in physical industries such as industry and transportation, as well as in fields such as mobile internet and video processing.

[0003] The rapid growth of data volumes across various industries poses significant data storage challenges for cloud-edge-device collaborative computing models. By 2025, the global IoT alone is expected to generate 75 zettabytes of data. This data growth presents numerous challenges. For example, some users or nodes experience high latency in accessing specific data within distributed systems, impacting user service experience (QoS) and task execution efficiency. Frequent data transmission creates network bandwidth pressure, and cold data is stored on hot nodes, resulting in resource waste. Furthermore, data placement requires comprehensive consideration of multiple factors, including user frequency of data use, initiation location, latency requirements, network bandwidth, network topology, and storage costs, before generating a placement strategy. Different placement strategies will vary depending on different requirements, and there is no one-size-fits-all approach. Optimizing data placement within the system to address these challenges for specific scenarios is a hot topic of ongoing research in academia and industry.

[0004] At present, data placement work can be mainly divided into the following types: one is based on optimization theory methods, the second is based on matching, and the third is based on machine learning.

[0005] Existing work on optimization theory typically builds mathematical models based on network conditions, node conditions, and access requirements; then uses heuristic or optimization algorithms to iteratively find the optimal solution. These methods include layout methods based on discrete differential evolution particle swarm optimization (PSO) and intelligent swarm optimization methods based on 0-1 integer programming problems.

[0006] For scoring and matching methods, existing work typically establishes placement rules based on data and node scores, or establishes virtual mapping relationships to determine placement. For example, some work considers data popularity, server capacity, data replacement rate, and replacement cost in data layout strategies. Other work sets up a node multi-attribute matrix and uses entropy weighting to calculate node scores to complete layout. Still other work sets the coordinates of switches and data in virtual space and then selects placement strategies that prioritize load balancing and short search paths based on the data's location.

[0007] Reinforcement learning is currently a mainstream research area in machine learning. Some approaches use real-time data streams and estimated value functions of recent delays as the input and output of reinforcement models to achieve data placement. Others employ Q-Learning-based approaches, but further tweak key parameters such as the learning rate and discount rate to accelerate convergence. Furthermore, some work utilizes a combined multi-arm bandit model, setting rewards based on data access efficiency and employing adaptive windows to accelerate training convergence.

[0008] Existing research has explored numerous possibilities, but remains focused on cloud or edge clusters, lacking research on collaborative cloud-edge-device environments. Furthermore, current data placement efforts rarely consider the need for data collection and upload. Unplanned initial placement can lead to uneven placement or high secondary placement overhead. In light of these challenges, this paper designs a data placement strategy for collaborative cloud-edge-device computing environments that differs from existing work. Summary of the Invention

[0009] To solve the above problems, the present invention proposes a data layout method for a cloud-edge-end collaborative environment, and studies and designs a data placement strategy based on the characteristics of the cloud-edge-end environment and application requirements. The overall idea of ​​the present invention is: by analyzing the node, network, and data characteristics of the cloud-edge-end environment, extracting and calculating the key factors affecting data distribution, and constructing a node-data model for subsequent analysis and use; establishing an overall optimization goal with constraints to provide mathematical tools for data layout tasks; based on the node-data model, achieving rapid initial data layout, reducing the overhead of a large number of optimization calculations, and alleviating the storage pressure of nodes connected to the terminals that collect data; and re-layouting data according to the actual needs of users to achieve layout optimization.

[0010] Specifically, the present invention consists of the following steps

[0011] Step (1): Create a node description vector.

[0012] Step (2): Create a data description vector.

[0013] Step (3): Establish the overall optimization objective with constraints.

[0014] Step (4): Calculate the initial layout configuration of the data for node-data matching.

[0015] Step (5): Calculate data layout configuration based on actual data usage.

[0016] Each step is further broken down into a series of sub-steps:

[0017] Step (1.1) describes the vector for node i. The first three dimensions are readily available and represent the node's ID, computing power, and storage capacity, respectively. The fourth dimension, connectivity, refers to the number of nodes directly connected to node i. The fifth dimension, centrality, is the sum of the connectivity of node i's directly connected nodes divided by the sum of the connectivity of the entire cluster. This dimension better describes the node's relative position within the hub. The specific formula can be written as

[0018] n i =[node i , stor i ,comp i ,conn i , cent i , net_cap i ] (1)

[0019] Nodes with the same connectivity can exhibit different centrality. The sixth dimension, network connectivity, is calculated by combining network bandwidth and node centrality. The sixth dimension states that a node has higher network connectivity when it has high total effective bandwidth, relatively balanced connections, and high bandwidth directly connected to nodes with higher centrality. The specific formula can be written as:

[0020]

[0021] Step (2.1) gives a description of the data k vector.

[0022] d k =[data k , data_sz k , pref k ] (3)

[0023] Step (3.1) gives the formula for the transmission cost of data k between node i and node j.

[0024]

[0025] Step (3.2) expresses the formula for the storage cost of all data of node i.

[0026]

[0027] Step (3.3) gives a mathematical description of the overall optimization objective. The goal of data relocation is to minimize transmission and storage overhead, ensure that all data accesses are completed within the specified upper limit, and achieve the requirement that the total amount of data stored in a single node does not exceed the storage capacity.

[0028]

[0029] Step (4.1) analyzes the relationship between data and storage. A node's storage capacity should cover the size of the data; otherwise, the data cannot be fully stored on the node. For node i and data k that may form a storage relationship, the storage capacity of node i should be greater than the space occupied by data k.

[0030] Step (4.2) analyzes the relationship between data, centrality, and network capabilities. For intelligent applications, to ensure consistent progress during training, large data sets need to be managed by nodes with strong computing power and relatively smooth network conditions.

[0031] Step (4.3) considers the overhead of data transmission in the initial distribution. Given that data k may be well matched on more than one node, data k chooses the node with the lowest transmission overhead when deciding on a node. Furthermore, considering that storing too much data on a single node may degrade its performance, the data will appropriately search for suboptimal solutions in the surrounding area. Specifically, a load factor is added to the node. When the load exceeds a certain value, the matching value of less compatible data will decrease, causing it to choose a different node.

[0032] In step (5.1), the present invention uses a heuristic algorithm, Particle Swarm Optimization (PSO), as the basis for the data relocation algorithm. The optimization objective must account for multiple dimensions, such as transmission overhead and storage overhead. Therefore, single-dimensional methods are not applicable. This invention introduces a multi-objective PSO optimization mechanism and considers Pareto dominance as a multi-objective approach.

[0033] In step (5.2), the present invention uses a discrete encoding method to encode particles and their velocities. The particles in the present invention are not individual data; their dimensions have been upgraded from individual data to the entire layout state. When calculating each particle's score, it is necessary to sum the transmission cost from the initial position to the current position and the storage cost of the current position.

[0034] Step (5.3) The present invention further utilizes the leader particle set screening method in CMOPSO, that is, adopts non-dominated sorting and crowding distance-based algorithms. Unlike CMOPSO, we have made certain modifications to the competition method and position update calculation, so that non-leader particles can integrate more leader particle information in each round of update. The data layout calculation based on the CMOPSO-like method is shown in formula (7), where ω represents the speed weight, φ and δ are random numbers between (0, 1), and v i，t+1 represents the velocity of particle i at time t+1, x i，t+1 represents the position of particle i at time t+1, p i，1，t and p i，2，t are the two leading particles at time t.

[0035]

[0036] Specifically, in discrete environments, we transform cosine similarity into the more user-friendly Manhattan distance. Specifically, the ease with which data propagates between nodes is an important metric for measuring inter-node distances, and continuous cosine similarity does not adequately reflect this relationship. Furthermore, when deciding the next direction for non-leader particles, we use four leader particles, rather than two, as a basis for comparison. Non-leader particles calculate their Manhattan distances with each of the four leader particles and use the first two as a basis for modifying their next direction. This approach prevents non-leader particles from moving too quickly toward a leader particle, allowing them to capture more information.

[0037] The advantages of the present invention are:

[0038] (1) A two-stage data layout method is proposed. Compared with existing methods, when the storage capacity of the node directly connected to the terminal is insufficient, this method can allow the data collected by the terminal to be placed in a relatively reasonable location during the initial layout. At the same time, the fast matching of this method can help reduce the computational complexity during the re-layout process; compared with existing methods, this method has lower computational overhead in the initial layout and re-layout stages.

[0039] (2) A CMOPSO-like relocation method is used. Compared to the typical MOPSO algorithm, this method avoids using historical best information, thereby reducing additional storage consumption. At the same time, the present invention introduces more dominant particles to form more competitive effects, so that non-leading particles have more references when updating, reducing the probability of falling into local optimality. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a background diagram of the data layout method proposed in the present invention.

[0041] Figure 2 This is the initial layout algorithm proposed by the present invention.

[0042] Figure 3 This is the re-layout algorithm proposed by the present invention. Specific implementation methods

[0043] Below, the present invention is further explained with reference to the accompanying drawings and tables.

[0044] Table 1 shows some basic symbols used in the formulas and models established in the present invention.

[0045] Table 1

[0046] symbol describe N Node Collection <![CDATA[node i ]]> Node i <![CDATA[stor i ]]> Node i storage capacity <![CDATA[comp i ]]> Node i computing power <![CDATA[conn i ]]> Connectivity of node i <![CDATA[o stor ]]> Storage unit overhead <![CDATA[o trans ]]> Transmission unit overhead <![CDATA[band i，j ]]> Network bandwidth between nodes i and j D Data Collection <![CDATA[data k ]]> Data k <![CDATA[data_sz k ]]> Data k scale <![CDATA[pref k ]]> Data Volunteer <![CDATA[avg i ]]> Average bandwidth of node i

[0047] Figure 1 The background diagram of the present invention is shown. The scenario targeted by the present invention is a cloud-edge-end collaborative computing environment. The end side may collect data or operate on the scenario; the edge side receives the data from the end side and performs fine-tuning or inference; the cloud side is responsible for training the original model and accepting the storage of data from the edge side. Because the storage capacity of edge side nodes varies and the actual execution tasks are different, it is necessary to distribute the data collected by the end side to a more appropriate location to speed up processing or reduce storage overhead.

[0048] Figure 2 The algorithm flow of the initial data layout in step (4) of the present invention is shown. The input of the algorithm is the basic information of the nodes and data, and the output is the result of the initial layout. The first step of the algorithm, that is, the first line, is to initialize all nodes and data. The second step of the algorithm, that is, lines 2 to 12, is to calculate the matching degree and transmission cost for all nodes and data, so as to obtain the final matching score. The third step of the algorithm, that is, lines 13 to 23, is to layout the data; among them, lines 14 and 15 are to perform data scrambling and set the storage loading threshold of the node; lines 17 to 21 are to load higher-scoring data into the node, and when the total storage exceeds the threshold, the unloaded data is placed on other nodes. Through the above operations, the initial layout of the data is completed.

[0049] Figure 3 The algorithm flow of data rearrangement in step (5) of the present invention is shown. The input of the algorithm is the state of the initial data layout, and the output is the result of the data rearrangement. The first step of the algorithm, that is, the first line, is to initialize all particles. The second step of the algorithm, that is, lines 2 to 12, is to screen the leader node and update the particle position and speed accordingly. Specifically, the second line performs the calculation of the first leader particle, and lines 5 to 8 are cyclic calculations for the updates of each particle. The third step of the algorithm, that is, line 13, is to select the rearrangement result that best meets the requirements from the last leader particle. Through the above operations, the data rearrangement is completed.

[0050] The data layout method for cloud-edge-end collaborative environment includes: extracting and calculating key factors affecting data distribution by analyzing the node, network and data characteristics of the cloud-edge-end environment, and constructing a node-data model for subsequent analysis and use; establishing an overall optimization target with constraints to provide mathematical tools for data layout tasks; realizing initial data layout based on the node-data model, reducing the overhead of optimization calculation, and alleviating the storage pressure of nodes connected to the terminals for data collection; and re-layouting data according to the data placement strategy for the cloud-edge-end environment characteristics and application requirements based on the actual use needs of users to achieve layout optimization; specifically comprising the following steps: step (1): establishing a node description vector; step (2): establishing a data description vector;

[0051] Step (3): Establish the overall optimization goal with constraints; Step (4): Calculate the initial data layout configuration of node-data matching; Step (5): Calculate the data layout configuration based on actual data usage.

[0052] The specific process of step (1) includes the following steps: Step (1.1) gives a description of the node i vector, the first dimension in the node i vector represents the location ID of the node i, the second dimension represents the computing power of the node i, the third dimension represents the storage capacity of the node i, the fourth dimension represents connectivity, which refers to how many nodes the node i is directly connected to, the fifth dimension represents centrality, which is obtained by summing the connectivity of the directly connected nodes of the node i and dividing it by the sum of the connectivity of the entire cluster to better describe the relative position of the node in the center, and the sixth dimension represents network connectivity, which is calculated by combining network bandwidth and node centrality;

[0053] The vector expression of node i is as follows:

[0054] n i =[node i , stor i ,comp i ,conn i , cent i, net_cap i ],

[0055]

[0056] where n i is the node i vector, node i is the position of node i, stor i is the computing power of node i, comp i is the storage capacity of node i, conn i is connectivity, cent i is the centrality of node i, net_cap i is the network connectivity capability of node i, node j is the location of node j, neigh i is the neighbor of node i, band i，j is the bandwidth between node i and node j, cent j is the centrality of node j, avg i is the average bandwidth of node i.

[0057] The specific process of step (4) is step (4.1), step (4.2) and step (4.3):

[0058] Step (4.1) analyzes the relationship between data and storage. The storage capacity of the node should cover the size of the data, otherwise the data cannot be fully stored on the node. For node i and data k that may form a storage relationship, the storage capacity of node i should be greater than the space occupied by data k. Step (4.2) analyzes the relationship between data and centrality and network capabilities. For intelligent applications, in order to ensure the consistency of progress during training, larger data needs to be managed by nodes with stronger computing power and relatively smooth network conditions. Step (4.3) considers the overhead of data transmission in the initial distribution. Since data k may achieve better matching on more than one node, data k chooses the node with the smallest transmission overhead when deciding the node. In addition, considering that storing too much data on a single node may cause the performance of the node to decline, the data will appropriately perform the task of searching for suboptimal solutions in the surrounding area. Specifically, a load factor is added to the node. When the load exceeds a certain value, the matching value of data with lower matching will decrease, causing it to choose to change the node.

[0059] The specific process of step (5) is step (5.1), step (5.2) and step (5.3):

[0060] Step (5.1) uses the PSO particle swarm optimization heuristic algorithm as the basis of the data re-layout algorithm. The optimization goal needs to take into account the multi-dimensional problems of transmission overhead and storage overhead. The multi-objective PSO optimization mechanism is introduced, and Pareto dominance is considered as a multi-objective processing method. Step (5.2) uses a discrete encoding method to encode particles and their velocities. Particles are not single data, but are upgraded from single data to the entire layout state. When calculating the score of each particle, the transmission overhead from the initial position to the current position and the storage overhead of the current position are summed separately. Step (5.3) uses the leader particle set screening method in CMOPSO, that is, adopts non-dominated sorting and crowding distance-based algorithms, and modifies the competition method and position update calculation, so that non-leader particles can integrate more leader particle information in each round of update.

[0061]

[0062] where v i，t+1 represents the velocity of particle i at time t+1, i is the time, ω represents the weight of the velocity, v i，t is the velocity of particle i at time t, φ and δ are both random numbers between (0, 1), p i，1，t and p i，2，t are the two leading particles at time t, x i，t is the position of particle i at time t, x i，t+1 is the position of particle i at time t+1.

[0063] The present invention proposes a data layout method for a cloud-edge-end collaborative environment, and studies and designs a data placement strategy based on the characteristics of the cloud-edge-end environment and application requirements. The overall idea of ​​the present invention is: by analyzing the node, network, and data characteristics of the cloud-edge-end environment, extracting and calculating the key factors affecting data distribution, and constructing a node-data model for subsequent analysis and use; establishing an overall optimization goal with constraints to provide mathematical tools for data layout tasks; based on the node-data model, achieving rapid initial data layout, reducing the overhead of a large number of optimization calculations, and alleviating the storage pressure of nodes connected to the terminals that collect data; and re-layouting data according to the actual needs of users to achieve layout optimization.

Claims

1. A data layout method for a cloud-edge-device collaborative environment, characterized by: The following steps are involved: Step (1): Create a node description vector; Step (2): Create a data description vector; Step (3): Establishing a constrained optimization goal, wherein the goal is to minimize transmission and storage overhead so that all data accesses are completed within a specified upper limit and do not exceed the storage capacity requirement; Step (4): Initial data layout configuration for computing node-data matching; Step (5): Calculate data layout configuration based on actual data usage; Step (4) includes step (4.1), step (4.2) and step (4.3): Step (4.1) analyzes the relationship between data and storage. For node i and data k that may form a storage relationship, the storage capacity of node i should be greater than the space occupied by data k. Step (4.2) analyzes the relationship between data, centrality, and network capabilities. For intelligent applications, to ensure consistency in training progress, large data needs to be managed by nodes with strong computing power and relatively smooth network conditions. Step (4.3) considers the overhead of data transmission in the initial distribution. Since data k may have a good match on more than one node, when deciding on a node, data k chooses the node with the lowest transmission overhead. A load factor is added to the node. When the load exceeds a certain value, the matching value of the less compatible data will decrease, causing it to choose a different node. The data performs the task of searching for a suboptimal solution in the surrounding area. Step (5) includes step (5.1), step (5.2) and step (5.3): Step (5.1) uses the PSO particle swarm optimization heuristic algorithm as the basis for the data re-layout algorithm. The optimization goal needs to take into account the multi-dimensional issues of transmission overhead and storage overhead. A multi-objective PSO optimization mechanism is introduced, and Pareto dominance is considered as a multi-objective processing method. Step (5.2) uses a discrete encoding method to encode particles and their velocities. Particles are not individual data, but are upgraded from individual data to the entire layout state. When calculating the score of each particle, the transmission cost from the initial position to the current position and the storage cost of the current position are summed separately. Step (5.3) uses the leader particle set screening method in CMOPSO, that is, adopts the non-dominated sorting and crowding distance-based algorithms, and modifies the competition method and position update calculation so that non-leader particles can integrate more leader particle information in each round of update; where v i,t+1 represents the velocity of particle i at time t+1, t is the time, ω represents the weight of the velocity, v i,t is the velocity of particle i at time t, φ and δ are both random numbers between (0,1), p i,1,t and p i,2,t are the two leading particles at time t, x i,t is the position of particle i at time t, x i,t+1 is the position of particle i at time t+1.

2. The data layout method for cloud-edge-device collaborative environment according to claim 1 is characterized in that: The specific process of step (1) includes the following steps: Step (1.1) describes the vector of node i. The first dimension of the vector represents the location ID of node i, the second dimension represents the computing power of node i, the third dimension represents the storage capacity of node i, the fourth dimension represents connectivity, which refers to the number of nodes directly connected to node i, and the fifth dimension represents centrality. Centrality is obtained by summing the connectivity of node i's directly connected nodes and dividing it by the sum of the connectivity of the entire cluster to better describe the relative position of the node within the center. The sixth dimension represents network connectivity, which is calculated by combining network bandwidth and node centrality. The vector expression of node i is as follows: n i =[node i ,stor i ,comp i ,conn i ,cent i ,net_cap i ], where n i is the node i vector, node i is the position of node i, stor i is the storage capacity of node i, comp i is the computing power of node i, conn i is connectivity, cent i is the centrality of node i, net_cap i is the network connectivity capability of node i, node j is the location of node j, neigh i is the nearest neighbor of node i, band i,j is the bandwidth between node i and node j, cent j is the centrality of node j, avg i is the average bandwidth of node i.

Images