Data layout method for cloud edge-end collaborative environment

Abstract

The invention provides a data layout method oriented to a cloud edge end collaborative environment. A data placement strategy for cloud edge end environment characteristics and application requirements is researched and designed. The overall thought of the method is as follows: key elements influencing data distribution are extracted and calculated by analyzing nodes, networks and data features of a cloud edge end environment, and a node-data model is constructed for subsequent analysis and use; establishing a total optimization target with constraint conditions, and providing a mathematical tool for a data layout task; according to a node-data model, rapid initial data layout is realized, a large amount of overhead of optimization calculation is reduced, and the storage pressure of nodes connected with a terminal for collecting data is relieved; and performing data re-layout according to actual use requirements of a user to realize layout optimization.

Description

Technical Field

[0001] The present invention relates to the field of computers, and particularly to a data layout method for a cloud-edge-end collaborative environment. Background Art

[0002] The advancement of computing technology has brought more application possibilities to more physical and virtual industries. Cloud computing integrates computing power and storage capacity in the form of a resource pool, providing on-demand computing and storage resources for many current applications. However, since cloud centers are usually geographically far away from users, long-distance network transmission prevents cloud services from responding to user needs in a timely manner, resulting in access delays. This contradiction has given rise to edge computing. Compared with cloud centers, edge centers are smaller and more flexible, closer to terminals and users, and more responsive. At the same time, with the widespread deployment of various types of terminals (such as cameras, smart sensors, etc.) in scenarios such as smart industry and smart transportation and the popularization of cloud-edge centers, this cloud-edge-end collaborative computing model is becoming a high-quality option for developing high-performance, intelligent applications. At present, cloud computing, edge computing and other models are playing an increasingly important role in physical industries such as industry and transportation, as well as in fields such as mobile Internet and video processing.

[0003] The rapid growth of data in various industries has brought significant challenges to the cloud-edge-end collaborative computing model in terms of data storage. By 2025, the global IoT alone may generate a total of 75ZB of data. The growth of data volume will bring many problems. For example, some users or nodes have a high delay in obtaining specified data in a distributed system, which affects the user's service experience (QoS) and task execution efficiency; frequent data transmission causes network bandwidth pressure; cold data is stored in hot nodes, resulting in resource waste. Furthermore, data layout needs to comprehensively consider multiple factors such as the frequency of data use, initiation location, latency requirements, and network bandwidth, network topology, and storage cost in order to generate a set of layout strategies. Under different requirements, the layout strategy will change, and there is no one-size-fits-all approach. How to optimize the placement of data in the system to better solve the above problems for specific scenarios is a hot topic of continuous research in academia and industry.

[0004] At present, data placement work can be mainly classified into the following types: one is based on optimization theory methods, the second is based on matching, and the third is based on machine learning.

[0005] For optimization theory methods, existing work usually establishes mathematical models based on network conditions, node conditions and access requirements, and then uses heuristic algorithms or optimization algorithms to iteratively solve the optimal solution. These methods include layout methods based on discrete differential evolution particle swarm optimization (Particle Swarm Optimization) algorithm, intelligent swarm optimization methods based on 0-1 integer programming problems, etc.

[0006] For scoring and matching methods, existing work usually establishes placement rules based on the scores of data and nodes, or establishes virtual mapping relationships to determine placement. For example, some work comprehensively considers data popularity, server capacity, data replacement rate, and replacement cost data layout strategies; for example, other work sets up a node multi-attribute matrix and uses the entropy weight method to calculate the node score to complete the layout; and some work sets the coordinates of switches and data in virtual space, and then selects load balancing and short search path placement strategies based on the location of the data.

[0007] For machine learning methods, reinforcement learning is currently the mainstream research direction. Some methods use real-time data streams and estimated value functions of recent delays as input and output of reinforcement models to achieve data placement; or use Q-Learning-based methods and additionally debug key parameters such as learning rate and discount rate to accelerate convergence; in addition, some work uses a combined multi-arm bandit model to set rewards based on the efficiency of data access and use adaptive windows to accelerate the convergence of training.

[0008] Existing work has made a lot of attempts, but it is still focused on cloud or edge clusters, lacking research on collaborative cloud-edge-device environments. In addition, current data placement work rarely considers the situation where data needs to be collected and uploaded. The lack of planning for the initial placement may cause uneven placement or high overhead during the secondary placement. In view of these situations, this paper designs a data placement strategy for collaborative cloud-edge-device computing environments that is different from existing work. Summary of the invention

[0009] To solve the above problems, the present invention proposes a data layout method for cloud-edge-end collaborative environment, and studies and designs data placement strategies for cloud-edge-end environment characteristics and application requirements. The overall idea of ​​the present invention is: by analyzing the node, network, and data characteristics of the cloud-edge-end environment, extracting and calculating the key factors affecting data distribution, and constructing a node-data model for subsequent analysis and use; establishing an overall optimization goal with constraints to provide mathematical tools for data layout tasks; based on the node-data model, achieving rapid initial data layout, reducing the overhead of a large number of optimization calculations, and reducing the storage pressure of nodes connected to the terminals that collect data; based on the actual user needs, re-layout the data to achieve layout optimization.

[0010] Specifically, the present invention consists of the following steps

[0011] Step (1): Create a node description vector.

[0012] Step (2): Create a data description vector.

[0013] Step (3): Establish the overall optimization objective with constraints.

[0014] Step (4): Initial layout configuration of data for computing node-data matching.

[0015] Step (5): Calculate data layout configuration based on actual data usage.

[0016] Each step is further broken down into a series of sub-steps:

[0017] Step (1.1) gives a description of the node i vector. The first three dimensions are easy to obtain, representing the node ID, the node's computing power, and the node's storage capacity. The fourth dimension, connectivity, refers to how many nodes node i is directly connected to. The fifth dimension, centrality, is obtained by summing the connectivity of the directly connected nodes of node i and dividing it by the sum of the connectivity of the entire cluster, which can better describe the relative position of the node in the center. The specific formula can be written as

[0018] n i =[node i , stor i ,comp i ,conn i , cent i , net_cap i ] (1)

[0019] Nodes with the same connectivity may have different performances in centrality. The sixth dimension of network connectivity is calculated by network bandwidth and node centrality. The purpose of the sixth dimension is that a node has higher network connectivity when its total effective bandwidth is high, its connections are relatively balanced, and its bandwidth directly connected to nodes with higher centrality is high. The specific formula can be written as

[0020]

[0021] Step (2.1) gives a description of the data k vector.

[0022] d k =[data k , data_sz k ,pref k ] (3)

[0023] Step (3.1) gives the formula for the transmission cost of data k between node i and node j.

[0024]

[0025] Step (3.2) expresses the formula for the storage cost of all data of node i.

[0026]

[0027] Step (3.3) gives a mathematical description of the overall optimization goal. The goal of data relocation is to use the minimum transmission and storage overhead to ensure that all data accesses are completed within the specified upper limit and that the total amount of data stored in a single node does not exceed the storage capacity.

[0028]

[0029] Step (4.1) analyzes the relationship between data and storage. The storage capacity of the node should cover the size of the data, otherwise the data cannot be completely stored on the node. For node i and data k that may form a storage relationship, the storage capacity of node i should be greater than the space occupied by data k.

[0030] Step (4.2) analyzes the relationship between data, centrality, and network capabilities. For intelligent applications, in order to ensure the consistency of progress during training, large-scale data needs to be managed by nodes with strong computing power and relatively smooth network conditions.

[0031] Step (4.3) considers the overhead of data transmission in the initial distribution. Since data k may have a good match on more than one node, when deciding the node, data k selects the node with the smallest transmission overhead. In addition, considering that too much data stored on a single node may cause the performance of the node to decline, the data will appropriately perform the task of looking for suboptimal solutions around. Specifically, a load factor is added to the node. When the load exceeds a certain value, the matching value of the data with lower matching will decrease, causing it to choose to change the node.

[0032] Step (5.1) The present invention uses a heuristic algorithm called PSO (particle swarm optimization) as the basis of the data re-layout algorithm. The optimization target needs to take into account multi-dimensional issues such as transmission overhead and storage overhead, so the single-dimensional method is no longer applicable. The present invention introduces a multi-objective PSO optimization mechanism and considers using Pareto dominance as a multi-objective processing method.

[0033] Step (5.2) The present invention uses a discrete encoding method to encode particles and their speed. The particles in the present invention are not single data, and their dimensions are upgraded from single data to the entire layout state. When calculating the score of each particle, it is necessary to sum the transmission overhead from the initial position to the current position and the storage overhead of the current position.

[0034] Step (5.3) The present invention further utilizes the leader particle set screening method in CMOPSO, that is, adopts non-dominated sorting and crowding distance-based algorithms. Unlike CMOPSO, we have made some modifications to the competition method and position update calculation, so that non-leader particles can integrate more leader particle information in each round of update. The data layout calculation based on CMOPSO is shown in formula (7), where ω represents the weight of speed, φ and δ are random numbers between (0, 1), and v i，t+1 represents the velocity of particle i at time t+1, x i，t+1 represents the position of particle i at time t+1, p i，1，t and p i，2，t are the two leading particles at time t.

[0035]

[0036] Specifically, in a discrete environment, we transform cosine similarity into a more user-friendly Manhattan distance. Specifically, the ease with which data propagates between nodes is an important basis for measuring the distance between nodes, and continuous cosine similarity cannot reflect this relationship well. In addition, in the process of deciding the next direction of non-leader particles, we use four rather than two leader particles as a basis for comparison. Non-leader particles will calculate their Manhattan distances with the four leader particles respectively, and select the first two as the basis for moving forward to modify their next direction. The advantage of this method is that it prevents non-leader particles from moving too quickly toward a leader particle, and can cover more information.

[0037] The advantages of the present invention are:

[0038] (1) A two-stage data layout method is proposed. Compared with the existing method, when the storage capacity of the node directly connected to the terminal is insufficient, this method can allow the data collected by the terminal to be placed in a relatively reasonable position during the initial layout. At the same time, the fast matching of this method can help reduce the amount of calculation during the re-layout process; compared with the existing method, this method has less computational overhead in the initial layout and re-layout stages.

[0039] (2) A CMOPSO-like re-layout method is adopted. Compared with the typical MOPSO algorithm, this method can avoid using historical best information, thereby reducing additional storage consumption. At the same time, the present invention introduces more dominant particles to form more competitive effects, so that non-leader particles have more references when updating, reducing the probability of falling into the local optimum. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A background diagram of the data layout method proposed in the present invention.

[0041] Figure 2 This is the initial layout algorithm proposed by the present invention.

[0042] Figure 3 This is the re-layout algorithm proposed by the present invention. Specific implementation methods

[0044] Below, the present invention is further explained with reference to the accompanying drawings and tables.

[0045] Table 1 shows some basic symbols used in the formulas and models established in the present invention.

[0046] Table 1

[0047] symbol describe N Node Collection <![CDATA[node i ]]> Node i <![CDATA[stor i ]]> Node i storage capacity <![CDATA[comp i ]]> Node i computing power <![CDATA[conn i ]]> The connectivity of node i <![CDATA[o stor ]]> Storage unit overhead <![CDATA[o trans ]]> Transmission unit overhead <![CDATA[band i，j ]]> Network bandwidth between nodes i and j D Data Collection <![CDATA[data k ]]> Data k <![CDATA[data_sz k ]]> Data k scale <![CDATA[pref k ]]> Data Volunteer <![CDATA[avg i ]]> Average bandwidth of node i

[0048] Figure 1 The background diagram of the present invention is shown. The scenario for the present invention is a cloud-edge-end collaborative computing environment. The end side may collect data or operate the scenario; the edge side accepts the data from the end side and performs fine-tuning or reasoning; the cloud side is responsible for training the original model and accepting the storage of data from the edge side. Since the storage capacity of edge nodes varies and the actual execution tasks are different, the data collected on the end side needs to be allocated to a more appropriate location to speed up the processing speed or reduce storage overhead.

[0049] Figure 2 The algorithm flow of the initial data layout of step (4) of the present invention is shown. The input of the algorithm is the basic information of the nodes and data, and the output is the result of the initial layout. The first step of the algorithm, that is, the first line, is to initialize all nodes and data. The second step of the algorithm, that is, lines 2 to 12, is to calculate the matching degree and transmission cost for all nodes and data, so as to obtain the final matching score. The third step of the algorithm, that is, lines 13 to 23, is to layout the data; among them, lines 14 and 15 are to beat the data and set the storage loading threshold of the node; lines 17 to 21 are to load higher-scoring data into the node, and when the total storage exceeds the threshold, the unloaded data is placed on other nodes. Through the above operations, the initial layout of the data is completed.

[0050] Figure 3 The algorithm flow of data re-layout in step (5) of the present invention is shown. The input of the algorithm is the state of the initial data layout, and the output is the result of data re-layout. The first step of the algorithm, i.e., the first line, is to initialize all particles. The second step of the algorithm, i.e., lines 2 to 12, is to screen the leader node and update the particle position and speed accordingly. Specifically, the second line performs the calculation of the first leader particle, and lines 5 to 8 calculate the update of each particle in a loop. The third step of the algorithm, i.e., line 13, is to select the re-layout result that best meets the requirements from the last leader particle. Through the above operations, the data re-layout is completed.

[0051] A data layout method for a cloud-edge-device collaborative environment includes: extracting and calculating key factors affecting data distribution by analyzing the node, network, and data characteristics of the cloud-edge-device environment, and constructing a node-data model for subsequent analysis; establishing an overall optimization target with constraints to provide mathematical tools for data layout tasks; implementing initial data layout based on the node-data model to reduce the overhead of optimization calculations and alleviate the storage pressure of nodes connected to terminals for data collection; re-layouting data according to data placement policies for cloud-edge-device environment characteristics and application requirements based on actual user needs to achieve layout optimization; specifically including the following steps: step (1): establishing a node description vector; step (2): establishing a data description vector;

[0052] Step (3): Establish an overall optimization goal with constraints; Step (4): Calculate the initial data layout configuration of node-data matching; Step (5): Calculate the data re-layout configuration based on actual data usage.

[0053] The specific process of step (1) includes the following steps: Step (1.1) gives a description of the node i vector, the first dimension in the node i vector represents the location id of the node i, the second dimension represents the computing power of the node i, the third dimension represents the storage capacity of the node i, the fourth dimension represents connectivity, connectivity refers to how many nodes the node i is directly connected to, the fifth dimension represents centrality, the centrality is obtained by summing the connectivity of the directly connected nodes of the node i and dividing it by the sum of the connectivity of the entire cluster, so as to better describe the relative position of the node in the center, the sixth dimension represents the network connection capacity, the network connection capacity is calculated by the network bandwidth and the node centrality;

[0054] The node i vector expression is as follows:

[0055] n i =[node i , stor i ,comp i ,conn i , cent i, net_cap i ],

[0056]

[0057] where n i is the node i vector, node i is the location of node i, storage i is the computing power of node i, comp i is the storage capacity of node i, conn i is connectivity, cent i is the centrality of node i, net_cap i is the network connectivity capability of node i, node j is the location of node j, neigh i is the neighbor of node i, band i，j is the bandwidth between node i and node j, cent j is the centrality of node j, avg i is the average bandwidth of node i.

[0058] The specific process of step (4) is step (4.1), step (4.2) and step (4.3):

[0059] Step (4.1) analyzes the relationship between data and storage. The storage capacity of the node should cover the size of the data, otherwise the data cannot be completely stored on the node. For node i and data k that may form a storage relationship, the storage capacity of node i should be greater than the space occupied by data k. Step (4.2) analyzes the relationship between data and centrality and network capabilities. For intelligent applications, in order to ensure the consistency of progress during training, data with larger sizes need to be managed by nodes with stronger computing power and relatively smooth network conditions. Step (4.3) considers the overhead of data transmission in the initial distribution. For data k, it may be possible to achieve better matching on more than one node. When deciding on a node, data k selects the node with the smallest transmission overhead. In addition, considering that storing too much data on a single node may cause the performance of the node to decline, the data will appropriately perform the task of looking for suboptimal solutions in the surrounding area. Specifically, a load factor is added to the node. When the load exceeds a certain value, the matching value of data with lower matching will decrease, causing it to choose to change the node.

[0060] The specific process of step (5) is step (5.1), step (5.2) and step (5.3):

[0061] Step (5.1) uses the PSO particle swarm optimization heuristic algorithm as the basis of the data re-layout algorithm. The optimization target needs to take into account the multi-dimensional problems of transmission overhead and storage overhead. A multi-objective PSO optimization mechanism is introduced, and Pareto dominance is considered as a multi-objective processing method; Step (5.2) uses a discrete encoding method to encode particles and their speeds. Particles are not single data, but are upgraded from single data to the entire layout state; when calculating the score of each particle, the transmission overhead from the initial position to the current position and the storage overhead of the current position are summed up separately; Step (5.3) uses the leader particle set screening method in CMOPSO, that is, non-dominated sorting and crowding distance-based algorithms are used to modify the competition method and position update calculation, so that non-leader particles can integrate more leader particle information in each round of updates;

[0062]

[0063] where v i，t+1 represents the velocity of particle i at time t+1, i is the time, ω represents the weight of the velocity, and v i，t is the velocity of particle i at time t, φ and δ are both random numbers between (0, 1), p i，1，t and p i，2，t are the two leading particles at time t, x i，t is the position of particle i at time t, x i，t+1 is the position of particle i at time t+1.

[0064] The present invention proposes a data layout method for a cloud-edge-device collaborative environment, and studies and designs a data placement strategy for cloud-edge-device environment characteristics and application requirements. The overall idea of ​​the present invention is: by analyzing the node, network, and data characteristics of the cloud-edge-device environment, extracting and calculating the key factors affecting data distribution, and constructing a node-data model for subsequent analysis and use; establishing an overall optimization goal with constraints to provide mathematical tools for data layout tasks; based on the node-data model, achieving rapid initial data layout, reducing the overhead of a large number of optimization calculations, and alleviating the storage pressure of nodes connected to the terminal for data collection; and re-layouting data based on the actual needs of users to achieve layout optimization.

Claims

1. A data layout method for a cloud-edge-device collaborative environment, characterized in that: include: By analyzing the node, network, and data characteristics of the cloud-edge environment, the key factors affecting data distribution are extracted and calculated, and a node-data model is constructed for subsequent analysis and use; an overall optimization goal with constraints is established to provide mathematical tools for data layout tasks; based on the node-data model, the initial data layout is achieved, the optimization calculation overhead is reduced, and the storage pressure of the nodes connected to the terminals that collect data is reduced; based on the actual user needs, data is re-layouted according to the data placement policy for the cloud-edge environment characteristics and application needs to achieve layout optimization; The specific steps include: Step (1): Establish a node description vector; Step (2): Establish a data description vector; Step (3): Establish the overall optimization objective with constraints; Step (4): initial layout configuration of data for computing node-data matching; Step (5): Calculate data layout configuration based on actual data usage.

2. The data layout method for cloud-edge-device collaborative environment according to claim 1 is characterized in that: The specific process of step (1) includes the following steps: Step (1.1) gives a description of the node i vector. The first dimension in the node i vector represents the location id of the node i. The second dimension represents the computing power of the node i. The third dimension represents the storage capacity of the node i. The fourth dimension represents connectivity. Connectivity refers to how many nodes the node i is directly connected to. The fifth dimension represents centrality. Centrality is obtained by summing the connectivity of the directly connected nodes of the node i and dividing it by the sum of the connectivity of the entire cluster to better describe the relative position of the node in the center. The sixth dimension represents network connectivity. The network connectivity is calculated by the network bandwidth and the node centrality. The node i vector expression is as follows: n i =[node i ,stor i ,comp i ,conn i ,cent i ,net_cap i ], where n i is the node i vector, node i is the location of node i, storage i is the computing power of node i, comp i is the storage capacity of node i, coun i is connectivity, cent i is the centrality of node i, net_cap i is the network connectivity capability of node i, node j is the location of node j, neigh i is the neighbor of node i, band i，j is the bandwidth between node i and node j, cent j is the centrality of node j, avg i is the average bandwidth of node i.

3. The data layout method for cloud-edge-device collaborative environment according to claim 1 is characterized in that: The specific process of step (4) is step (4.1), step (4.2) and step (4.3): Step (4.1) Analyze the relationship between data and storage. The storage capacity of the node should cover the size of the data, otherwise the data cannot be completely stored on the node. For node i and data k that may form a storage relationship, the storage capacity of node i should be greater than the space occupied by data k. Step (4.2) analyzes the relationship between data, centrality, and network capabilities. For intelligent applications, in order to ensure the consistency of progress during training, large-sized data needs to be managed by nodes with strong computing power and relatively smooth network conditions; Step (4.3) considers the overhead of data transmission in the initial distribution. Since data k may have a better match on more than one node, data k will select the node with the smallest transmission overhead when deciding the node. In addition, considering that storing too much data on a single node may cause the performance of the node to decline, the data will appropriately perform the task of looking for suboptimal solutions in the surrounding area. Specifically, a load factor is added to the node. When the load exceeds a certain value, the matching value of the data with lower matching will decrease, causing it to choose to change the node.

4. The data layout method for a cloud-edge-device collaborative environment according to claim 1 is characterized in that: The specific process of step (5) is step (5.1), step (5.2) and step (5.3): Step (5.1) PSO particle swarm optimization is used as the basis of the data re-layout algorithm. The optimization goal needs to take into account the multi-dimensional problems of transmission overhead and storage overhead. The multi-objective PSO optimization mechanism is introduced, and Pareto dominance is considered as a multi-objective processing method. Step (5.2) uses a discrete encoding method to encode particles and their speeds. Particles are not single data, but are upgraded from single data to the entire layout state. When calculating the score of each particle, the transmission cost from the initial position to the current position and the storage cost of the current position are summed up separately. Step (5.3) uses the leader particle set screening method in CMOPSO, that is, adopts the non-dominated sorting and crowding distance-based algorithms, and modifies the competition method and position update calculation, so that non-leader particles can integrate more leader particle information in each round of update; where v i,t+1 represents the velocity of particle i at time t+1, t is the time, ω represents the weight of the velocity, v i,t is the velocity of particle i at time t, φ and δ are both random numbers between (0,1), p i,1,t and p i,2,t are the two leading particles at time t, x i,t is the position of particle i at time t, x i,t+1 is the position of particle i at time t+1.

Images