A method for workflow data layout and storage medium in a cloud-edge environment

Abstract

This invention discloses a method and storage medium for workflow data layout in a cloud-edge environment. The cloud-edge environment is mathematically represented, and based on replica generation and data transmission overhead, the data layout problem is modeled as a 0-1 integer programming problem with the goal of minimizing total latency, resulting in a mathematical problem model. A nonlinear inertial weighted discrete particle swarm optimization algorithm based on genetic algorithm operators is employed. The crossover and mutation operators of the genetic algorithm are introduced into the particle swarm algorithm, and the inertial weights are adaptively adjusted according to the differences between the current particle and the global particle to solve the mathematical problem model. Workflow data layout is performed based on the solution results. This effectively reduces latency. Furthermore, the introduction of crossover and mutation operators of the genetic algorithm into the particle swarm algorithm enhances the search capability of the particle swarm algorithm, avoids premature convergence, and the adaptive adjustment of the inertial weights according to the differences between the current particle and the global particle makes the optimization process more efficient.

Description

Technical Field

[0001] This invention relates to the field of workflow data layout technology, and in particular to a method and storage medium for workflow data layout in a cloud-edge environment. Background Technology

[0002] Workflow models are an effective way to describe business processes, consisting of many interrelated tasks. Workflows are widely used in scientific fields such as astronomy, physics, and bioinformatics. As data-intensive applications, the deployment of scientific workflows places stringent requirements on the computing power and storage capacity of the environment.

[0003] Cloud computing offers powerful storage and computing capabilities, providing personalized services and ensuring resource availability for scientific workflows. However, the operation of scientific workflows involves large-scale data transmission, and using cloud computing deployed remotely can lead to significant data transmission latency. Edge computing moves computation closer to the user's location at the network edge, reducing data transmission latency while storing users' private data. However, edge computing resources are limited and cannot store all the data needed and generated during the execution of scientific workflows. Combining cloud computing and edge computing can provide a secure and efficient way to deploy scientific workflows.

[0004] Due to the presence of privacy-sensitive data, scientific workflows involve significant data transfers, causing substantial latency. As storage costs decrease, data replication is increasingly used in cloud and edge computing, reducing data transfer frequency by accessing replicas more conveniently. However, deploying data replicas in cloud-edge environments presents several challenges. Specifically, replica generation, transmission, and storage incur overhead, requiring the selection of appropriate data to generate a suitable number of replicas, and choosing the optimal location for replica placement is also a challenge.

[0005] Therefore, it is particularly important to know how to layout data replicas to reduce latency. Summary of the Invention

[0006] The technical problem to be solved by this invention is to provide a method and storage medium for workflow data layout in a cloud-edge environment, which can effectively reduce latency.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0008] A method for workflow data layout in a cloud-edge environment includes the following steps:

[0009] S1. The cloud-edge environment is mathematically represented. Based on the replica generation overhead and data transmission overhead, the data layout problem is modeled as a 0-1 integer programming problem with the goal of minimizing the total latency, and the mathematical problem model is obtained.

[0010] S2. A nonlinear inertia-weighted discrete particle swarm optimization algorithm based on genetic algorithm operators is adopted. The crossover and mutation operators of the genetic algorithm are introduced into the particle swarm algorithm, and the inertia weight is adaptively adjusted according to the difference between the particles and the global particles to solve the mathematical problem model.

[0011] S3. Lay out the workflow data based on the solution results.

[0012] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is as follows:

[0013] A storage medium for workflow data layout in a cloud-edge environment, wherein a computer program is stored thereon, and when the computer program is executed, it implements the steps in the above-mentioned method for workflow data layout in a cloud-edge environment.

[0014] The beneficial effects of this invention are as follows: The method and storage medium for workflow data layout in a cloud-edge environment of this invention aim to minimize the total latency. It models the data replica layout as a 0-1 integer programming problem and adopts a nonlinear inertial weight discrete particle swarm optimization algorithm based on genetic algorithm operators to solve the data layout problem, effectively reducing latency. Furthermore, the crossover and mutation operators of the genetic algorithm are introduced into the particle swarm algorithm to enhance the search capability of the particle swarm algorithm, avoid premature convergence, and adaptively adjust the inertial weight according to the difference between the current particle and the global particle, making the optimization process more efficient. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of a scientific workflow example of a workflow data layout method in a cloud-edge environment according to an embodiment of the present invention;

[0016] Figure 2 This is a schematic diagram illustrating a scientific workflow data layout example of a workflow data layout method in a cloud-edge environment according to an embodiment of the present invention.

[0017] Figure 3 This is a schematic diagram illustrating a one-dimensional encoding example of a workflow data layout method in a cloud-edge environment according to an embodiment of the present invention.

[0018] Figure 4 This is a schematic diagram illustrating a two-dimensional encoding example of a workflow data layout method in a cloud-edge environment according to an embodiment of the present invention.

[0019] Figure 5 This is a schematic diagram of a mutation operator example for a workflow data layout method in a cloud-edge environment according to an embodiment of the present invention;

[0020] Figure 6 This is a schematic diagram of a cross operator example of a workflow data layout method in a cloud-edge environment according to an embodiment of the present invention;

[0021] Figure 7 This is a flowchart illustrating a method for workflow data layout in a cloud-edge environment, according to an embodiment of the present invention. Detailed Implementation

[0022] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0023] Please refer to Figure 1 , Figure 2 as well as Figures 4 to 7 A method for workflow data layout in a cloud-edge environment, comprising the following steps:

[0024] S1. The cloud-edge environment is mathematically represented. Based on the replica generation overhead and data transmission overhead, the data layout problem is modeled as a 0-1 integer programming problem with the goal of minimizing the total latency, and the mathematical problem model is obtained.

[0025] S2. A nonlinear inertia-weighted discrete particle swarm optimization algorithm based on genetic algorithm operators is adopted. The crossover and mutation operators of the genetic algorithm are introduced into the particle swarm algorithm, and the inertia weight is adaptively adjusted according to the difference between the particles and the global particles to solve the mathematical problem model.

[0026] S3. Lay out the workflow data based on the solution results.

[0027] As can be seen from the above description, the beneficial effects of the present invention are as follows: The present invention provides a method and storage medium for workflow data layout in a cloud-edge environment, aiming to minimize total latency. It models the data replica layout as a 0-1 integer programming problem and employs a nonlinear inertial weight discrete particle swarm optimization algorithm based on genetic algorithm operators to solve the data layout problem, effectively reducing latency. Furthermore, the introduction of crossover and mutation operators from the genetic algorithm into the particle swarm algorithm enhances the search capability of the particle swarm algorithm, avoids premature convergence, and adaptively adjusts the inertial weights according to the difference between the current particle and the global particle, making the optimization process more efficient.

[0028] Furthermore, the mathematical representation of the cloud edge environment described in step S1 specifically involves:

[0029] The cloud edge environment is represented as:

[0030] S={S cld ,S edg};

[0031] Among them, cloud computing S cld It contains j data centers, represented as:

[0032] S cld ={s1,s2,…,sj};

[0033] Edge computing S edg It contains k data centers, represented as:

[0034] S edg ={s j+1 ,s j+2 ,…,s j+k};

[0035] Each data center i Represented as:

[0036] s i = <c i γ i a i >;

[0037] Among them, c i Represents its storage capacity, γ i Represents data center type, γ i ∈{0,1},γ i =0 indicates that the data center is a cloud data center and can only store public data. i =1 indicates that the data center is an edge data center, which can store public data and private data in fixed locations. i Represents the speed at which data is copied in the data center;

[0038] Network bandwidth between data centers is expressed as:

[0039]

[0040] Among them, b ij Represents data center s i With data centers j bandwidth;

[0041] Scientific workflow is represented as:

[0042] G = (V, E, D);

[0043] Where V represents the set of tasks in the scientific workflow:

[0044] V = {v1, v2, ..., v} w};

[0045] E represents the set of task dependencies in a scientific workflow:

[0046]

[0047] D represents the data replica set:

[0048] D = {d1, d2, ..., d}m};

[0049] Each task v i The relevant dataset is represented as <D i D o >, D i D represents its input dataset. o This indicates that its output dataset, input dataset, and output dataset each consist of one or more data sets, and the dependencies between tasks are e. ij ∈E, representing task v j It is task v i The subsequent task needs to be completed in task v. i Task v can only be executed after completion; otherwise, it cannot be executed. j For task v i No dependencies, each data replica set d i Contains several copies of the i-th data, d ij It indicates that it is the j-th copy of the i-th data, d i1 Let represent the i-th original data, and each data copy contains attributes. <z i1 ,n i1 ,f i1 ,l i1 >,z i1 Indicates the data size, n i1 Indicates the number of data replicas, which is a positive integer. If n i1 =1 indicates that there are no other copies of the data, f i1 Indicates the generated data d i1 The task, if the data is the initial data, f i1 Denote as 0, l i1 Record data d ij Privacy information, if data d ij For private data, then l i1 Record the data center to which it belongs; if it is public data, then l i1 It is 0.

[0050] As described above, the cloud edge environment is mathematically represented through the above steps.

[0051] Furthermore, the modeling of the mathematical problem model in step S1 is specifically as follows:

[0052] Data d i1 In data centers k Replication overhead t copy for:

[0053]

[0054] Among them, z i1 It is data dij Size, a k It is a data center k The speed of data copying;

[0055] Data d ij From data centers k Transmitted to data center l Transmission overhead t tran for:

[0056]

[0057] Where b kl It is a data center k With data centers l The bandwidth between them is such that if the replica is copied and deployed to the current data center, there is no transmission overhead;

[0058] The data layout is represented as {S,D,Y,T} total}, where S is the set of data centers, D is the dataset, Y is the set of data layout locations, and all data d ij ∈D, each corresponds to a unique data center:

[0059]

[0060] T total Let T be the total latency corresponding to the data layout scheme, and T be the data replication time. copy With data transmission time T tran sum:

[0061] T total =T copy +T tran ;

[0062] Data replication time T copy Represented as:

[0063]

[0064] in, Representing data d i1 The layout position, n i1 For data d i Number of copies;

[0065] Data transmission time T tran Represented as:

[0066]

[0067] Where h(i,j,k,l)∈{0,1}, h(i,j,k,l)=1 represents the l-th copy d of data k. klThere exists from data center s i To data center j The transmission is complete; otherwise, h(i,j,k,l) ​​= 0.

[0068] The objective of the data layout strategy is expressed as:

[0069]

[0070] Where β(i,j,k)∈{0,1}, β(i,j,k)=1 means that the kth copy of data j is stored on data center i.

[0071] As can be seen from the above description, through the above steps, a mathematical model of a data layout strategy with the objective of minimizing total latency is obtained.

[0072] Further, step S2 includes the following steps:

[0073] A two-dimensional array is used to encode the data layout strategy to construct candidate particles:

[0074] Data layout scheme for particle i in the t-th iteration as follows:

[0075]

[0076] Each This represents the location of the replica set of data j for the i-th particle in the t-th iteration:

[0077]

[0078] Where, q k ∈{0,1},q k =1 indicates that a replica of data j is deployed on data center k; otherwise, it indicates that no replica of data j is deployed on data center k. tij Chinese q k =1 represents the number of copies of data j.

[0079] As can be seen from the above description, two issues should be considered when using data replicas: (1) how to represent different replicas of data, and (2) how to represent the storage location of data replicas. The above steps solve these two issues, taking into account both completeness and non-redundancy.

[0080] Furthermore, solving the mathematical problem model using the nonlinear inertial weighted discrete particle swarm optimization algorithm based on the genetic algorithm operator includes the following steps:

[0081] Analyze the scientific workflow, perform topological sorting on the tasks, and obtain a task queue that can be executed sequentially;

[0082] Initialize the maximum capacity of the data center, generate an initial population based on the privacy dataset, and distribute the privacy data in the initial population to the corresponding data center, while the public data is distributed randomly without generating other copies;

[0083] Simulate the data layout process, determine whether the particle is a feasible solution, if it is a feasible solution, calculate the total time delay, if it is an infeasible solution, record the infeasible dataset;

[0084] Set all individuals in the initial population as the best in their historical history, set the best in the population history as the particle with the best fitness in the initial population, and calculate the particle fitness.

[0085] Iterate the population, mutate the population according to the inertia weight factor w, cross the population with the historical best population of individuals according to the acceleration factor α1, cross the population with the historical best population according to the acceleration factor α2, calculate the fitness of the new population, and update the global information.

[0086] When the iteration ends, output the total delay of the population's historical best.

[0087] As described above, the data replica layout strategy based on NPSO-GA can be implemented by following the steps outlined above.

[0088] Furthermore, the calculation of fitness includes:

[0089] Based on the comparison of the fitness values ​​F of particles with feasible and infeasible solutions, a fitness function is established:

[0090] If both compared particles are feasible solutions, the particle with the lower total time delay has a better fitness. The fitness function is defined as follows:

[0091] F = T total ;

[0092] If both particles being compared are infeasible solutions, then the infeasible dataset D is... inf Shorter particles have better fitness, indicating that more data is distributed in feasible positions, making them more likely to become feasible solution particles in subsequent iterations. The fitness function is as follows:

[0093] F = |D inf |;

[0094] If feasible solution particles are compared with infeasible solution particles, the feasible solution is selected, and the fitness function is as follows:

[0095]

[0096] As can be seen from the above description, since the encoding of the data layout strategy in this invention is not robust, infeasible solution particles will be generated. Therefore, different fitness values ​​need to be defined according to different situations.

[0097] Furthermore, the data layout process includes the following steps:

[0098] Perform initialization operations, including initializing the task location list used to record the execution location of all tasks and the over-limit flag used to record whether any data center exceeds its capacity limit during task execution;

[0099] Calculate the data center capacity after the initial dataset is laid out, traverse the task queue, calculate the execution position of the task and record it in the task position list;

[0100] When a task generates an output dataset, the input and output datasets of the task are temporarily stored on the data center. It is then determined whether the data center has exceeded its capacity limit. The output data of the task is then distributed to the data center specified by the task, and the capacity of the data center is updated.

[0101] If a data center exceeds its capacity limit during task execution, the data located in the data center exceeding the capacity limit will be recorded in the unsolvable dataset D. inf Otherwise, calculate and record the total delay.

[0102] As described above, the data layout process is achieved through the above steps.

[0103] Furthermore, in the nonlinear inertia-weighted discrete particle swarm optimization algorithm based on genetic algorithm operators described in step S2, introducing the crossover and mutation operators of the genetic algorithm into the particle swarm algorithm includes the following steps:

[0104] Iterate over the particle's velocity and position:

[0105]

[0106] The update strategy for the i-th particle at the t-th time is:

[0107]

[0108] Among them, C g and C p It is the crossover operator, M u It is a mutation operator. It is the individual historical best of particle i at the t-th iteration, g t It is the population's historical best at the t-th iteration, where α1, α2, and w are between 0 and 1, representing the acceleration factor and the inertia weight factor.

[0109] The mutation operator of the genetic algorithm is used to replace the inertial part of the particle swarm optimization algorithm:

[0110]

[0111] Generate a random number r between 0 and 1 w If it is less than the inertia weighting factor w, then the particle mutates:

[0112] Obtain the unsolvable dataset D of particle X i inf According to the unsolvable dataset D inf and privacy dataset D fix Find the mutation location:

[0113] If D inf If there is no data in D, then select "not in D". fix The quantile corresponding to a data point, if D inf If there is data, select D. inf One of the common data quantiles;

[0114] The number of copies (copyCount) of data corresponding to the position of particle X i to be mutated is calculated.

[0115] X i [muIndex][j] = 1;

[0116] This indicates that the data corresponding to the position of particle Xi to be mutated exists as a copy in data center j;

[0117] Update the number of replicas (copyCount). Based on the original number, increase or decrease the number of replicas (copyCount) according to probability, and ensure that there is at least one replica and the number of replicas (copyCount) does not exceed the number of data centers.

[0118] For particle X i Generate a data replica layout scheme with a replica count of copyCount at the position to be mutated;

[0119] The crossover operator of the genetic algorithm is used to replace the individual cognition and social cognition components in the particle swarm optimization algorithm.

[0120]

[0121] in, This represents the optimal historical intersection of particles and individuals. This represents the historical optimal crossover between particles and the population.

[0122] As described above, by introducing the crossover and mutation operators of the genetic algorithm into the particle swarm optimization algorithm through the above steps, the search capability of the particle swarm optimization algorithm is enhanced and premature convergence is avoided.

[0123] Furthermore, in the nonlinear inertia weighted discrete particle swarm optimization algorithm based on genetic algorithm operators described in step S2, the adaptive adjustment of the inertia weight according to the difference between the particle and the global particle includes the following steps:

[0124] A nonlinear adjustment strategy for inertia weights is adopted, which adjusts the inertia weights based on the difference between the current particle and the global particles:

[0125]

[0126] This represents the difference between a particle and the best particle in the population.

[0127] Use a linear variation strategy to adjust the acceleration factors α1 and α2:

[0128]

[0129] As described above, the inertial weights are adaptively adjusted based on the differences between the current particle and the global particles through the above steps, making the optimization process more efficient.

[0130] A storage medium for workflow data layout in a cloud-edge environment, wherein a computer program is stored thereon, and when the computer program is executed, it implements the steps in the above-mentioned method for workflow data layout in a cloud-edge environment.

[0131] The present invention provides a method and storage medium for workflow data layout in a cloud-edge environment, which is applicable to the layout of workflow data in a cloud-edge environment.

[0132] Please refer to Figures 1 to 7 Embodiment 1 of the present invention is as follows:

[0133] A method for workflow data layout in a cloud-edge environment includes the following steps:

[0134] S1. The cloud-edge environment is mathematically represented. Based on the replica generation overhead and data transmission overhead, the data layout problem is modeled as a 0-1 integer programming problem with the goal of minimizing the total latency, and the mathematical problem model is obtained.

[0135] The mathematical representation of the cloud edge environment described in step S1 specifically includes:

[0136] The cloud edge environment is represented as:

[0137] S={S cld ,S edg};

[0138] Among them, cloud computing S cld It contains j data centers, represented as:

[0139] S cld ={s1,s2,…,sj};

[0140] Edge computing S edg It contains k data centers, represented as:

[0141] S edg ={s j+1 ,s j+2 ,…,s j+k};

[0142] Each data center i Represented as:

[0143] s i = <c i γ i a i >;

[0144] Among them, c i Represents its storage capacity, γ i Represents data center type, γ i ∈{0,1},γ i =0 indicates that the data center is a cloud data center and can only store public data. i =1 indicates that the data center is an edge data center, which can store public data and private data in fixed locations. i Represents the speed at which data is copied in the data center;

[0145] Network bandwidth between data centers is expressed as:

[0146]

[0147] Among them, b ij Represents data center s i With data centers j bandwidth;

[0148] Scientific workflow is represented as:

[0149] G = (V, E, D);

[0150] Where V represents the set of tasks in the scientific workflow:

[0151] V = {v1, v2, ..., v} w};

[0152] E represents the set of task dependencies in a scientific workflow:

[0153]

[0154] D represents the data replica set:

[0155] D = {d1, d2, ..., d}m};

[0156] A task is a unit of computation that can be performed in a data center. It is executed in a specific order, using a dataset as input and producing new datasets. Each task v i The relevant dataset is represented as <D i D o >, D i D represents its input dataset. o This indicates that its output dataset, input dataset, and output dataset each consist of one or more data sets, and the dependencies between tasks are e. ij ∈E, representing task v j It is task v i The subsequent task needs to be completed in task v. i Task v can only be executed after completion; otherwise, it cannot be executed. j For task v i No dependencies, each data replica set d i Contains several copies of the i-th data, d ij It indicates that it is the j-th copy of the i-th data, d i1 Let represent the i-th original data, and each data copy contains attributes. <z i1 ,n i1 ,f i1 ,l i1 >,z i1 Indicates the data size, n i1 Indicates the number of data replicas, which is a positive integer. If n i1 =1 indicates that there are no other copies of the data, f i1 Indicates the generated data d i1 The task, if the data is the initial data, f i1 Denote as 0, l i1 Record data d ij Privacy information, if data d ij For private data, then l i1 Record the data center to which it belongs; if it is public data, then l i1 It is 0.

[0157] Different replicas are deployed across different data centers to reduce data transmission latency. No replicas are created for sensitive data. Using data replicas incurs additional overhead, including data replication costs. copy and data copy transfer overhead t tran This will also consume data center storage resources. (Data d) i1 In data centers k Replication overhead t copy for:

[0158]

[0159] Among them, z i1 It is data d ij Size, a k It is a data center k The speed of data copying;

[0160] Data d ij From data centers k Transmitted to data center l Transmission overhead t tran for:

[0161]

[0162] Where b kl It is a data center k With data centers l There is no transmission overhead if the copy is replicated and deployed to the current data center.

[0163] In this embodiment, copying all public data would incur significant overhead. Therefore, the number of copies of each data item is dynamic in this invention, and the number of times data is used as task input will affect the number of copies. Figure 1 The present invention demonstrates a data replication model that selectively replicates data, trading the overhead of generating copies for transmission overhead, thereby reducing overall latency.

[0164] The data layout is represented as {S,D,Y,T} total}, where S is the set of data centers, D is the dataset, Y is the set of data layout locations, and all data d ij ∈D, each corresponds to a unique data center:

[0165]

[0166] Before a task in a scientific workflow is executed, all input data required for that task should be transmitted to the data center where the task will be executed. Because the amount of data in a scientific workflow is enormous, task scheduling time is much shorter than data transmission time; therefore, task scheduling time is ignored. total Let T be the total latency corresponding to the data layout scheme, and T be the data replication time. copy With data transmission time T tran sum:

[0167] T total =T copy +T tran ;

[0168] Data replication time T copy Represented as:

[0169]

[0170] in, Representing data d i1 The layout position, n i1 For data d i Number of copies;

[0171] Data transmission time T tran Represented as:

[0172]

[0173] Where h(i,j,k,l)∈{0,1}, h(i,j,k,l)=1 represents the l-th copy d of data k. kl There exists from data center s i To data center j The transmission is complete; otherwise, h(i,j,k,l) ​​= 0.

[0174] The objective of the data layout strategy is expressed as:

[0175]

[0176] Where β(i,j,k)∈{0,1}, β(i,j,k)=1 means that the kth copy of data j is stored on data center i.

[0177] In this embodiment, as Figure 2 The data layout shown is from the scientific workflow. Figure 1 This scientific workflow comprises a task set V = {v1, v2, v3, v4, v5, v6, v7} and a dataset D = {d1, d2, d3, d4, d5, d6, d7}, with data sizes ranging from {6GB, 10GB, 4GB, 3GB, 3GB, 5GB, 11GB}. The dataset is divided into a public dataset D. flex ={d2,d6,d7} and privacy dataset D fix ={d1,d3,d4,d5}. The data center comprises two edge data centers, each with a capacity of 25GB, and a cloud data center with unlimited storage. Let the bandwidth between the data centers be {b}. 12 ,b 13 ,b 23The speeds are set to {10M / s, 20M / s, 100M / s}, and the data replication speed in the data center is set to 800M / s. Privacy data d1 and d3 are deployed on edge data center 2, and privacy data d4 and d5 are deployed on edge data center 3. Since all tasks involve privacy data as input or output, tasks v1, v2, v3, and v6 are executed on edge data center 2, and tasks v4, v5, and v7 are executed on edge data center 3.

[0178] in Figure 2 a and Figure 2 b represents two layout schemes that do not use data replicas. The difference is that scheme a places data d2 in data center 2, while scheme b places data d2 in data center 3. Both schemes will result in two cross-data center transmissions of data d2 and one cross-data center transmission of data d7, causing a latency of approximately 6144 seconds. Figure 2 Scheme c, proposed in this invention, uses a dynamic replica count. Data replication occurs only once when data d2 is generated from v2, and one copy is transferred to data center 3. This requires only one replication of data d2 and one cross-data center transfer, as well as one cross-data center transfer for data d7, resulting in a latency of approximately 5427 seconds. Furthermore, replicating all public data would not only incur unnecessary time overhead but could also exceed the capacity limits of the edge data center. This invention, within the constraints of available capacity, trades the overhead of replica generation for transmission overhead, reducing overall latency through the rational use of data replicas.

[0179] S2. A nonlinear inertia-weighted discrete particle swarm optimization algorithm based on genetic algorithm operators is adopted. The crossover and mutation operators of the genetic algorithm are introduced into the particle swarm algorithm, and the inertia weight is adaptively adjusted according to the difference between the particles and the global particles to solve the mathematical problem model.

[0180] The overall goal of the data layout strategy is to map dataset D to data center S, minimizing total latency within the limits of data center capacity. This embodiment proposes a data replication layout strategy based on the Nonlinear Inertial Weight Discrete Particle Swarm Optimization algorithm (NPSO-GA) using genetic algorithm operators. This strategy considers the cost of generating data replicas, selectively replicates data according to task requirements, and determines the data's layout location.

[0181] Problem coding:

[0182] Two issues should be considered when using data copies: (1) how to represent different copies of data, and (2) how to represent the storage location of data copies. The coding of these issues should take into account completeness, non-redundancy, and soundness as much as possible.

[0183] Figure 3 One existing static replication factor encoding method generates the same number of replicas for public data (2 replicas in the diagram). A one-dimensional array is used to represent the data layout scheme of a scientific workflow in a cloud-edge environment, where each bit represents the placement position of a data replica. This encoding method has completeness, as each candidate solution in the problem space can be encoded as a particle. However, it lacks non-redundancy and robustness. For example, particles X1 = (2,2,3,2,3,3,2,3,1,1) and X2 = (2,3,2,2,3,3,2,3,1,1) correspond to the same solution in the problem space, both representing a copy of data d2, with the two replicas placed in data centers 2 and 3 respectively. Furthermore, this encoding method requires the number of replicas to be determined in advance and cannot adjust the number of data replicas based on the frequency of data usage.

[0184] In this embodiment, a novel encoding scheme is proposed, which uses a two-dimensional array to construct candidate solution particles.

[0185] Step S2 includes the following steps:

[0186] A two-dimensional array is used to encode the data layout strategy to construct candidate particles:

[0187] Data layout scheme for particle i in the t-th iteration as follows:

[0188]

[0189] Each This represents the location of the replica set of data j for the i-th particle in the t-th iteration:

[0190]

[0191] Where, q k ∈{0,1},q k =1 indicates that a replica of data j is deployed on data center k; otherwise, it indicates that no replica of data j is deployed on data center k. tij Chinese q k =1 represents the number of copies of data j.

[0192] This encoding scheme is complete and non-redundant, and the number and position of copies can be changed as the particles iterate. Figure 2 The encoding scheme corresponding to the c data layout is as follows: Figure 4 (Assume the number of data centers is 3.)

[0193] Fitness function:

[0194] The research objective of this invention is to reduce the total latency of scientific workflow data layout, assuming that particles with lower total latency have higher quality. However, the encoding of this invention lacks robustness and may produce infeasible solution particles. Different fitness levels need to be defined for different situations. There are two reasons for infeasible solutions: privacy leakage and failure to meet capacity constraints. Privacy leakage indicates that at least one piece of privacy data has been copied or deployed to a non-corresponding data center. Failure to meet capacity constraints indicates that at least one edge data center has stored data exceeding its capacity limit, using an invalid dataset D. inf This describes the set of data that causes a particle to become an infeasible solution. The comparison of the fitness values ​​F of particles for both feasible and infeasible solutions is divided into three cases.

[0195] Based on the comparison of the fitness values ​​F of particles with feasible and infeasible solutions, a fitness function is established:

[0196] If both compared particles are feasible solutions, the particle with the lower total time delay has a better fitness. The fitness function is defined as follows:

[0197] F = T total ;

[0198] If both particles being compared are infeasible solutions, then the infeasible dataset D is... inf Shorter particles have better fitness, indicating that more data is distributed in feasible positions, making them more likely to become feasible solution particles in subsequent iterations. The fitness function is as follows:

[0199] F = |D inf |;

[0200] If feasible solution particles are compared with infeasible solution particles, the feasible solution is selected, and the fitness function is as follows:

[0201]

[0202] Particle update strategy:

[0203] Particle Swarm Optimization (PSO) uses particles to represent each solution in the search space. The velocity of a particle determines its direction and distance of flight. The optimal solution is obtained by iteratively analyzing the velocity and position of the particles.

[0204] Iterate over the particle's velocity and position:

[0205]

[0206] In this embodiment, NPSO-GA is an improvement on the PSO algorithm. The update strategy for the i-th particle in NPSO-GA for the t-th time is as follows:

[0207] The update strategy for the i-th particle at the t-th time is:

[0208]

[0209] Among them, C g and C p It is the crossover operator, M u It is a mutation operator. It is the individual historical best of particle i at the t-th iteration, g t It is the population's historical best at the t-th iteration, where α1, α2, and w are between 0 and 1, representing the acceleration factor and the inertia weight factor.

[0210] The mutation operator of the genetic algorithm is used to replace the inertial part of the particle swarm optimization algorithm:

[0211]

[0212] Generate a random number r between 0 and 1 w If it is less than the inertia weighting factor w, then the particle undergoes a mutation process M. u As shown in Algorithm 1:

[0213]

[0214]

[0215] In Algorithm 1, the unsolvable dataset D of particle Xi is first obtained. inf According to the unsolvable dataset D inf and privacy dataset D fix Find the mutation location:

[0216] If D inf If there is no data in D, then select "not in D". fix The quantile corresponding to a data point, if D inf If there is data, select D. inf One of the common data quantiles;

[0217] Count the number of copies (copyCount) of data corresponding to the position of particle Xi to be mutated.

[0218] X i [muIndex][j] = 1;

[0219] This indicates that the data corresponding to the position of particle Xi to be mutated exists as a copy in data center j;

[0220] Update the number of replicas (copyCount). Based on the original number, increase or decrease the number of replicas (copyCount) according to probability, and ensure that there is at least one replica and the number of replicas (copyCount) does not exceed the number of data centers.

[0221] For particle X i Generate a data replica layout scheme with a replica count of copyCount for the position to be mutated.

[0222] The entire mutation process not only causes changes in the data layout location, but also alters the number of replicas. Figure 5 It is an example of a mutation process.

[0223] The crossover operator of the genetic algorithm is used to replace the individual cognition and social cognition components in the particle swarm optimization algorithm.

[0224]

[0225]

[0226] in, This represents the optimal historical intersection of particles and individuals. This represents the historical optimal crossover between particles and the population.

[0227] The process of crossover between a particle and its historical best (population historical best) is as follows: After the mutation operation, a random number r1 (r2) between 0 and 1 is generated. If this number is less than or equal to the acceleration factor α1 (α2), two quantiles of the particle are randomly selected. The segment between these two quantiles is used as the crossover interval. The segment in the crossover interval is replaced with the corresponding segment of p (or g), such as... Figure 6 The image shows an example of a crossover process.

[0228] Parameter update:

[0229] A larger inertia weight factor is beneficial for global search, helping to escape local optima; while a smaller w is beneficial for local search, allowing the algorithm to converge quickly to the optimal solution. To achieve a balance between search speed and accuracy, this invention employs a strategy of nonlinearly adjusting the inertia weight w:

[0230] A nonlinear adjustment strategy for inertia weights is adopted, which adjusts the inertia weights based on the difference between the current particle and the global particles:

[0231]

[0232] This represents the difference between the current particle and the optimal particle in the population. When its value is large, it means that the current particle is far behind the optimal particle in the population, and the inertia weight should be increased for a global search. Conversely, the inertia weight should be decreased for a local search, allowing the algorithm to converge to the optimal solution quickly.

[0233] Use a linear variation strategy to adjust the acceleration factors α1 and α2:

[0234]

[0235] As the number of iterations increases, α1 decreases while α2 increases, resulting in a larger acceleration factor α1 and a smaller acceleration factor α2 in the early stage of iteration, allowing for a more refined search for local optima within a smaller range. In the later stage of iteration, a smaller acceleration factor α1 and a larger acceleration factor α2 are obtained, improving the global cooperation ability among particles and making it easier for particles to escape local optima.

[0236] Overview of data replica layout strategies:

[0237] Algorithm 2 introduces the overall process of the data replica layout strategy, which is based on the process of the traditional PSO algorithm:

[0238] In Algorithm 2, the system is first initialized (lines 1-5), the scientific workflow is parsed, the tasks are topologically sorted, and a task queue that can be executed sequentially is obtained (line 1).

[0239] Initialize the maximum capacity of the data center (line 2). Generate an initial population based on the privacy dataset. Privacy data in the initial population will be distributed to the corresponding data center, while public data will be distributed randomly without generating other copies (line 3).

[0240] In this embodiment, the data layout process is simulated by the DataPlacement() function to determine whether the particle is a feasible solution. If it is a feasible solution, the total delay is calculated. If it is an infeasible solution, the infeasible dataset is recorded (line 4).

[0241] Set all individuals in the initial population as the best in their historical history, set the best in the population history as the particle with the best fitness in the initial population (line 5), and calculate the particle fitness.

[0242] Iterate the population (lines 6-12), mutate the population according to the inertia weight factor w (line 8), cross the population with the historical best population of individuals according to the acceleration factor α1, cross the population with the historical best population according to the acceleration factor α2 (line 9), calculate the fitness of the new population, and update the global information (lines 10-11).

[0243] When the iteration ends, output the total delay of the population's historical best (line 12).

[0244] Data layout process:

[0245] Algorithm 3 provides the data layout process for encoding particles and records the fitness of the particles.

[0246]

[0247]

[0248] In this embodiment, the data placement function DataPlacement() returns the fitness information of the population. For particles with feasible solutions, it records their total delay; for particles with unsolvable solutions, it records their unsolvable dataset D. inf .

[0249] The data layout process includes the following steps:

[0250] Perform initialization operations, including initializing the task location list taskLocList, which records the execution location of all tasks, and the over-limit flag flagOverflow, which records whether any data center exceeds its capacity limit during task execution (lines 1-4).

[0251] Calculate the data center capacity after the initial dataset is laid out (lines 5-7), calculate the data center capacity during task execution, traverse the task queue (lines 8-17), calculate the task execution position and record it in the task position list (lines 9-13).

[0252] When a task generates an output dataset, the input and output datasets of the task will be temporarily stored on the data center. It is then determined whether the data center exceeds its capacity limit (lines 14-15). The output data of the task is then laid out on the data center specified by the task, and the capacity of the data center is updated (line 16).

[0253] If a data center exceeds its capacity limit during task execution, the data located in the data center exceeding the capacity limit will be recorded in the unsolvable dataset D. inf If yes (lines 18-19), otherwise calculate and record the total latency (lines 20-22), including data replication latency and data transmission latency.

[0254] S3. Lay out the workflow data based on the solution results.

[0255] Embodiment 2 of the present invention is as follows:

[0256] A storage medium for workflow data layout in a cloud-edge environment, wherein a computer program is stored thereon, characterized in that, when the computer program is executed, it implements the steps of the method for workflow data layout in a cloud-edge environment according to any one of claims 1-9.

[0257] In summary, this invention provides a method and storage medium for workflow data layout in a cloud-edge environment. Considering factors such as transmission bandwidth, data replica generation cost, data center capacity, and privacy data, it adaptively generates data replicas to optimize transmission latency during scientific workflow operation. With the goal of minimizing total latency, the data replica layout is modeled as a 0-1 integer programming problem. Based on the topology of the scientific workflow, replicas are generated for frequently used data, trading the cost of replica generation for transmission overhead, thereby reducing total latency. A nonlinear inertial weighted discrete particle swarm optimization algorithm based on genetic algorithm operators is proposed to solve the data layout problem. The crossover and mutation operators of the genetic algorithm are introduced into the particle swarm algorithm to enhance its search capability, avoid premature convergence, and adaptively adjust the inertial weights based on the difference between the current particle and the global particle, making the optimization process more efficient.

[0258] The core objective of this invention is to minimize latency while satisfying data privacy and data center storage capacity limitations during the execution of scientific workflows.

[0259] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for workflow data layout in a cloud-edge environment, the method comprising: The method comprises the steps of: S1, mathematically representing a cloud-edge environment, modeling a data layout problem into a 0-1 integer programming problem based on copy generation overhead and data transmission overhead, and obtaining a mathematical problem model, with the objective of minimizing total latency; S2, using a nonlinear inertia weight discrete particle swarm optimization algorithm based on a genetic algorithm operator to introduce crossover and mutation operators of the genetic algorithm into the particle swarm algorithm, and adaptively adjusting an inertia weight according to differences between particles and global particles, so as to solve the mathematical problem model; S3, performing workflow data layout according to a solution result; The mathematical representation of the cloud-edge environment in step S1 is specifically as follows: The cloud-edge environment is represented as: S = {S cld ,S edg}; wherein the cloud computing S cld comprises j data centers, denoted as: S cld = {s1, s2,..., s j}; Edge computing S edg comprising k data centers, denoted as: S edg = {s j+1 ,s j+2 ,…,s j+k} Each data center s i is represented as: s i = <c i , γ i , a i > ; wherein c i represents the storage capacity of the data center, γ i represents the type of the data center, γ i ∈{0,1}, γ i = 0 represents that the data center is a cloud data center and can only store public data, γ i = 1 represents that the data center is an edge data center and can store public data and privacy data of fixed storage locations, a i represents the speed of the data center to replicate data; The network bandwidth between data centers is represented as: where b ij represents the bandwidth of the data center s i and the data center s j . The scientific workflow is represented as: G=(V, E, D); V represents a task set in the scientific workflow; V = {v1, v2,..., v w}; E represents a set of task dependency relationships in the scientific workflow; D represents a data copy set; D = {d1, d2,..., d m}; Each task v i The relevant dataset is represented as <D i D o >, D i D represents its input dataset. o This indicates that its output dataset, input dataset, and output dataset each consist of one or more data sets, and the dependencies between tasks are e. ij ∈E, representing task v j It is task v i The subsequent task needs to be completed in task v. i Task v can only be executed after completion; otherwise, it cannot be executed. j For task v i No dependencies, each data replica set d i Contains several copies of the i-th data, d ij It indicates that it is the j-th copy of the i-th data, d i1 Let represent the i-th original data, and each data copy contains attributes. <z i1 ,n i1 ,f i1 ,l i1 >,z i1 Indicates the data size, n i1 Indicates the number of data replicas, which is a positive integer. If n i1 =1 indicates that there are no other copies of the data, f i1 Indicates the generated data d i1 The task, if the data is the initial data, f i1 Denote as 0, l i1 Record data d ij Privacy information, if data d ij For private data, then l i1 Record the data center to which it belongs; if it is public data, then l i1 =0; The modeling of the mathematical problem model in step S1 is specifically as follows: Data d i1 In data center s k The replication overhead t copy is: where z i1 is the size of the data d ij and a k is the speed at which the data is replicated at the data center s k . Data d ij From data center s k To data center s l Transmission overhead t tran Is: where b kl is the bandwidth between data center s k and data center s l if the replica is replicated and then placed to the current data center, there is no transmission overhead. The data layout is represented as {S, D, Y, T}, where S is a set of data centers, D is a set of data, Y is a set of layout locations of data, and all data d total ∈D corresponds to a unique data center: ij ∈Y. T total is the total latency corresponding to the data layout scheme, is the data replication time T copy and the data transmission time T tran sum: T total = T copy + T tran ; Data duplication time T copy is represented as: wherein, represents the layout position of data d i1 n i1 is the number of copies of data d i . Data transmission time T tran is represented as: where h(i,j,k,l) e {0,1}, h(i,j,k,l) = 1 indicates that the lth copy d kl There is a transmission from data center s i to data center s j , otherwise h(i,j,k,l) = 0; The objective of the data layout strategy is represented as: Wherein, β(i,j,k)∈{0,1}, β(i,j,k)=1 represents that the kth copy of data j is stored on the data center i.

2. The method for workflow data layout in a cloud-edge environment according to claim 1, wherein, The step S2 comprises the steps of: A two-dimensional array is used to encode the data layout strategy, and a candidate particle is constructed; Data layout scheme of particle i at the tth iteration As follows: Each bit Represents the storage location of the replica set of data j of the i-th particle in the t-th iteration: where q k ∈ {0, 1}, q k = 1 indicates that a replica of data j is located on data center k, otherwise indicates that no replica of data j is located on data center k, x tij where q k = 1 represents the number of replicas of data j.

3. The method of claim 2, wherein, Solving the mathematical problem model by using the nonlinear inertia weight discrete particle swarm optimization algorithm based on the genetic algorithm operator comprises the steps of: Analyzing the scientific workflow, topologically sorting the tasks, and obtaining a task queue that can be sequentially executed; Initializing the maximum capacity of the data center, generating an initial population according to the private data set, and initializing the population, wherein the private data is laid out on the corresponding data center, and the public data is randomly laid out without generating other copies; Simulating the data layout process, judging whether the particle is a feasible solution, if it is a feasible solution, calculating the total latency, if it is not a feasible solution, recording the infeasible data set; Setting all individuals in the initial population as individual historical optimum, setting the population historical optimum as the particle with the best fitness in the initial population, and calculating the fitness of the particle; Iterating the population, mutating the population according to the inertia weight factor w, crossing the population with the individual historical optimum population according to the acceleration factor α1, crossing the population with the population historical optimum according to the acceleration factor α2, calculating the fitness of the new population, and updating the global information; Outputting the total latency of the population historical optimum when the iteration is ended.

4. The method of claim 3, wherein, The calculation of the fitness comprises: Comparing the fitness values F of the two types of particles based on the feasible solution and the infeasible solution, and establishing a fitness function: If both particles are feasible solutions, the particle with lower total latency has better fitness, and the fitness function is defined as follows: F = T total ; If both particles of the comparison are infeasible solutions, then the infeasible dataset D inf The smaller the length of the particle, the better the fitness, indicating that more data is placed in a feasible position, and it is easier to become a feasible solution particle in subsequent iterations. The fitness function is as follows: F = |D inf |; If the feasible solution particle and the infeasible solution particle are compared, the feasible solution is selected, and the fitness function is as follows:

5. The method of claim 3, wherein, The data layout process comprises the steps of: Performing initialization operations, including initializing a task position list for recording the execution positions of all tasks and a limit identifier for recording whether a data center exceeds its capacity limit during task execution; Capacity of the data center after data layout of the initial data set is calculated, the task queue is traversed, and the execution position of the task is calculated and recorded to the task position list; When the task generates an output data set, the input data set and the output data set of the task are temporarily stored on the data center, whether the data center exceeds the capacity limit is judged, and then the output data of the task is laid out on the data center specified by the task, and the capacity of the data center is updated; If the data center where the task is executed exceeds the capacity limit, the data records arranged on the data center exceeding the capacity limit are recorded in the unresolvable data set D inf Otherwise, the total latency is calculated and recorded.

6. The method of claim 5, wherein, In the nonlinear inertia weight discrete particle swarm optimization algorithm based on the genetic algorithm operator in step S2, the cross and mutation operators of the genetic algorithm are introduced into the particle swarm algorithm, including the following steps: The velocity and position of the particle are iterated: The i-th particle is updated for the t-th time: where C g and C p is a crossover operator, M u is a mutation operator, is the individual history optimum of particle i at iteration t, g t is the population history optimum at iteration t, and a1, a2, and w are between 0 and 1, representing acceleration and inertia weight factors; The inertia part in the particle swarm algorithm is replaced by the mutation operator of the genetic algorithm: generating a random number r between 0 and 1 w if it is less than the inertia weight factor w, the particle undergoes mutation: Obtaining an intractable dataset D of particles X i inf from the intractable dataset D inf and the private dataset D fix Variation positions are found: If there is no data in D inf , select the quantile not corresponding to one data in D fix ; if there is data in D inf , select the common quantile of one data in D inf ; The copy number copyCount of the data corresponding to the position to be mutated of the particle Xi is counted, and if X i [muIndex][j] = 1; It is indicated that the data corresponding to the position to be mutated of the particle Xi exists in the data center j. The copy number copyCount is updated, and on the basis of the original number, the copy number copyCount is increased or decreased according to the probability, and it is ensured that there is at least one copy and the copy number copyCount does not exceed the number of data centers. To generate a data replica placement scheme with a copy count of copyCount for the mutation position of particle X i To generate a data replica placement scheme with a copy count of copyCount for the mutation position of particle X The individual cognition and social cognition part in the particle swarm algorithm is replaced by the cross operator of the genetic algorithm: wherein, represents particle and individual history best crossover, represents particle and population history best crossover.

7. The method of claim 3, wherein, In the nonlinear inertia weight discrete particle swarm optimization algorithm based on the genetic algorithm operator in step S2, the inertia weight is adaptively adjusted according to the difference between the particle and the global particle, including the following steps: The inertia weight is adjusted based on the difference between the current particle and the global particle using a nonlinear adjustment strategy: represents the difference between the particle and the best particle in the population; The acceleration factors a1 and a2 are adjusted using a linear change strategy:

8. A storage medium of a workflow data layout in a cloud-edge environment, having stored thereon a computer program, characterized in that, The computer program is executed to realize the steps in the method of workflow data layout in the cloud edge environment in any one of claims 1-7.

Images