Layout method of scientific workflow data in hybrid cloud environment

Abstract

The invention discloses a layout method of scientific workflow data in a hybrid cloud environment. A scientific workflow data layout problem is encoded in a discrete particle swarm encoding manner; variation and cross operations of a genetic algorithm are imported to update particles so as to effectively overcome the premature convergence problem of discrete particle swarm optimization; an updatemethod for adaptively adjusting an inertia weight factor based on the degree of difference between global historical optimal particles and current particles is constructed to effectively meet the complex and variable nature of data problems; and a fitness function between infeasible particles with data center capacity deposit excess and feasible particles with remaining capacity is defined to reduce the data movement times between data centers and compress the data transmission amount, so that the execution efficiency of scientific workflow can be effectively improved.

Description

technical field [0001] The invention relates to a scientific workflow data layout method in the field of parallel and distributed high-performance computing, in particular to a scientific workflow data layout method in a hybrid cloud environment. Background technique [0002] In scientific research fields such as astronomy and bioinformatics, scientists analyze data from existing data sources or collect data from physical devices by performing thousands of tasks, and generate a large amount of new data, such as intermediate data or final data results , These data are often in the order of TB or even PB. In the past, scientists typically used simple methods such as the Perl scripting language to orchestrate tasks and manage data, but this approach was time-consuming and error-prone. Computational tasks and scientific applications are therefore usually modeled as workflows that automate data analysis through data or control dependencies. Scientific workflows involving a larg...

AI Technical Summary

Problems solved by technology

Among them, the most relevant research work considers data dependencies, screens out non-private datasets that are highly dependent on private datasets, and puts these datasets together in the corresponding private cloud data center, thus effectively shortening the time spent on data. transfer time, however this work does not consider the effect of size differences between datasets on data layout

Images