Remote sensing image parallel cluster processing scheduling strategy suitable for Internet scene

A remote sensing image and scheduling strategy technology, applied in the direction of electrical digital data processing, multi-program device, program control design, etc., can solve the problem that the parallel cluster scheduling strategy cannot adapt to remote sensing image heterogeneous parallel cluster processing scenarios, etc., and achieve a focus on communication Cost, good adaptability, and the effect of improving computing efficiency

Active Publication Date: 2020-12-22
HENAN UNIVERSITY
View PDF8 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] In order to solve the problem that the current parallel cluster scheduling strategy cannot adapt to the heterogeneous parallel cluster processing scenario of remote sensing images combined with the Internet, the present invention provides a scheduling strategy for parallel cluster processing of remote sensing images adapted to Internet scenarios, and introduces a directed acyclic graph Workflow model, combined with this model, proposes a parallel partition method adapted to the remote sensing image processing algorithm of the present invention, which has wide adaptability to the changing needs of the remote sensing image processing algorithm in the Internet scene; and then designs a two-level combined scheduling strategy , the first-level scheduling strategy performs workflow initialization scheduling according to the results of the parallel partition method, and the second-level scheduling strategy performs scheduling error compensation scheduling in the parallel process, so that the entire scheduling strategy fully adapts to the characteristics of dynamic workflow in the Internet scene, making Remote sensing image processing tasks with large-scale data, dynamic irregular requests, and multi-user and multi-demand characteristics can be reasonably scheduled in multi-node parallel clusters, maximizing the use of computing resources in parallel clusters, and ultimately improving computing speed

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Remote sensing image parallel cluster processing scheduling strategy suitable for Internet scene
  • Remote sensing image parallel cluster processing scheduling strategy suitable for Internet scene
  • Remote sensing image parallel cluster processing scheduling strategy suitable for Internet scene

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0064] Such as figure 2 As shown, the embodiment of the present invention provides a scheduling strategy for parallel cluster processing of remote sensing images adapted to Internet scenarios, including:

[0065] S101. Using a directed acyclic graph algorithm parallel design model to form a directed acyclic graph with the target remote sensing image processing algorithm;

[0066] Specifically, the target remote sensing image processing algorithm combined with the directed acyclic graph algorithm parallel design model is divided into operator sets, which record the necessary data such as input data volume, output data volume, overall calculation scale, and calculation volume constants, and form a directed acyclic graph algorithm. The necessary data such as the input data volume, output data volume, overall calculation scale, and calculation volume constants include: the level of the operator in the directed acyclic graph, the pre-operator dependency of the operator, and the po...

Embodiment 2

[0078] Embodiments of the present invention also provide a scheduling strategy for parallel cluster processing of remote sensing images adapted to Internet scenarios, including the following steps:

[0079] S201. Using the directed acyclic graph algorithm parallel design model to form the target remote sensing image processing algorithm into a directed acyclic graph, specifically:

[0080] S2011. Obtain the specific flow of the target remote sensing image processing algorithm by drawing an algorithm flow chart;

[0081] S2012. Based on the drawn flow chart, determine the boundary between logical operations and numerical operations, the boundary between two logical operations and the boundary between two numerical operations, and divide the target remote sensing image processing algorithm into multiple operators from the boundary , forming an operator set;

[0082] S2013. Form the divided operator set into a directed acyclic graph according to the logical relationship between ...

Embodiment 3

[0125] On the basis of Embodiment 1 and Embodiment 2, the representative NDVI and LSWI combined vegetation high-precision recognition remote sensing image processing algorithm is used as the target remote sensing image processing algorithm, and the process of algorithm division and scheduling is carried out as a specific embodiment A detailed description.

[0126] The normalized vegetation index (Normalized Vegetation Index, NDVI) is described in remote sensing images as the sum of the ratio of the difference between the reflectance value of the near-infrared band and the reflectance value of the red band. The atomic data of the NDVI calculation process is a single pixel, and the NDVI calculation of one area has nothing to do with the calculation process of other areas. The calculation process needs to iteratively calculate the pixels one by one, which has obvious parallel characteristics and is suitable for data parallelism and functional parallelism. Combined parallel design...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a remote sensing image parallel cluster processing scheduling strategy suitable for an internet scene. The strategy comprises the following steps: step 1, forming a directed acyclic graph from a target remote sensing image processing algorithm by adopting a directed acyclic graph algorithm parallel design model; step 2, determining an operator with a relatively large overallcalculation scale in the directed acyclic graph according to a preset rule, and performing data dilution on the operator with the relatively large overall calculation scale by adopting an algorithm data volume dilution method; 3, based on a directed acyclic graph dynamic workflow scheduling strategy, scheduling operators in the operator set after data dilution to a parallel cluster node server one by one for calculation; and step 4, after the directed acyclic graph dynamic workflow scheduling strategy is executed and before the tasks of all the node servers are completely executed, schedulingthe tasks of the node servers with relatively large loads to the node servers with relatively small loads based on the task stealing type error compensation strategy optimized by the communication cost.

Description

technical field [0001] The invention relates to the technical field of parallel computing, in particular to a scheduling strategy for parallel cluster processing of remote sensing images adapted to Internet scenarios. Background technique [0002] With the rapid development of my country's remote sensing industry and the explosive growth of the number of remote sensing satellites, the scale of remote sensing image data is also rising rapidly. The traditional stand-alone processing mode can no longer meet the needs of remote sensing image processing, while parallel computing can Take advantage of it. With the development of computer hardware and the improvement of software framework compatibility, remote sensing images can be more easily introduced into multiple parallel frameworks for parallel processing. Some remote sensing image processing algorithms in the existing research adopt a data parallel design mode, that is, the input remote sensing image data is divided into mul...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F9/50
CPCG06F9/5083G06F9/5038
Inventor左宪禹苏岳瀚王倩倩周兵葛强杜莹
OwnerHENAN UNIVERSITY