Coflow collaborative job flow scheduling perception data flow division method and device

Abstract

The invention relates to a Coflow collaborative job flow scheduling perception data flow division method and device, computer equipment and a storage medium. The method comprises the following steps:obtaining a to-be-divided DDSP application program data stream; extracting a complex calculation task flow of the DDSP application program; according to a preset Coflow collaborative job flow scheduling model, obtaining a Coflow collaborative job flow scheduling model, a complex calculation task flow in a DDSP application program is divided into a plurality of sub-task flows; further analyzing thedata access demand and the data dependence of each sub-task flow; carrying out multi-dimensional data segmentation on the DDSP application program data stream; reducing cross access to the data blocks among different tasks as much as possible and the dependence between the data blocks d; finally, according to the data access requirements of the sub-task flows in each computing node, taking data communication minimization between computing nodes and workload equalization of the computing nodes as optimization objectives, and allocating the segmented data blocks to the most appropriate computing nodes, so that the communication overhead between the distributed computing nodes is effectively reduced, the utilization rate of the data blocks is increased, the access speed is increased, and themethod is suitable for distributed data flow processing.

Description

technical field [0001] The present application relates to the field of information technology, in particular to a data flow division method, device, computer equipment and storage medium with Coflow collaborative job flow scheduling awareness. Background technique [0002] With the rapid development of the Internet, sensor networks, and mobile Internet technologies, various application fields continue to generate large amounts of data sets in the form of streams. Streaming computing is a highly real-time computing model and an effective way of big data computing. Practical application fields such as financial markets, network monitoring, telecommunications, and sensor networks all generate and store massive streaming data sets. DDSP (Distributed Data Stream Processing, distributed data stream processing) is an effective method to improve the performance of large-scale data stream processing. [0003] However, several key challenges are faced in DDSP applications at present:...

AI Technical Summary

Problems solved by technology

[0004] However, distributed data stream processing applications are different from traditional static data processing. They usually involve a collection of parallel task flows executed on distributed machines. There are different logical dependencies and data dependencies between task flows. Simply follow the traditional static data Data flow division corresponding to the processing method It is obviously unreasonable to perform data flow division processing on distributed data streams

Images