A method and system for constructing a distributed collaborative traffic simulation environment

Abstract

The invention discloses a method and a system for constructing a distributed cooperative flow simulation environment. Based on distributed cluster computing directed acyclic graph information, network environment topology information and cooperative traffic statistical distribution data, the invention generates simulated collaborative traffic conforming to real computing traffic characteristics. The simulation environment constructed by this method allows users to define message structure, network structure and job characteristics, and provides an accurate and controllable experimental environment for users to conduct research on Coflow scheduling. The invention shields the randomness and complexity of the communication phase of computing tasks in a real computing cluster environment, can realize communication environment simulation and traffic replay scenarios on demand, and effectively supports the experiment and test work of Coflow scheduling research.

Description

technical field [0001] The invention belongs to the technical field of computer network traffic simulation, and in particular relates to a method and system for constructing a distributed collaborative traffic simulation environment. Background technique [0002] The new generation of information technology is deeply integrated with the manufacturing industry, and more and more IoT sensors are embedded in industrial production lines and used to collect industrial information data of various equipment in various links in the manufacturing process. With the development of digital twin technology and the continuous collection of industrial data, the development and application of industrial big data has become an inevitable trend. Among them, most computing and analysis tasks require the use of cluster resources, and these tasks often include a large amount of data exchange. A typical application of distributed clusters decomposes complex computing tasks into different computi...

AI Technical Summary

Problems solved by technology

However, the communication components of a typical distributed cluster computing framework cannot generate data packets with coflow identifiers, and the distributed cluster computing framework dynamically determines the task execution process according to the status of cluster resources, so it is difficult to implement controllable repetitive scenarios. To train, iteratively schedule the model

[0004] At present, Coflow scheduling research mostly adopts distributed cluster network capture and replay methods for training or simulation to generate traffic statistics to train models, and lacks a complete set of technologies and methods for collaborative traffic structure simulation in the network communication stage in distributed cluster computing scenarios

Images