A data broadcasting system, data broadcasting method and device

Abstract

The embodiment of the invention discloses a data broadcasting system, a data broadcasting method and equipment, which are used to reduce the occupation of system network IO and memory resources. The data broadcast system in the embodiment of the present invention includes a control node and at least one data node, and at least one task executor runs on each data node; the control node generates broadcast data, and for each data node, a task executor is set as the master Task executor; the main task executor obtains the broadcast data, saves the broadcast data to the off-heap memory on the data node, and sends the address of the off-heap memory to other task executors on the data node; other task executors read from the heap Get the broadcast data from the address of the external memory. In the embodiment of the present invention, on the same data node, only one copy of broadcast data needs to be distributed to the main task executor, thereby reducing the occupation of system network IO and memory resources.

Description

technical field [0001] The present invention relates to the field of big data, in particular to a data broadcasting system, a data broadcasting method and equipment. Background technique [0002] With the advent of the big data era, the memory-based parallel computing platform Spark has widely become a popular framework for processing massive data in the industry. Compared with Hadoop, Spark is more suitable for iterative-based machine learning algorithms and graph algorithms, and the Spark open source community is very Active, the ecosystem based on the Spark parallel framework is also increasingly rich, such as Spark-SQL, Spark-Streaming, etc. [0003] There are several modes of running Spark: for example: local, standalone, yarn, mesos, etc. Resilient Distributed Dataset (English: Resilient Distributed Dataset, referred to as: RDD) is one of the core concepts of Spark, which means read-only, partitionable, fault-tolerant, can be cached in whole or in part in memory, and ...

AI Technical Summary

Problems solved by technology

[0005] In the prior art, when a Spark cluster mode is running, when a variable is broadcast, N shares of the broadcast variable will be distributed to the data node according to the number N of task executors (executors) started on the data node, resulting in system network IO (Input / Output) and memory resources are excessively occupied, causing task failure

Images