Method for processing cross task data in distributive network system

Abstract

The invention belongs to the technical field of network information, relating to a method for processing cross task data in a distributive network system including a main node and a plurality of slave nodes. The method comprises the following steps of: providing a corresponding cache on each slave node; saving the data of the current task executed in the slave node in the corresponding cache, generating a message comprising the saved data and the position information of the saved data, and transmitting the message to the main node; requiring to execute a subsequent task at any one slave node of the network, wherein the main node generates a scheduling instruction to a slave node saving the same data when the data required by the subsequent task is same as the pre-saved data, and commands the slave node to execute the subsequent task by the pre-saved same data; therefore, the repeated utilization of the cross task data is realized, the repeated loading of the same data input from a distributive file system is avoided, the speed of accessing the same data many times is increased, and the network communication traffic is effectively reduced.

Description

technical field [0001] The invention belongs to the technical field of network information, and in particular relates to a method for processing cross-task data in a distributed network system. Background technique [0002] With the growth of user business needs and the development of network technology, the storage and calculation of massive data pose new challenges to traditional computer systems. Massive data refers to a data collection with a huge amount of data (often on the order of terabytes or more), which is much larger than general-purpose databases and far exceeds the storage and processing capabilities of a single computer. In order to solve the management and access problems of massive data, people divide such a task that requires huge computing and storage capabilities into many subtasks through a distributed network system, and then assign and schedule these subtasks to multiple interconnected computer nodes for parallelism. processing, and finally combine th...

AI Technical Summary

Problems solved by technology

The current design and implementation of map / reduce parallel programming models (such as Hadoop) do not take into account data locality across tasks, so in such applications they must reload each task from the distributed file system over the network. Repeated loading of the same input data limits the calculation speed of the task

Images