Distributed parallel calculating system and method based on dynamic data division

Abstract

The invention relates to a distributed parallel computing system based on dynamic data splitting and a method thereof. The system comprises a splitting unit, an operational unit, an environmental unit and a load balancing unit, wherein the splitting unit is responsible for splitting a data item to be computed and dividing data to be computed into a plurality of relative independent computational items according to division rules; the operational unit is responsible for performing distributed parallel computation on the computational items and comprises a plurality of calculators; the environmental unit is responsible for deploying the executive environment in the calculators before parallel computation and reducing the calculators to the state before execution after computational execution is completed; and the load balancing unit is responsible for inspecting loading conditions of all the calculators and dynamically distributing corresponding number of computational items to various calculators according to resource consumption conditions of the various calculators. The system and the method have the advantages that: firstly, the computing speed is accelerated to the maximum degree; secondly, spare resources of a computer are utilized to the maximum degree; thirdly, convenience is provided for controlling, monitoring and examining data computation; and fourthly, the operational unit is independently disaster-tolerant.

Description

technical field [0001] The invention relates to the fields of computer and network communication, in particular to a distributed parallel computing system and method based on dynamic data splitting. Background technique [0002] With the popularization of computers and networks, due to the increasingly frequent network communication activities, there are more and more servers providing various services for computers in the network, such as various portal websites, various instant messaging service providers, and various search engines etc., and various servers that provide these network services not only receive massive amounts of data every day, but also perform calculations on these data under various complex calculation models. Taking instant messaging service providers as an example, the total amount of data they receive may exceed 500G per day, and reach 16T per month; and for the received data, dozens of calculations need to be performed by computing servers (computing...

AI Technical Summary

Problems solved by technology

However, this method of splitting static data according to tasks has the following disadvantages: 1. Various tasks are large and small, and the required computing resources are distributed unevenly, so the utilization rate of computing machines is not high; 2. Additional work must be done. Disaster recovery makes the utilization rate of computing machines lower; 3. It is not possible to flexibly add or delete computing machines to increase computing efficiency or separate computing machines; 4. It is difficult to switch the computing environment of computing machines, and the computing environment of computing machines corresponds to the tasks it performs. fixed

The disadvantages of this solution are: 1. This solution is only suitable for data transmission on the Internet, not suitable for intranet data transmission; 2. This technical solution can only speed up the transmission of information, but cannot perform model calculations on the data, which can improve the efficiency of information. transfer efficiency without improving the computational efficiency of data

Images