Parallel message processing method, system and related device

Abstract

The invention provides a parallel message processing method, which comprises the following steps that: when a first arithmetic unit completes the calculation of parameter units, sending a reduction operation request to a main controller, so as to enable the main controller to globally broadcast the parameter units corresponding to the reduction operation request to all arithmetic units, and enabling each arithmetic unit to cache the parameter units; after the second arithmetic unit completes calculation of the parameter units to obtain reduction data, enabling the second arithmetic unit to compare the reduction data with the parameter units in the cache of the second arithmetic unit to generate bit vector information corresponding to the second arithmetic unit; when new bit vector information is generated each time, carrying out bit AND operation on the bit vector information corresponding to each arithmetic unit; and executing reduction operation according to the bit and the operationresult. According to the application, the defect that the arithmetic unit frequently communicates and interacts with the main controller is avoided, the communication pressure on the main controlleris obviously reduced, and the bandwidth is fully utilized. The invention further provides a parallel message processing method and system, a computer readable storage medium and deep learning equipment, which have the above beneficial effects.

Description

technical field [0001] The present application relates to the field of data processing, in particular to a parallel message processing method, system and related devices. Background technique [0002] Large-scale distributed deep learning training tasks, especially for complex models and large-scale data sets, will generate a large number of message bodies and message queues during the training process. For data parallel distributed training using parallel message queue processing, multiple GPUs are independent Run the respective training data, then wait for the GPU of this node or other nodes to complete the training, and then adopt different strategies to synchronize the model parameters. In this process, a large number of message bodies and lengthy message processing waiting time are generated. The more computing units, the process takes up the longer the time. Nowadays, many mainstream deep learning frameworks use the message queue generated by the message parallel proc...

AI Technical Summary

Problems solved by technology

Nowadays, many mainstream deep learning frameworks use message parallel processing to generate message queues for data parallel distributed training, but

Images