Data parallel processing method and device, computer equipment and readable storage medium

Abstract

Embodiments of the invention disclose a data parallel processing method and apparatus, a computer device and a readable storage medium. The method comprises the steps of obtaining real-time to-be-processed data; generating unique identification information corresponding to the to-be-processed data; according to a preset calculation model, to-be-processed data is split to obtain multiple pieces of sub-data corresponding to the to-be-processed data, each piece of sub-data is associated with blood relationship metadata corresponding to the to-be-processed data, and each piece of blood relationship metadata comprises the unique identification information; according to a preset grouping rule, distributing all the sub-data to different data processors for parallel computing; and according to a preset ordered dictionary and the blood relationship metadata, combining processing results fed back by the data processors to obtain a target processing result corresponding to the to-be-processed data. According to the method and the device, the blood relationship metadata are associated in the data splitting, calculating and merging processes, and large-scale real-time data are quickly and orderly processed in parallel according to a grouping modeling mode.

Description

technical field [0001] The present invention relates to the field of data processing, and in particular, to a data parallel processing method, apparatus, computer equipment and readable storage medium. Background technique [0002] In the existing real-time data processing engine, the parallel processing method is usually used to process data. The existing real-time data processing engine usually performs numbering processing according to the splitting sequence of the real-time data, and processes the processing results according to the numbering sequence. merge. [0003] The existing real-time data processing engines have two main problems: one is that there may be strict timing requirements before and after the calculation of the same type of data. For example, the user's operation sequence is login-browse-add to shopping cart-order-logout , corresponding to the five operation logs with a fixed time sequence. However, if the current real-time data processing engine is ex...

AI Technical Summary

Problems solved by technology

[0003] There are two major problems with existing real-time data processing engines: one is that there may be strict timing requirements before and after the calculation of the same type of data, for example, the user's operation sequence is login-browse-add shopping cart-order-logout , corresponding to five operation logs with fixed timing

[0004] The second is that the existing real-time data processing engines often store a large amount of data fields in one computing unit for calculation when dealing with large-scale parallel computing problems, resulting in a long calculation time for a single piece of data and blocking subsequent data processing.

Images