Data processing node management method and device, equipment and storage medium

Abstract

The invention discloses a data processing node management method, device and equipment and a storage medium, and belongs to the field of cloud technology and big data. The method comprises the steps of obtaining abnormal state information of an abnormal processing node when it is detected that the abnormal processing node exists in a processing node cluster corresponding to a data processing task;if the abnormal state information meets the condition, determining to start auxiliary nodes except the processing node cluster to replace the abnormal processing node; under the condition that the auxiliary node is determined to be started, adjusting an execution strategy of the data processing task; based on an execution strategy, determining data processing sub-tasks corresponding to the auxiliary nodes and the remaining processing nodes except the exception processing nodes in the processing node cluster; and sending corresponding task execution instructions to the auxiliary nodes and theremaining processing nodes. According to the scheme provided by the invention, the reliability of data processing can be improved, and the data processing task can be smoothly and efficiently completed.

Description

technical field [0001] The present application relates to the fields of cloud technology and big data, and in particular to a management method, device, equipment and storage medium of a data processing node. Background technique [0002] Big data refers to a collection of data that cannot be captured, managed and processed by conventional software tools within a certain period of time. , high growth rates and diverse information assets. [0003] In related technologies, computer equipment uses parallel computing (Parallel Computing) to process big data. Among them, parallel computing refers to the process of using multiple computing resources to solve computing problems at the same time. The basic idea of ​​parallel computing is to use multiple processors to solve the same problem cooperatively, that is, the problem to be solved is decomposed into several parts, and each part is calculated by an independent processor. That is to say, for the same data processing task, co...

AI Technical Summary

Problems solved by technology

[0004] However, in related technologies, if a certain processing node is abnormal during the processing of data processing tasks, the corresponding data processing tasks cannot be completed normally, and the reliability of data processing is low.

Images