Data processing method and device, computer equipment and storage medium

Abstract

The embodiment of the invention provides a data processing method and a device, computer equipment and a storage medium, and the method comprises the steps: receiving a to-be-processed data processing task through a first-layer server, splitting the data processing task into a plurality of sub-tasks, and distributing the sub-tasks to a plurality of second-layer servers; loading the sub-tasks from a database through the second-layer servers, and distributing the identity marks of the sub-tasks to a third-layer server; and adopting the third-layer server to load the corresponding data processing task from the database according to the identity marks of the sub-tasks and execute the data processing task. According to the technical scheme, a three-layer distribution asynchronous task processing architecture is constructed, the processing resources of the server cluster can be fully utilized, the time-consuming and resource-consuming data processing tasks are decomposed into the sub-tasks to be rapidly processed, the processing efficiency of a large number of data processing tasks high in time consumption is improved, a large number of blocking is avoided, and the server pressure is reduced.

Description

technical field [0001] The present application relates to the technical field of data processing, and in particular, the present application relates to a data processing method and device, as well as a computer device and a computer-readable storage medium. Background technique [0002] With the development of computer technology, big data processing capabilities are becoming more and more important. In the database processing business, such a scenario is often encountered, that is, a large number of data processing requests flood into the server system in a short period of time, and these data processing requests will trigger some Time-consuming data update operations. In fact, these operations often do not have high real-time requirements. If these data processing requests are processed directly in a synchronous manner, it will cause a large number of data processing requests to wait or block, and current limiting measures are adopted It will also cause a large number of r...

AI Technical Summary

Problems solved by technology

[0002] With the development of computer technology, big data processing capabilities are becoming more and more important. In the database processing business, such a scenario is often encountered, that is, a large number of data processing requests flood into the server system in a short period of time, and these data processing requests will trigger some Time-consuming data update operations. In fact, these operations often do not have high real-time requirements. If these data processing requests are processed directly in a synchronous manner, it will cause a large number of data processing requests to wait or block, and current limiting measures are adopted It will also cause a large number of requests to be intercepted, resulting in unavailable services; if these data processing requests are processed in an asynchronous manner, the thread pool will be full, the CPU load will be high, and in severe cases, the server system will even be paralyzed

[0003] It can be seen that the conventional synchronous or asynchronous data processing methods are prone to defects such as massive request blocking and excessive server pressure when processing a large number of data processing request tasks

Images