Data processing method and device, storage medium and electronic terminal

Abstract

The invention relates to the technical field of computers, in particular to a data processing method and device, a storage medium and an electronic terminal. The method comprises the steps of receiving to-be-processed data uploaded by a data production end; wherein the to-be-processed data comprises a data type identifier; storing the first type of data to a database based on the data type identifier, and storing the second type of data to a task table; extracting a first type of data from the database and carrying out data analysis so as to store a data primary key into a redis database queue according to a data analysis result, and storing corresponding data into a redis database key value pair; and in response to a service starting instruction, obtaining a target data primary key from the redis database queue, pulling corresponding data in batches based on the target data primary key, and executing the pulled data. According to the scheme, million-level TPS reading can be supported.

Description

technical field [0001] The present disclosure relates to the field of computer technology, and in particular to a data processing method, a data processing device, a storage medium, and an electronic terminal. Background technique [0002] With the rapid development of Internet technology, more and more high-traffic and high-concurrency application scenarios have emerged. For example, during the time period of merchant promotions and shopping festivals, there are a large number of merchants and users, and the amount of data is huge. The amount of data generated during the transaction process may be several times that of usual. In the face of high-traffic and high-concurrency application scenarios, the method of configuring more computer rooms and multi-cluster application deployment is generally used to load the required data into the system memory as much as possible, and directly perform memory calculations when the traffic peak comes. [0003] When pulling data in the pr...

AI Technical Summary

Problems solved by technology

However, there are certain defects in this method. For example, when using the message queue, batch processing cannot be performed, and the amount of data is large. When the update is frequent, it can only be processed one by one.

Slow processing speed, easy to cause data backlog

Zookeeper is not very suitable for large files and objects. When data changes frequently, data pressure will increase sharply in the case of multiple clients.

However, the RPC pull method is easy to pull redundant data, and it is difficult to distinguish differences

Images