Data processing method and device

Abstract

The invention discloses a data processing method and device, electronic equipment and a computer storage medium. The method comprises the following steps: storing data in a storage system into a to-be-processed queue; combining the data in the queue to be processed into one or more data sets, wherein each data set comprises a plurality of pieces of data; and calling an interface to process the data in the data set in batches. The data information processing is changed from synchronization to asynchronization, the data is read from the storage system and is inserted into the to-be-processed queue; the data in the to-be-processed queue is combined into a data set of multiple pieces of data, and the interface is called to process data in the data set in batches; therefore, the frequency of calling the interface is greatly reduced; the method reduces database pressure, improves data processing capacity in unit time, improves data processing efficiency, meets timeliness requirements of dataprocessing, can improve CPU utilization rate, system stability and data safety, enables the system to stably, efficiently and orderly process mass data, and stably processes huge data volume every day.

Description

technical field [0001] The present application relates to the technical field of data processing, and in particular to data processing methods and devices, electronic equipment, and computer storage media. Background technique [0002] In today's big data era, industries such as logistics, e-commerce, and the Internet are often faced with the situation of processing massive data. In the process of processing massive data, it is often necessary to consider data duplication and high concurrency when processing data. The usual method is to traverse and import Data, after querying the database to remove duplicates, store the data in the database, then traverse the data again for data processing, and finally store the processed data and data results. [0003] In the process of data processing in the prior art, most of the data is processed for a single piece of data. Usually, an interface needs to be called once to process each piece of data. The greater the amount of data, the g...

AI Technical Summary

Problems solved by technology

[0004] The purpose of this application is to provide a data processing method and device, electronic equipment, and computer storage media to solve the deficiencies of the prior art, to perform batch processing on data, to call an interface to process multiple pieces of data, to reduce the number of calls to the interface, and to reduce the pressure on the database , to overcome the problem of data processing timeout due to the large amount of imported data per unit time

Images