Data processing method and device

Abstract

The invention discloses a data processing method and device, electronic equipment and a computer storage medium. The method comprises the following steps: storing data in a storage system into a to-be-processed queue; combining the data in the queue to be processed into one or more data sets, wherein each data set comprises a plurality of pieces of data; and calling an interface to process the data in the data set in batches. The data information processing is changed from synchronization to asynchronization, the data is read from the storage system and is inserted into the to-be-processed queue; the data in the to-be-processed queue is combined into a data set of multiple pieces of data, and the interface is called to process data in the data set in batches; therefore, the frequency of calling the interface is greatly reduced; the method reduces database pressure, improves data processing capacity in unit time, improves data processing efficiency, meets timeliness requirements of dataprocessing, can improve CPU utilization rate, system stability and data safety, enables the system to stably, efficiently and orderly process mass data, and stably processes huge data volume every day.

Description

technical field

[0001] The present application relates to the technical field of data processing, and in particular to data processing methods and devices, electronic equipment, and computer storage media. Background technique

[0002] In today's big data era, industries such as logistics, e-commerce, and the Internet are often faced with the situation of processing massive data. In the process of processing massive data, it is often necessary to consider data duplication and high concurrency when processing data. The usual method is to traverse and import Data, after querying the database to remove duplicates, store the data in the database, then traverse the data again for data processing, and finally store the processed data and data results.

[0003] In the process of data processing in the prior art, most of the data is processed for a single piece of data. Usually, an interface needs to be called once to process each piece of data. The greater the amount of data, the g...

Examples

Embodiment Construction

[0075] Below, the present application will be further described in conjunction with the accompanying drawings and specific implementation methods. It should be noted that, on the premise of not conflicting, the various embodiments described below or the technical features can be combined arbitrarily to form a new embodiment. .

[0076] see Figure 1 to Figure 3 , the first embodiment provides a data processing method, the method includes steps S100-S300.

[0077] The data processing method in this embodiment can be applied to data processing systems in industries such as logistics, e-commerce, and the Internet, or other scenarios that require processing massive amounts of data.

[0078] Step S100: Store data in the storage system into a queue to be processed.

[0079] The storage system may be a pika database or a redis database. The types of data are, for example, waybill data, order data, consulting demand data, and the like. Specifically, the data in the storage system ...

technical field

[0001] The present application relates to the technical field of data processing, and in particular to data processing methods and devices, electronic equipment, and computer storage media. Background technique

[0002] In today's big data era, industries such as logistics, e-commerce, and the Internet are often faced with the situation of processing massive data. In the process of processing massive data, it is often necessary to consider data duplication and high concurrency when processing data. The usual method is to traverse and import Data, after querying the database to remove duplicates, store the data in the database, then traverse the data again for data processing, and finally store the processed data and data results.

[0003] In the process of data processing in the prior art, most of the data is processed for a single piece of data. Usually, an interface needs to be called once to process each piece of data. The greater the amount of data, the g...

Examples

Embodiment Construction

[0075] Below, the present application will be further described in conjunction with the accompanying drawings and specific implementation methods. It should be noted that, on the premise of not conflicting, the various embodiments described below or the technical features can be combined arbitrarily to form a new embodiment. .

[0076] see Figure 1 to Figure 3 , the first embodiment provides a data processing method, the method includes steps S100-S300.

[0077] The data processing method in this embodiment can be applied to data processing systems in industries such as logistics, e-commerce, and the Internet, or other scenarios that require processing massive amounts of data.

[0078] Step S100: Store data in the storage system into a queue to be processed.

[0079] The storage system may be a pika database or a redis database. The types of data are, for example, waybill data, order data, consulting demand data, and the like. Specifically, the data in the storage system ...

Images