Data processing method and device, computer equipment and storage medium

Abstract

The invention relates to the field of artificial intelligence, time division is carried out on a to-be-processed data table, and to-be-processed data in a time division range is sequentially stored ina target database, so that the data query performance and the data storage efficiency of a system are improved. The invention relates to the field of data processing, in particular to a data processing method and device, computer equipment and a storage medium, and the data processing method comprises the steps: obtaining to-be-processed data, and distributing the to-be-processed data to a to-be-processed data table according to a preset distribution strategy; determining time segmentation information corresponding to the to-be-processed data table according to the acquisition time corresponding to the to-be-processed data and the data volume corresponding to the to-be-processed data table; and according to the time division information corresponding to the to-be-processed data table, determining a plurality of to-be-executed tasks corresponding to the to-be-processed data table, and executing the plurality of to-be-executed tasks so as to store the to-be-processed data in the to-be-processed data table into the target database. In addition, the invention also relates to a blockchain technology, and the to-be-processed data table can be stored in the blockchain.

Description

technical field [0001] The present application relates to the field of artificial intelligence, and in particular to a data processing method, device, computer equipment and storage medium. Background technique [0002] When the existing business data system synchronizes massive business data, it often needs to synchronize the business data from the big data platform to the postgresql database, and then merge the business data from the postgresql database into the target database. [0003] In the process of processing business data, due to the large amount of data, the business data system cannot process all business data at one time, and needs to be processed in batches. When the existing business data system processes business data, there are the following problems: when processing business data in batches, in order to improve performance, the processed business data will be deleted, but frequent deletion operations cause the vacuum process in the postgresql library to fai...

AI Technical Summary

Problems solved by technology

When the existing business data system processes business data, there are the following problems: when processing business data in batches, in order to improve performance, the processed business data will be deleted, but frequent deletion operations cause the vacuum process in the postgresql library to fail to recycle The table space continuously reduces the performance of the business data system and affects the efficiency of the business data system to process data

Images