A data processing method, device and storage medium based on time series

Abstract

The present invention provides a data processing method, device and storage medium based on time series. The method includes: a slicing step, performing slicing processing on the acquired order data of the same store within a time period based on the time series to obtain a plurality of data Fragmentation; synchronization step, based on message queue MQ, synchronize some or all of the multiple data fragments to the distributed cache according to the task type; persistence step, use asynchronous multithreading to synchronize the distributed Some or all of the multiple data shards in the cache are persisted to the database. In the present invention, the real-time changing order data of the same store is processed by the fragmentation mechanism, which solves the technical problems of low order synchronization efficiency and poor timeliness, and avoids the omission of order data in the grabbing process. The mechanism of metadata and asynchronous multi-threading solves the problem that a large number of order data in the same store cannot be concurrent and cannot be expanded horizontally, which improves the user experience.

Description

Technical field[0001]The present invention relates to the technical field of data security, in particular to a data processing method, device and storage medium based on time series.Background technique[0002]In the prior art, for specific e-commerce store data, an expanded query time interval and a single-threaded processing mechanism are adopted. As a result, in the process of processing synchronization orders on the e-commerce platform, due to the large amount of orders on the e-commerce platform, loss of orders, Missing orders.[0003]In emergencies, such as erp server downtime and high concurrency pressure, server processing efficiency decreases, leading to untimely order synchronization data processing, server downtime is too long, and the order query time interval is interrupted after restart, and the part of the interruption is the leakage. The reason for the order: the server processing efficiency is slow, and the transaction is not submitted successfully when the database sav...

AI Technical Summary

Problems solved by technology

[0002] In the existing technology, for specific e-commerce store data, the extended query time interval and single-threaded processing mechanism are used, resulting in the process of processing synchronous orders on the e-commerce platform, due to the large number of orders on the e-commerce platform, it is easy to lose orders, Situation of missed order

[0003] In emergencies, such as erp server downtime, server processing efficiency drops under high concurrency pressure, resulting in untimely order synchronization data processing, server downtime is too long, and order query time intervals are interrupted after restarting, and the outages are leaks The reason for the order; the server processing efficiency is slow, and the transaction is not successfully submitted when the database saves the data, which eventually leads to the loss of the order, which leads to data statistics or processing errors

[0004] Under normal circumstances, an e-commerce store can only be an order grabbing task (singleton mode). This design greatly affects the order synchronization efficiency and timeliness of the e-commerce platform, and ultimately affects the user experience

Images