Repeated data processing method and device and medium

Abstract

The invention discloses a duplicated data processing method and device and a medium, and is suitable for the field of data processing. Receiving a data change request and acquiring multiple pieces of repeated data of the request; storing the plurality of repeated data into a cache, filtering, and obtaining final data; and storing the final data into the index of the search middleware to complete index updating. According to the method, the repeated data are stored in the cache to be filtered to obtain the final data, so that the final data are stored in the search middleware, the excessive data throughput of the repeated data to the search middleware is reduced, and a buffering effect on a large amount of instantaneous data is achieved. And meanwhile, additional fixed resource cost is avoided, and the data processing amount of the search middleware is increased only under limited resources.

Description

technical field [0001] The invention relates to the field of data processing, in particular to a method, device and medium for repeated data processing. Background technique [0002] In the database, related data is queried through a variety of different information. In a scenario with a large amount of data and high concurrency, the amount of data is large, and the entire query speed will be very slow. It is necessary to use the search middleware (Elastic Search, ES) to Complete your query. Take the query of order data as an example on the list page. In the notification of data change, the search middleware will obtain the order document data according to the identity (Identity document, ID) data. Since the data in the database is associated data, a document data contains The order data and multiple audit record data are collected, and the search middleware processes the order ID data multiple times. During the processing, there will be a lot of invalid and repeated data ...

AI Technical Summary

Problems solved by technology

[0003] Usually, the data will be synchronized frequently, which will generate a large amount of repeated document data, which is frequently written to the search middleware in the instantaneous interval, causing the search middleware's central processing unit (Central Processing Unit, CPU) to soar due to invalid repeated operations , which caused a performance bottleneck. At present, the expansion search middleware is used to solve technical problems. The throughput of a machine is limited, and the repeated data for multiple requests is processed at the same time in a limited time, which makes the CPU saturated or even downtime. Therefore, it is necessary to increase the number of machine to process a large amount of invalid duplicate data, but incurs an increase in the cost of fixed resources

Images