Data extraction method and device, equipment and storage medium

Abstract

The invention discloses a data extraction method and device, equipment and a storage medium. The method comprises the following steps: regularly obtaining target data in a target data state in a pre-created data summary table; grouping and assembling according to the operation type of the target data to obtain corresponding to-be-extracted data; and extracting the to-be-extracted data to the corresponding target search engine. According to the embodiment, the target data in the target data state in the data summary sheet is obtained regularly, and the target data is grouped and assembled according to the operation type of the target data, namely, the target data is extracted in batches and imported into the corresponding target search engine, so that the phenomenon that the pressure of a server is too large due to excessive data extracted at a time is avoided, and the efficiency of the server is improved. Therefore, the stability of data extraction of the server is ensured.

Description

technical field [0001] The embodiments of the present invention relate to the field of data processing, and in particular, to a data extraction method, device, device, and storage medium. Background technique [0002] With the rapid development of social informatization, the informatization construction of traditional industries is imminent, as well as the construction of e-commerce in various industries. In the construction of these systems, there is a crucial module that is the search module. Traditional search uses traditional relational databases to provide regular services. However, with the increase in data volume, diversification of user search behavior and system For statistics, monitoring and management of searches, etc., traditional databases are already overwhelmed with these functions. Then came the non-relational database of Elasticsearch for search services. However, Elasticsearch cannot provide all the functions of the persistent database, so the data in it a...

AI Technical Summary

Problems solved by technology

[0003] Conventional search engine Elasticsearch data import implementation: general data synchronization needs to be manually triggered, specifically requesting data from other systems first, importing the obtained data after obtaining the data to be updated, specifically deleting all data in the previous search engine database, Then import all the data from other systems to the search engine at one time. During the processing process, data loss is not eliminated, and data that needs to be changed is not processed as needed, but all are processed, resulting in repeated data processing, and a huge amount of data may be generated at the same time. Cause server downtime, the system does not make corresponding log records, and automatic data extraction cannot be realized

Images