Elasticsearch search engine index construction method and device

Abstract

The invention provides an Elasticsearch search engine index construction method and device. The method comprises the steps that an Flink cluster is used to export full index target data from a database at regular time, and a new index database is created; near-real-time index services monitor a business data change message notification, read latest data from the database, update the latest data to an existing index database, detect whether batch index construction is carried out or not, and update batch index data to the new index database; and alias switching is carried out on Elasticsearch indexes, and an index target is pointed to the new index database. Compared with the prior art, the Elasticsearch search engine index construction method provided by the invention has the advantages that the data synchronization efficiency is improved, and the data consistency in and after the index batch processing construction process is ensured.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a method and device for constructing an Elasticsearch search engine index. Background technique [0002] The index of Elasticsearch is usually based on the Logstash synchronization technology, and adopts the index synchronization scheme of regularly executing SQL statements. This solution has complex processing and low synchronization efficiency when crossing multiple business data sources. In addition, the timing execution mechanism cannot be guaranteed in real time, and data changes between two synchronization time points cannot be reflected in the index in time. [0003] Another common processing method is to synchronize by listening to database log change messages (such as MySql's binlog log). This solution can indeed solve the lack of real-time performance, but a single document in the retrieval system often spans multiple Multiple database instances and tables in the bus...

AI Technical Summary

Problems solved by technology

This solution has complex processing and low synchronization efficiency when crossing multiple business data sources. In addition, the timing execution mechanism cannot be guaranteed in real time, and data changes between two synchronization time points cannot be reflected in the index in time

[0003] Another common processing method is to synchronize by listening to database log change messages (such as MySql's binlog log). This solution can indeed solve the lack of real-time performance, but a single document in the retrieval system often spans multiple Multiple database instances and tables in the business field, such as the product information of a typical e-commerce system, not only have the basic attributes of the product, but also include information such as its category, seller, evaluation, sales volume, etc., and these information will be stored in the corresponding entity domain In the business database, the database change log of a single table greatly increases the complexity of data processing, and it is impossible to simply determine the range of index documents it affects, such as:

[0004] Only some fields in most monitoring tables need to be included in the index. Changes to fields that are not included in the index will also trigger database change logs. How to deal with the screening and identification of this part of the change log? The validity of the document is affected. One-to-many or many-to-many associated database change logs will generate multiple data change records in multiple associated tables. These change records are continuously generated, and the listener cannot know the status of a certain thing. Whether all the logs have been obtained, the integrity cannot be guaranteed

Even if the obtained records are complete and mapped to index documents, different business scenarios will correspond to different operations, and these operations are difficult to handle uniformly

Images