Unlock instant, AI-driven research and patent intelligence for your innovation.

ES index reconstruction method and storage medium

A technology of indexing and consumption groups, applied in the field of database search, can solve the problems of waiting for several hours to query the latest data, bad user experience, and users unable to search the latest data.

Pending Publication Date: 2020-06-30
FUJIAN TIANQUAN EDUCATION TECH LTD
View PDF2 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

This may take a few minutes when the amount of old index data is not large, but when the amount of old index data is hundreds of gigabytes or even several terabytes, users may have to wait for several hours to query the latest data. C-end search services with high real-time requirements on the Internet cannot be tolerated
[0004] Therefore, it is necessary to provide an effective solution to the problem that users may not be able to search for the latest data for a long time due to rebuilding the index and bring users a bad experience

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • ES index reconstruction method and storage medium
  • ES index reconstruction method and storage medium
  • ES index reconstruction method and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0067] Please refer to figure 2 , this embodiment provides an ES index rebuilding method, which brings users a senseless index rebuilding experience.

[0068] The method records the index operations on the old index through the message queue, and consumes queue messages to the new index for data appending after the new index is rebuilt, without stopping the use of the old index in the whole process.

[0069] The message queue involved in this embodiment is a dual consumer group, which can be a kafka topic, a rabbitmq topic or an activemq topic. In this embodiment, the kafka topic is used for illustration.

[0070] The methods include:

[0071] S1: Create a consumer group thread waiting to trigger consumption;

[0072] Specifically, create a thread corresponding to the consumption group new_group of the kafka topic, and the consumption mode of the consumption group is to consume messages only after being triggered.

[0073] S2: The index operation corresponding to the old ...

Embodiment corresponding Embodiment 1

[0090] This embodiment corresponds to Embodiment 1, and provides a specific application scenario:

[0091] The topic corresponding to Kafka is: topic_order, and the corresponding consumer group is: consumer_order_group;

[0092] The name of the old order index is: index_order, and the corresponding alias is: alias_order;

[0093] The new order index name is: index_order_new.

[0094] Methods include:

[0095] 1. The old order program queries the old order index index_order according to the alias_order, and writes the order data; at the same time, the newly added order data will also be written into the newly created consumer_order_group waiting to trigger consumption.

[0096] That is to say, after the transformation, the old order program can still query index_order according to the alias_order and write the order data. At this point, the order program also needs to write the order data into the topic_order topic of Kafka.

[0097] 2. Perform the operation of rebuilding t...

Embodiment 3

[0108] This embodiment corresponds to Embodiment 1, and provides another specific application scenario:

[0109] Business scenario (order):

[0110] In large-scale e-commerce platforms, user order data is often stored in the order index order in elasticsearh, and this part of data may reach more than one billion pieces of data, occupying 2 to 3T of disk space. By utilizing the distributed search feature of elasticsearch, users can search for their own orders among more than one billion orders within milliseconds.

[0111] Business needs:

[0112] Some users of online orders reported that the order cannot be searched.

[0113] Technical solution:

[0114] Because there is a problem with the field mapping setting of the online order index, users cannot search for the correct order. However, there are already more than one billion pieces of data in online orders, and the conventional index rebuilding scheme will cause users to not be able to search for the latest orders withi...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides an ES index reconstruction method and a storage medium. The method comprises the steps of creating a consumption group thread waiting for triggering consumption; synchronously writing the index operation corresponding to the old index into the consumption group; configuring setting and mapping fields of a new index; reconstructing an index; triggering the consumption group to consume, and consuming the index operation in the consumption group to a newly established index; and when the consumption delay of the consumption group is lower than a threshold value, switching the association between the old index and the alias of the old index into the association between the new index and the alias. According to the invention, ES index switching can be carried out withoutstopping service, and nearly non-inductive experience is brought to a user; meanwhile, the method has the advantages of being efficient in implementation, steady and low in cost.

Description

technical field [0001] The invention relates to the field of database search, in particular to a method and a storage medium for rebuilding an index by an ES. Background technique [0002] With the vigorous development of the mobile Internet, business systems are faced with complex search scenarios of big data, while the traditional relational database MySQL is no longer suitable for complex search scenarios of big data. [0003] Elasticsearch is a distributed full-text search engine based on the underlying technology of Lucene, which provides near-real-time solutions for complex search conditions. The specific principle is divided into the following steps: first, the user submits the data to the Elasticsearch database, and then the corresponding sentence is segmented through the word segmentation controller, and its weight and word segmentation results are stored together. Using the principle of inverted index, when the user When searching for data, rank and score accordin...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/22
CPCG06F16/2228
Inventor 刘德建林伟郭玉湖陈宏
Owner FUJIAN TIANQUAN EDUCATION TECH LTD