Data generation technology full-text retrieval method based on elastsearch

Abstract

The invention relates to the technical field of information retrieval, in particular to a data generation technology full-text retrieval method based on elasticsearch. The innovation point of the scheme lies in how to share generations and how to manage the generations, the business insight is combined with the underlying technology, and the insight and abstraction of metadata are realized. By extracting metadata, complete data driving can be realized. Therefore, the system can support more accurate retrieval of larger-scale data, and the technical problems of low business handling efficiency of business personnel and long retrieval time are solved. And after the full-text retrieval efficiency is improved, the system stability is improved, and the operation cost is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of information retrieval, and in particular to a full-text retrieval method based on elasticsearch data generation technology. Background Art

[0002] In the field of full-text search, elasticsearch is currently a popular and recognized open source solution in the industry. The query load of Elasticsearch is distributed to each shard node, and the underlying layer of elasticsearch is the lucene framework. When there are too many segment files in lucene, it will affect the query performance and needs to be merged regularly, which will affect the query performance.

[0003] Elasticsearch data is accessed in two main ways: write and read. Index writes will cause changes in the number of segments.

[0004] The prior art is described as follows: 1. In the elasticsearch architecture, data can be divided into different indexes, and a query solution using aliases is also provided. However, the common practice in business is to divide the index by a single dimension, such as the time dimension of year or month. However, in the case of historical data updates, the affected data range cannot be controlled. Moreover, if one index is divided per month, too many indexes will cause a sharp drop in query performance.

[0005] 2. In elasticsearch, the underlying storage of data is based on lucene technology, and will eventually be stored as segment data. If there are frequent data updates, a large number of segment fragments will be generated. Too many segment fragments will affect the performance of queries and statistics, and segment merging needs to be performed regularly. Segment merging can be performed in stages, such as merging from 100 segments to 50 segments, and then from 50 segments to 1 segment. However, no matter how the stages are divided, the time consumed by the process is proportional to the number of fragments and inversely proportional to the number of target segments. The more fragments there are, the longer it takes. Merging segments for index files with large amounts of data is a very time-consuming process. The locks generated during the segment merging process and when the merging is not completed will affect the query performance. This leads to a problem. It is necessary to trigger segment merging in a timely manner when there are many fragments, but large-scale segment merging processes may cause query business performance to degrade or even service unavailability.

[0006] 3. Combining the above 1 and 2, it can be deduced that based on the current technology, when performing index query on large amount of text, it is impossible to effectively solve the performance degradation problem caused by updating the index when updating the data.

[0007] 4. Through business insights, we can conclude that the more recently generated data is updated, the more frequently it is updated. When querying, all data needs to be integrated to obtain results, including the oldest and latest data. Combining situations 1, 2, and 3, current technology cannot effectively solve the scenario where historical data query and update are required at the same time, and new data is updated frequently. Summary of the invention

[0008] The purpose of this invention is to design a general architecture and process, separate the data through generation technology, and adopt different data optimization and update strategies for each generation, so as to improve the full-text search performance as a whole and solve the problem of the impact of data update on query performance in massive data query scenarios.

[0009] The full-text retrieval method based on elasticsearch's data generation technology includes the following steps: S1. Vertical division by update frequency: Combined with business characteristics, vertical division is performed by data update frequency; divided into different categories of elasticsearch indexes; different generations are new generation, middle generation, and old generation; update frequency new generation> middle generation> old generation; S2. Move records to corresponding generations according to business through status monitoring or scheduled tasks; S3. Perform index merging operations on the corresponding generational data during non-business hours according to the business; S4. Incrementally update each generation index at regular time according to the business; S5. Create an index alias to point to all generational data; S6. When querying, directly specify the index alias for unified query; S7. After following the above process, compare the amount of fragmented data in the new generation, middle generation, and old generation, and assign the generation with the least amount of fragmented data to the non-business time to perform the index merge operation in step 3.

[0010] In S6, long and short keywords are processed separately: for long keywords, the search target adopts the following word segmentation method, ngram segmentation is performed in the indexing stage, and ngram and wildcard fuzzy matching are combined in the query stage; first, the data is preliminarily screened using the ngram method, and then the screened result set is full-text matched; for short keywords, the conventional phrase query match_phrase is used to implement fuzzy search.

[0011] The significance of the present invention is that it provides a generalizable process and method, which has never been mentioned in the current public technical solutions and is completely designed through business insight innovation. It also solves practical problems that have not been solved after many attempts in actual projects. The innovation of this solution lies in how to divide and manage generations, in combining business insights with underlying technologies, and in the insight and abstraction of metadata. By refining metadata, it can be completely data-driven.

[0012] Improve system stability: By optimizing the search process, the risk of system crash or performance degradation can be reduced, and maintenance costs can be reduced. Reduce operating costs: Reduce server resource consumption, reduce the use of CPU, memory and storage resources, and thus reduce hardware costs. Improve business handling experience: Business personnel can get matching search results in a shorter time, reduce the waiting time for handling, and meet the diverse search needs of users, thereby improving the overall efficiency of business handling.

[0013] In summary, the application of this process method in this embodiment enables the system to support more accurate retrieval of larger-scale data, solve the business pain point of low efficiency of business personnel in handling cases, and the technical difficulty of long retrieval time. The improvement of full-text retrieval efficiency helps to improve system stability, reduce operating costs, and improve the business handling experience, ultimately improving the overall quality and efficiency of business personnel in handling cases. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a system framework diagram of the present invention; Figure 2 To query the logical diagram; Figure 3 This is a schematic diagram of index merging. DETAILED DESCRIPTION

[0015] The full-text retrieval method based on elasticsearch's data generation technology includes the following steps: S1. Vertical division by update frequency: Combined with business characteristics, vertical division is performed by data update frequency; divided into different categories of elasticsearch indexes; different generations are new generation, middle generation, and old generation; update frequency new generation> middle generation> old generation; S2. Move records to corresponding generations according to business through status monitoring or scheduled tasks; S3. Perform index merging operations on the corresponding generational data during non-business hours according to the business; S4. Incrementally update each generation index at regular time according to the business; S5. Create an index alias to point to all generational data; S6. When querying, directly specify the index alias for unified query; S7. After following the above process, compare the amount of fragmented data in the new generation, middle generation, and old generation, and assign the generation with the least amount of fragmented data to the non-business time to perform the index merge operation in step 3.

[0016] In S6, long and short keywords are processed separately: for long keywords, the search target adopts the following word segmentation method, ngram segmentation is performed in the indexing stage, and ngram and wildcard fuzzy matching are combined in the query stage; first, the data is preliminarily screened using the ngram method, and then the screened result set is full-text matched; for short keywords, the conventional phrase query match_phrase is used to implement fuzzy search.

[0017] Vertical division by update frequency: Combined with business characteristics, vertical division is performed based on the frequency of data update. Different elasticsearch indexes are used.

[0018] Scenarios where old data is read more and written less, while new data is read more and written more For example, in the case query category, the data of completed (archived) cases (old data) will not be updated unless they are activated and reprocessed. This type of data is classified as the old generation.

[0019] Data created before today and still being processed is considered as relatively new data, which will be updated but not frequently. This type of data is classified as mid-age.

[0020] The data newly added on the day is updated most actively, whether it is created or updated, and this type of data is classified as the new generation.

[0021] Query the optimization details on the application side: From the analysis of user behavior, there are two situations: short keyword input (users are not sure of the complete text string) and long keyword input (users know the complete text or the main text content, and hope to reversely search for other features corresponding to the text, enter the message person, etc.). Considering the probability characteristics that long keywords have few hit results and short keywords have many hit results, long and short keywords are processed separately: for long keywords, the search target adopts the following word segmentation method, ngram word segmentation is performed in the index stage, and ngram and wildcard fuzzy matching are combined in the query stage. The details are as follows: first, the data is preliminarily screened using the ngram method, and then the filtered result set is fully matched. By adding this layer of preliminarily screening, the balance between performance, precision and completeness is achieved. For short keywords, the words entered by users are usually regular words, and incomplete words or stop words are less likely to appear. For example, when users search for "incomplete words appear", they are unlikely to enter "now not" or "not finished, the whole" ("," is a stop word). At this time, the conventional phrase query (match_phrase) can achieve the effect of fuzzy search.

[0022] In recent years, more and more netizens like to report problems and make suggestions to the government through the Internet. At present, the government-citizen interaction platform has gathered tens of millions of netizens' message data. Business personnel often need to search the existing message data in the process of handling cases. Therefore, an efficient and accurate retrieval method is of great significance to improving the quality of business handling and improving the efficiency of handling cases. At the same time, it can better provide strong technical support for the government to "listen" to the people's voice, "answer" the people's doubts, and "resolve" the people's worries.

[0023] At present, the platform has gathered about 75 million netizens' message data, including about 74.4 million historical data, about 500,000 in-transit data, and about 100,000 daily active new data. Business personnel often use some keywords for combined query in the process of handling cases, and the message flow also synchronizes the real-time update of data. Such high-frequency data operation will lead to a decrease in retrieval efficiency. The present invention has created a new process method for full-text retrieval, which applies data generation technology to full-text retrieval based on elasticsearch to improve the retrieval efficiency of tens of millions of message data.

[0024] The main points to be protected in this invention are: This solution introduces data generation technology into the full-text retrieval solution based on elasticsearch, and abstracts a generalizable architecture and process.

[0025] Introduce business policy interfaces into data routing (data flows to different generations according to business status) to convert business logic into data logic, thereby achieving business service-oriented and data and process standardization.

[0026] The core technologies that need to be protected are the data generation division rules and data migration business plans.

[0027] Business metadata abstraction: abstract business metadata and map it to data columns. For example, in the embodiment, the core metadata is the completion status and business timestamp. Generational index management: move the new generation index to the middle generation on a daily basis; explicitly merge the index segments of each index during non-working hours.

[0028] The existing conventional technologies usually divide the data into specific dimensions, such as time or channel. However, no matter how they are divided, they cannot effectively solve the situation where data updates and queries exist simultaneously across time periods or channels. Moreover, as time passes and channels expand, after a certain stage, data partitions will expand to the point where maintenance costs skyrocket and performance drops sharply.

[0029] The focus of this solution design is to simplify the division, and the premise of simplification is the following business insights: 1. All data needs to be queried, and the integrity of the data needs to be ensured; 2. The largest amount of data is relatively static data, and the most active data is a small amount of data generated recently; 3. Do not affect performance by causing a large amount of data to be fragmented due to updates to a small amount of data.

[0030] Because the division is simplified, as long as the basic business nature remains unchanged and the above three points are met, there will be no singularity problem in the conventional solution. The data generation division and movement in this solution is based on business metadata rather than a single dimension. It is a dynamic division rather than a static division. It is fundamentally different from the division scheme of the existing technology. The idea of ​​business generation is projected to the actual technology, first projected to the logical index of the middle layer, and then landed at the final index segment file level; seamlessly combined with the multi-index query, segment merging, and alias query provided by elasticsearch.

[0031] Glossary Full-text searching: Full-text retrieval refers to a retrieval method in which a computer indexing program scans every word in an article and creates an index for each word, indicating the number of times and location of the word in the article. When a user queries, the retrieval program searches based on the pre-established index and feeds the search results back to the user.

[0032] Inverted Index: Inverted indexes are derived from the need to find records based on attribute values ​​in practical applications. Each item in this index table includes an attribute value and the address of each record with this attribute value. Since the attribute value is not determined by the record, but the location of the record is determined by the attribute value, it is called an inverted index.

[0033] PR curve (Precision Recall curve): The PR curve is a function graph with P as the horizontal axis and R as the vertical axis. P stands for precision, and R stands for recall. The PR curve represents the relationship between precision and recall. Generally, recall is set as the horizontal axis and precision is set as the vertical axis.

[0034] Recall rate: Recall Rate (also called recall rate) is the ratio of the number of relevant documents retrieved to the number of all relevant documents in the document library.

[0035] Precision rate: Precision is an indicator that measures the signal-to-noise ratio of a retrieval system, that is, the percentage of relevant documents retrieved to all documents retrieved. It is generally expressed as: Precision = (the amount of relevant information retrieved / the total amount of information retrieved) x 100%.

[0036] Fuzzy matching: Enter a complete keyword string, search in the target result set, and return a set of result records containing the keyword. The string and order of the keyword in the result record must be exactly the same as the input condition.

[0037] Index writing: The process of calling elasticsearch's index record addition or modification interface to write index data.

[0038] Merge index: Lucene indexes are composed of multiple segments, each of which is a logical searchable and retrievable unit. If there are too many segments in the index, the search efficiency will be affected. The merging of segments is essentially the process of merging multiple sets of inverted indexes or forward indexes.

[0039] Index alias: Elasticsearch provides the index alias function. An index alias is like a shortcut or soft link that can point to one or more indexes.

Claims

1. Full-text retrieval method based on elasticsearch data generation technology, characterized by The steps include: S1. Vertical division by update frequency: Combined with business characteristics, vertical division is performed by data update frequency; divided into different categories of elasticsearch indexes; different generations are new generation, middle generation, and old generation; update frequency new generation> middle generation> old generation; S2. Move records to corresponding generations according to business through status monitoring or scheduled tasks; S3. Perform index merging operations on the corresponding generational data during non-business hours according to the business; S4. Incrementally update each generation index at regular time according to the business; S5. Create an index alias to point to all generational data; S6. When querying, directly specify the index alias for unified query; S7. After following the above process, compare the amount of fragmented data in the new generation, middle generation, and old generation, and assign the generation with the least amount of fragmented data to the non-business time to perform the index merge operation in step 3.

2. The full-text retrieval method based on elasticsearch data generation technology according to claim 1 is characterized in that In S6, long and short keywords are processed separately: for long keywords, the search target adopts the following word segmentation method, ngram segmentation is performed in the indexing stage, and ngram and wildcard fuzzy matching are combined in the query stage; first, the data is preliminarily screened using the ngram method, and then the screened result set is full-text matched; for short keywords, the conventional phrase query match_phrase is used to implement fuzzy search.

Images