A method and system for updating an incremental index

Abstract

The invention discloses an incremental index updating method and a system. The invention determines inverted index incremental data and forward index incremental data by separating and processing total index data. The inverted index incremental data is deleted in the total index data, and the inverted index incremental data is loaded into the incremental index query service module for updating; forward incremental index data are obtained and the forward incremental index data are updated in the full amount of index data. As the incremental index query service module is introduced, separating the inverted index incremental data and the forward index incremental data, storing the inverted index incremental data to the incremental index query service module for updating and indexing, and updating the forward index incremental data in real time are performed, so as to reduce the risk of the server and improve the stability of the server. In addition, it is possible to respond more quicklyto a user's search trends or intentions in order to improve the user's search experience.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a method and system for updating an incremental index. Background technique [0002] With the further development of the network, the amount of search indexes has grown explosively at a rate of 800%. The generation and push of index data every day has become the bottleneck of data processing, and the rapid update and iteration of index data has become a great challenge. [0003] If all global data is calculated, generated, and pushed every day, it will consume a lot of computing resources, bandwidth resources, time costs, and labor costs. After analysis, most of the current 500 million full data has not changed much in the past 7 days, so there is no need to recalculate, generate and push these data every day, only that there are changes in the calculation, generation and push every day Incremental index data, regular reconstruction of the full index, can achieve rapid u...

AI Technical Summary

Problems solved by technology

[0004] The current common method is to merge the incremental index data with the full index online. During data merging, the performance requirements of the server are relatively high, such as the disk, memory and processor of the server. Under the condition of tight resources, the server There are certain risks, and it is not conducive to the stability of the server

Images