A method, device and storage medium for updating data in a search database

Abstract

The present invention relates to the field of big data search technology, and discloses a data updating method, device and storage medium for a retrieval database, wherein the data updating method for a retrieval database comprises a base database and an incremental database, and stores base database intermediate files and incremental database intermediate files respectively generated by the base database and the incremental database in the process of indexing; merges the data of the incremental database into the base database, matches the incremental database intermediate files with the base database intermediate files, and balances the index data of the base database. The data updating method for a retrieval database of the present invention avoids a large amount of repeated calculations in the process of merging data between the incremental database and the base database, shortens the time of data merging, and has low risk.

Description

Technical Field

[0001] The present invention relates to the field of big data search technology, and in particular to a method, device and storage medium for updating data in a retrieval database. Background Art

[0002] The retrieval database generally includes a full database and an incremental database. The base database provides the data basis for the base index service, which is used to query all documents up to a certain date; the incremental database provides the data basis for the incremental index service. Many search engines focus on time-sensitive search. For example, for a hot event that has just happened, the search engine needs to be able to search for the page of the hot event as soon as possible. How can this be achieved? The incremental index service can solve the problem of time-sensitive search.

[0003] The process of merging data between the existing incremental database and the basic database is as follows:

[0004] a) When the incremental database is almost full, the data in the incremental database is equally divided into each base shard of the base database;

[0005] b) The data before each base sharding and the data from the incremental database are combined to rebuild the base database.

[0006] See also Figure 1 As shown, the schematic diagram of the structure of the inverted chain of the inverted index in the existing base database, the search keyword key corresponds to multiple node nodes, each node node contains the document number TID, the document brief and the document weight weight. When the incremental database and the base database are merged, the inverted chain of the key generated by the newly added document in the incremental database is directly added to the inverted chain of the corresponding key in the base database, or the inverted chain of the entire new key is added to the base database; and when the incremental database modifies the existing documents in the base database, it is necessary to re-run the term, invert, balance and store the inverted chain data of the modified document containing the key in the base database. It can be seen from this that the problem with the above-mentioned method of merging data between the incremental database and the base database for document modification is that a large amount of repeated calculations are required in the process of building the base database, building the incremental database and merging data between the incremental database and the base database, which is time-consuming and high-risk.

[0007] In view of this, the present invention is proposed. Summary of the invention

[0008] In order to solve the above problems, the present invention proposes

[0009] A method for updating data in a retrieval database, wherein the retrieval database includes a base database and an incremental database.

[0010] Storing the intermediate files of the base database and the intermediate files of the incremental database respectively generated in the process of indexing the base database and the incremental database;

[0011] Merge the data of the incremental database into the base database, match the intermediate files of the incremental database with the intermediate files of the base database, and balance the index data of the base database.

[0012] As an optional implementation manner of the present invention, the base database and the incremental database respectively generate the base database intermediate file and the incremental database intermediate file during the indexing process, including:

[0013] The basic database generates a basic database intermediate file item in the process of establishing an inverted index. The file structure of the basic database intermediate file item is TID: (key1, offset1), (key2, offset2), ..., (keym, offsetm), wherein the TID is a document ID, the key1, key2, ..., keym are search keywords for document segmentation, and the offset1, offset2, ..., offsetm are the positions of the document on the inverted index of each search keyword key;

[0014] The incremental database generates an incremental library intermediate file item' in the process of establishing an inverted index. The file structure of the incremental library intermediate file item' is TID': (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn'), wherein the TID is a document ID, the key1', key2', ..., keyn' are search keywords for document segmentation, and the offset1', offset2', ..., offsetn' are the positions of the document on the inverted index of each search keyword key.

[0015] As an optional implementation manner of the present invention, when merging the data of the incremental database into the base database, matching the intermediate file of the incremental database with the intermediate file of the base database, and balancing the index data of the base database includes:

[0016] Match the TID' in the incremental library intermediate file item' with the TID in the base library intermediate file item;

[0017] After a successful match, the postings chain of the search keyword key contained in TID in the base database is balanced according to (key1, offset1), (key2, offset2), ..., (keym, offsetm) corresponding to TID and (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn') corresponding to TID'.

[0018] As an optional implementation manner of the present invention, balancing data on the postings chain of the search keyword key contained in TID in the base database according to (key1, offset1), (key2, offset2), ..., (keym, offsetm) corresponding to TID and (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn') corresponding to TID' includes:

[0019] Match the keym of TID with the keyn' of TID'. If they match, replace the postings index node data of the search keyword key contained in TID in the base database with the postings index node data of the search keyword key contained in TID' in the incremental database according to the offsetm of TID.

[0020] As an optional implementation manner of the present invention, balancing data on the postings chain of the search keyword key contained in TID in the base database according to (key1, offset1), (key2, offset2), ..., (keym, offsetm) corresponding to TID and (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn') corresponding to TID' includes:

[0021] Match the keym of TID with the keyn' of TID'. If the matches are not the same, add / delete the inverted index of the search keyword key for TID in the base database according to the offsetm of TID and the offsetn' of TID'.

[0022] As an optional embodiment of the present invention, a method for updating data in a retrieval database of the present invention includes:

[0023] If TID' adds a new search keyword keyn' relative to TID, then the postings list of the search keyword key' in the incremental database is added to the base database;

[0024] If TID' deletes the search keyword keym relative to TID, then the postings list of the search keyword keym in the base database is deleted.

[0025] As an optional implementation manner of the present invention, when merging the data of the incremental database into the base database, matching the intermediate file of the incremental database with the intermediate file of the base database, and balancing the index data of the base database includes:

[0026] Match the TID' in the incremental library intermediate file item' with the TID in the base library intermediate file item;

[0027] If the match fails, the postings index of the search keyword keyn' contained in the newly added TID in the incremental database is added to the base database.

[0028] As an optional implementation manner of the present invention, the TID of the intermediate file item in the base library further includes a TID weight weight, and the TID' of the intermediate file item' in the incremental library further includes a TID' weight weight.

[0029] The present invention also provides a data updating device for a retrieval database, wherein the retrieval database includes a base database and an incremental database, and the data updating device includes:

[0030] An intermediate file output module, wherein the base database and the incremental database respectively output a base database intermediate file and an incremental database intermediate file during the indexing process;

[0031] And a data balancing module, when merging the data of the incremental database into the base database, matches the intermediate files of the incremental database with the intermediate files of the base database to balance the index data of the base database.

[0032] The present invention also provides a storage medium storing a computer executable program, and when the computer executable program is executed, the method for updating data in a retrieval database is implemented.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] The data updating method of the retrieval database of the present invention generates a base database intermediate file and an incremental database intermediate file respectively during the process of establishing indexes in the base database and the incremental database, and persistently stores the base database intermediate file and the incremental database intermediate file. When merging the data of the incremental database into the base database, the incremental database intermediate file is matched with the base database intermediate file, and the index data of the base database is balanced according to the matching result. In this way, a large amount of repeated calculations are avoided in the process of merging data between the incremental database and the base database, the time of data merging is shortened, and the risk is low.

[0035] The TID' in the intermediate file item' of the incremental database of the present invention successfully matches the TID in the intermediate file item of the base database, indicating that the document corresponding to TID' in the incremental database is a document that already exists in the base database (the document corresponding to TID), and is a modification of the document already existing in the base database. In this embodiment, according to (key1, offset1), (key2, offset2), ..., (keym, offsetm) corresponding to TID and (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn') corresponding to TID, the inverted index of the search keyword key contained in TID in the base database is balanced, thereby avoiding a large amount of repeated calculations, shortening the time for data merging, and having low risk.

[0036] For the same search keyword key in the same document, the present invention only needs to replace the postings index node data of the search keyword key contained in TID in the base database with the postings index node data of the search keyword key contained in TID' in the incremental database according to the offsetm of TID, thereby merging the data of the incremental database into the base database. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of the structure of an inverted chain of an inverted index in a base database in the background art;

[0038] Figure 2 A schematic diagram of the workflow of the data updating method for the retrieval database of the present invention;

[0039] Figure 3 A schematic diagram of the structure of the intermediate file item of the basic quantity library of the present invention;

[0040] Figure 4 A schematic diagram of the structure of the incremental library intermediate file item' of the present invention. DETAILED DESCRIPTION

[0041] To make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be described clearly and completely in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them.

[0042] Therefore, the following detailed description of the embodiments of the present invention is not intended to limit the scope of the invention claimed for protection, but merely represents some embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features and technical solutions in the embodiments may be combined with each other.

[0044] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0045] In the description of the present invention, it should be noted that the orientation or positional relationship indicated by the terms "upper", "lower", etc. is based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the invention product is usually placed when in use, or the orientation or positional relationship commonly understood by those skilled in the art. Such terms are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0046] See also Figure 2 As shown, this embodiment provides a method for updating data in a retrieval database, wherein the retrieval database includes a base database and an incremental database, and stores base database intermediate files and incremental database intermediate files generated by the base database and the incremental database in the process of indexing.

[0047] Merge the data of the incremental database into the base database, match the intermediate files of the incremental database with the intermediate files of the base database, and balance the index data of the base database.

[0048] In search engines, after data is crawled, an index is created to facilitate retrieval. The index includes forward index and inverted index. Forward index refers to the document ID as the key, and the number of times each keyword appears is recorded in the table. When searching, the word information in each document in the table is scanned until all documents containing the query keyword are found.

[0049] Inverted index (English: Inverted index), also often called reverse index, insertion file or reverse file, is an indexing method used to store the mapping of the storage location of a word in a document or a group of documents under full-text search. It is the most commonly used data structure in document retrieval systems. Through the inverted index, you can quickly get a list of documents containing the word based on the word. The inverted index mainly consists of two parts:

[0050] 1. An index table consisting of different index terms is called a "lexicon". It contains various words and their statistical information (such as frequency of occurrence nDocs), which can be directly used in various ranking algorithms.

[0051] 2. It consists of a collection of documents where each index term appears, as well as information such as the hit position. It is also called a "record table". It is the table generated by the forward index. Of course, this part can be omitted.

[0052] The intermediate files of the base library and the intermediate files of the incremental library of this embodiment can adopt the above-mentioned "record table".

[0053] The data updating method of the retrieval database of the present embodiment generates a base database intermediate file and an incremental database intermediate file respectively during the process of indexing the base database and the incremental database, and persistently stores the base database intermediate file and the incremental database intermediate file. When merging the data of the incremental database into the base database, the incremental database intermediate file is matched with the base database intermediate file, and the index data of the base database is balanced according to the matching result. In this way, a large amount of repeated calculations are avoided in the process of merging data between the incremental database and the base database, the time of data merging is shortened, and the risk is low.

[0054] As an optional implementation of this embodiment, the base database and the incremental database described in this embodiment generate the base database intermediate file and the incremental database intermediate file respectively during the indexing process, including:

[0055] See also Figure 3 As shown, the basic database generates a basic database intermediate file item in the process of establishing an inverted index. The file structure of the basic database intermediate file item is TID: (key1, offset1), (key2, offset2), ..., (keym, offsetm), wherein the TID is a document ID, the key1, key2, ..., keym are search keywords for document segmentation, and the offset1, offset2, ..., offsetm are the positions of the document on the inverted index of each search keyword key.

[0056] See also Figure 4As shown, the incremental database generates an incremental library intermediate file item' in the process of establishing an inverted index. The file structure of the incremental library intermediate file item' is TID': (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn'), wherein the TID is a document ID, the key1', key2', ..., keyn' are search keywords for document segmentation, and the offset1', offset2', ..., offsetn' are the positions of the document on the inverted index of each search keyword key.

[0057] Furthermore, in this embodiment, when merging the data of the incremental database into the base database, matching the intermediate file of the incremental database with the intermediate file of the base database, and balancing the index data of the base database include:

[0058] Match the TID' in the incremental library intermediate file item' with the TID in the base library intermediate file item;

[0059] After a successful match, the postings chain of the search keyword key contained in TID in the base database is balanced according to (key1, offset1), (key2, offset2), ..., (keym, offsetm) corresponding to TID and (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn') corresponding to TID'.

[0060] In this embodiment, the TID' in the intermediate file item' of the incremental database matches the TID in the intermediate file item of the base database successfully, indicating that the document corresponding to TID' in the incremental database is a document that already exists in the base database (the document corresponding to TID), and is a modification of the existing document in the base database. In this embodiment, according to (key1, offset1), (key2, offset2), ..., (keym, offsetm) corresponding to TID and (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn') corresponding to TID, the inverted index of the search keyword key contained in TID in the base database is balanced, thereby avoiding a large amount of repeated calculations, shortening the time for data merging, and reducing the risk.

[0061] As an optional implementation manner of this embodiment, according to this embodiment, according to (key1, offset1), (key2, offset2), ..., (keym, offsetm) corresponding to TID and (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn') corresponding to TID', data balancing of the postings chain of the search keyword key contained in TID in the base database includes:

[0062] Match the keym of TID with the keyn' of TID'. If they match, replace the postings index node data of the search keyword key contained in TID in the base database with the postings index node data of the search keyword key contained in TID' in the incremental database according to the offsetm of TID.

[0063] For the same search keyword key in the same document, we only need to replace the postings index node data of the search keyword key contained in TID in the base database with the postings index node data of the search keyword key contained in TID' in the incremental database according to the offsetm of TID, thus merging the data in the incremental database into the base database.

[0064] Furthermore, in this embodiment, according to (key1, offset1), (key2, offset2), ..., (keym, offsetm) corresponding to TID and (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn') corresponding to TID', data balancing of the postings chain of the search keyword key contained in TID in the base database includes:

[0065] Match the keym of TID with the keyn' of TID'. If the matches are different, it means that the document corresponding to TID' in the incremental database has changed relative to the base database. Then, according to the offsetm of TID and the offsetn' of TID', add / delete the inverted index of the search keyword key of TID in the base database.

[0066] Specifically, in a method for updating data in a retrieval database of this embodiment:

[0067] If TID' adds a new search keyword keyn' relative to TID, then the postings list of the search keyword key' in the incremental database is added to the base database;

[0068] If TID' deletes the search keyword keym relative to TID, then the postings list of the search keyword keym in the base database is deleted.

[0069] As an optional implementation of this embodiment, when merging the data of the incremental database into the base database, matching the intermediate file of the incremental database with the intermediate file of the base database, and balancing the index data of the base database described in this embodiment include:

[0070] Match the TID' in the incremental library intermediate file item' with the TID in the base library intermediate file item;

[0071] If the match fails, the postings index of the search keyword keyn' contained in the newly added TID' in the incremental database is added to the base database.

[0072] As an optional implementation of this embodiment, the TID of the intermediate file item in the base library in this embodiment also includes a TID weight weight, and the TID' of the intermediate file item' in the incremental library also includes a TID' weight weight.

[0073] Specifically, this embodiment takes a retrieval database capable of realizing question search as an example, and provides a method for updating data in a question retrieval database, wherein the question retrieval database includes a question base database and a question increment database.

[0074] The test question base database and the test question increment database respectively generate a test question base database intermediate file and a test question increment database intermediate file during the indexing process;

[0075] When the data of the incremental question database is merged into the basic question database, the intermediate files of the incremental question database are matched with the intermediate files of the basic question database to balance the index data of the basic question database.

[0076] The data updating method of the test question retrieval database of the present embodiment generates a test question base database intermediate file and a test question incremental database intermediate file respectively during the process of establishing indexes in the test question base database and the test question incremental database, and persistently stores the test question base database intermediate file and the test question incremental database intermediate file; when merging the data of the test question incremental database into the test question base database, the test question incremental database intermediate file is matched with the test question base database intermediate file, and the index data of the test question base database is balanced according to the matching result; in this way, a large amount of repeated calculation is avoided in the process of merging data between the test question incremental database and the test question base database, thereby shortening the time of data merging and reducing the risk.

[0077] As an optional implementation mode of this embodiment, the question base database and the question increment database described in this embodiment respectively generate the question base database intermediate file and the question increment database intermediate file in the process of indexing, including:

[0078] The test question base database generates a test question base library intermediate file item in the process of establishing an inverted index. The file structure of the test question base library intermediate file item is TID: (key1, offset1), (key2, offset2), ..., (keym, offsetm), wherein the TID is the test question ID, the key1, key2, ..., keym are the search keywords for test question segmentation, and the offset1, offset2, ..., offsetm are the positions of the test questions on the inverted chain of each search keyword key.

[0079] The test question incremental database generates a test question incremental library intermediate file item' during the process of establishing an inverted index. The file structure of the test question incremental library intermediate file item' is TID': (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn'), wherein the TID is the test question ID, the key1', key2', ..., keyn' are the search keywords for test question segmentation, and the offset1', offset2', ..., offsetn' are the positions of the test questions on the inverted chain of each search keyword key.

[0080] Furthermore, in this embodiment, when merging the data of the incremental question database into the basic question database, matching the intermediate file of the incremental question database with the intermediate file of the basic question database, and balancing the index data of the basic question database include:

[0081] Match the TID' in the intermediate file item' of the test question increment library with the TID in the intermediate file item of the test question base library;

[0082] After the match is successful, the inverted index of the search keyword key contained in TID in the test question database is balanced according to (key1, offset1), (key2, offset2), ..., (keym, offsetm) corresponding to TID and (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn') corresponding to TID'.

[0083] In this embodiment, the TID' in the intermediate file item' of the test question incremental database matches the TID in the intermediate file item of the test question base database successfully, indicating that the test question corresponding to TID' in the test question incremental database is a test question that already exists in the test question base database (the test question corresponding to TID), and is a modification of the test question already existing in the test question base database. In this embodiment, according to (key1, offset1), (key2, offset2), ..., (keym, offsetm) corresponding to TID and (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn') corresponding to TID, the inverted chain of the search keyword key contained in TID in the test question base database is balanced, thereby avoiding a large amount of repeated calculations, shortening the time for data merging, and having low risk.

[0084] As an optional implementation manner of this embodiment, according to this embodiment, according to (key1, offset1), (key2, offset2), ..., (keym, offsetm) corresponding to TID and (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn') corresponding to TID', data balancing is performed on the postings chain of the search keyword key contained in TID in the test question base database, including:

[0085] Match the keym of TID with the keyn' of TID'. If they match, then replace the postings index node data of the search keyword key contained in TID in the basic question database with the postings index node data of the search keyword key contained in TID' in the incremental question database according to the offsetm of TID.

[0086] For the same search keyword key in the same test question, it is only necessary to replace the postings index node data of the search keyword key contained in TID in the test question base database with the postings index node data of the search keyword key contained in TID' in the test question increment database according to the offsetm of TID, thereby merging the data in the test question increment database into the test question base database.

[0087] Furthermore, in this embodiment, according to (key1, offset1) corresponding to TID,

[0088] (key2, offset2), ..., (keym, offsetm) and (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn') corresponding to TID', balancing the inverted index of the search keyword key contained in TID in the test question database includes:

[0089] Match the keym of TID with the keyn' of TID'. If the matches are different, it means that the test question corresponding to TID' in the test question incremental database has changed relative to the test question base database. Then, according to the offsetm of TID and the offsetn' of TID', add / delete the inverted index of the search keyword key for TID in the test question base database.

[0090] Specifically, in a method for updating data in a test question retrieval database of this embodiment:

[0091] If TID' adds a new search keyword keyn' relative to TID, then the postings index of the search keyword key' in the test question incremental database is added to the test question base database;

[0092] If TID' deletes the search keyword keym relative to TID, then the postings list of the search keyword keym in the test question database will be deleted.

[0093] As an optional implementation mode of this embodiment, when merging the data of the test question increment database into the test question base database, matching the intermediate file of the test question increment database with the intermediate file of the test question base database, and balancing the index data of the test question base database as described in this embodiment includes:

[0094] Match the TID' in the intermediate file item' of the test question increment library with the TID in the intermediate file item of the test question base library;

[0095] If the match fails, the postings index of the search keyword keyn' contained in the newly added TID' in the test question incremental database is added to the test question base database.

[0096] As an optional implementation of this embodiment, the TID of the intermediate file item of the test question base library described in this embodiment also includes a TID weight weight, and the TID' of the intermediate file item' of the test question increment library described in this embodiment also includes a TID' weight weight.

[0097] This embodiment also provides a data updating device for a retrieval database, wherein the retrieval database includes a base database and an incremental database, and the data updating device includes:

[0098] An intermediate file generation module generates a base database intermediate file and an incremental database intermediate file respectively during the indexing process of the base database and the incremental database;

[0099] And a data balancing module, when merging the data of the incremental database into the base database, matches the intermediate files of the incremental database with the intermediate files of the base database to balance the index data of the base database.

[0100] The intermediate file output module of the data updating device for the retrieval database of the present embodiment respectively outputs the intermediate files of the base database and the intermediate files of the incremental database during the process of indexing the base database and the incremental database, and persistently stores the intermediate files of the base database and the intermediate files of the incremental database in the storage module. When merging the data of the incremental database into the base database, the data balancing module matches the intermediate files of the incremental database with the intermediate files of the base database, and balances the index data of the base database according to the matching results. In this way, the intermediate files of the data updating device for the retrieval database of the present embodiment avoid a large amount of repeated calculations during the process of merging data between the incremental database and the base database, shorten the time of data merging, and have low risk.

[0101] As an optional implementation of this embodiment, the intermediate file output module in this embodiment generates the intermediate files of the base database and the intermediate files of the incremental database respectively during the indexing process of the base database and the incremental database, including:

[0102] The intermediate file output module generates a base database intermediate file item during the process of establishing an inverted index in the base database. The file structure of the base database intermediate file item is TID: (key1, offset1), (key2, offset2), ..., (keym, offsetm), where the TID is the document ID, the key1, key2, ..., keym are the search keywords for document segmentation, and the offset1, offset2, ..., offsetm are the positions of the document on the inverted index of each search keyword key.

[0103] The intermediate file generation module generates an incremental library intermediate file item' during the process of establishing an inverted index in the incremental database. The file structure of the incremental library intermediate file item' is TID': (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn'), wherein the TID is a document ID, the key1', key2', ..., keyn' are search keywords for document segmentation, and the offset1', offset2', ..., offsetn' are the positions of the document on the inverted index of each search keyword key.

[0104] Furthermore, when merging the data of the incremental database into the base database, the data balancing module of this embodiment matches the intermediate file of the incremental database with the intermediate file of the base database, and balancing the index data of the base database includes:

[0105] Match the TID' in the incremental library intermediate file item' with the TID in the base library intermediate file item;

[0106] After a successful match, the postings chain of the search keyword key contained in TID in the base database is balanced according to (key1, offset1), (key2, offset2), ..., (keym, offsetm) corresponding to TID and (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn') corresponding to TID'.

[0107] In this embodiment, the TID' in the intermediate file item' of the incremental database matches the TID in the intermediate file item of the base database successfully, indicating that the document corresponding to TID' in the incremental database is a document that already exists in the base database (the document corresponding to TID), and is a modification of the existing document in the base database. In this embodiment, according to (key1, offset1), (key2, offset2), ..., (keym, offsetm) corresponding to TID and (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn') corresponding to TID, the inverted index of the search keyword key contained in TID in the base database is balanced, thereby avoiding a large amount of repeated calculations, shortening the time for data merging, and reducing the risk.

[0108] As an optional implementation manner of this embodiment, the data balancing module of this embodiment performs data balancing on the postings chain of the search keyword key contained in TID in the base database according to (key1, offset1), (key2, offset2), ..., (keym, offsetm) corresponding to TID and (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn') corresponding to TID', including:

[0109] The data balancing module matches the keym of TID with the keyn' of TID'. If they match, the postings index node data of the search keyword key contained in TID in the base database is replaced with the postings index node data of the search keyword key contained in TID' in the incremental database according to the offsetm of TID.

[0110] For the same search keyword key in the same document, we only need to replace the postings index node data of the search keyword key contained in TID in the base database with the postings index node data of the search keyword key contained in TID' in the incremental database according to the offsetm of TID, thus merging the data in the incremental database into the base database.

[0111] Furthermore, the data balancing module of this embodiment performs data balancing on the postings chain of the search keyword key contained in TID in the base database according to (key1, offset1), (key2, offset2), ..., (keym, offsetm) corresponding to TID and (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn') corresponding to TID', including:

[0112] The data balancing module matches the keym of TID with the keyn' of TID'. If the matches are different, it means that the document corresponding to TID' in the incremental database has changed relative to the base database. Then, the inverted index of the search keyword key of TID in the base database is added / deleted according to the offsetm of TID and the offsetn' of TID'.

[0113] Specifically, in a data updating device for searching a database in this embodiment:

[0114] If TID' adds a new search keyword keyn' relative to TID, the data balancing module adds the postings index of the search keyword key' in the incremental database to the base database;

[0115] If TID' deletes the search keyword keym relative to TID, the data balancing module deletes the postings list of the search keyword keym in the base database.

[0116] As an optional implementation of this embodiment, when the data balancing module of this embodiment merges the data of the incremental database into the base database, the incremental database intermediate file is matched with the base database intermediate file, and the index data balancing of the base database includes:

[0117] The data balancing module matches the TID' in the intermediate file item' of the incremental library with the TID in the intermediate file item of the base library;

[0118] If the match fails, the postings index of the search keyword keyn' contained in the newly added TID' in the incremental database is added to the base database.

[0119] As an optional implementation of this embodiment, the TID of the intermediate file item in the base library in this embodiment also includes a TID weight weight, and the TID' of the intermediate file item' in the incremental library also includes a TID' weight weight.

[0120] This embodiment also provides an electronic device, including a processor and a memory, wherein the memory is used to store a computer executable program. When the computer program is executed by the processor, the processor executes the method for updating data in a retrieval database.

[0121] The electronic device is presented in the form of a general-purpose computing device. The processor may be one or more than one and work in coordination. The present invention does not exclude distributed processing, that is, the processor may be dispersed in different physical devices. The electronic device of the present invention is not limited to a single entity, but may also be the sum of multiple physical devices.

[0122] The memory stores a computer executable program, which is usually a machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least part of the steps in the method.

[0123] The memory includes a volatile memory, such as a random access memory unit (RAM) and / or a cache memory unit, and may also be a non-volatile memory, such as a read-only memory unit (ROM).

[0124] It should be understood that the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as display screens, and some electronic devices also include human-computer interaction elements such as buttons, keyboards, etc. As long as the electronic device can execute the computer-readable program in the memory to implement the method of the present invention or at least part of the steps of the method, it can be considered as an electronic device covered by the present invention.

[0125] This embodiment also provides a computer-readable storage medium storing a computer-executable program. When the computer-executable program is executed, the method for updating data in a retrieval database is implemented.

[0126] The computer-readable storage medium described in the present embodiment may include a data signal propagated in a baseband or as a part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0127] Through the above description of the implementation mode, it is easy for those skilled in the art to understand that the present invention can be implemented by hardware capable of executing a specific computer program, such as the system of the present invention, and the electronic processing unit, server, client, mobile phone, control unit, processor, etc. contained in the system. The present invention can also be implemented by computer software that executes the method of the present invention, such as control software executed by a microprocessor, an electronic control unit, a client, a server, etc. However, it should be noted that the computer software that executes the method of the present invention is not limited to being executed by one or a specific hardware entity, and it can also be implemented in a distributed manner by unspecified specific hardware. For computer software, the software product can be stored in a computer-readable storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), and can also be distributed and stored on the network, as long as it can enable the electronic device to execute the method according to the present invention.

[0128] The above embodiments are only used to illustrate the present invention and are not intended to limit the technical solutions described in the present invention. Although the present invention has been described in detail with reference to the above embodiments, the present invention is not limited to the above specific implementation methods. Therefore, any modification or equivalent replacement of the present invention; and all technical solutions and improvements thereof that do not depart from the spirit and scope of the invention are included in the scope of the claims of the present invention.

Claims

1. A method for updating data in a retrieval database, wherein the retrieval database includes a base database and an incremental database, characterized in that: Persistently storing the intermediate files of the base database and the intermediate files of the incremental database respectively generated in the process of indexing the base database and the incremental database; Merge the data of the incremental database into the base database, match the intermediate files of the incremental database with the intermediate files of the base database, and balance the index data of the base database; The base database and the incremental database generate base database intermediate files and incremental database intermediate files respectively during the indexing process, including: The basic database generates a basic database intermediate file item in the process of establishing an inverted index. The file structure of the basic database intermediate file item is TID: (key1, offset1), (key2, offset2), ..., (keym, offsetm), wherein the TID is a document ID, the key1, key2, ..., keym are search keywords for document segmentation, and the offset1, offset2, ..., offsetm are the positions of the document on the inverted index of each search keyword key; The incremental database generates an incremental database intermediate file item' in the process of establishing an inverted index. The file structure of the incremental database intermediate file item' is TID': (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn'), wherein TID' is a document ID, key1', key2', ..., keyn' are search keywords for document segmentation, and offset1', offset2', ..., offsetn' are the positions of the document on the inverted index of each search keyword key; When merging the data of the incremental database into the base database, matching the intermediate file of the incremental database with the intermediate file of the base database, and balancing the index data of the base database includes: Match the TID' in the incremental library intermediate file item' with the TID in the base library intermediate file item; After the match is successful, the postings chain of the search keyword key contained in TID in the base database is balanced according to (key1, offset1), (key2, offset2), ..., (keym, offsetm) corresponding to TID and (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn') corresponding to TID'; The step of balancing the postings index of the search keyword key contained in TID in the base database according to (key1, offset1), (key2, offset2), ..., (keym, offsetm) corresponding to TID and (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn') corresponding to TID' includes: Match the keym of TID with the keyn' of TID'. If they match, replace the postings index node data of the search keyword key contained in TID in the base database with the postings index node data of the search keyword key contained in TID' in the incremental database according to the offsetm of TID.

2. A method for updating data in a retrieval database according to claim 1, characterized in that: The step of balancing the postings index of the search keyword key contained in TID in the base database according to (key1, offset1), (key2, offset2), ..., (keym, offsetm) corresponding to TID and (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn') corresponding to TID' includes: Match the keym of TID with the keyn' of TID'. If the matches are not the same, add / delete the inverted index of the search keyword key for TID in the base database according to the offsetm of TID and the offsetn' of TID'.

3. A method for updating data in a retrieval database according to claim 2, characterized in that: If TID' adds a new search keyword keyn' relative to TID, then the postings list of the search keyword key' in the incremental database is added to the base database; If TID' deletes the search keyword keym relative to TID, then the postings list of the search keyword keym in the base database is deleted.

4. A method for updating data in a retrieval database according to claim 1, characterized in that: The step of merging the data of the incremental database into the base database, matching the intermediate files of the incremental database with the intermediate files of the base database, and balancing the index data of the base database includes: Match the TID' in the incremental library intermediate file item' with the TID in the base library intermediate file item; If the match fails, the postings index of the search keyword keyn' contained in the newly added TID' in the incremental database is added to the base database.

5. A method for updating data in a retrieval database according to claim 1, characterized in that: The TID of the intermediate file item in the base library also includes a TID weight weight, and the TID' of the intermediate file item' in the incremental library also includes a TID' weight weight.

6. A data updating device for searching a database, characterized in that: The retrieval database includes a base database and an incremental database, and the data updating device includes: An intermediate file output module, wherein the base database and the incremental database respectively generate a base database intermediate file and an incremental database intermediate file during the indexing process for persistent storage; and a data balancing module, which matches the intermediate files of the incremental database with the intermediate files of the base database when merging the data of the incremental database into the base database, and balances the index data of the base database; The base database and the incremental database generate base database intermediate files and incremental database intermediate files respectively during the indexing process, including: The intermediate file output module generates a base database intermediate file item in the process of establishing an inverted index in the base database. The file structure of the base database intermediate file item is TID: (key1, offset1), (key2, offset2), ..., (keym, offsetm), wherein the TID is a document ID, the key1, key2, ..., keym are search keywords for document segmentation, and the offset1, offset2, ..., offsetm are the positions of the document on the inverted index of each search keyword key; The intermediate file generation module generates an incremental database intermediate file item' in the process of establishing an inverted index in the incremental database. The file structure of the incremental database intermediate file item' is TID': (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn'), wherein TID' is a document ID, key1', key2', ..., keyn' are search keywords for document segmentation, and offset1', offset2', ..., offsetn' are the positions of the document on the inverted index of each search keyword key; When the data balancing module merges the data of the incremental database into the base database, the incremental database intermediate file is matched with the base database intermediate file, and the index data balancing of the base database includes: Match the TID' in the incremental library intermediate file item' with the TID in the base library intermediate file item; After the match is successful, the postings chain of the search keyword key contained in TID in the base database is balanced according to (key1, offset1), (key2, offset2), ..., (keym, offsetm) corresponding to TID and (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn') corresponding to TID'; The data balancing module performs data balancing on the postings chain of the search keyword key contained in TID in the base database according to (key1, offset1), (key2, offset2), ..., (keym, offsetm) corresponding to TID and (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn') corresponding to TID', including: Match the keym of TID with the keyn' of TID'. If they match, replace the postings index node data of the search keyword key contained in TID in the base database with the postings index node data of the search keyword key contained in TID' in the incremental database according to the offsetm of TID.

7. A storage medium, characterized in that: A computer executable program is stored, and when the computer executable program is executed, a method for updating data in a retrieval database as described in any one of claims 1 to 5 is implemented.

Images