A multi-level database building method, device and storage medium for searching database

Abstract

The present invention relates to the field of big data search technology, and discloses a multi-level database construction method, device and storage medium for retrieval database, wherein the multi-level database construction method for retrieval database includes real-time database construction, which is used to store data information changed from a first time node in the past to a current time node, and the first time node is regularly updated so that the time interval between the first time node and the current time node remains unchanged; the real-time database construction includes: writing the data information of business changes in the retrieval database into a publish-subscribe system in real time; subscribing to the publish-subscribe system to obtain real-time changed data information; generating positive index data according to the real-time changed data information, and pushing it into the real-time database shard. The present invention realizes a multi-level database construction scheme that at least includes a real-time database, which can not only ensure the efficiency and stability of retrieval, but also ensure the timeliness of retrieval, meet the retrieval needs of users, and enhance the retrieval experience of users.

Description

Technical Field

[0001] The present invention relates to the field of big data search technology, and in particular to a multi-level database building method, device and storage medium for a retrieval database. Background Art

[0002] In the era of big data, search engine services based on retrieval databases are widely used. With the increase of data, there are more complex requirements for building retrieval databases.

[0003] The following issues need to be considered when building an existing search database: large-capacity search databases generally have higher search processing efficiency and better search stability, but due to the large capacity of the search database, the update frequency and efficiency of the search database are relatively low. In particular, for the increasing number of search databases that can store billions of data, when building a search database, it is necessary not only to consider the search efficiency and search stability, but also the timeliness of the search, and to meet the search needs of users and improve the user's search experience.

[0004] In view of this, the present invention aims to provide a solution for building a retrieval database that can take into account retrieval efficiency, stability and data update frequency. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes a multi-level database construction method for a retrieval database, including real-time database construction, for storing data information changed from a first time node in the past to a current time node, wherein the first time node is regularly updated so that the time interval between the first time node and the current time node remains unchanged;

[0006] The real-time database construction includes:

[0007] Write the data information of business changes in the retrieved database into the publish-subscribe system in real time;

[0008] Subscribe to the publish-subscribe system to obtain real-time changing data information;

[0009] The forward index data is generated according to the real-time changed data information and pushed into the real-time database shard.

[0010] As an optional implementation manner of the present invention, generating forward index data according to the real-time changed data information and pushing it into the real-time retrieval database shard includes:

[0011] Perform word segmentation processing on data information that changes in real time;

[0012] Call the index_parser module to generate positive index data from the data information after word segmentation;

[0013] The forward index data is pushed into the real-time database shard through the message queue.

[0014] As an optional implementation manner of the present invention, the retrieval database is a test question database, and the real-time database construction includes:

[0015] The test question database receives the test question data written by each business party;

[0016] The changed test data in the test database is written into the distributed publish-subscribe messaging system kafka in real time;

[0017] Subscribe to the real-time changing test data in the kafka;

[0018] Perform word segmentation processing on test data;

[0019] Call the index_parser module to generate positive index data from the test data after word segmentation;

[0020] The forward index data is pushed into the real-time database shard through the message queue.

[0021] As an optional embodiment of the present invention, the multi-level database construction method of the retrieval database of the present invention includes building a base database for storing data information stored before a second time node in the past, the second time node is earlier than the first time node, and the second time node is updated irregularly;

[0022] The basic quantity database construction includes:

[0023] Write the data information of business changes in the retrieved database into the publish-subscribe system in real time;

[0024] Subscribe to the publish-subscribe system to obtain real-time changed data information and store it in a folder;

[0025] Regularly perform word segmentation processing on the data information in the folder and then store it in a new folder;

[0026] Regularly generate positive index data from the data information after word segmentation in the new folder and store it in hbase;

[0027] Hbase reader obtains data from the Hbase from time to time, generates inverted index data, and pushes it into the base database shard.

[0028] As an optional implementation manner of the present invention, the retrieval database is a test question database, and the basic database construction includes:

[0029] The test question database receives the test question data written by each business party;

[0030] The changed test data in the test database is written into the distributed publish-subscribe messaging system kafka in real time;

[0031] The PHP script subscribes to the real-time changing test data in the kafka and stores it in the folder FILE;

[0032] The python script periodically obtains the test data in the folder, performs word segmentation processing, and then stores it in a new folder;

[0033] The index_parser module is executed regularly to generate positive index data for the test data that changed between the first time node and the current time node in the new folder and store it in hbase;

[0034] Hbase reader obtains data from the Hbase irregularly, calls inverter to generate inverted index data, and pushes it into the base database shard.

[0035] As an optional embodiment of the present invention, the multi-level database construction method of the retrieval database of the present invention includes incremental database construction, which is used to store data information between the second time node and the first time node in the past;

[0036] The incremental database construction includes:

[0037] Use hive to regularly obtain the data information of business changes in the database after each update of the first time node;

[0038] Perform word segmentation on the data information of the business change and store it in hbase;

[0039] The hbase reader obtains the data in the hbase, generates inverted index data, and pushes it to the incremental database shard.

[0040] As an optional implementation manner of the present invention, in the basic quantity retrieval database construction, the hbase reader obtains data from the hbase irregularly, including:

[0041] Monitor the data stored in the incremental retrieval database. When the stored data exceeds the set value, the hbase reader in the basic retrieval database construction obtains the data from the hbase.

[0042] The second time node is updated according to the time interval at which the hbase reader obtains data from the hbase.

[0043] As an optional implementation manner of the present invention, the incremental database shards are updated regularly to obtain the latest incremental database shard data;

[0044] The base database shards are updated from time to time, and the latest base database shard data is obtained according to the selected shard host.

[0045] The present invention also provides a multi-level database building device for a retrieval database, including a real-time database building module for storing data information changed from a first time node in the past to a current time node, wherein the first time node is regularly updated so that the time interval between the first time node and the current time node remains unchanged;

[0046] The real-time database building module includes:

[0047] The publish-subscribe module writes and retrieves data information about business changes in the database in real time;

[0048] Real-time database building module, subscription and publishing module to obtain real-time changing data information;

[0049] and an index module, which generates positive index data according to the real-time changed data information and pushes it into the real-time retrieval database shard.

[0050] The present invention also provides a storage medium storing a computer executable program, and when the computer executable program is executed, the multi-level database building method of the retrieval database is implemented.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] The multi-level database construction method of the retrieval database of the present invention includes real-time database construction, and the real-time database is mainly for the retrieval of real-time data, and has a better retrieval effect on real-time data. The real-time database capacity of the real-time database construction of the present embodiment is relatively small, and can have higher update efficiency and update frequency, ensuring the timeliness of the retrieval data. Therefore, the multi-level database construction method of the retrieval database of the present embodiment can combine the real-time database construction with the existing data construction method, which can not only ensure the efficiency and stability of the retrieval, but also ensure the timeliness of the retrieval, meet the user's retrieval needs, and enhance the user's retrieval experience.

[0053] The multi-level database construction method of the retrieval database of the present invention realizes the three-level database construction of the real-time database, the incremental database and the base database. The three databases are actually the division of all data at the current moment, and they cover the data of different time periods respectively. In essence, there is no need to merge the real-time database, the incremental database and the base database, and only needs to be updated regularly. During the retrieval, the data of the real-time database, the incremental database and the base database will not be distinguished, and all will be retrieved, and the user of the data is unaware of the difference between the libraries.

[0054] The multi-level database construction method of the retrieval database of the present invention realizes the three-level database construction of the base database, the incremental database and the real-time database: the base database: persistent database construction, almost immobile, extremely large capacity; the incremental database: efficient database construction scheme, daily update, weekly balance, general capacity; the real-time database: real-time database construction scheme, small capacity, automatic discarding of non-important data. The multi-level database construction method of the retrieval database of the present invention can ensure the efficiency and stability of the retrieval, as well as the timeliness of the retrieval, meet the retrieval needs of users, and enhance the retrieval experience of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A flowchart of a multi-level database building method for searching a database according to the first embodiment of the present invention;

[0056] Figure 2 A schematic diagram of the relationship between the real-time database, the incremental database and the base database in the first embodiment of the present invention;

[0057] Figure 3 A schematic diagram of the workflow of a method for updating data in a retrieval database according to a second embodiment of the present invention;

[0058] Figure 4 A schematic diagram of the structure of the intermediate file item of the basic quantity library according to the second embodiment of the present invention;

[0059] Figure 5 A schematic diagram of the structure of the incremental library intermediate file item' according to the second embodiment of the present invention. DETAILED DESCRIPTION

[0060] To make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be described clearly and completely in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them.

[0061] Therefore, the following detailed description of the embodiments of the present invention is not intended to limit the scope of the invention claimed for protection, but merely represents some embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features and technical solutions in the embodiments may be combined with each other.

[0063] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0064] In the description of the present invention, it should be noted that the orientation or positional relationship indicated by the terms "upper", "lower", etc. is based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the invention product is usually placed when in use, or the orientation or positional relationship commonly understood by those skilled in the art. Such terms are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0065] Embodiment 1

[0066] See also Figure 1 As shown, a multi-level database construction method for retrieving a database includes real-time database construction, which is used to save data information changed from a first time node in the past to a current time node. The first time node is regularly updated so that the time interval between the first time node and the current time node remains unchanged.

[0067] The real-time database construction described in this embodiment includes:

[0068] Write the data information of business changes in the retrieved database into the publish-subscribe system in real time;

[0069] Subscribe to the publish-subscribe system to obtain real-time changing data information;

[0070] The forward index data is generated according to the real-time changed data information and pushed into the real-time database shard.

[0071] The real-time database of this embodiment is used to store data information from the first time node in the past to the current time node. The first time node is generally set to a time relatively close to the current time node to avoid the need to build a database for too much data, which results in a long database building time and the inability to ensure the timeliness of retrieval. In general, the real-time database of this embodiment stores all data at the daily level, such as all data of the most recent day or the most recent few days. Since the capacity of the real-time database of this embodiment is relatively small, in order to ensure the real-time update of the real-time database for real-time data, the first time node of the real-time database of this embodiment is regularly updated so that the time interval between the first time node and the current time node remains unchanged. When the first time node changes, the data originally stored in the real-time database will be deleted to store the data after the changed first time node.

[0072] The real-time database construction of this embodiment generates positive index data according to the real-time changed data information, so that the index establishment of the retrieval database can be carried out more quickly, and the efficiency of building the real-time database is improved; and because the amount of data in the real-time database is small, when the user searches, although it is a positive index, the impact on the retrieval efficiency is not significant.

[0073] The multi-level database construction method of the retrieval database of the present embodiment includes the construction of a real-time database. The real-time database is mainly for the retrieval of real-time data, and has a better retrieval effect on real-time data. The real-time database capacity of the real-time database construction of the present embodiment is relatively small, and can have a higher update efficiency and update frequency, ensuring the timeliness of the retrieval data. Therefore, the multi-level database construction method of the retrieval database of the present embodiment can combine the real-time database construction with the existing data construction method, which can not only ensure the efficiency and stability of the retrieval, but also ensure the timeliness of the retrieval, meet the user's retrieval needs, and enhance the user's retrieval experience.

[0074] Furthermore, in this embodiment, generating forward index data according to the real-time changed data information and pushing it into the real-time retrieval database shard includes:

[0075] Perform word segmentation processing on data information that changes in real time;

[0076] Call the index_parser module to generate positive index data from the data information after word segmentation;

[0077] The forward index data is pushed into the real-time database shard through the message queue.

[0078] The index_parser module of this embodiment is an index parser, which is mainly used for generating positive index data. The positive index data of this embodiment is pushed into the real-time database shard through the message queue. The message queue MQ can realize the decoupling between the generation and storage of index data, and the positive index data can be stored in each real-time database shard in an orderly manner through the message queue MQ.

[0079] As an optional implementation of this embodiment, the retrieval database described in this embodiment is a test question database, and the real-time database construction includes:

[0080] The test question database receives the test question data written by each business party;

[0081] The changed test data in the test database is written into the distributed publish-subscribe messaging system kafka in real time;

[0082] Subscribe to the real-time changing test data in the kafka;

[0083] Perform word segmentation processing on test data;

[0084] Call the index_parser module to generate positive index data from the test data after word segmentation;

[0085] The forward index data is pushed into the real-time database shard through the message queue.

[0086] Specifically, the real-time database construction of this embodiment includes a real-time database construction module realtime_plant, and the real-time database construction module realtime_plant has the following functions:

[0087] (1) Real-time consumption of Kafka data;

[0088] (2) Flow control, controlling the number of databases to be built;

[0089] (3) Data type control, selecting data according to the set data type;

[0090] (3) Word segmentation: perform simple word segmentation on the data that needs to be built into the database.

[0091] Therefore, in this embodiment, the real-time database construction is as follows: each business party will add the topic data that needs to be put online to certain tables of the search database (DB), and the big data team will write the topic change information in the DB into Kafka in real time. The real-time database construction module realtime_plant will monitor the data of Kakfa in real time, perform simple word segmentation, and then call the index_parser module, which will generate the positive row data and finally push the data into the realtime_search segment.

[0092] Further, see Figure 1 As shown, a multi-level database construction method for a retrieval database in this embodiment includes a base database construction, which is used to save data information entered into the database before a second time node in the past. The second time node is earlier than the first time node, and the second time node is updated irregularly.

[0093] The basic quantity database construction of this embodiment includes:

[0094] Write the data information of business changes in the retrieved database into the publish-subscribe system in real time;

[0095] Subscribe to the publish-subscribe system to obtain real-time changed data information and store it in a folder;

[0096] Regularly perform word segmentation processing on the data information in the folder and then store it in a new folder;

[0097] Regularly generate positive index data from the data information after word segmentation in the new folder and store it in hbase;

[0098] Hbase reader obtains data from the Hbase from time to time, generates inverted index data, and pushes it into the base database shard.

[0099] The basic database of this embodiment is built to store data information stored before the second time node in the past, and can store billions of data, ensuring the efficiency and stability of retrieval.

[0100] As an optional implementation of this embodiment, the retrieval database described in this embodiment is a test question database, and the basic database construction includes:

[0101] The test question database receives the test question data written by each business party;

[0102] The changed test data in the test database is written into the distributed publish-subscribe messaging system kafka in real time;

[0103] The PHP script subscribes to the real-time changing test data in the kafka and stores it in the folder FILE;

[0104] The python script periodically obtains the test data in the folder, performs word segmentation processing, and then stores it in a new folder;

[0105] The index_parser module is executed regularly to generate positive index data for the test data that changed between the first time node and the current time node in the new folder and store it in hbase;

[0106] hbase: distributed database;

[0107] Hbase reader obtains data from the Hbase irregularly, calls inverter to generate inverted index data, and pushes it into the base database shard;

[0108] inverter: The core process of the database construction process, mainly used to generate inverted data.

[0109] The consumer in the basic database construction of this embodiment is a PHP script, which is used to consume Kafka data and save the consumed data to a file FILE.

[0110] The plant in the basic database construction of this embodiment is a python script, which is executed regularly and is mainly used to perform simple word segmentation on the data that needs to be built.

[0111] Hbase reader in this embodiment: It is executed irregularly and is used to read Hbase data. It is executed when the amount of data in the add database exceeds a certain amount.

[0112] Therefore, the basic database of this embodiment is built as follows: each business party will add the topic data that needs to be put online to certain tables in the DB, and the big data team will write the topic change information in the DB into kafka in real time. The consumer script writes the topic change information into the FILE in real time. The plant is executed regularly, and writes the topic change information into a new FILE after a simple word segmentation. The index_parser is executed regularly, and the topic change information between the previous time point and this time point is stored in the hbase. The hbase reader is executed irregularly, obtains data from the hbase, and then calls the inverter to create inverted data. The base shard is manually updated from time to time to obtain the latest shard data from a fixed machine.

[0113] Further, see Figure 1 As shown, a multi-level database construction method for a retrieval database of this embodiment includes an incremental database construction for storing data information between a second time node and a first time node in the past. The incremental database of this embodiment can store data information for the time period between the base database and the real-time database, and the base database, the incremental database and the real-time database realize the front coverage of the data in the database up to the current time point. The data volume stored in the incremental database is between the base database and the real-time database, and can be set to the tens of millions level.

[0114] The incremental database construction described in this embodiment includes:

[0115] Use hive to regularly obtain the data information of business changes in the database after each update of the first time node;

[0116] Perform word segmentation on the data information of the business change and store it in hbase;

[0117] The hbase reader obtains the data in the hbase, generates inverted index data, and pushes it to the incremental database shard.

[0118] In this embodiment, the incremental database construction obtains data information of business changes in the retrieval database for a period of time through Hive, and generates inverted index data to ensure the efficiency of retrieval.

[0119] In the incremental database construction of this embodiment:

[0120] Hive plant: This is executed regularly and runs on Hive. It is the core process of the database building process and is mainly used to perform simple word segmentation on the data that needs to be built.

[0121] hbase: distributed database;

[0122] hbase reader: used to read hbase data and send data to index_parser;

[0123] inverter: The core process of the database construction process, mainly used to generate inverted data.

[0124] Incremental database construction in this embodiment: Each business party will add the topic data that needs to be put online to certain tables in the DB. The hive plant module will be executed regularly to obtain the topic change information between the last execution and this execution, and then push the data to hbase. After the execution is completed, the hbase reader will be notified to obtain the data, and then index_parser will be called to create the forward data. After the forward data is created, inverter will be called to create the reverse data. The add shard will be restarted regularly every day to obtain the latest add shard data.

[0125] As an optional implementation of this embodiment, in the basic quantity retrieval database construction, the hbase reader obtains data from the hbase irregularly, including:

[0126] Monitor the data stored in the incremental retrieval database. When the stored data exceeds the set value, the hbase reader in the basic retrieval database construction obtains the data from the hbase.

[0127] The second time node is updated according to the time interval at which the hbase reader obtains data from the hbase.

[0128] As an optional implementation of this embodiment, the incremental database shards are updated regularly to obtain the latest incremental database shard data;

[0129] The base database shards are updated from time to time, and the latest base database shard data is obtained according to the selected shard host.

[0130] See also Figure 2 As shown, the multi-level database construction method of the retrieval database of this embodiment realizes the three-level database construction of the real-time database, the incremental database and the base database. The three databases are actually the division of all data at the current moment, and they cover data in different time periods respectively. In essence, the real-time database, the incremental database and the base database do not need to be merged, but only need to be updated regularly.

[0131] As can be seen from the flow chart, the base database has its own separate database construction process. When the data in the incremental database exceeds a certain amount, the base database will be put online once, that is, the hbase reader module that the base database points to from time to time. At this time, the data in the base database will overwrite the data in the incremental database. In this way, the second time node A in the figure will migrate, and the data in the incremental database will decrease. As the first time node B migrates, the data in the incremental database will gradually increase.

[0132] During retrieval, there will be no distinction between the data in the real-time database, incremental database, and base database, and all data will be retrieved. The data users are unaware of the difference between the databases.

[0133] The multi-level database construction method of the retrieval database of this embodiment realizes the three-level database construction of the base database, the incremental database, and the real-time database: the base database: persistent database construction, almost immobile, and extremely large capacity; the incremental database: efficient database construction plan, daily update, weekly balance, and general capacity; the real-time database: real-time database construction plan, small capacity, and automatic discarding of non-important data. The multi-level database construction method of the retrieval database of the present invention can ensure the efficiency and stability of the retrieval, as well as the timeliness of the retrieval, meet the user's retrieval needs, and enhance the user's retrieval experience.

[0134] This embodiment also provides a multi-level database building device for a search database, including a real-time database building module, for storing data information changed from a first time node in the past to a current time node, wherein the first time node is regularly updated so that the time interval between the first time node and the current time node remains unchanged;

[0135] The real-time database building module includes:

[0136] The publish-subscribe module writes and retrieves data information about business changes in the database in real time;

[0137] Real-time database building module, subscription publishing and subscription system to obtain real-time changing data information;

[0138] and an index module, which generates positive index data according to the real-time changed data information and pushes it into the real-time retrieval database shard.

[0139] The multi-level database building device for the retrieval database of the present embodiment includes a real-time database building module, which is mainly for the retrieval and database building of real-time data, and has a better retrieval effect on real-time data. The real-time database capacity of the real-time database building of the present embodiment is relatively small, and it can have a higher update efficiency and update frequency, ensuring the timeliness of the retrieved data. Therefore, the multi-level database building device for the retrieval database of the present embodiment can combine the real-time database building with the existing data building method, which can not only ensure the efficiency and stability of the retrieval, but also ensure the timeliness of the retrieval, meet the user's retrieval needs, and enhance the user's retrieval experience.

[0140] As an optional implementation of this embodiment, the retrieval database described in this embodiment is a test question database, and the real-time database building module includes:

[0141] Kafka, the changed test data in the test database is written into the distributed publish-subscribe messaging system Kafka in real time;

[0142] The real-time database building module realtime_plant subscribes to the real-time changing test data in the kafka and performs word segmentation processing on the test data;

[0143] The index_parser module generates positive index data from the test data after word segmentation;

[0144] The message queue MQ pushes the forward index data into the real-time database shard.

[0145] Specifically, the real-time database building module of this embodiment includes a real-time database building module realtime_plant, and the real-time database building module realtime_plant has the following functions:

[0146] (1) Real-time consumption of Kafka data;

[0147] (2) Flow control, controlling the number of databases to be built;

[0148] (3) Data type control, selecting data according to the set data type;

[0149] (3) Word segmentation: perform simple word segmentation on the data that needs to be built into the database.

[0150] Therefore, in this embodiment, the real-time database construction is as follows: each business party will add the topic data that needs to be put online to certain tables of the search database (DB), and the big data team will write the topic change information in the DB into Kafka in real time. The real-time database construction module realtime_plant will monitor the data of Kakfa in real time, perform simple word segmentation, and then call the index_parser module, which will generate the positive row data and finally push the data into the realtime_search segment.

[0151] Furthermore, a multi-level database building device for a retrieval database in this embodiment includes a basic database building module, which is used to save data information entered into the database before a second time node in the past. The second time node is earlier than the first time node, and the second time node is updated irregularly.

[0152] The basic database building module of this embodiment includes:

[0153] The test question database receives the test question data written by each business party;

[0154] Kafka, the changed test data in the test database is written into the distributed publish-subscribe messaging system Kafka in real time;

[0155] The Consumer module subscribes to the real-time changing test data in the Kafka file and stores it in the FILE folder;

[0156] The Plant module periodically obtains the test data in the folder and performs word segmentation processing, and then stores it in a new folder;

[0157] The index_parser module is executed regularly to generate positive index data for the test data that changed between the first time node and the current time node in the new folder, and store it in hbase;

[0158] hbase: distributed database;

[0159] The hbase reader module obtains data from the hbase from time to time, calls the inverter to generate inverted index data, and pushes it into the base database shard;

[0160] The inverter module is the core process of the database construction process and is mainly used to generate inverted data.

[0161] The consumer module in the basic database construction of this embodiment is a PHP script, which is used to consume Kafka data and save the consumed data to a file FILE.

[0162] The plant module in the basic database construction of this embodiment is a python script, which is executed regularly and is mainly used to perform simple word segmentation on the data that needs to be built.

[0163] Hbase reader in this embodiment: It is executed irregularly and is used to read Hbase data. It is executed when the amount of data in the add database exceeds a certain amount.

[0164] Therefore, the basic database of this embodiment is built as follows: each business party will add the topic data that needs to be put online to certain tables in the DB, and the big data team will write the topic change information in the DB into kafka in real time. The consumer script writes the topic change information into the FILE in real time. The plant is executed regularly, and writes the topic change information into a new FILE after a simple word segmentation. The index_parser is executed regularly, and the topic change information between the previous time point and this time point is stored in the hbase. The hbase reader is executed irregularly, obtains data from the hbase, and then calls the inverter to create inverted data. The base shard is manually updated from time to time to obtain the latest shard data from a fixed machine.

[0165] Furthermore, a multi-level database building device for a retrieval database of the present embodiment includes an incremental database building module for storing data information between a second time node and a first time node in the past. The incremental database of the present embodiment can store data information for a time period between a base database and a real-time database, and the base database, the incremental database and the real-time database realize the front coverage of the data in the database up to the current time point. The data volume stored in the incremental database is between the base database and the real-time database, and can be set to the tens of millions level.

[0166] The incremental database building module in this embodiment includes:

[0167] The Hive plant module retrieves the data information of business changes in the database after each update of the first time node through Hive regularly, performs word segmentation processing on the data information of business changes, and stores it in Hbase;

[0168] The hbase reader module obtains the data in the hbase, generates inverted index data, and pushes it to the incremental database shard.

[0169] In this embodiment, the incremental database construction obtains data information of business changes in the retrieval database for a period of time through Hive, and generates inverted index data to ensure the efficiency of retrieval.

[0170] In the incremental database building module of this embodiment:

[0171] Hive plant module: This module is executed regularly and runs on Hive. It is the core process of the database building process and is mainly used to perform simple word segmentation on the data that needs to be built.

[0172] hbase: distributed database;

[0173] Hbase reader module: used to read Hbase data and send data to index_parser;

[0174] Inverter module: The core process of the database construction process, mainly used for generating inverted data.

[0175] Incremental database construction in this embodiment: Each business party will add the topic data that needs to be put online to certain tables in the DB. The hive plant module will be executed regularly to obtain the topic change information between the last execution and this execution, and then push the data to hbase. After the execution is completed, the hbase reader module will be notified to obtain the data, and then the index_parser module will be called to establish the forward data. After the forward data is established, the inverter module will be called to establish the reverse data. The add shard will be restarted regularly every day to obtain the latest add shard data.

[0176] As an optional implementation of this embodiment, the hbase reader module in the basic quantity retrieval database construction periodically obtains data from the hbase including:

[0177] Monitor the data stored in the incremental retrieval database. When the stored data exceeds the set value, the hbase reader module in the basic retrieval database construction obtains the data from the hbase.

[0178] The second time node is updated according to the time interval at which the hbase reader module obtains data from the hbase.

[0179] As an optional implementation of this embodiment, the incremental database shards are updated regularly to obtain the latest incremental database shard data;

[0180] The base database shards are updated from time to time, and the latest base database shard data is obtained according to the selected shard host.

[0181] The multi-level database construction device of the retrieval database of this embodiment realizes the three-level database construction of the real-time database, the incremental database and the basic database. The three databases are actually the division of all data at the current moment, and they cover data in different time periods. In essence, the real-time database, the incremental database and the basic database do not need to be merged, but only need to be updated regularly.

[0182] As can be seen from the flow chart, the base database has its own separate database construction process. When the data in the incremental database exceeds a certain amount, the base database will be put online once, that is, the hbase reader module that the base database points to from time to time. At this time, the data in the base database will overwrite the data in the incremental database. In this way, the second time node A in the figure will migrate, and the data in the incremental database will decrease. As the first time node B migrates, the data in the incremental database will gradually increase.

[0183] During retrieval, there will be no distinction between the data in the real-time database, incremental database, and base database, and all data will be retrieved. The data users are unaware of the difference between the databases.

[0184] The multi-level database construction device for the retrieval database of this embodiment realizes three-level database construction: basic database: persistent database construction, almost immobile, and extremely large capacity; incremental database: efficient database construction solution, daily updates, weekly balancing, and average capacity; real-time database: real-time database construction solution, small capacity, and automatic discarding of non-important data.

[0185] This embodiment also provides a storage medium storing a computer executable program. When the computer executable program is executed, the multi-level database building method of the retrieval database is implemented.

[0186] The computer-readable storage medium described in the present embodiment may include a data signal propagated in a baseband or as a part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0187] Embodiment 2

[0188] See also Figure 3As shown, this embodiment provides a method for updating data in a retrieval database, wherein the retrieval database includes a base database and an incremental database, and stores base database intermediate files and incremental database intermediate files generated by the base database and the incremental database in the process of indexing.

[0189] Merge the data of the incremental database into the base database, match the intermediate files of the incremental database with the intermediate files of the base database, and balance the index data of the base database.

[0190] In search engines, after data is crawled, an index is created to facilitate retrieval. The index includes forward index and inverted index. Forward index refers to the document ID as the key, and the number of times each keyword appears is recorded in the table. When searching, the word information in each document in the table is scanned until all documents containing the query keyword are found.

[0191] Inverted index (English: Inverted index), also often called reverse index, insertion file or reverse file, is an indexing method used to store the mapping of the storage location of a word in a document or a group of documents under full-text search. It is the most commonly used data structure in document retrieval systems. Through the inverted index, you can quickly get a list of documents containing the word based on the word. The inverted index mainly consists of two parts:

[0192] 1. An index table consisting of different index terms is called a "lexicon". It contains various words and their statistical information (such as frequency of occurrence nDocs), which can be directly used in various ranking algorithms.

[0193] 2. It consists of a collection of documents where each index term appears, as well as information such as the hit position. It is also called a "record table". It is the table generated by the forward index. Of course, this part can be omitted.

[0194] The intermediate files of the base library and the intermediate files of the incremental library of this embodiment can adopt the above-mentioned "record table".

[0195] The data updating method of the retrieval database of the present embodiment generates a base database intermediate file and an incremental database intermediate file respectively during the process of indexing the base database and the incremental database, and persistently stores the base database intermediate file and the incremental database intermediate file. When merging the data of the incremental database into the base database, the incremental database intermediate file is matched with the base database intermediate file, and the index data of the base database is balanced according to the matching result. In this way, a large amount of repeated calculations are avoided in the process of merging data between the incremental database and the base database, the time of data merging is shortened, and the risk is low.

[0196] As an optional implementation of this embodiment, the base database and the incremental database described in this embodiment generate the base database intermediate file and the incremental database intermediate file respectively during the indexing process, including:

[0197] See also Figure 4 As shown, the basic database generates a basic database intermediate file item in the process of establishing an inverted index. The file structure of the basic database intermediate file item is TID: (key1, offset1), (key2, offset2), ..., (keym, offsetm), wherein the TID is a document ID, the key1, key2, ..., keym are search keywords for document segmentation, and the offset1, offset2, ..., offsetm are the positions of the document on the inverted index of each search keyword key.

[0198] See also Figure 5 As shown, the incremental database generates an incremental library intermediate file item' in the process of establishing an inverted index. The file structure of the incremental library intermediate file item' is TID': (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn'), wherein the TID is a document ID, the key1', key2', ..., keyn' are search keywords for document segmentation, and the offset1', offset2', ..., offsetn' are the positions of the document on the inverted index of each search keyword key.

[0199] Furthermore, in this embodiment, when merging the data of the incremental database into the base database, matching the intermediate file of the incremental database with the intermediate file of the base database, and balancing the index data of the base database include:

[0200] Match the TID' in the incremental library intermediate file item' with the TID in the base library intermediate file item;

[0201] After a successful match, the postings chain of the search keyword key contained in TID in the base database is balanced according to (key1, offset1), (key2, offset2), ..., (keym, offsetm) corresponding to TID and (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn') corresponding to TID'.

[0202] In this embodiment, the TID' in the intermediate file item' of the incremental database matches the TID in the intermediate file item of the base database successfully, indicating that the document corresponding to TID' in the incremental database is a document that already exists in the base database (the document corresponding to TID), and is a modification of the existing document in the base database. In this embodiment, according to (key1, offset1), (key2, offset2), ..., (keym, offsetm) corresponding to TID and (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn') corresponding to TID, the inverted index of the search keyword key contained in TID in the base database is balanced, thereby avoiding a large amount of repeated calculations, shortening the time for data merging, and reducing the risk.

[0203] As an optional implementation manner of this embodiment, according to this embodiment, according to (key1, offset1), (key2, offset2), ..., (keym, offsetm) corresponding to TID and (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn') corresponding to TID', data balancing of the postings chain of the search keyword key contained in TID in the base database includes:

[0204] Match the keym of TID with the keyn' of TID'. If they match, replace the postings index node data of the search keyword key contained in TID in the base database with the postings index node data of the search keyword key contained in TID' in the incremental database according to the offsetm of TID.

[0205] For the same search keyword key in the same document, we only need to replace the postings index node data of the search keyword key contained in TID in the base database with the postings index node data of the search keyword key contained in TID' in the incremental database according to the offsetm of TID, thus merging the data in the incremental database into the base database.

[0206] Furthermore, in this embodiment, according to (key1, offset1), (key2, offset2), ..., (keym, offsetm) corresponding to TID and (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn') corresponding to TID', data balancing of the postings chain of the search keyword key contained in TID in the base database includes:

[0207] Match the keym of TID with the keyn' of TID'. If the matches are different, it means that the document corresponding to TID' in the incremental database has changed relative to the base database. Then, according to the offsetm of TID and the offsetn' of TID', add / delete the inverted index of the search keyword key of TID in the base database.

[0208] Specifically, in a method for updating data in a retrieval database of this embodiment:

[0209] If TID' adds a new search keyword keyn' relative to TID, then the postings list of the search keyword key' in the incremental database is added to the base database;

[0210] If TID' deletes the search keyword keym relative to TID, then the postings list of the search keyword keym in the base database is deleted.

[0211] As an optional implementation of this embodiment, when merging the data of the incremental database into the base database, matching the intermediate file of the incremental database with the intermediate file of the base database, and balancing the index data of the base database described in this embodiment include:

[0212] Match the TID' in the incremental library intermediate file item' with the TID in the base library intermediate file item;

[0213] If the match fails, the postings index of the search keyword keyn' contained in the newly added TID' in the incremental database is added to the base database.

[0214] As an optional implementation of this embodiment, the TID of the intermediate file item in the base library in this embodiment also includes a TID weight weight, and the TID' of the intermediate file item' in the incremental library also includes a TID' weight weight.

[0215] Specifically, this embodiment takes a retrieval database capable of realizing question search as an example, and provides a method for updating data in a question retrieval database, wherein the question retrieval database includes a question base database and a question increment database.

[0216] The test question base database and the test question increment database respectively generate a test question base database intermediate file and a test question increment database intermediate file during the indexing process;

[0217] When the data of the incremental question database is merged into the basic question database, the intermediate files of the incremental question database are matched with the intermediate files of the basic question database to balance the index data of the basic question database.

[0218] The data updating method of the test question retrieval database of the present embodiment generates a test question base database intermediate file and a test question incremental database intermediate file respectively during the process of establishing indexes in the test question base database and the test question incremental database, and persistently stores the test question base database intermediate file and the test question incremental database intermediate file; when merging the data of the test question incremental database into the test question base database, the test question incremental database intermediate file is matched with the test question base database intermediate file, and the index data of the test question base database is balanced according to the matching result; in this way, a large amount of repeated calculation is avoided in the process of merging data between the test question incremental database and the test question base database, thereby shortening the time of data merging and reducing the risk.

[0219] As an optional implementation mode of this embodiment, the question base database and the question increment database described in this embodiment respectively generate the question base database intermediate file and the question increment database intermediate file in the process of indexing, including:

[0220] The test question base database generates a test question base library intermediate file item in the process of establishing an inverted index. The file structure of the test question base library intermediate file item is TID: (key1, offset1), (key2, offset2), ..., (keym, offsetm), wherein the TID is the test question ID, the key1, key2, ..., keym are the search keywords for test question segmentation, and the offset1, offset2, ..., offsetm are the positions of the test questions on the inverted chain of each search keyword key.

[0221] The test question incremental database generates a test question incremental library intermediate file item' during the process of establishing an inverted index. The file structure of the test question incremental library intermediate file item' is TID': (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn'), wherein the TID is the test question ID, the key1', key2', ..., keyn' are the search keywords for test question segmentation, and the offset1', offset2', ..., offsetn' are the positions of the test questions on the inverted chain of each search keyword key.

[0222] Furthermore, in this embodiment, when merging the data of the incremental question database into the basic question database, matching the intermediate file of the incremental question database with the intermediate file of the basic question database, and balancing the index data of the basic question database include:

[0223] Match the TID' in the intermediate file item' of the test question increment library with the TID in the intermediate file item of the test question base library;

[0224] After the match is successful, the inverted index of the search keyword key contained in TID in the test question database is balanced according to (key1, offset1), (key2, offset2), ..., (keym, offsetm) corresponding to TID and (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn') corresponding to TID'.

[0225] In this embodiment, the TID' in the intermediate file item' of the test question incremental database matches the TID in the intermediate file item of the test question base database successfully, indicating that the test question corresponding to TID' in the test question incremental database is a test question that already exists in the test question base database (the test question corresponding to TID), and is a modification of the test question already existing in the test question base database. In this embodiment, according to (key1, offset1), (key2, offset2), ..., (keym, offsetm) corresponding to TID and (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn') corresponding to TID, the inverted chain of the search keyword key contained in TID in the test question base database is balanced, thereby avoiding a large amount of repeated calculations, shortening the time for data merging, and having low risk.

[0226] As an optional implementation manner of this embodiment, according to this embodiment, according to (key1, offset1), (key2, offset2), ..., (keym, offsetm) corresponding to TID and (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn') corresponding to TID', data balancing is performed on the postings chain of the search keyword key contained in TID in the test question base database, including:

[0227] Match the keym of TID with the keyn' of TID'. If they match, then replace the postings index node data of the search keyword key contained in TID in the basic question database with the postings index node data of the search keyword key contained in TID' in the incremental question database according to the offsetm of TID.

[0228] For the same search keyword key in the same test question, it is only necessary to replace the postings index node data of the search keyword key contained in TID in the test question base database with the postings index node data of the search keyword key contained in TID' in the test question increment database according to the offsetm of TID, thereby merging the data in the test question increment database into the test question base database.

[0229] Furthermore, in this embodiment, according to (key1, offset1), (key2, offset2), ..., (keym, offsetm) corresponding to TID and (key1', offset1'), (key2', offset2'), ..., (keyn', offsetn') corresponding to TID', data balancing is performed on the postings chain of the search keyword key contained in TID in the basic database of test questions, including:

[0230] Match the keym of TID with the keyn' of TID'. If the matches are different, it means that the test question corresponding to TID' in the test question incremental database has changed relative to the test question base database. Then, according to the offsetm of TID and the offsetn' of TID', add / delete the inverted index of the search keyword key for TID in the test question base database.

[0231] Specifically, in a method for updating data in a test question retrieval database of this embodiment:

[0232] If TID' adds a new search keyword keyn' relative to TID, then the postings index of the search keyword key' in the test question incremental database is added to the test question base database;

[0233] If TID' deletes the search keyword keym relative to TID, then the postings list of the search keyword keym in the test question database will be deleted.

[0234] As an optional implementation mode of this embodiment, when merging the data of the test question increment database into the test question base database, matching the intermediate file of the test question increment database with the intermediate file of the test question base database, and balancing the index data of the test question base database as described in this embodiment includes:

[0235] Match the TID' in the intermediate file item' of the test question increment library with the TID in the intermediate file item of the test question base library;

[0236] If the match fails, the postings index of the search keyword keyn' contained in the newly added TID' in the test question incremental database is added to the test question base database.

[0237] As an optional implementation of this embodiment, the TID of the intermediate file item of the test question base library described in this embodiment also includes a TID weight weight, and the TID' of the intermediate file item' of the test question increment library described in this embodiment also includes a TID' weight weight.

[0238] Through the above description of the implementation mode, it is easy for those skilled in the art to understand that the present invention can be implemented by hardware capable of executing a specific computer program, such as the system of the present invention, and the electronic processing unit, server, client, mobile phone, control unit, processor, etc. contained in the system. The present invention can also be implemented by computer software that executes the method of the present invention, such as control software executed by a microprocessor, an electronic control unit, a client, a server, etc. However, it should be noted that the computer software that executes the method of the present invention is not limited to being executed by one or a specific hardware entity, and it can also be implemented in a distributed manner by unspecified specific hardware. For computer software, the software product can be stored in a computer-readable storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), and can also be distributed and stored on the network, as long as it can enable the electronic device to execute the method according to the present invention.

[0239] The above embodiments are only used to illustrate the present invention and are not intended to limit the technical solutions described in the present invention. Although the present invention has been described in detail with reference to the above embodiments, the present invention is not limited to the above specific implementation methods. Therefore, any modification or equivalent replacement of the present invention; and all technical solutions and improvements thereof that do not depart from the spirit and scope of the invention are included in the scope of the claims of the present invention.

Claims

1. A multi-level database construction method for a retrieval database, characterized in that: It includes building a real-time database for storing data information changed from a first time node in the past to a current time node, wherein the first time node is regularly updated so that the time interval between the first time node and the current time node remains unchanged; The real-time database construction includes: Write the data information of business changes in the retrieved database into the publish-subscribe system in real time; Subscribe to the publish-subscribe system to obtain real-time changing data information; Generate forward index data according to the real-time changed data information, and push it into the real-time database shard; The multi-level database construction method includes a base database construction, which is used to store data information stored before a second time node in the past, the second time node is earlier than the first time node, and the second time node is updated irregularly; The basic quantity database construction includes: Write the data information of business changes in the retrieved database into the publish-subscribe system in real time; Subscribe to the publish-subscribe system to obtain real-time changed data information and store it in a folder; Regularly perform word segmentation processing on the data information in the folder and then store it in a new folder; Regularly generate positive index data from the data information after word segmentation in the new folder and store it in hbase; Hbase reader obtains data from the Hbase irregularly, generates inverted index data, and pushes it into the base database shard; The multi-level database construction method includes incremental database construction, which is used to store data information between the second time node and the first time node in the past; The incremental database construction includes: Use hive to regularly obtain the data information of business changes in the database after each update of the first time node; Perform word segmentation on the data information of the business change and store it in hbase; Hbase reader obtains data from the Hbase, generates inverted index data, and pushes it to the incremental database shard; The real-time database, base database, and incremental database are divisions of all data at the current moment, covering data in different time periods respectively. A separate database construction process is provided for the base database. When the data in the incremental database exceeds a certain amount, the base database is put online once. The base database points to the hbase reader from time to time. The data in the base database covers the data in the incremental database. When the second time node is migrated, the data in the incremental database is reduced. With the migration of the first time node, the data in the incremental database gradually increases.

2. A multi-level database construction method for a retrieval database according to claim 1, characterized in that: The step of generating forward index data according to the real-time changed data information and pushing the data into the real-time retrieval database shard comprises: Perform word segmentation processing on data information that changes in real time; Call the index_parser module to generate positive index data from the data information after word segmentation; The forward index data is pushed into the real-time database shard through the message queue.

3. A multi-level database construction method for a retrieval database according to claim 2, characterized in that: The retrieval database is a test question database, and the real-time database construction includes: The test question database receives the test question data written by each business party; The changed test data in the test database is written into the distributed publish-subscribe messaging system kafka in real time; Subscribe to the real-time changing test data in the kafka; Perform word segmentation processing on test data; Call the index_parser module to generate positive index data from the test data after word segmentation; The forward index data is pushed into the real-time database shard through the message queue.

4. A multi-level database construction method for a retrieval database according to claim 1, characterized in that: The retrieval database is a test question database, and the basic database construction includes: The test question database receives the test question data written by each business party; The changed test data in the test database is written into the distributed publish-subscribe messaging system kafka in real time; The PHP script subscribes to the real-time changing test data in the kafka and stores it in the folder FILE; The python script periodically obtains the test data in the folder, performs word segmentation processing, and then stores it in a new folder; The index_parser module is executed regularly to generate positive index data for the test data that changed between the first time node and the current time node in the new folder and store it in hbase; The hbase reader obtains data from the hbase irregularly, calls the inverter to generate inverted index data, and pushes it into the base database shard.

5. A multi-level database building method for a retrieval database according to claim 1, characterized in that: In the basic retrieval database construction, the hbase reader obtains data from the hbase irregularly, including: Monitor the data stored in the incremental retrieval database. When the stored data exceeds the set value, the hbase reader in the basic retrieval database construction obtains the data from the hbase. The second time node is updated according to the time interval at which the hbase reader obtains data from the hbase.

6. A multi-level database building method for a retrieval database according to claim 1, characterized in that: The incremental database shards are updated regularly to obtain the latest incremental database shard data; The base database shards are updated from time to time, and the latest base database shard data is obtained according to the selected shard host.

7. A multi-level database building device for searching a database, characterized in that: It includes a real-time database building module, which is used to store data information changed from a first time node in the past to a current time node, and the first time node is regularly updated so that the time interval between the first time node and the current time node remains unchanged; The real-time database building module includes: The publish-subscribe module writes and retrieves data information about business changes in the database in real time; Real-time database building module, subscription and publishing module to obtain real-time changing data information; and an index module, which generates positive index data according to the real-time changed data information and pushes it into the real-time retrieval database shard; The multi-level database building device includes a basic database building module, which is used to store data information stored before a second time node in the past, the second time node is earlier than the first time node, and the second time node is updated irregularly; The basic quantity database building module includes: The test question database receives the test question data written by each business party; Kafka, the changed test data in the test database is written into the distributed publish-subscribe messaging system Kafka in real time; The Consumer module subscribes to the real-time changing test data in the Kafka file and stores it in the FILE folder; The Plant module periodically obtains the test data in the folder and performs word segmentation processing, and then stores it in a new folder; The index_parser module is executed regularly to generate positive index data for the test data that changed between the first time node and the current time node in the new folder, and store it in hbase; hbase: distributed database; The hbase reader module obtains data from the hbase from time to time, calls the inverter to generate inverted index data, and pushes it into the base database shard; The inverter module is the core process of the database construction process and is mainly used to generate inverted data. The multi-level database building device includes an incremental database building module for storing data information between a second time node and a first time node in the past; The incremental database building module includes: The Hive plant module retrieves the data information of business changes in the database after each update of the first time node through Hive regularly, performs word segmentation processing on the data information of business changes, and stores it in Hbase; The hbase reader module obtains the data in the hbase, generates inverted index data, and pushes it to the incremental database shard; The real-time database, base database, and incremental database are divisions of all data at the current moment, covering data in different time periods respectively. A separate database construction process is provided for the base database. When the data in the incremental database exceeds a certain amount, the base database is put online once. The base database points to the hbase reader module from time to time. The data in the base database covers the data in the incremental database. When the second time node is migrated, the data in the incremental database is reduced. With the migration of the first time node, the data in the incremental database gradually increases.

8. A storage medium, characterized in that: A computer executable program is stored, and when the computer executable program is executed, a multi-level database building method for a retrieval database as described in any one of claims 1 to 6 is implemented.

Images