Index updating method, device, equipment and storage medium of question-answering system

Abstract

The present invention relates to the field of big data technology, and discloses an index update method for a question-and-answer system, including: receiving an index update request, and determining a target question-and-answer engine to be updated according to configuration information preset by the question-and-answer system; based on the index update request, setting the target question-and-answer engine to an index update state, and reading the index sequence number preset in Redis to obtain the first index name of the corresponding index; based on the index name, obtaining the corresponding Annoy index file from Redis and loading it into memory; using the index name as a search formula, querying the second index name of the search formula from Elasticsearch, and using the second index name as the latest ES index of the target question-and-answer engine; resetting the index update state of the target question-and-answer engine and determining the next target question-and-answer engine to be updated according to the configuration information. The present invention also discloses an index update device, equipment, and computer-readable storage medium for a question-and-answer system. The present invention provides users with a more accurate index update service to ensure the correct progress of the index update process.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to an index updating method, device, equipment and computer-readable storage medium for a question-answering system. Background Art

[0002] In recent years, with the continuous development of artificial intelligence, in the existing index update technology, the general architecture of the question-answering system in the intelligent dialogue system is mainly literal plus semantic recall. The search type usually uses the Annoy algorithm for large-scale retrieval, and the search type uses Elasticsearch as literal recall, and the results of literal recall and semantic recall are re-arranged to obtain the final result. The search recall of the Annoy algorithm depends on the semantic Annoy index, and the literal recall of the ES index is based on the data to be retrieved that has been stored in Elasticsearch.

[0003] However, because the current index generation relies on offline manual processing, the entire index update process is completed by restarting the service, resulting in excessive cost of index update. Therefore, it is urgent to provide an index update method that does not rely on offline manual processing to solve the above problem. Summary of the invention

[0004] The main purpose of the present invention is to provide an index updating method, device, equipment and computer-readable storage medium for a question-answering system, aiming to solve the technical problem that the current index generation relies on offline manual processing, resulting in excessive update costs.

[0005] To achieve the above object, the present invention provides an index updating method for a question-answering system, the index updating method for the question-answering system comprising the following steps:

[0006] Receiving an index update request, and determining a target question-and-answer engine to be updated based on configuration information preset in the question-and-answer system;

[0007] Based on the index update request, the target question-answering engine is set to an index update state, and the index serial number preset in Redis is read to obtain the first index name of the index corresponding to the index serial number;

[0008] Based on the first index name, obtain the corresponding Annoy index file from Redis and load it into memory;

[0009] Using the first index name as a search formula, querying Elasticsearch for a second index name that matches the search formula, and using the second index name as the latest ES index of the target question-answering engine;

[0010] The index update state of the target question-and-answer engine is reset and the next target question-and-answer engine to be updated is determined based on the configuration information to perform index update.

[0011] Optionally, before the step of receiving the index update request and determining the target question-and-answer engine to be updated based on the configuration information preset in the question-and-answer system, the following steps are included:

[0012] Get the original corpus;

[0013] Collecting and annotating the original corpus, and outputting annotated corpus;

[0014] Verifying the annotated corpus and outputting the verification result;

[0015] Based on the verification result, storing the annotated corpus that meets the eligibility assessment in a corpus management platform;

[0016] Based on a preset interface, calling a question-answering engine to obtain the annotated corpus in the corpus management platform and storing it in Redis;

[0017] Obtain the annotated corpus from Redis, preprocess the annotated corpus, and obtain a semantic vector of the annotated corpus;

[0018] An Annoy index file is generated based on the semantic vector of the annotated corpus.

[0019] Optionally, after generating the Annoy index file based on the semantic vector of the corpus, the method further includes:

[0020] Write the Annoy index file into Redis and save the intermediate corpus used when generating the Annoy index file;

[0021] Based on the Annoy index file and the intermediate corpus, construct the request parameter information of the ES index calling interface;

[0022] Based on the request parameter information, the ES index calling interface is called to generate an ES index;

[0023] The Annoy index file is stored in Redis, and the ES index is stored in Elasticsearch.

[0024] Optionally, the step of receiving an index update request and determining a target question-and-answer engine to be updated based on configuration information preset in the question-and-answer system includes:

[0025] Receive an index update request and obtain the IP list information corresponding to the question-and-answer engine;

[0026] Based on each IP address in the IP list information corresponding to the question and answer engine, each target question and answer engine to be updated is determined.

[0027] Optionally, after the step of resetting the index update state of the target question-and-answer engine and determining the next target question-and-answer engine to be updated based on the configuration information to perform index update, the step further includes:

[0028] If the target question-and-answer engine index update fails, setting the target question-and-answer engine to an index update failure state;

[0029] Based on the health detection mechanism of the Q&A engine, detect whether there is a Q&A engine in the state of index update failure;

[0030] If it exists, resend the index update request to the question-and-answer engine in the index update failure state;

[0031] If the index update times of the target question-and-answer engine exceed the preset times and the index update is still not completed, a reminder email is sent to the preset recipient mailbox.

[0032] Optionally, after the step of sending a reminder email to a preset recipient mailbox if the index update times of the target question-and-answer engine exceed a preset number and the index update is still not completed, the method further includes:

[0033] Based on the IP list information corresponding to the question-and-answer engines, determine whether the index version numbers corresponding to the question-and-answer engines are consistent;

[0034] If the index version numbers corresponding to the question-and-answer engines are inconsistent, the index version numbers corresponding to the question-and-answer engines are counted respectively;

[0035] Based on the index version numbers corresponding to the question and answer engines, the question and answer engines whose corresponding index version numbers are in the minority are determined, and the question and answer engines whose corresponding index version numbers are in the minority are set to an unavailable state.

[0036] Furthermore, the present invention also provides an index updating device for a question-answering system, the index updating device for the question-answering system comprising:

[0037] A receiving module, configured to receive an index update request and determine a target question-and-answer engine to be updated based on configuration information preset in the question-and-answer system;

[0038] The acquisition module sets the target question-answering engine to an index update state based on the index update request, and reads the index serial number preset in Redis to obtain the index name of the latest first index corresponding to the index serial number;

[0039] A loading module, used to obtain the corresponding Annoy index file from Redis based on the first index name and load it into the memory;

[0040] An update module, configured to use the first index name as a search formula, query the second index name of the index from Elasticsearch, and use the second index name as the latest ES index of the target question-answering engine;

[0041] A reset module is used to reset the index update state of the target question-answering engine and determine the next target question-answering engine to be updated based on the configuration information to perform index update.

[0042] The index updating device of the question-answering system further includes:

[0043] A corpus storage module is used to obtain original corpus, collect and annotate the original corpus, and output annotated corpus; verify the annotated corpus and output the verification result, and based on the verification result, store the annotated corpus that meets the qualification assessment in the corpus management platform;

[0044] A calling module, used to call the question-answering engine based on a preset interface to obtain the annotated corpus in the corpus management platform and store it in Redis;

[0045] A preprocessing module, used for obtaining the annotated corpus from Redis, preprocessing the annotated corpus, and obtaining a semantic vector of the annotated corpus;

[0046] Annoy index generation module is used to generate an Annoy index file based on the semantic vector of the corpus.

[0047] The index updating device of the question-answering system further includes:

[0048] A writing module, used to write the Annoy index file into Redis and save the intermediate corpus used when the Annoy index file is generated;

[0049] A construction module, used to construct request parameter information of the ES index calling interface based on the Annoy index file and the intermediate corpus;

[0050] An ES index generation module, used to call the ES index calling interface based on the request parameter information to generate an ES index;

[0051] The index storage module is used to store the Annoy index file in Redis and store the ES index in Elasticsearch.

[0052] Optionally, the receiving module is specifically used for:

[0053] An index update request is received, and the IP list information corresponding to the question and answer engine is obtained; and target question and answer engines to be updated are determined based on the IP addresses in the IP list information corresponding to the question and answer engine.

[0054] Optionally, the update module is further used for:

[0055] When the index update of the target question and answer engine fails, the target question and answer engine is set to an index update failure state; based on the health detection mechanism of the question and answer engine, it is detected whether there is a question and answer engine in an index update failure state; when the question and answer engine is in an index update failure state, an index update request is resent to the question and answer engine in the index update failure state; when the index update times of the target question and answer engine exceed the preset times and the index update is still not completed, a reminder email is sent to the preset recipient mailbox.

[0056] Furthermore, the index updating device of the question-answering system further includes:

[0057] The version number unification module is used to determine whether the index version numbers corresponding to the various question and answer engines are consistent based on the IP list information corresponding to the question and answer engines; if the index version numbers corresponding to the various question and answer engines are inconsistent, the index version numbers corresponding to the various question and answer engines are counted respectively; based on the index version numbers corresponding to the various question and answer engines, the question and answer engines whose corresponding index version numbers are in the minority are determined, and the question and answer engines whose corresponding index version numbers are in the minority are set to an unavailable state.

[0058] Furthermore, to achieve the above-mentioned purpose, the present invention also provides an index update device for a question-and-answer system, the index update device for the question-and-answer system comprising a memory, a processor, and an index update program for the question-and-answer system stored in the memory and executable on the processor, the index update program for the question-and-answer system, when executed by the processor, implements the steps of the index update method for the question-and-answer system as described in any one of the above items.

[0059] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which an index update program of a question-and-answer system is stored, and when the index update program of the question-and-answer system is executed by a processor, the steps of the index update method of the question-and-answer system as described in any of the above items are implemented.

[0060] The present invention determines the target question-and-answer engine to be updated through the configuration information preset in the question-and-answer system. When the target question-and-answer engine receives an index update request, the corresponding target question-and-answer engine is set to an index update state, and the index sequence number preset in Redis is read to obtain the corresponding Annoy index file; the index name of the index file is used as a search formula to query the latest ES index of the corresponding target question-and-answer engine from Elasticsearch. After the index update is completed, the state of the corresponding target question-and-answer engine is reset, and the next question-and-answer engine to be updated is determined according to the configuration information, and the update of all question-and-answer engine indexes is completed. The present invention reduces the current index update's reliance on offline manual processing, ensures the correct execution of the index update process, and further reduces the cost of development and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 A schematic diagram of the structure of the device hardware operating environment involved in the embodiment of the index updating device of the question-answering system of the present invention;

[0062] Figure 2 It is a flowchart of a first embodiment of an index updating method of a question-answering system of the present invention;

[0063] Figure 3 It is a flowchart of a second embodiment of the index updating method of the question-answering system of the present invention;

[0064] Figure 4 A schematic diagram of a flow chart of a third embodiment of an index updating method of a question-answering system according to the present invention;

[0065] Figure 5 for Figure 2 A detailed flow chart of step S110 in an embodiment;

[0066] Figure 6 A flowchart of a fourth embodiment of an index updating method of a question-answering system according to the present invention;

[0067] Figure 7 A flowchart of a fifth embodiment of an index updating method of a question-answering system according to the present invention;

[0068] Figure 8 Schematic diagram of functional modules of an index updating device of a question-answering system according to an embodiment of the present invention.

[0069] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0070] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0071] The invention provides an index updating device for a question-answering system.

[0072] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of the device hardware operating environment involved in the embodiment of the index updating device of the question-answering system of the present invention.

[0073] like Figure 1 As shown, the index update device of the question-answering system may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may optionally include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may optionally be a storage device independent of the aforementioned processor 1001.

[0074] Those skilled in the art will understand that Figure 1 The hardware structure of the index updating device of the question-answering system shown in the figure does not constitute a limitation on the index updating device of the question-answering system, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0075] like Figure 1 As shown, the memory 1005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and an index update program of the question-answering system. The operating system is a program for managing and controlling the index update device and software resources of the question-answering system, and supports the operation of the network communication module, the user interface module, the index update program of the question-answering system, and other programs or software; the network communication module is used to manage and control the network interface 1004; and the user interface module is used to manage and control the user interface 1003.

[0076] exist Figure 1 In the hardware structure of the index update device of the question-and-answer system shown, the network interface 1004 is mainly used to connect to the system background and communicate data with the system background; the user interface 1003 is mainly used to connect to the client (user end) and communicate data with the client; the index update device of the question-and-answer system calls the index update program of the question-and-answer system stored in the memory 1005 through the processor 1001, and executes the operations of the various embodiments of the index update method of the question-and-answer system below.

[0077] Based on the hardware structure of the index updating device of the question-answering system described above, various embodiments of the index updating method of the question-answering system of the present invention are proposed.

[0078] Reference Figure 2 , Figure 2 This is a flow chart of the first embodiment of the index updating method of the question-answering system of the present invention. In this embodiment, the index updating method of the question-answering system includes the following steps:

[0079] Step S110, receiving an index update request, and determining a target question-and-answer engine to be updated based on configuration information preset in the question-and-answer system;

[0080] In this embodiment, index is a database term, and its foreign name is index. In a relational database, an index is a separate, physical storage structure that sorts the values ​​of one or more columns in a database table. It is a collection of values ​​of one or more columns in a table. For example, the role of an index is equivalent to the catalog of a book, and the required content can be quickly found according to the page number in the catalog.

[0081] In this embodiment, the index update request is an index update instruction. According to the configuration information in the question-answering system, the IP information of each question-answering engine is obtained, and the question-answering engine receives the index update request sent by the system and starts to update the index.

[0082] In this embodiment, the question-answering system is mainly divided into four parts: faq-Index, question-answering engine (faq-engine), persistent storage Redis and inverted index Elasticsearch (abbreviated as es). Among them, faq-Index generates indexes online; the question-answering engine (faq-engine) supports online matching indexes; persistent storage Redis saves Annoy indexes; Elasticsearch saves literal indexes.

[0083] Step S120: based on the index update request, setting the target question-answering engine to an index update state, and reading the index number preset in Redis to obtain the first index name of the index corresponding to the index number;

[0084] In this embodiment, when a certain question-answering engine (FAQ-engine) receives an index update request, it will first set itself to the index update state to stop accepting new index update requests and complete the current index update work. When a new index update request is received, a status bit will be set to false. Each time a new index update request comes, this status bit will be checked. If it is false, an HTTP 500 error will be returned.

[0085] In this embodiment, http 500 is an error code of the http protocol, which means an internal error of the server. Further, after the current index update is completed, the status bit will be set to true, and a new index update request will be received to complete the index update.

[0086] In this embodiment, Redis is a high-performance database. When notifying to update the index, faq-index will write the index to Redis and Elasticsearch. There will be an increasing serial number in Redis. After the index is generated, it will be added to the backend of the index according to the remainder obtained by dividing the increasing serial number by 2. For example: if the index name is elife and the serial number is 3, the generated Annoy index name is elife_1, and the index name in es is elife_1. When the index is generated again, the serial number becomes 4, and the generated Annoy index name is elife_0, and the index name in es is elife_0.

[0087] In this embodiment, the preset logic refers to the logic of Faq-index, that is, "the incremental serial number is only used to distinguish two versions, and two values ​​0 and 1 can also be used. When the current value is 0, it is set to 1 after the update request is completed, and it is updated to 0 next time.

[0088] Step S130, based on the first index name, obtain the corresponding Annoy index file from Redis and load it into the memory;

[0089] In this embodiment, the index file is composed of a data file, which is a sequential file with an index. To access records randomly in the file, you need to know the address of the record. For example, a customer wants to check a bank account, and neither the customer nor the cashier knows the address of the customer's record. The customer can only give the cashier his own account number (key). Here, the index file can associate the account number with the record address.

[0090] In this embodiment, the index name generally refers to the name of the index, where the index is named according to certain rules in order to distinguish the indexes of different versions. For example, if the index name is elife and the sequence number is 3, the generated Annoy index name is elife_1, and the index name in es is elife_1. When the index is generated again, the sequence number becomes 4, and the generated Annoy index name is elife_0, and the index name in es is elife_0. According to the index name, the index name is queried from Redis to obtain the corresponding index file.

[0091] Step S140, using the first index name as a search formula, querying Elasticsearch for a second index name that matches the search formula, and using the second index name as the latest ES index of the target question-answering engine;

[0092] In this embodiment, the search formula is the instruction issued by the searcher to the computer, and is also the language of human-computer dialogue. The search formula expresses the searcher's search intention and is usually composed of search terms, logical operators, wildcards, etc.

[0093] Among them, the search term is a concentrated reflection of the searcher's search intent and is also the main component of the search language. The search term is a general term. In subject search, the search term categories include free words, keywords, subject terms, etc. When searching with more than two search terms, the relationship between the terms must be connected with logical operators to express the searcher's search intent. Logical operations in the database usually use "or", "and", and "not" to represent the relationship between search terms. The three logical operators can also be combined according to search requirements to form complex search formulas.

[0094] The search formula contains brackets, which means the combination in the brackets will be calculated first. For example, to find literature on hawthorn food processing, the search formula can be: (hawthorn + red fruit + mountain red) 4 food processing. In the search formula without priority, some search systems perform operations from left to right according to the word order, while some systems use priority to artificially determine the order of operations.

[0095] Wildcards mainly include: truncation, substitution and position operators. Among them, truncation is represented by "" or "?". When searching, truncation is added at different positions of the stem to represent the variable parts of the word, thereby reducing the input of search terms with the same stem and improving the recall rate. At the same time, English vocabulary often has multiple characters in one word due to the difference between British spelling and American spelling, and these changes are only caused by different pronunciations. In order to ensure that the search terms are not missed, a substitution is put in the changed position to replace the changed letters. Furthermore, the position operator is used to indicate the positional relationship between two search terms.

[0096] In this embodiment, the index file is read into the memory through the Annoy interface, and Elasticsearch searches through the corresponding index name to retrieve the new index name of the index file in Elasticsearch, and replaces the new index name as the index name of the index file to complete the index update.

[0097] In this embodiment, the literal meaning of loading is to increase the load capacity. In this embodiment, it refers to the loading of files or information when starting a program, that is, reading the required files and information into the memory.

[0098] In this embodiment, the memory is one of the important components in the computer, also known as storage, and its function is to temporarily store the calculation data in the CPU.

[0099] In this embodiment, Elasticsearch is a distributed, highly scalable, and highly real-time search and data analysis engine.

[0100] Step S150: resetting the index update state of the target question-answering engine and determining the next target question-answering engine to be updated based on the configuration information to perform index update.

[0101] In this embodiment, the target Q&A engine refers to the Q&A engine that has currently received the index update request and completed the index update. When the Q&A engine receives the Q&A update request, in order to complete the update of the current index, it will no longer receive new index update requests, so it will set its own state to the index update state, making the host corresponding to the current Q&A engine in an unavailable state, so as to refuse to receive new index update requests.

[0102] If and only if the current index update is completed, the Q&A engine will return to normal and wait for the next index update request. After the current Q&A engine is updated successfully, according to the IP list information of the Q&A engine in the configuration information, it will notify the other Q&A engines to obtain the latest index from Redis and Elasticsearch, accept the index update request, and complete the index update.

[0103] In this embodiment, reset means resetting, which can be extended to: resetting, and restoring to the original state. After the target question-answering engine completes the index update, the state will be restored from the index update state to the original state to receive a new index update request.

[0104] This embodiment determines the target question-and-answer engine to be updated through the configuration information preset in the question-and-answer system. When the target question-and-answer engine receives an index update request, the corresponding target question-and-answer engine is set to the index update state, and the index sequence number preset in Redis is read to obtain the corresponding Annoy index file; the index name of the index file is used as the search formula to query the latest ES index of the corresponding target question-and-answer engine from Elasticsearch. After the index update is completed, the state of the corresponding target question-and-answer engine is reset, and the next question-and-answer engine to be updated is determined according to the configuration information, and the update of all question-and-answer engine indexes is completed. The present invention reduces the current index update's reliance on offline manual processing, ensures the correct execution of the index update process, and further reduces the cost of development and maintenance.

[0105] Reference Figure 3 , Figure 3 This is a flow chart of a second embodiment of the index updating method of the question-answering system of the present invention. In this embodiment, before the above step S110, the following steps are also included:

[0106] Step S210, obtaining original corpus;

[0107] In this embodiment, the corpus refers to all the collected question-answer pairs, which can be understood as a collection of all historical question-answer records.

[0108] In this embodiment, when there is a need to update the corpus, the original text captured from the relevant website or the text information collected manually is converted into a unified electronic text format to obtain the original corpus.

[0109] Step S220, collecting and annotating the original corpus, and outputting the annotated corpus;

[0110] In this embodiment, the collected original corpus is parsed and processed, entities, relationships and events are selected in turn, and their specific categories are determined and marked. At the same time, the marked content is automatically saved in a unified format according to a predefined encoding method to obtain the corpus after parsing.

[0111] Step S230, verifying the annotated corpus and outputting the verification result;

[0112] In this embodiment, the corpus can only be entered into the corpus management platform when it meets the entry specifications of the corpus management platform. Therefore, before entering, the corpus that has been annotated must be checked for eligibility and the verification results must be output. Among them, the eligibility check of the corpus includes but is not limited to: whether it is repeated, whether it contains special characters, and whether the entry format meets the specifications of the corpus management platform. For example, whether it contains special characters. Here, special characters are processed in a blacklist manner, and only Chinese, English letters, numbers and common punctuation marks are allowed. All other characters are special characters; or, format verification: the corpus entry of the corpus management platform must be in the form of question-answer pairs. Only the form of question-answer pairs can meet the entry requirements.

[0113] Step S240, based on the verification result, storing the annotated corpus that meets the eligibility assessment in a corpus management platform;

[0114] In this embodiment, the corpus management platform is a platform for inputting corpus and managing the input corpus.

[0115] In this embodiment, according to the verification result, if the verification result is qualified, the annotated corpus that meets the qualification assessment is stored in the corpus management platform; if the verification result is unqualified, the annotated corpus is removed.

[0116] This embodiment does not limit the storage location of the annotated corpus, and the corpus can be stored in the corpus management platform or in a local database.

[0117] Step S250, based on a preset interface, calling a question-answering engine to obtain the annotated corpus in the corpus management platform and storing it in Redis;

[0118] In this embodiment, the account interface is the interface of the corpus management platform, through which the annotated corpus can be synchronously transmitted to the question-answering engine. After the annotated corpus of the corpus management platform is entered, the system will send a corpus synchronization instruction to the question-answering engine, and further call a preset interface that can synchronously transmit corpus, receive the annotated corpus of the corpus management platform through the interface, and complete the reading and writing of the corpus by the question-answering system.

[0119] In this embodiment, after the corpus is successfully entered, the index module of the question-answering engine will register the notification update interface address to the corpus management platform when it starts, notify the question-answering engine to synchronize and update the corpus, and store the corpus in Redis. According to the timestamp of the historically pulled corpus, the data is pulled in pages and batches to complete the synchronization of the corpus. The timestamp of the historically pulled corpus refers to the maximum value of the last update time of the last pulled corpus.

[0120] In this embodiment, Redis is a high-performance storage, that is, a database. After the question-answering engine completes the synchronization of the corpus, the corpus is stored in Redis.

[0121] Step S260, obtaining the annotated corpus from Redis, preprocessing the annotated corpus, and obtaining a semantic vector of the annotated corpus;

[0122] In this embodiment, as mentioned above, Redis is a high-performance storage, that is, a database. Updated corpus is obtained from Redis, and the corpus is preprocessed to obtain the semantic vector of the corpus.

[0123] In this embodiment, the semantic vector has two meanings: one is a vector, which is to identify text by a vector; and the semantic vector refers to a vector with a certain meaning, which is mainly manifested in that words with similar semantics are close in distance on the vector. For example, the two sentences [Can I drink water] and [Can I drink water] have the same meaning. In this case, the semantic similarity is high, and the distance on the vector is also close.

[0124] Step S270: Generate an Annoy index file based on the semantic vector of the annotated corpus.

[0125] In this embodiment, according to the semantic vector of the corpus, the dynamic link library of Annoy is called to generate the Annoy index file.

[0126] In this embodiment, a dynamic link library (DLL) is a program module that can be shared by other application programs, and encapsulates some shared routines and resources.

[0127] Reference Figure 4 , Figure 4This is a flow chart of the third embodiment of the index updating method of the question-answering system of the present invention. In this embodiment, after the above step S270, it also includes:

[0128] Step S310, writing the Annoy index file into Redis, and saving the intermediate corpus used when generating the Annoy index file;

[0129] In this embodiment, the generated Annoy index file is stored in Redis, and the Annoy index file will have an increasing serial number in Redis. For example, if the index name is elife and the serial number is 3, the generated Annoy index name is elife_1, and the index name in es is elife_1. When the index is generated again, the serial number becomes 4, and the generated Annoy index name is elife_0, and the index name in es is elife_0.

[0130] In this embodiment, the corpus order used when the Annoy index file is generated is the intermediate corpus.

[0131] In this embodiment, the Annoy algorithm search will only return the position in the Annoy index. For example, if the Annoy index has a total of 100 data items, the Annoy algorithm search will only return the position information, such as 10. In order to find the corresponding text information through this position information, it is necessary to follow the corpus order used when the Annoy index file was generated, that is, the intermediate corpus.

[0132] Step S320: constructing request parameter information of the ES index calling interface based on the Annoy index file and the intermediate corpus;

[0133] In this embodiment, the relevant data that needs to be stored in elasticsearch is supplemented according to the Annoy index file and the corpus order used when the file is generated. Since the Annoy index file contains question-answer pairs, which only have text information, the ES index contains more related information, such as fields such as organization and object.

[0134] In this embodiment, request parameter information of a calling interface of an ES index is constructed, and the interface is called according to the request parameter information to generate an ES index.

[0135] Step S330: Based on the request parameter information, call the ES index calling interface to generate an ES index;

[0136] In this embodiment, the ES index uses elasticsearch (search server) as a database, stores data in elasticsearch according to elasticsearch rules, and then retrieves the stored data set through elasticsearch.

[0137] In this embodiment, a single question-and-answer engine will correspond to multiple knowledge bases, and the entire update process is executed in parallel; that is, the two major parts of updating the corpus and generating the index file are carried out in parallel. Through the message notification mechanism, it is ensured that each process can be carried out sequentially without affecting each other.

[0138] It should be noted that the Elasticsearch index can only be notified after the Annoy index file is successfully updated to ensure the consistency of the data in Elasticsearch and Annoy index.

[0139] Step S340, storing the Annoy index file in Redis, and storing the ES index in Elasticsearch.

[0140] In this embodiment, the ES index uses Elasticsearch as a database, stores data in Elasticsearch according to the storage rules of Elasticsearch, and then retrieves the data set through Elasticsearch.

[0141] In this embodiment, Redis is a high-performance storage device, that is, a database.

[0142] In this embodiment, Elasticsearch is a distributed, highly scalable, and highly real-time search and data analysis engine. Although it is not a database, it is often used as a document database.

[0143] In this embodiment, the Annoy index and the ES index are stored in Redis and Elasticsearch, where the Annoy index file will have an increasing serial number in Redis. For example, if the index name is elife and the serial number is 3, the generated Annoy index name is elife_1, and the index name in ES is elife_1. When the index is generated again, the serial number becomes 4, and the generated Annoy index name is elife_0, and the index name in ES is elife_0.

[0144] In this embodiment, the incremented serial number is only used to distinguish the two versions, and can also use two values ​​0 and 1. When the current value is 0, it is set to 1 after the update request, and is updated to 0 next time. According to the incremented serial number and the same logic as the faq-index module, the latest index name is obtained.

[0145] Reference Figure 5 , Figure 5 for Figure 2Based on the above embodiment, in this embodiment, the above step S110 further includes:

[0146] Step S1101, receiving an index update request and obtaining IP list information corresponding to the question-and-answer engine;

[0147] In this embodiment, since it is a distributed service, there will be multiple question-and-answer engines, and the IP information of each question-and-answer engine is obtained.

[0148] Step S1102: determining target question-and-answer engines to be updated based on the IP addresses in the IP list information corresponding to the question-and-answer engines.

[0149] In this embodiment, after the index file is generated according to the IP information of each question-and-answer engine, the interface of each question-and-answer engine is called in sequence, relying on the health detection mechanism of springcloud. When a request to update the index is received, the incremental serial number in Redis is first read, and then the index name of the latest index is obtained using the same logic as the faq-index module, whether it ends with 0 or 1. The Annoy index file is then obtained from Redis through the index name, saved locally and loaded into memory to complete the update, and the literal matching logic is switched to query the index name from Elasticsearch as the latest index name.

[0150] Reference Figure 6 , Figure 6 This is a flowchart of a fourth embodiment of the index updating method of the question-answering system of the present invention. In this embodiment, after the above step S160, the following steps are further included:

[0151] Step S410: if the target question-and-answer engine index update fails, setting the target question-and-answer engine to an index update failure state;

[0152] In this embodiment, if the index update of the target question and answer engine fails, the target question and answer engine is set to the index update failure state, so as to receive the index update request again to update the index. For example, when question and answer engine No. 1 receives the index update request to update the index, question and answer engine No. 1 will be in the index update state. At this time, question and answer engine No. 1 no longer receives new index update requests. If the index update fails due to a fault, question and answer engine No. 1 is set to the index update failure state, so as to redistribute the index update request to the host corresponding to question and answer engine No. 1.

[0153] In this embodiment, when the update of the question-answer engine fails, the FAQ index will try to update again until the update of the question-answer engine is completed. The interval waiting time of the retry increases with the number of update attempts. For example, when the update of the question-answer engine fails, the FAQ index will try to update again. If the first attempt fails, it will retry the second time after 5 seconds. If the second attempt fails, it will retry again after 10 seconds. If the third attempt fails, it will wait for 15 seconds before retrying the fourth time. The fourth and fifth attempts are separated by 20 seconds. The waiting time of each interval in the whole process is increasing.

[0154] Step S420, based on the health detection mechanism of the question-answering engine, detecting whether there is a question-answering engine in a state of index update failure;

[0155] In this embodiment, the health check of the question-and-answer engine is a detection method that can ensure whether the target question-and-answer engine is available. By detecting the health status of the question-and-answer engine corresponding to each service, it is determined whether there is a question-and-answer engine that is currently in an index update failure state. Among them, the detection methods of the health monitoring mechanism are mainly divided into three types:

[0156] Reporting: How often will each service report whether the current service status is available?

[0157] Active detection: The registration center actively detects whether the service is available at regular intervals.

[0158] Here, detection and reporting are both done through interfaces.

[0159] Another supplementary measure is to judge whether the return result of the business is normal. The method here depends on whether the business is proceeding normally.

[0160] Step S430: if it exists, resend the index update request to the question-and-answer engine in the index update failure state;

[0161] In this embodiment, the registration center will perform a health check on the health of the Q&A engine corresponding to each service, and detect whether there is a Q&A engine that is currently in a state of index update failure. If there is a Q&A engine that is in a state of index update failure, the index update request is resent to the Q&A engine to complete the index update. For example, when it is detected that Q&A engine No. 1 is in a state of update failure, the index update request is resent to Q&A engine No. 1 to continue to complete the index update.

[0162] Step S440: If the index update times of the target question-and-answer engine exceed the preset times and the index update is still not completed, a reminder email is sent to a preset recipient mailbox.

[0163] In this embodiment, if the target Q&A engine fails to update its index, and then receives an index update request again to complete the update, but the number of retries exceeds the preset number and still fails to complete the update, an email is sent to the corresponding person in charge's mailbox to remind the Q&A engine that the index update is abnormal. For example, after the Q&A engine No. 1 fails to update its index, it receives an index update request again to try to complete the update, but the update is still not completed after five retries, and then the update is no longer attempted. The system will directly send an email to the person in charge of the business system and the person in charge of the FAQ engine to remind them that the index update failed and ask the relevant person in charge to check the index update process.

[0164] In this embodiment, when the index update of a certain question-and-answer engine fails, the question-and-answer engine will retry the update of the index. However, as the number of attempts to update increases, the time interval between retries increases. For example, if the first attempt fails, wait 5 seconds before retrying the second time, wait 10 seconds before retrying the second time, wait 15 seconds before retrying the third time, and the fourth and fifth times are 20 seconds apart. The waiting time of each interval in the whole process increases.

[0165] Reference Figure 7 , Figure 7 This is a flowchart of a fifth embodiment of the index updating method of the question-answering system of the present invention. In this embodiment, after the above step S440, the following steps are further included:

[0166] Step S510, based on the IP list information corresponding to the question-and-answer engines, determining whether the index version numbers corresponding to the question-and-answer engines are consistent;

[0167] In this embodiment, the IP list information corresponding to the question and answer engine refers to the logical address corresponding to each question and answer engine.

[0168] In this embodiment, since the system has multiple question-and-answer engines and the updates of each question-and-answer engine are not always successful, there may be inconsistent indexes on multiple FAQ engine machines. At this time, the system will determine the index version corresponding to each current question-and-answer engine and output the determination result.

[0169] In this embodiment, index inconsistency means that some question-and-answer engines use the latest version of the index, while other parts still use the old version of the index due to notification failure. The new and old index versions are inconsistent, and the contents are inconsistent, that is, the index is inconsistent.

[0170] Step S520: if the index version numbers corresponding to the question-and-answer engines are inconsistent, the index version numbers corresponding to the question-and-answer engines are counted respectively;

[0171] In this embodiment, since there are multiple Q&A engines in the system, and not all Q&A engines can successfully complete the index update, this results in some Q&A engines using the new version of the index, while other Q&A engines use the old version of the index, so it is necessary to determine whether the index version corresponding to each Q&A engine is consistent. For example, there are 50 Q&A engines in the system, of which only 35 Q&A engines successfully complete the index update, which results in 35 of the 50 Q&A engines using the new version of the index, while the remaining 15 Q&A engines use the old version of the index, and the index versions corresponding to the 50 Q&A engines are inconsistent.

[0172] Step S530: based on the index version numbers corresponding to the question and answer engines, determine the question and answer engines whose corresponding index version numbers are in the minority, and set the question and answer engines whose corresponding index version numbers are in the minority to an unavailable state.

[0173] In this embodiment, if the index versions corresponding to each question-and-answer engine are inconsistent, the judgment result is used to see which version of the index is in the majority, and then the question-and-answer engine in the minority is set to an unavailable state, thereby removing the question-and-answer engine in the minority from the load through a distributed health detection mechanism.

[0174] In this embodiment, the method of setting the question-answering engine corresponding to the service to an unavailable state is as follows:

[0175] Since all services depend on the framework, the framework has a registration center, and services developed using the framework will be registered in the registration center;

[0176] The registration center will check the health status of the Q&A engine corresponding to each service. When it is detected that a certain Q&A engine is unavailable, the unhealthy Q&A engine in this service will be excluded.

[0177] Relying on the framework to implement calls between services will rely on the registration center to obtain the IP list of the question-and-answer engine of the called service;

[0178] When a question-and-answer engine corresponding to a service is unavailable, other services call this service through the framework. The registration center will only return healthy services to the caller to ensure the normal calling process.

[0179] In this embodiment, there are three detection methods of the health monitoring mechanism:

[0180] (1) Reporting: How often does each service report whether the current service status is available?

[0181] (2) Detection: The registration center actively detects whether the service is available at regular intervals;

[0182] Here, detection and reporting are done through interfaces;

[0183] (3) Determine whether the service is available normally by checking whether the return result of the business is normal.

[0184] Reference Figure 8 , Figure 8 This is a functional module diagram of an embodiment of an index updating device of a question-answering system of the present invention. In this embodiment, the index updating device of the question-answering system includes:

[0185] A receiving module 10, configured to receive an index update request and determine a target question-and-answer engine to be updated based on configuration information preset in the question-and-answer system;

[0186] The acquisition module 20 sets the target question-answering engine to an index update state based on the index update request, and reads the index serial number preset in Redis to obtain the index name of the latest first index corresponding to the index serial number;

[0187] A loading module 30, used to obtain the corresponding Annoy index file from Redis based on the first index name and load it into the memory;

[0188] An updating module 40 is configured to query a second index name of the index from Elasticsearch using the first index name as a search formula, and use the second index name as the latest ES index of the target question-answering engine;

[0189] The reset module 50 resets the index update state of the target question-answering engine and determines the next target question-answering engine to be updated based on the configuration information to perform index update.

[0190] Optionally, in a specific embodiment, the index updating device of the question-answering system further includes:

[0191] The corpus storage module is used to obtain the original corpus, collect and annotate the original corpus, and output the annotated corpus; verify the annotated corpus and output the verification result. Based on the verification result, the annotated corpus that meets the qualification assessment is stored in the corpus management platform.

[0192] A calling module, used to call the question-answering engine based on a preset interface to obtain the annotated corpus in the corpus management platform and store it in Redis;

[0193] A preprocessing module, used for obtaining the annotated corpus from Redis, preprocessing the annotated corpus, and obtaining a semantic vector of the annotated corpus;

[0194] Annoy index file generation module is used to generate an Annoy index file based on the semantic vector of the annotated corpus.

[0195] Optionally, in a specific embodiment, the index updating device of the question-answering system further includes:

[0196] A writing module, used to write the Annoy index file into Redis and save the intermediate corpus used when generating the Annoy index file;

[0197] A calling module, used to construct request parameter information of the ES index calling interface based on the Annoy index file and the intermediate corpus;

[0198] An ES index generation module, used to call the ES index calling interface based on the request parameter information to generate an ES index;

[0199] The index storage module is used to store the Annoy index file in Redis and store the ES index in Elasticsearch.

[0200] Optionally, in a specific embodiment, the receiving module is specifically used for:

[0201] An index update request is received, and IP list information corresponding to the question and answer engine is obtained, and target question and answer engines to be updated are determined based on the IP addresses in the IP list information corresponding to the question and answer engine.

[0202] Furthermore, the index updating device of the question-answering system further includes:

[0203] The sending module is used to set the target question and answer engine to an index update failure state when the index update of the target question and answer engine fails; based on the health detection mechanism of the question and answer engine, detect whether there is a question and answer engine in an index update failure state; when the question and answer engine is in an index update failure state, re-send an index update request to the question and answer engine in an index update failure state; when the index update times of the target question and answer engine exceed the preset times and the index update is still not completed, send a reminder email to a preset recipient mailbox.

[0204] Furthermore, the index updating device of the question-answering system further includes:

[0205] The version number unification module is used to determine whether the index version numbers corresponding to each question and answer engine are consistent based on the IP list information of the question and answer engine; when the index version numbers corresponding to each question and answer engine are inconsistent, the version numbers of the indexes corresponding to each question and answer engine are counted respectively; based on the version numbers of the indexes corresponding to each question and answer engine, the question and answer engine with a minority corresponding index version is determined, and the question and answer engine with a minority index version is set to an unavailable state.

[0206] This embodiment determines the target question-and-answer engine to be updated through the configuration information preset in the question-and-answer system. When the target question-and-answer engine receives an index update request, the corresponding target question-and-answer engine is set to the index update state, and the index sequence number preset in Redis is read to obtain the corresponding Annoy index file; the index name of the index file is used as the search formula to query the latest ES index of the corresponding target question-and-answer engine from Elasticsearch. After the index update is completed, the state of the corresponding target question-and-answer engine is reset, and the next question-and-answer engine to be updated is determined according to the configuration information, and the update of all question-and-answer engine indexes is completed. The present invention reduces the current index update's reliance on offline manual processing, ensures the correct execution of the index update process, and further reduces the cost of development and maintenance.

[0207] Based on the same embodiment description as the index updating method of the question-answering system of the present invention, this embodiment does not elaborate on the embodiment content of the index updating device of the question-answering system.

[0208] The present invention also provides a computer-readable storage medium.

[0209] In this embodiment, the computer-readable storage medium stores an index update program for the question-answering system, and when the index update program for the question-answering system is executed by the processor, the steps of the index update method for the question-answering system described in any of the above embodiments are implemented. The method implemented when the index update program for the question-answering system is executed by the processor can refer to the various embodiments of the index update method for the question-answering system of the present invention, so it will not be described in detail.

[0210] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM) and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0211] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims. All equivalent structures or equivalent process changes made using the contents of the specification and drawings of the present invention, or directly or indirectly used in other related technical fields, are protected by the present invention.

Claims

1. A method for updating an index of a question-answering system, characterized in that: The index updating method of the question-answering system comprises the following steps: Receiving an index update request, and determining a target question-and-answer engine to be updated based on configuration information preset in the question-and-answer system; Based on the index update request, the target question-answering engine is set to an index update state, and the index number preset in Redis is read to obtain the first index name of the index corresponding to the index number; Based on the first index name, obtain the corresponding Annoy index file from Redis and load it into memory; Using the first index name as a search formula, querying Elasticsearch for a second index name that matches the search formula, and using the second index name as the latest ES index of the target question-answering engine. The second index name is the new index name of the Annoy index file in Elasticsearch. The index update state of the target question-and-answer engine is reset and the next target question-and-answer engine to be updated is determined based on the configuration information to perform index update.

2. The index updating method of the question-answering system according to claim 1, characterized in that: Before the step of receiving the index update request and determining the target question-and-answer engine to be updated based on the configuration information preset in the question-and-answer system, the method includes: Get the original corpus; Collecting and annotating the original corpus, and outputting annotated corpus; Verifying the annotated corpus and outputting the verification result; Based on the verification results, the annotated corpus that meets the eligibility assessment is stored in the corpus management platform; Based on a preset interface, calling a question-answering engine to obtain the annotated corpus in the corpus management platform and storing it in Redis; Obtain the annotated corpus from Redis, preprocess the annotated corpus, and obtain a semantic vector of the annotated corpus; An Annoy index file is generated based on the semantic vector of the annotated corpus.

3. The index updating method of the question-answering system according to claim 2, characterized in that: After the step of generating the Annoy index file based on the semantic vector of the annotated corpus, the method further includes: Write the Annoy index file into Redis and save the intermediate corpus used when generating the Annoy index file; Based on the Annoy index file and the intermediate corpus, construct the request parameter information of the ES index calling interface; Based on the request parameter information, the ES index calling interface is called to generate an ES index; The Annoy index file is stored in Redis, and the ES index is stored in Elasticsearch.

4. The index updating method of the question-answering system according to claim 1, characterized in that: The receiving the index update request and determining the target question-and-answer engine to be updated based on the configuration information preset in the question-and-answer system includes: Receive an index update request and obtain the IP list information corresponding to the question-and-answer engine; Based on each IP address in the IP list information corresponding to the question and answer engine, each target question and answer engine to be updated is determined.

5. The index updating method of the question-answering system according to claim 1, characterized in that: After the step of resetting the index update state of the target question-answering engine and determining the next target question-answering engine to be updated based on the configuration information for index update, the method further includes: If the target question-and-answer engine index update fails, setting the target question-and-answer engine to an index update failure state; Based on the health detection mechanism of the Q&A engine, detect whether there is a Q&A engine in the state of index update failure; If it exists, resend the index update request to the question-and-answer engine in the index update failure state; If the index update times of the target question-and-answer engine exceed the preset times and the index update is still not completed, a reminder email is sent to the preset recipient mailbox.

6. The index updating method of the question-answering system according to claim 5, characterized in that: After the step of sending a reminder email to a preset recipient mailbox if the index update times of the target question-and-answer engine exceed the preset times and the index update is still not completed, the method further includes: Based on the IP list information corresponding to the question-and-answer engines, determine whether the index version numbers corresponding to the question-and-answer engines are consistent; If the index version numbers corresponding to the question-and-answer engines are inconsistent, the index version numbers corresponding to the question-and-answer engines are counted respectively; Based on the index version numbers corresponding to the question and answer engines, the question and answer engines whose corresponding index version numbers are in the minority are determined, and the question and answer engines whose corresponding index version numbers are in the minority are set to an unavailable state.

7. An index updating device for a question-answering system, characterized in that: The index updating device of the question-answering system comprises: A receiving module, configured to receive an index update request and determine a target question-and-answer engine to be updated based on configuration information preset in the question-and-answer system; An acquisition module, configured to set the target question-answering engine to an index update state based on the index update request, and read the index serial number preset in Redis to obtain the first index name of the index corresponding to the index serial number; A loading module, used to obtain the corresponding Annoy index file from Redis based on the first index name and load it into the memory; An update module is configured to use the first index name as a search formula, query a second index name matching the search formula from Elasticsearch, and use the second index name as the latest ES index of the target question-answering engine, where the second index name is the new index name of the Annoy index file in Elasticsearch; A reset module is used to reset the index update state of the target question-answering engine and determine the next target question-answering engine to be updated based on the configuration information to perform index update.

8. The index updating device of the question-answering system according to claim 7, characterized in that: The index updating device of the question-answering system further includes: A corpus storage module is used to obtain original corpus, collect and annotate the original corpus, and output annotated corpus; verify the annotated corpus and output the verification result, and based on the verification result, store the annotated corpus that meets the qualification assessment in the corpus management platform; A calling module, used to call the question-answering engine based on a preset interface to obtain the annotated corpus in the corpus management platform and store it in Redis; A preprocessing module, used for obtaining the annotated corpus from Redis, preprocessing the annotated corpus, and obtaining a semantic vector of the annotated corpus; Annoy index file generation module is used to generate an Annoy index file based on the semantic vector of the corpus.

9. An index updating device for a question-answering system, characterized in that: The index update device of the question-and-answer system includes a memory, a processor, and an index update program of the question-and-answer system stored in the memory and executable on the processor. When the index update program of the question-and-answer system is executed by the processor, the steps of the index update method of the question-and-answer system as described in any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an index update program of the question-answering system, and when the index update program of the question-answering system is executed by the processor, the steps of the index update method of the question-answering system according to any one of claims 1 to 6 are implemented.

Images