A synonym search method and device

Abstract

The present application provides a synonym search method and device, the method comprising: obtaining a query statement from a first application system; determining a synonym list corresponding to the first application system from the synonym lists corresponding to several application systems respectively maintained in the synonym cache as a target synonym list; obtaining synonyms and synonym weights corresponding to the search terms from the target synonym list, and generating a new query statement based on the search terms, the synonyms corresponding to the search terms, and the synonym weights; based on the new query statement and the index information corresponding to the first application system, searching for information on the first application system. The present application maintains a synonym list in a synonym cache, so that when establishing an index, it is only necessary to establish index information for each word contained in the word segmenter word library, saving disk space occupied by the index information, and being able to distinguish the importance of the original word and the synonym based on the synonym weight, and improving the recall rate of the search results.

Description

Technical Field

[0001] The present application relates to the field of retrieval, and in particular to a synonym retrieval method and device. Background Art

[0002] With the rapid development of the Internet, the business scenarios of information retrieval are becoming more and more diverse, among which synonym retrieval plays an increasingly important role in information retrieval.

[0003] The existing synonym retrieval method indexes the original word and synonyms in the target document in the full-text retrieval engine at the same time, that is, the indexes of the original word and the synonym point to the same position in the document, so that when the user searches for the original word or the synonym, the document containing the original word can be matched.

[0004] Existing synonym retrieval methods can greatly improve the recall rate of retrieval results and enhance user experience. However, the method of indexing both the original word and the synonym in the target document in the full-text retrieval engine requires the additional storage of a large number of synonym indexes, which makes the disk space consumption proportional to the number of documents containing synonyms, wasting a lot of disk space. In addition, when the existing synonym retrieval method searches based on synonyms and original words, the original word and the synonym have the same weight, which makes it impossible to distinguish the importance of the original word and the synonym. Summary of the invention

[0005] In view of this, the present application provides a synonym search method and device for solving the problems in the prior art that the synonym index wastes disk space and the original word and the synonym have the same weight, which leads to the inability to distinguish the importance of the original word and the synonym. The technical solution is as follows:

[0006] A synonym search method, comprising:

[0007] Obtaining a query statement from the first application system, wherein the query statement includes a search term;

[0008] Determine a synonym list corresponding to the first application system from the synonym lists corresponding to the several application systems respectively maintained in the synonym cache as a target synonym list, wherein the synonym list maintains a correspondence between a target word, a synonym corresponding to the target word, and a synonym weight, and the target word is an original word or a synonym of the original word on the application system;

[0009] Obtain synonyms and synonym weights corresponding to the search terms from the target synonym list, and generate new query statements based on the search terms, the synonyms corresponding to the search terms, and the synonym weights;

[0010] Based on the new query statement and the index information corresponding to the first application system, the information on the first application system is retrieved, wherein the index information corresponding to the first application system includes index information established for each word corresponding to the first application system in the word segmenter word library.

[0011] Optionally, it also includes: updating a synonym list maintained by a synonym cache;

[0012] Updates the synonym list maintained by the synonym cache, including:

[0013] Monitoring whether there is a new synonym task, wherein the new synonym task includes a new word and system indication information, the new word refers to a synonym added to the original word on the second application system, and the second application system is the application system indicated by the system indication information included in the new synonym task;

[0014] If there is a new synonym task, the synonym list corresponding to the second application system in the synonym cache is updated based on the new word and the synonym weight corresponding to the new word.

[0015] Optionally, the method further includes: updating index information corresponding to a plurality of application systems respectively;

[0016] Update the index information corresponding to several application systems, including:

[0017] If there is a new synonym task, determine whether the new word is an unregistered word based on the word segmenter vocabulary;

[0018] If yes, then update the word segmenter vocabulary based on the newly added word to obtain an updated word segmenter vocabulary;

[0019] The index is rebuilt according to the updated word segmenter word library and the index in the original index library corresponding to the second application system to obtain a rebuilt new index corresponding to the second application system.

[0020] Optionally, the word segmenter word library includes a word segmenter word library corresponding to each node in the search engine cluster;

[0021] Update the word segmentation vocabulary based on the newly added words, including:

[0022] Generate a new word library file and a word library update identification file based on the new word, and write the new word into the new word library file;

[0023] Monitor whether the word library update identification file is updated. If so, load the newly added words in the new word library file into the word segmenter word library corresponding to each node of the search engine cluster.

[0024] Optionally, re-indexing is performed based on the updated word segmenter word library and the index in the original index library corresponding to the second application system, including:

[0025] Create a new index library corresponding to the second application system;

[0026] According to the updated word segmenter word library and the index in the original index library corresponding to the second application system, the index is rebuilt in the new index library.

[0027] Optionally, also include:

[0028] If a new index needs to be added during the index rebuilding process, a transitional index library is created, and after the transitional index library is set to the default properties, the new index is written to the transitional index library;

[0029] After the index reconstruction is completed, the new index library is switched to the default attribute, and the newly added index in the transitional index library is written into the new index library, so as to obtain the reconstructed new index contained in the newly added index library.

[0030] A synonym search device includes: a query statement acquisition module, a target synonym list determination module, a query statement generation module and a search module;

[0031] A query statement acquisition module, used to acquire a query statement from the first application system, wherein the query statement includes a search term;

[0032] A target synonym list determination module is used to determine a synonym list corresponding to the first application system from the synonym lists corresponding to the plurality of application systems respectively maintained in the synonym cache as the target synonym list, wherein the synonym list maintains a correspondence between a target word, a synonym corresponding to the target word, and a synonym weight, and the target word is an original word or a synonym of the original word on the application system;

[0033] A query statement generation module is used to obtain synonyms and synonym weights corresponding to the search terms from the target synonym list, and generate a new query statement according to the search terms, the synonyms corresponding to the search terms, and the synonym weights;

[0034] The retrieval module is used to retrieve information on the first application system based on a new query statement and index information corresponding to the first application system, wherein the index information corresponding to the first application system includes index information established for each word corresponding to the first application system in the word segmenter word library.

[0035] Optionally, it also includes: a synonym list update module, used to update the synonym list maintained by the synonym cache;

[0036] A synonym list update module, including: a task monitoring submodule and a synonym list update submodule;

[0037] The task monitoring submodule is used to monitor whether there is a new synonym task, wherein the new synonym task contains a new word and system indication information, the new word refers to a synonym added to the original word on the second application system, and the second application system is the application system indicated by the system indication information contained in the new synonym task;

[0038] The synonym list updating submodule is used to update the synonym list corresponding to the second application system in the synonym cache based on the newly added word and the synonym weight corresponding to the newly added word if there is a synonym adding task.

[0039] Optionally, it also includes: an index information updating module, used to update the index information corresponding to the several application systems respectively;

[0040] An index information updating module, including: an unregistered word judging submodule, a word segmenter word library updating submodule and an index rebuilding submodule;

[0041] The unregistered word judgment submodule is used to judge whether the newly added word is an unregistered word based on the word segmenter word library if there is a synonym addition task;

[0042] A word segmenter word library updating submodule, for updating the word segmenter word library based on the newly added word to obtain an updated word segmenter word library if the newly added word is determined as an unregistered word by the unregistered word judging submodule;

[0043] The index reconstruction submodule is used to rebuild the index according to the updated word segmenter word library and the index in the original index library corresponding to the second application system, so as to obtain a reconstructed new index corresponding to the second application system.

[0044] Optionally, the word segmenter word library includes a word segmenter word library corresponding to each node in the search engine cluster;

[0045] The word segmenter word library update submodule includes: a file generation submodule and a file monitoring submodule;

[0046] A file generation submodule, for generating a new word library file and a word library update identification file based on the new word, and writing the new word into the new word library file;

[0047] The file monitoring submodule is used to monitor whether the word library update identification file is updated. If so, the newly added words in the new word library file are loaded into the word segmenter word library corresponding to each node of the search engine cluster.

[0048] Through the above technical solution, it can be known that the synonym search method provided by the present application first obtains the query statement from the first application system, and then determines the synonym list corresponding to the first application system from the synonym lists corresponding to several application systems respectively maintained by the synonym cache as the target synonym list, and then obtains the synonyms and synonym weights corresponding to the search terms from the target synonym list, and generates a new query statement based on the search terms, the synonyms corresponding to the search terms and the synonym weights, and finally retrieves the information on the first application system based on the new query statement and the index information corresponding to the first application system. The present application can maintain the synonym lists corresponding to several application systems in the synonym cache, so that when establishing an index, it is only necessary to establish index information for each word contained in the word segmenter thesaurus, which greatly saves the disk space occupied by the index information, and the present application simultaneously considers the search terms, the synonyms corresponding to the search terms and the synonym weights during the search, so that the importance of the original word and the synonym can be distinguished based on the synonym weights, and the present application improves the recall rate of the search results. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0050] Figure 1 A flowchart of a synonym search method provided in an embodiment of the present application;

[0051] Figure 2 A schematic diagram of the structure of a synonym search device provided in an embodiment of the present application;

[0052] Figure 3 A schematic diagram of a synonym search architecture with configurable weights provided in an embodiment of the present application;

[0053] Figure 4 A hardware structure block diagram of the synonym retrieval device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] The present application provides a synonym search method, which can be implemented based on a search engine cluster, so that when one or some search engine nodes in the search engine cluster are temporarily unable to provide services, other search engine nodes can also provide synonym search services. In order to enable those skilled in the art to better understand the present application, the synonym search method provided by the present application is described in detail through the following embodiments.

[0056] See also Figure 1 , shows a flow chart of a synonym search method provided in an embodiment of the present application, and the synonym search method may include:

[0057] Step S101: Obtain a query statement from a first application system.

[0058] In the present application, the search engine cluster can provide synonym search services for several application systems, including the first application system. When the user enters a query statement through the first application system, this step can obtain the query statement from the first application system. Here, the query statement includes a search term, and information in the first application system can be searched based on the search term in the query statement.

[0059] It is understandable that when a user inputs a query statement, the first application system may generate a query request based on the query statement. In an optional embodiment, the process of this step may include: obtaining the query request from the first application system, and parsing the query request to parse out a query statement that can be recognized by the underlying search engine.

[0060] Optionally, before parsing the query request, the query request may be subjected to a permission check, including a check of the user's permission to read the index in the specified index library and a check of the synonym search permission. After the permission check passes, the query request may be parsed.

[0061] Optionally, after obtaining the query statement in this step, the query statement can also be verified. The verification here includes verification of the query method, query fields, return fields, highlighted fields, etc. After the verification passes, the following steps can be executed.

[0062] Step S102: Determine the synonym list corresponding to the first application system from the synonym lists corresponding to the several application systems maintained in the synonym cache as the target synonym list.

[0063] In the prior art, if an original word contained in a document has a synonym, it is necessary to store the index of the original word (the original word refers to the word contained in the information on the application system) and the index of the synonym in the index library at the same time. Since the index of the original word and the index of the synonym point to the same position in the document, a large amount of disk space is wasted and the retrieval efficiency is affected to a certain extent.

[0064] To solve this problem, the inventor of this case thought of setting up a synonym cache, in which a synonym list corresponding to several application systems is maintained. In this way, only the index of the original word can be stored on the disk, thereby greatly saving disk space.

[0065] Here, the synonym list maintains the correspondence between the target word, the synonym corresponding to the target word, and the synonym weight. In this step, data can be stored in the synonym cache in the form of key-value pairs, and the target word refers to the word at the key "key" position in the key-value pair, and the synonym corresponding to the target word refers to the word at the value "value" position in the key-value pair.

[0066] Optionally, the target word can be only the original word; preferably, considering that in the key-value pair, the retrieval speed of the "key" position is faster, while it is difficult to obtain the synonyms and weights of the "value" position in the cache, based on this, the target word can be either the original word or a synonym of the original word. This method can ensure that when the application system retrieves the original word or synonym, it can quickly obtain the synonym list corresponding to the application system in the synonym cache, and the synonyms between different application systems do not affect each other.

[0067] An example is given to illustrate the storage method of "the target word can be either the original word or a synonym". Assuming that there are two synonyms for "original word 1" in application system A, namely "synonym 1" and "synonym 2", this step maintains three sets of synonym dictionary mapping relationships and three sets of different weight relationships in the synonym cache. When the target word is "original word 1", the weight of "synonym 1" is 0.5, and the weight of "synonym 2" is 0.3; when the target word is "synonym 1", the weight of "original word 1" is 0.5, and the weight of "synonym 2" is 0.2; when the target word is "synonym 2", the weight of "synonym 1" is 0.4, and the weight of "original word 1" is 0.1. For example, the logical relationship is as follows:

[0068] {"Application System A":[

[0069] {"Original word 1":[{"Synonym 1":"0.5"},{"Synonym 2":"0.3"}]},

[0070] {"Synonym 1":[{"Original word 1":"0.5"},{"Synonym 2":"0.2"}]},

[0071] {"Synonym 2":[{"Synonym 1":"0.4"},{"Original word 1":"0.1"}]}

[0072] ]}

[0073] It should be noted that the above-mentioned values ​​of 0.5, 0.3, 0.2, 0.4, 0.1, etc. are only examples and are not intended to limit the weights of synonyms. In addition, in the synonym list, the weight of the target word is always 1. If the synonym weight is not configured, the synonym weight defaults to 1.

[0074] It should also be noted that the "synonyms" mentioned in the above application include synonyms commonly used in daily life, different words used for the same concept by products in different fields, and synonyms defined by different application systems based on their own working fields and users' search habits.

[0075] Since the synonym cache only needs to maintain the correspondence between the target word, the synonym corresponding to the target word, and the synonym weight, and the number of the target word and the synonym of the target word is relatively limited, the application occupies a relatively small cache space. At the same time, the cache has the advantage of fast query speed, so that the application can query more efficiently and improve the retrieval efficiency.

[0076] Step S103: Obtain synonyms and synonym weights corresponding to the search terms from the target synonym list, and generate a new query statement according to the search terms, the synonyms corresponding to the search terms, and the synonym weights.

[0077] As described above, the "synonyms" provided in this application include both traditional synonyms and different words used for the same concept by products in different fields. It is understandable that searching based only on search terms may result in incomplete search results, or the search results may not be the information the user wants.

[0078] In order to improve the recall rate of search results (generally refers to the completeness rate, that is, the ratio of the amount of information in the search results to the total amount of the index, which is one of the important indicators for search engine evaluation), the search terms and synonyms of the search terms can be used simultaneously for searching. Based on this, the synonyms and synonym weights corresponding to the search terms can be obtained from the target synonym list through this step, and a new query statement can be generated based on the search terms (weight is 1), the synonyms corresponding to the search terms and the synonym weights.

[0079] For example, assuming that the original query statement (i.e., the query statement in step S101) is: query the result of "search term: computer, weight: 1", assuming that in the target synonym list, synonym 1 corresponding to "computer" is "computer", with a weight of 0.5, and synonym 2 is "notebook", with a weight of 0.3, then the new query statement is: query the results of "search keyword: computer, weight: 1", "search keyword: computer, weight: 0.5", and "search keyword: notebook, weight: 0.3" at the same time.

[0080] Step S104: Retrieve information on the first application system based on the new query statement and the index information corresponding to the first application system.

[0081] In this step, the new query statement may be matched with the index information corresponding to the first application system, so as to obtain the information required by the user on the first application system based on the matched index.

[0082] Here, the index information corresponding to the first application system includes index information established for each word corresponding to the first application system in the word segmenter word library; accordingly, the index information corresponding to any application system among the several application systems includes index information established for each word corresponding to the application system in the word segmenter word library. In other words, the word segmenter word library may contain words corresponding to several application systems respectively, and this step can establish index information for each word in the word segmenter word library, so as to obtain index information corresponding to several application systems respectively.

[0083] Since the present application establishes index information based on each word in the word segmenter vocabulary, when searching based on a new query statement, the search results corresponding to the keyword may be different from the search results corresponding to the keyword's synonyms, thereby improving the recall rate of the search results.

[0084] The synonym search method provided by the present application first obtains a query statement from the first application system, then determines the synonym list corresponding to the first application system from the synonym lists corresponding to several application systems respectively maintained in the synonym cache as the target synonym list, then obtains the synonyms and synonym weights corresponding to the search terms from the target synonym list, and generates a new query statement based on the search terms, the synonyms corresponding to the search terms and the synonym weights, and finally retrieves the information on the first application system based on the new query statement and the index information corresponding to the first application system. The present application can maintain the synonym lists corresponding to several application systems in the synonym cache, so that when establishing an index, it is only necessary to establish index information for each word contained in the word segmenter word library, which greatly saves the disk space occupied by the index information, and the present application simultaneously considers the search terms, the synonyms corresponding to the search terms and the synonym weights during the search, so that the importance of the original terms and synonyms can be distinguished based on the synonym weights, and the present application improves the recall rate of the search results.

[0085] An embodiment of the present application describes an update process of a synonym list maintained by a synonym cache, and an update process of index information corresponding to several application systems.

[0086] It is understandable that there may be a need to add synonyms for the original words on an application system. Optionally, this embodiment can generate a synonym addition task when it is necessary to add synonyms for the original words on an application system. When the existence of the synonym addition task is monitored, the newly added synonyms are updated to the synonym cache. That is, based on the newly added synonyms and the corresponding synonym weights, the synonym list corresponding to the application system in the synonym cache is updated.

[0087] Optionally, it is rather cumbersome to generate a synonym addition task and update the corresponding synonym list of the application system every time a new synonym needs to be added, considering that it is rather cumbersome. In order to solve this problem, the present application can maintain a synonym dictionary, which can store the target words maintained by the synonym cache and the corresponding synonyms and synonym weights, and can also store the synonyms that need to be added and the synonym weights corresponding to the new synonyms. After storing the newly added synonyms multiple times in the synonym dictionary, the synonym list is updated in batches based on the multiple newly added synonyms; of course, if necessary, it is also possible to store the newly added synonym only once, and then update the synonym list based on the newly added synonym.

[0088] Specifically, for the sake of ease of description, one or more newly added synonyms are referred to as newly added words, and the application system corresponding to the newly added words is referred to as the second application system (included in the second application system), that is, the newly added words refer to the synonyms newly added to the original words on the second application system.

[0089] Then the process of "updating the synonym list in batches based on the multiple newly added synonyms" in this application may include: monitoring whether there is a synonym addition task, and if there is a synonym addition task, updating the synonym list corresponding to the second application system in the synonym cache based on the newly added words and the synonym weights corresponding to the newly added words. Here, the synonym addition task contains the newly added words and system indication information, and optionally, the newly added words refer to the synonyms newly added to the original words on the second application system, and the second application system is the application system indicated by the system indication information contained in the synonym addition task.

[0090] It is worth noting that, in this embodiment, after the newly added word is updated to the synonym cache, if the newly added word is not an unregistered word, since the index of the newly added word already exists in the index library corresponding to the second application system, the newly added word can take effect directly; if the newly added word is an unregistered word, since there is no index of the newly added word in the index library corresponding to the second application system, the newly added word has not taken effect yet, and the index information corresponding to the second application system needs to be updated before it can take effect, and only after it takes effect can information be retrieved based on the newly added word. Here, unregistered words refer to words that are not included in the word segmenter word library but must be segmented.

[0091] Optionally, the process of updating the index information corresponding to the second application system may include: if there is a new synonym task, determining whether the new word is an unregistered word based on the word segmenter vocabulary; if so, updating the word segmenter vocabulary based on the new word to obtain an updated word segmenter vocabulary; rebuilding the index based on the updated word segmenter vocabulary and the index in the original index library corresponding to the second application system to obtain a reconstructed new index corresponding to the second application system.

[0092] Here, the word segmenter word library refers to the word library maintained inside the word segmenter, which can store original words on several application systems, and, when the original words have synonyms, store the synonyms of the original words.

[0093] It is worth noting that the above embodiment provides a process for updating the index information corresponding to the second application system. The process for updating the index information corresponding to other application systems among the multiple application systems is the same as the process for updating the index information corresponding to the second application system, and this application will not repeat it.

[0094] In addition, it should be noted that the sequence of steps mentioned in the above-mentioned update process of the synonym list maintained by the synonym cache and the update process of the index information corresponding to several application systems are only examples and are not intended to limit the present application.

[0095] For example, in an actual application, the updating process of the synonym list maintained by the synonym cache and the updating process of the index information corresponding to several application systems can be as follows:

[0096] First, monitor whether there is a new synonym task. If not, continue to monitor. If so, determine whether the new word is an unregistered word. If the new word is an unregistered word, update the word segmenter vocabulary based on the new word to obtain an updated word segmenter vocabulary. After updating the word segmenter vocabulary, rebuild the index according to the updated word segmenter vocabulary and the index in the original index library corresponding to the second application system to obtain a reconstructed new index corresponding to the second application system. After rebuilding the index, update the synonym list corresponding to the application system based on the newly added synonyms and the corresponding synonym weights. If the new word is not an unregistered word, skip the steps of updating the word segmenter vocabulary and index reconstruction, and directly execute the step of adding synonyms to the cache.

[0097] The above-mentioned embodiments have introduced that the synonym search method provided by the present application can be implemented based on a search engine cluster. Based on this, the above-mentioned word segmenter vocabulary can include the word segmenter vocabulary corresponding to each node in the search engine cluster.

[0098] Then, in an optional embodiment, the above process of "updating the word segmenter word library based on the newly added word" may include:

[0099] A1. Generate a new word library file and a word library update identification file based on the new word, and write the new word into the new word library file.

[0100] Optionally, after writing the newly added words into the newly added word library file, this step may store the newly added word library file in the designated word segmenter word library directory.

[0101] Optionally, in this step, when it is necessary to update the word segmenter word library based on the newly added word, a new synonym word library task may be first created, and then a new word library file and a word library update identification file may be generated based on the newly added word.

[0102] A2. Monitor whether the word library update identification file is updated. If so, load the newly added words in the new word library file into the word segmenter word library corresponding to each node of the search engine cluster.

[0103] In this step, the word segmenter of each node in the search engine cluster can start a listening thread to listen to whether the word library update identification file is updated. If it is (that is, A1 currently generates a word library update identification file based on the new words), it means that there are new words. Then the new words in the new word library file can be loaded into the word segmenter word library corresponding to each node of the search engine cluster, so as to facilitate the subsequent index reconstruction based on the words in the updated word segmenter word library.

[0104] It should be noted that this step loads the newly added words in the newly added word library file into the word segmenter word libraries corresponding to each node of the search engine cluster, so that the word segmenter word libraries corresponding to each node of the search engine cluster remain consistent.

[0105] Optionally, based on A1 and A2 above, taking into account that loading failure may occur when loading the new words in the new vocabulary file into the word segmenter vocabulary corresponding to each node, based on this, after loading the new words, each node in the search engine cluster returns the loading result. If the loading result returned by any node indicates a loading failure, the update task status is failed. If each node in the search engine cluster returns a loading result indicating a successful loading, the update task status is successful.

[0106] Optionally, when a node returns a loading result indicating a loading failure, the word segmenter vocabulary corresponding to the node can also be reloaded with the new words. If the loading still fails after a set number of retries, it is considered that the word segmenter vocabulary of the node has failed to load the new words.

[0107] The following introduces the process of "rebuilding the index based on the updated word segmenter vocabulary and the index in the original index library corresponding to the second application system".

[0108] Optionally, the process of this embodiment may include the following steps:

[0109] B1. Create a new index library corresponding to the second application system.

[0110] Those skilled in the art should understand that after loading new words into the word segmenter vocabulary, it is not possible to simply create an index for the new words in the original index library corresponding to the second application system, but it is necessary to rebuild the index. Therefore, it is necessary to first create a new index library corresponding to the second application system through this step to store the rebuilt new index.

[0111] Optionally, when index reconstruction is required, you can first generate a reindexing task and then create a new index library.

[0112] B2. Rebuild the index in the new index library according to the updated word segmenter vocabulary and the index in the original index library corresponding to the second application system.

[0113] Optionally, after rebuilding the index in the new index library in this step, the original index library corresponding to the second application system may be closed.

[0114] The process of index reconstruction in this step is the same as that in the prior art and will not be described in detail here.

[0115] If there is no new index operation during the execution of the index reconstruction task, the index reconstruction task is completed through B1 and B2; if there is a new index operation during the execution of the index reconstruction task, the new index can be written into the new index library through the following B3 and B4.

[0116] B3. If a new index needs to be added during the index reconstruction process, a transitional index library is created, and after the transitional index library is set to the default attribute, the new index is written into the transitional index library.

[0117] Considering that in the prior art, when a new index operation is performed during the execution of a reindexing task, the new index will be written into the original index library. If the location where the new index is written has been reindexed, the new index cannot be written into the new index library, resulting in inconsistency between the new and old index libraries. To avoid this situation, the prior art usually directly shuts down the new index service during the index reconstruction process.

[0118] Based on this, the inventor of this case thought of creating a transition index library and setting the transition index library as the default attribute (external indexes can be written to the index library only when the index library is the default attribute). At this time, the index can be read through the transition index library and the original index library, and the index can be written through the transition index library.

[0119] Then, if a new index is needed during the index reconstruction process, the new index can be written into the transitional index library.

[0120] B4. After the index reconstruction is completed, the new index library is switched to the default attribute, and the newly added index in the transitional index library is written into the new index library, so as to obtain the reconstructed new index contained in the newly added index library.

[0121] In this step, after the index reconstruction is completed, the new index library can be switched to the default attribute. At this time, the index is read through the transition index library and the new index library, and the index is written through the new index library. After that, the newly added index in the transition index library can be written to the new index library to complete the entire index reconstruction task. Optionally, the task status can be updated afterwards.

[0122] Optionally, after this step is completed, the transitional index library can be deleted.

[0123] It can be seen that this embodiment can realize the re-indexing of the index library without shutting down the newly added index service, and can ensure the integrity and consistency of the indexes in the new and old index libraries after the re-indexing.

[0124] The embodiment of the present application also provides a synonym search device. The synonym search device provided in the embodiment of the present application is described below. The synonym search device described below and the synonym search method described above can be referenced to each other.

[0125] See also Figure 2 , which shows a schematic diagram of the structure of the synonym search device provided in the embodiment of the present application, such as Figure 2 As shown, the synonym search device may include: a query statement acquisition module 201 , a target synonym list determination module 202 , a query statement generation module 203 and a search module 204 .

[0126] The query statement acquisition module 201 is used to acquire a query statement from the first application system, wherein the query statement includes a search term.

[0127] The target synonym list determination module 202 is used to determine the synonym list corresponding to the first application system from the synonym lists corresponding to several application systems maintained in the synonym cache as the target synonym list, wherein the synonym list maintains the correspondence between the target word, the synonym corresponding to the target word and the synonym weight, and the target word is the original word or a synonym of the original word on the application system.

[0128] The query statement generating module 203 is used to obtain synonyms and synonym weights corresponding to the search terms from the target synonym list, and generate a new query statement according to the search terms, the synonyms corresponding to the search terms, and the synonym weights.

[0129] The retrieval module 204 is used to retrieve information on the first application system based on the new query statement and the index information corresponding to the first application system, wherein the index information corresponding to the first application system includes index information established for each word corresponding to the first application system in the word segmenter vocabulary.

[0130] The synonym search device provided by the present application first obtains a query statement from the first application system, then determines the synonym list corresponding to the first application system from the synonym lists corresponding to several application systems respectively maintained in the synonym cache as the target synonym list, then obtains the synonyms and synonym weights corresponding to the search terms from the target synonym list, and generates a new query statement based on the search terms, the synonyms corresponding to the search terms and the synonym weights, and finally retrieves the information on the first application system based on the new query statement and the index information corresponding to the first application system. The present application can maintain the synonym lists corresponding to several application systems in the synonym cache, so that when establishing an index, it is only necessary to establish index information for each word contained in the word segmenter word library, which greatly saves the disk space occupied by the index information, and the present application simultaneously considers the search terms, the synonyms corresponding to the search terms and the synonym weights during the search, so that the importance of the original terms and synonyms can be distinguished based on the synonym weights, and the present application improves the recall rate of the search results.

[0131] In a possible implementation, the synonym search device provided by the present application may further include: a synonym list updating module.

[0132] The synonym list updating module is used to update the synonym list maintained by the synonym cache.

[0133] Optionally, the synonym list updating module may include: a task monitoring submodule and a synonym list updating submodule.

[0134] Among them, the task monitoring submodule is used to monitor whether there is a new synonym task, wherein the new synonym task contains a new word and system indication information, the new word refers to a synonym added to the original word on the second application system, and the second application system is the application system indicated by the system indication information contained in the new synonym task.

[0135] The synonym list updating submodule is used to update the synonym list corresponding to the second application system in the synonym cache based on the newly added word and the synonym weight corresponding to the newly added word if there is a synonym adding task.

[0136] In a possible implementation, the synonym search device provided by the present application may further include: an index information updating module.

[0137] The index information updating module is used to update the index information corresponding to several application systems respectively.

[0138] Optionally, the index information updating module may include: an unregistered word judging submodule, a word segmenter word library updating submodule and an index rebuilding submodule.

[0139] Among them, the unregistered word judgment submodule is used to judge whether the newly added word is an unregistered word based on the word segmenter vocabulary if there is a synonym addition task.

[0140] The word segmenter word library updating submodule is used to update the word segmenter word library based on the newly added word if the unregistered word judgment submodule determines that the newly added word is an unregistered word, so as to obtain an updated word segmenter word library.

[0141] The index reconstruction submodule is used to rebuild the index according to the updated word segmenter word library and the index in the original index library corresponding to the second application system, so as to obtain a reconstructed new index corresponding to the second application system.

[0142] In a possible implementation, the word segmenter vocabulary includes a word segmenter vocabulary corresponding to each node in the search engine cluster.

[0143] Based on this, the above-mentioned word segmenter vocabulary updating submodule may include: a file generating submodule and a file monitoring submodule.

[0144] The file generation submodule is used to generate a new word library file and a word library update identification file based on the new word, and write the new word into the new word library file;

[0145] The file monitoring submodule is used to monitor whether the word library update identification file is updated. If so, the newly added words in the new word library file are loaded into the word segmenter word library corresponding to each node of the search engine cluster.

[0146] In a possible implementation, the index reconstruction submodule may include: a new index library creation module and a reconstruction index writing module.

[0147] The new index library creation module is used to create a new index library corresponding to the second application system.

[0148] The index reconstruction writing module is used to rebuild the index in the new index library according to the updated word segmenter word library and the index in the original index library corresponding to the second application system.

[0149] In a possible implementation, the above-mentioned index reconstruction submodule may also include: a transition index library creation module and a new index writing module.

[0150] The transition index library creation module is used to create a transition index library if a new index needs to be added during the index reconstruction process, and write the new index into the transition index library after setting the transition index library to the default attribute;

[0151] A new index writing module is added to switch the new index library to the default attribute after the index reconstruction is completed, and write the newly added index in the transition index library to the new index library to obtain the reconstructed new index contained in the new index library.

[0152] In order to enable those skilled in the art to better understand the present application, a specific synonym search scenario is used as an example for explanation below. It should be noted that the following embodiment is only an example and is not intended to limit the present application.

[0153] Optionally, the synonym search method and device provided by the present application can be applied to Figure 3 The weights shown are configurable on the synonym search architecture. Figure 3 As shown, the weight-configurable synonym retrieval architecture includes a vocabulary management unit, a reindexing unit, a synonym cache, and a synonym retrieval unit.

[0154] Among them, the thesaurus management unit is mainly used for the maintenance of the synonym dictionary and loading the newly added words in the synonym dictionary into the word segmenter; the synonym cache maintains the target words of several application systems, the synonyms corresponding to the target words and the synonym weights in the cache respectively, to serve the subsequent retrieval of synonyms; since the newly added synonym dictionary may contain unregistered words, it is necessary to rebuild the index unit, which is used to rebuild the index in the original index library based on the updated word segmenter thesaurus. During the index reconstruction process, the new index operation can still be provided; the synonym retrieval unit is mainly used to analyze the query statements of the application system, query the synonym cache, and generate query statements containing synonyms and their weights.

[0155] like Figure 3 As shown, the thesaurus management unit may include: a task generator, a thesaurus listener, a thesaurus loader and a word segmenter.

[0156] Optionally, the unregistered word judgment submodule provided in the present application can be arranged on the task generator, and the word segmenter vocabulary update submodule provided in the present application can be arranged on the task generator, the vocabulary listener and the vocabulary loader.

[0157] Among them, the unregistered word judgment submodule and the file generation submodule included in the word segmenter vocabulary update submodule can be deployed on the task generator, so that the task generator in the vocabulary management unit can be used to create a new synonym vocabulary task, and when the newly added word is judged to be an unregistered word, a new vocabulary file and a vocabulary update identification file are generated for the unregistered word, and the new word is written into the new vocabulary file.

[0158] The file listening submodule included in the word segmenter vocabulary update submodule can be deployed on the vocabulary listener and the vocabulary loader, so that the vocabulary listener can be used to enable the word segmenter of each node in the cluster to start a listening thread to listen to whether the vocabulary update identification file is updated to determine whether there are new words; the vocabulary loader can be used to load the new words in the new vocabulary file into the word segmenter after listening to the update of the vocabulary update identification file, that is, to load the new words in the new vocabulary file into the word segmenter vocabulary corresponding to each node of the search engine cluster.

[0159] The word segmenter in the above-mentioned word library management unit is used for subsequent index reconstruction.

[0160] like Figure 3 As shown, the reindexing unit may include: a task generator and a task executor.

[0161] The task generator in the reindexing unit may be responsible for creating an index library reindexing task, wherein the reindexing task includes a task of migrating data from a transitional index library to a new index library.

[0162] Optionally, the above-mentioned index reconstruction sub-module can be deployed on the task executor, that is, the new index library creation module, the reconstruction index writing module, the transition index library creation module and the new index writing module can be deployed on the task executor, so that the task executor can be responsible for executing the index library re-indexing, creating a transition index library, executing the data migration from the transition index library to the new index library, and after the index reconstruction task is completed, closing the original index library, deleting the transition index library, and updating the task status.

[0163] like Figure 3 As shown, the synonym retrieval unit may include: an authority controller, a query parser, and a query generator.

[0164] The permission controller is used to perform permission verification on user query requests (ie, query requests containing query statements), including the user's permission to read indexes in the specified index library and the permission to retrieve synonyms.

[0165] The query parser is used to verify the query statement from the parsing result of the user query request message, including the query method, query field, return field and highlighted field.

[0166] Optionally, the target synonym list determination module and query statement generation module provided in the present application can be deployed on a query generator, so that the query generator can match the search terms in the parsed query statement with the synonym list in the synonym cache, and splice all synonyms and synonym weights in the matched synonym list into the query statement to generate a new query statement.

[0167] The embodiment of the present application also provides a synonym search device. Optionally, Figure 4 The hardware structure diagram of the synonym search device is shown in FIG. Figure 4 , the hardware structure of the synonym retrieval device may include: at least one processor 401, at least one communication interface 402, at least one memory 403 and at least one communication bus 404;

[0168] In the embodiment of the present application, the number of the processor 401, the communication interface 402, the memory 403, and the communication bus 404 is at least one, and the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404;

[0169] The processor 401 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention;

[0170] The memory 403 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0171] The memory 403 stores a program, and the processor 401 can call the program stored in the memory 403, and the program is used to:

[0172] Obtaining a query statement from the first application system, wherein the query statement includes a search term;

[0173] Determine a synonym list corresponding to the first application system from the synonym lists corresponding to the several application systems respectively maintained in the synonym cache as a target synonym list, wherein the synonym list maintains a correspondence between a target word, a synonym corresponding to the target word, and a synonym weight, and the target word is an original word or a synonym of the original word on the application system;

[0174] Obtain synonyms and synonym weights corresponding to the search terms from the target synonym list, and generate new query statements according to the search terms, the synonyms corresponding to the search terms, and the synonym weights;

[0175] Based on the new query statement and the index information corresponding to the first application system, the information on the first application system is retrieved, wherein the index information corresponding to the first application system includes index information established for each word corresponding to the first application system in the word segmenter word library.

[0176] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0177] The embodiment of the present application further provides a readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the above-mentioned synonym search method is implemented.

[0178] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0179] Finally, it should be noted that, in this article, relational terms such as and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprises a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0180] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0181] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A synonym search method, characterized in that: include: Acquire a query statement from the first application system, wherein the query statement includes a search term; Determine, from the synonym lists corresponding to several application systems respectively maintained in the synonym cache, the synonym list corresponding to the first application system as the target synonym list, wherein the synonym list maintains a correspondence between a target word, a synonym corresponding to the target word, and a synonym weight, and the target word is an original word or a synonym of the original word on the application system; Obtaining synonyms and synonym weights corresponding to the search term from the target synonym list, and generating a new query statement according to the search term, the synonyms and synonym weights corresponding to the search term; Retrieving information on the first application system based on the new query statement and index information corresponding to the first application system, wherein the index information corresponding to the first application system includes index information established for each word corresponding to the first application system in the word segmenter word library; Updating the synonym list maintained by the synonym cache, including: monitoring whether there is a synonym adding task, wherein the synonym adding task includes a new word and system indication information, the new word refers to a synonym added to the original word on the second application system, and the second application system is the application system indicated by the system indication information included in the synonym adding task; if the synonym adding task exists, updating the synonym list corresponding to the second application system in the synonym cache based on the new word and the synonym weight corresponding to the new word; The index information corresponding to the several application systems is updated respectively; the updating of the index information corresponding to the several application systems includes: if the synonym adding task exists, judging whether the newly added word is an unregistered word based on the word segmenter word library, and the unregistered word refers to a word that is not included in the word segmenter word library but must be segmented; if so, updating the word segmenter word library based on the newly added word to obtain an updated word segmenter word library; reconstructing the index according to the updated word segmenter word library and the index in the original index library corresponding to the second application system to obtain a reconstructed new index corresponding to the second application system.

2. The synonym search method according to claim 1, characterized in that: The word segmenter word library includes the word segmenter word library corresponding to each node in the search engine cluster; The updating of the word segmenter word library based on the newly added word comprises: Generate a new word library file and a word library update identification file based on the new word, and write the new word into the new word library file; Monitor whether the word library update identification file is updated, and if so, load the newly added words in the newly added word library file into the word segmenter word library corresponding to each node of the search engine cluster.

3. The synonym search method according to claim 1, characterized in that: The index reconstruction according to the updated word segmenter word library and the index in the original index library corresponding to the second application system includes: Creating a new index library corresponding to the second application system; According to the updated word segmenter word library and the index in the original index library corresponding to the second application system, index reconstruction is performed in the new index library.

4. The synonym search method according to claim 3, characterized in that: Also includes: If a new index needs to be added during the index reconstruction process, a transitional index library is created, and after the transitional index library is set as a default attribute, the new index is written into the transitional index library; After the index reconstruction is completed, the new index library is switched to the default attribute, and the newly added index in the transitional index library is written into the new index library to obtain the reconstructed new index contained in the newly added index library.

5. A synonym search device, characterized in that: include: A query statement acquisition module, a target synonym list determination module, a query statement generation module and a retrieval module; The query statement acquisition module is used to acquire a query statement from the first application system, wherein the query statement includes a search term; The target synonym list determination module is used to determine the synonym list corresponding to the first application system from the synonym lists corresponding to the several application systems respectively maintained in the synonym cache as the target synonym list, wherein the synonym list maintains the correspondence between the target word, the synonym corresponding to the target word and the synonym weight, and the target word is the original word or the synonym of the original word on the application system; The query statement generation module is used to obtain the synonyms and synonym weights corresponding to the search term from the target synonym list, and generate a new query statement according to the search term, the synonyms and synonym weights corresponding to the search term; The retrieval module is used to retrieve information on the first application system based on the new query statement and index information corresponding to the first application system, wherein the index information corresponding to the first application system includes index information established for each word corresponding to the first application system in the word segmenter word library; A synonym list updating module is used to update the synonym list maintained by the synonym cache; the synonym list updating module includes: Task monitoring submodule and synonym list update submodule; The task monitoring submodule is used to monitor whether there is a new synonym task, wherein the new synonym task contains a new word and system indication information, the new word refers to a synonym added to the original word on the second application system, and the second application system is the application system indicated by the system indication information contained in the new synonym task; The synonym list updating submodule is used to update the synonym list corresponding to the second application system in the synonym cache based on the newly added word and the synonym weight corresponding to the newly added word if the synonym adding task exists; An index information updating module, used to update the index information corresponding to the several application systems respectively; The index information updating module includes: an unregistered word judging submodule, a word segmenter word library updating submodule and an index rebuilding submodule; The unregistered word judgment submodule is used to judge whether the newly added word is an unregistered word based on the word segmenter word library if the synonym new addition task exists; the unregistered word refers to a word that is not included in the word segmenter word library but must be segmented. The word segmenter word library updating submodule is used for updating the word segmenter word library based on the newly added word to obtain an updated word segmenter word library if the unregistered word judging submodule judges that the newly added word is an unregistered word; The index reconstruction submodule is used to perform index reconstruction based on the updated word segmenter vocabulary and the index in the original index library corresponding to the second application system to obtain a reconstructed new index corresponding to the second application system.

6. The synonym search device according to claim 5, characterized in that: The word segmenter word library includes the word segmenter word library corresponding to each node in the search engine cluster; The word segmenter word library update submodule includes: a file generation submodule and a file monitoring submodule; The file generation submodule is used to generate a new word library file and a word library update identification file based on the new word, and write the new word into the new word library file; The file monitoring submodule is used to monitor whether the word library update identification file is updated. If so, the newly added words in the newly added word library file are loaded into the word segmenter word library corresponding to each node of the search engine cluster.

Images