Document search method and device based on document knowledge base, equipment and medium

By combining locally sensitive hash function and document knowledge graph, a document search method is constructed, which solves the problem of low document search efficiency in the existing technology, and achieves more efficient and accurate document search results.

CN119025731APending Publication Date: 2024-11-26GUANGZHOU FANGZHOU INFORMATION TECHNOLOGY CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411091028.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing knowledge base search methods rely on vector similarity retrieval, resulting in low document search efficiency and ineffective improvement of search efficiency and accuracy.

Method used

Use locally sensitive hash function to determine the hash value of the document search data, and build the first document collection and the second document collection based on the preset target hash interval and document knowledge graph, thereby determining the search results, avoiding the search one by one The similarity between vectors and document vectors.

Benefits of technology

The efficiency and accuracy of document search is improved, and the accuracy and efficiency of search results are significantly improved through the combination of hash intervals and knowledge graphs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025731A_ABST
    Figure CN119025731A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a document searching method and device based on a document knowledge base, equipment and a medium. The method comprises the following steps: acquiring document search data, and determining a current search vector corresponding to the document search data; determining a current hash value of the current search vector based on a locality sensitive hash function, and determining a first document set according to the current hash value and a preset target hash interval; wherein the target hash interval is constructed based on a locality sensitive hash function and candidate document vectors corresponding to candidate documents in a preset document knowledge base; determining a second document set according to the current search vector and a preset document knowledge graph; wherein the document knowledge graph is constructed based on document vector similarity between candidate document vectors corresponding to candidate documents in a document knowledge base; and determining a search result corresponding to the document search data according to the first document set and the second document set. And the document searching efficiency and accuracy are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of data processing technology, and in particular to a document search method, device, equipment and medium based on a document knowledge base. Background Art

[0002] Most of the current knowledge base searches rely on vector similarity retrieval. For the document knowledge base, the user's query is encoded into a multi-dimensional vector, and then the similarity between the current search vector and the document vector in the document knowledge base is calculated for matching, and the top k are taken as the search results. The above method has low search efficiency. Summary of the invention

[0003] The present invention provides a document search method, device, equipment and medium based on a document knowledge base to improve document search efficiency.

[0004] According to one aspect of the present invention, a document search method based on a document knowledge base is provided, comprising:

[0005] Acquire document search data, and determine a current search vector corresponding to the document search data;

[0006] Based on a locality-sensitive hash function, determine a current hash value of the current search vector, and determine a first document set according to the current hash value and a preset target hash interval; wherein the target hash interval is constructed based on the locality-sensitive hash function and candidate document vectors corresponding to each candidate document in a preset document knowledge base;

[0007] Determine a second document set according to the current search vector and a preset document knowledge graph; wherein the document knowledge graph is constructed based on the document vector similarity between the candidate document vectors corresponding to each candidate document in the document knowledge base;

[0008] A search result corresponding to the document search data is determined according to the first document set and the second document set.

[0009] According to another aspect of the present invention, there is provided a document search device based on a document knowledge base, comprising:

[0010] A search vector determination module, used to obtain document search data and determine a current search vector corresponding to the document search data;

[0011] A first set determination module is used to determine a current hash value of the current search vector based on a local sensitive hash function, and determine a first document set according to the current hash value and a preset target hash interval; wherein the target hash interval is constructed based on the local sensitive hash function and candidate document vectors corresponding to each candidate document in a preset document knowledge base;

[0012] A second set determination module, configured to determine a second document set according to the current search vector and a preset document knowledge graph; wherein the document knowledge graph is constructed based on the document vector similarity between the candidate document vectors corresponding to each candidate document in the document knowledge base;

[0013] The search result determination module is used to determine the search results corresponding to the document search data according to the first document set and the second document set.

[0014] According to another aspect of the present invention, there is provided an electronic device, comprising:

[0015] one or more processors;

[0016] A memory for storing one or more programs;

[0017] When one or more programs are executed by one or more processors, the one or more processors can execute any one of the document search methods based on the document knowledge base provided by the embodiments of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions, and the computer instructions are used to enable a processor to implement any document search method based on a document knowledge base provided by an embodiment of the present invention when executed.

[0019] The embodiment of the present invention provides a document search solution based on a document knowledge base, which obtains document search data and determines the current search vector corresponding to the document search data; determines the current hash value of the current search vector based on a local sensitive hash function, and determines the first document set according to the current hash value and a preset target hash interval; wherein the target hash interval is constructed based on the local sensitive hash function and the candidate document vectors corresponding to each candidate document in the preset document knowledge base; determines the second document set according to the current search vector and a preset document knowledge graph; wherein the document knowledge graph is constructed based on the document vector similarity between the candidate document vectors corresponding to each candidate document in the document knowledge base; determines the search results corresponding to the document search data according to the first document set and the second document set. The above scheme, by introducing the target hash interval and the document knowledge graph, determines the first document set and the second document set respectively, and then determines the search results, thereby avoiding determining the similarity between the current search vector and each candidate document vector in the document knowledge base one by one, and improving the efficiency and accuracy of document search.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a flowchart of a document search method based on a document knowledge base provided in Embodiment 1 of the present invention;

[0023] Figure 2 is a flowchart of a document search method based on a document knowledge base provided in Embodiment 2 of the present invention;

[0024] Figure 3 It is a structural diagram of a document search device based on a document knowledge base provided in Embodiment 3 of the present invention;

[0025] Figure 4 It is a structural diagram of an electronic device for implementing a document search method based on a document knowledge base provided by a fourth embodiment of the present invention. DETAILED DESCRIPTION

[0026] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0027] Embodiment 1

[0028] Figure 1 This is a flowchart of a document search method based on a document knowledge base provided in Example 1 of the present invention. This embodiment is applicable to the situation of searching for documents in a document knowledge base. The method can be executed by a document search device based on a document knowledge base. The device can be implemented in software and / or hardware and can be configured in an electronic device that carries a document search function based on a document knowledge base.

[0029] See also Figure 1 The document search method based on the document knowledge base shown includes:

[0030] S110: Acquire document search data, and determine a current search vector corresponding to the document search data.

[0031] The document search data refers to data used to search for documents, and can be used to indicate which documents are needed. Exemplarily, the document search data can be document demand data provided by the user. The current search vector refers to a vector of the document search data. Exemplarily, the document search data is encoded to obtain the corresponding current search vector.

[0032] S120: Determine a current hash value of a current search vector based on a locality sensitive hash function, and determine a first document set according to the current hash value and a preset target hash interval.

[0033] The current hash value refers to the hash value corresponding to the current search vector. Exemplarily, a local sensitive hash function is used to calculate a hash value corresponding to the current search vector, which is the current hash value. It should be noted that there is a one-to-one correspondence between the document search data, the current search vector and the current hash value.

[0034] The target hash interval can be used to divide the candidate documents in the document knowledge base. Exemplarily, the target hash interval is constructed based on a local sensitive hash function and a candidate document vector corresponding to each candidate document in the preset document knowledge base.

[0035] The document knowledge base refers to a pre-set database for storing candidate documents. The candidate document refers to a document stored in the document knowledge base. The candidate document vector refers to a vector of a candidate document. Exemplarily, the candidate document is encoded to obtain a corresponding candidate document vector.

[0036] The first document set refers to a document set determined according to the target hash interval.

[0037] Specifically, based on the locality sensitive hash function, a current hash value corresponding to the current search vector is determined; a current sub-interval of the current hash value in the target hash interval is determined, and a first document set is determined based on candidate documents corresponding to the determined current sub-interval.

[0038] In an optional embodiment, the target hash interval is determined based on the following method: determining a candidate document vector for each candidate document in a preset document knowledge base, and determining a document hash value corresponding to each candidate document vector based on a local sensitive hash function; determining a candidate hash value from the document hash value, and determining an initial hash interval based on the candidate hash value; determining the local distribution of candidate documents corresponding to each document hash value in each sub-interval in the initial hash interval, and adjusting the initial hash interval based on the local distribution to obtain a target hash interval.

[0039] The document hash value refers to the hash value corresponding to the candidate document vector. Exemplarily, a locality-sensitive hash function is used to calculate a hash value corresponding to each candidate document vector, which is the document hash value.

[0040] The candidate hash values ​​may include the largest document hash value and the smallest document hash value. The initial hash interval refers to the hash interval preliminarily determined based on the candidate hash values. The local distribution refers to the number of candidate documents distributed in each subinterval of the initial hash interval.

[0041] Exemplarily, based on the candidate hash values, an initial hash interval with the same hash value interval is determined (i.e., the intervals of each sub-interval in the initial hash interval are the same); the number of candidate documents corresponding to each sub-interval in the initial hash interval is determined, and each sub-interval in the initial hash interval is adjusted to obtain a target hash interval.

[0042] The hash value intervals of each subinterval in the target hash interval may be different, but the number of candidate documents corresponding to each subinterval is as close as possible. It should be noted that adjusting each subinterval in the initial hash interval may include at least one of increasing the hash value interval of the subinterval, decreasing the hash value interval of the subinterval, merging the subinterval, and splitting the subinterval.

[0043] It should be noted that there is a one-to-one correspondence between candidate documents, candidate document vectors, and document hash values.

[0044] It can be understood that by introducing the initial hash interval and the local distribution to determine the target hash interval, the accuracy and practicality of the determined target hash interval are improved.

[0045] S130. Determine a second document set according to the current search vector and a preset document knowledge graph.

[0046] Among them, the document knowledge graph is constructed based on the document vector similarity between the candidate document vectors corresponding to each candidate document in the document knowledge base.

[0047] The document knowledge graph can be used to characterize the similarity between candidate document vectors in the document knowledge base. The second document set refers to a document set determined based on the document knowledge graph.

[0048] In an optional embodiment, the document knowledge graph is determined based on the following method: determining the document vector similarity between candidate document vectors corresponding to any two candidate documents in the document knowledge base, and constructing the document knowledge graph based on the document vector similarity and a preset similarity threshold.

[0049] The document vector similarity can be used to characterize the similarity between any two candidate document vectors. The embodiment of the present invention does not impose any limitation on the value of the preset similarity threshold, which can be set by technicians based on experience. For example, the preset similarity threshold can be 60%.

[0050] Exemplarily, if the document vector similarity is greater than a preset similarity threshold, it is determined that the similarity between the two candidate document vectors corresponding to the document vector similarity is high, there is a correlation relationship between the two candidate document vectors, and the two candidate document vectors can be connected by an edge; if the document vector similarity is less than or equal to the preset similarity threshold, it is determined that the similarity between the two candidate document vectors corresponding to the document vector similarity is low, and there is no correlation relationship between the two candidate document vectors; based on the above scheme, a document knowledge graph including multiple candidate document links is constructed.

[0051] It can be understood that by constructing a document knowledge graph based on document vector similarity and a preset similarity threshold, the accuracy of the determined document knowledge graph is improved.

[0052] S140: Determine search results corresponding to the document search data according to the first document set and the second document set.

[0053] The search result refers to at least one candidate document determined based on the document search data.

[0054] Exemplarily, determining search results corresponding to document search data based on the first document set and the second document set includes: taking the union of the first document set and the second document set as the search result corresponding to the document search data; or taking the intersection of the first document set and the second document set as the search result corresponding to the document search data.

[0055] The embodiment of the present invention provides a document search solution based on a document knowledge base, which obtains document search data and determines the current search vector corresponding to the document search data; determines the current hash value of the current search vector based on a local sensitive hash function, and determines the first document set according to the current hash value and a preset target hash interval; wherein the target hash interval is constructed based on the local sensitive hash function and the candidate document vectors corresponding to each candidate document in the preset document knowledge base; determines the second document set according to the current search vector and a preset document knowledge graph; wherein the document knowledge graph is constructed based on the document vector similarity between the candidate document vectors corresponding to each candidate document in the document knowledge base; determines the search results corresponding to the document search data according to the first document set and the second document set. The above scheme, by introducing the target hash interval and the document knowledge graph, determines the first document set and the second document set respectively, and then determines the search results, thereby avoiding determining the similarity between the current search vector and each candidate document vector in the document knowledge base one by one, and improving the efficiency and accuracy of document search.

[0056] Embodiment 2

[0057] Figure 2 It is a flowchart of a document search method based on a document knowledge base provided in the second embodiment of the present invention. Based on the above embodiments, this embodiment further refines the operation of "determining the first document set according to the current hash value and the preset target hash interval" into "taking the subinterval of the current hash value in the target hash interval as the current subinterval; determining the first vector similarity between the current search vector and each candidate document vector corresponding to the current subinterval, and determining the first document set according to the first vector similarity" to improve the determination mechanism of the first document set. It should be noted that for the part not described in detail in the embodiment of the present invention, please refer to the description of other embodiments.

[0058] See also Figure 2 The document search method based on the document knowledge base shown includes:

[0059] S210: Acquire document search data, and determine a current search vector corresponding to the document search data.

[0060] S220: Determine a current hash value of the current search vector based on a locality sensitive hash function, and use a subinterval of the current hash value in the target hash interval as a current subinterval.

[0061] The target hash interval is constructed based on a local sensitive hash function and a candidate document vector corresponding to each candidate document in a preset document knowledge base.

[0062] The current subinterval refers to the subinterval in the target hash interval where the current hash value is located. Exemplarily, the target hash interval includes multiple subintervals, each of which corresponds to a different hash value range, and the subinterval in the target hash interval where the current hash value is located is determined, and the determined subinterval is used as the current subinterval.

[0063] For example, if the target hash interval includes sub-interval A, sub-interval B, and sub-interval C, and there is a current hash value in sub-interval A, then sub-interval A is used as the current sub-interval.

[0064] S230: Determine a first vector similarity between the current search vector and each candidate document vector corresponding to the current subinterval, and determine a first document set according to the first vector similarity.

[0065] The first vector similarity refers to the vector similarity used to determine the first document set. Exemplarily, the first vector similarity may be the similarity between the current search vector and the candidate document vectors corresponding to each document hash value in the current subinterval.

[0066] Exemplarily, determining the first vector similarity between the current search vector and each candidate document vector corresponding to the current subinterval, and determining the first document set based on the first vector similarity, includes: taking the candidate document vector corresponding to each document hash value in the current subinterval as the current document vector; determining the first vector similarity between the current search vector and each current document vector respectively; sorting the corresponding current document vectors from large to small based on each first vector similarity; taking the candidate documents corresponding to the first preset number of current document vectors in the sorting result as the first target document; generating the first document set including the first target document. Among them, the current document vector refers to a candidate document vector that can be used to calculate the similarity with the current search vector. The embodiment of the present invention does not impose any limitation on the size of the preset first document number, which can be set by the technician according to experience or needs, or determined repeatedly through a large number of experiments. The first target document refers to the candidate document used to generate the first document set.

[0067] In an optional embodiment, after taking the subinterval of the current hash value in the target hash interval as the current subinterval, the method also includes: determining a reference subinterval based on the current subinterval; wherein the reference subinterval is an adjacent subinterval of the current subinterval in the target hash interval; accordingly, determining the first vector similarity between the current search vector and each candidate document vector corresponding to the current subinterval, including: determining the first vector similarity between the current search vector and each candidate document vector corresponding to the current subinterval and the reference subinterval, respectively.

[0068] The reference subinterval refers to the subintervals that are adjacent to the current subinterval in the target hash interval. For example, if the subintervals in the target hash interval are ordered as subinterval A, subinterval B, subinterval C, and subinterval D, and the current subinterval is subinterval B, then the reference subinterval of the current subinterval includes subinterval A and subinterval C; if the current subinterval is A, then the reference subinterval of the current subinterval is subinterval B.

[0069] Exemplarily, determining the first vector similarities between the current search vector and each candidate document vector corresponding to the current subinterval and the reference subinterval, respectively, includes: taking the candidate document vector corresponding to each document hash value in the current subinterval and the reference subinterval as the current document vector; determining the first vector similarities between the current search vector and each current document vector, respectively.

[0070] Furthermore, based on the first vector similarity, a first document set is determined, including: sorting the corresponding current document vectors from large to small based on the similarities of each first vector; taking the candidate documents corresponding to a preset first number of current document vectors in the sorting result as the first target document; and generating a first document set including the first target document.

[0071] It can be understood that by introducing the reference sub-interval, the comprehensiveness and accuracy of the generated first document set are improved.

[0072] S240: Determine a second document set according to the current search vector and a preset document knowledge graph.

[0073] Among them, the document knowledge graph is constructed based on the document vector similarity between the candidate document vectors corresponding to each candidate document in the document knowledge base.

[0074] In an optional embodiment, a second document set is determined based on a current search vector and a preset document knowledge graph, including: determining candidate document links in the document knowledge graph, and selecting any candidate document vector from each candidate document link as a reference document vector; determining a reference similarity between the current search vector and the reference document vector, and selecting a reference document link from the candidate document link based on the reference similarity; determining a second vector similarity between the current search vector and each candidate document vector in the reference document link, and determining a second document set based on the second vector similarity.

[0075] Among them, the candidate document link refers to the link composed of candidate document vectors with high similarity (i.e., the document vector similarity is greater than the preset similarity threshold) in the document knowledge graph. The reference document vector refers to the candidate document vector in the candidate document link that can be used to calculate the similarity with the current search vector. The reference similarity refers to the degree of similarity between the current search vector and the reference document vector. The reference document link refers to the candidate document link that can participate in the calculation of the second vector similarity. The second vector similarity refers to the vector similarity used to determine the second document set.

[0076] Exemplarily, at least one candidate document link in the document knowledge graph is determined, and any candidate document vector is selected from each candidate document link as a reference document vector; the reference similarity between the current search vector and the reference document vector is determined; the corresponding candidate document links are sorted from large to small according to the reference similarity, and the first preset link number of candidate document links are selected from the sorting result as the reference document link; the second vector similarity between the current search vector and each candidate document vector in the reference document link is determined; the corresponding candidate document vectors are sorted from large to small according to each second vector similarity; the candidate documents corresponding to the first preset second document number of candidate document vectors in the sorting result are used as the second target document; and a second document set including the second target document is generated. The embodiment of the present invention does not impose any limitation on the size of the preset link number and / or the preset second document number, which can be set by a technician based on experience or needs, or repeatedly determined through a large number of experiments.

[0077] It should be noted that the preset first document quantity and the preset second document quantity may be the same or different, and the embodiment of the present invention does not specifically limit this.

[0078] It can be understood that by selecting reference document links from candidate document links, the candidate documents are screened, avoiding selecting the required candidate documents one by one from the document knowledge base, and improving the efficiency of generating the second document set, that is, improving the efficiency of document search.

[0079] S250: Determine search results corresponding to the document search data according to the first document set and the second document set.

[0080] It should be noted that the search results can be determined based on the preset first document quantity, the preset second document quantity and the preset document quantity threshold. The embodiment of the present invention does not impose any limitation on the size of the preset document quantity threshold, which can be set by technicians based on experience or needs, or can be repeatedly determined through a large number of experiments.

[0081] Exemplarily, if the preset number of first documents and / or the preset number of second documents is greater than or equal to a preset document number threshold, the intersection of the first document set and the second document set is used as the search result; if the preset number of first documents and / or the preset number of second documents is less than the preset document number threshold, the union of the first document set and the second document set is used as the search result.

[0082] The embodiment of the present invention provides a document search scheme based on a document knowledge base, which refines the operation of determining the first document set according to the current hash value and the preset target hash interval into taking the subinterval of the current hash value in the target hash interval as the current subinterval; determining the first vector similarity between the current search vector and each candidate document vector corresponding to the current subinterval, and determining the first document set according to the first vector similarity, thereby improving the determination mechanism of the first document set. The above scheme improves the efficiency of generating the first document set, that is, improves the efficiency of document search, by determining the current subinterval in the target hash interval according to the current hash value and determining the first document set according to the current subinterval.

[0083] On the basis of the above technical solution, an embodiment of the present invention proposes a retrieval method that combines a hash-based search method with a graph search method.

[0084] Exemplarily, a hash-based search method uses a local sensitive hash function to divide candidate document vectors. By using the local sensitive hash function, a corresponding document hash value can be calculated for each candidate document vector. For candidate document vectors that are closer in distance, their corresponding document hash values ​​are also closer. This method divides the value range of the local sensitive hash function into several intervals, so that each candidate document vector has a subinterval corresponding to it in response to a specific local sensitive hash function. This method divides the candidate document vectors by the target hash interval. If the subintervals where the document hash values ​​corresponding to the two candidate document vectors are located are the same, then the two candidate document vectors belong to the same category. When searching, the current subinterval can be calculated by the same local sensitive hash function and interval division method. Then, the similarity of the first vector of all candidate document vectors in the current subinterval and the reference subinterval of the current subinterval with the current search vector can be calculated in turn, and the candidate document vector with the smallest first vector similarity is selected.

[0085] The hash-based method can effectively exclude a large number of candidate document vectors with low similarity to the current search vector by calculating the current subinterval and the reference subinterval where the current search vector is located, thus reducing the number of vector similarity calculations. However, this method can usually only divide the vector space evenly, while in actual applications, the distribution of vectors in space is usually uneven, resulting in a huge difference in the number of vectors in each subinterval, which further affects the efficiency and accuracy of the search.

[0086] Therefore, a graph search method is further proposed. The graph-based search method does not divide the vector space. The method pre-calculates the document vector similarity between each candidate document vector, and maintains the similarity relationship between the candidate document vectors in the form of a graph. Specifically, in the graph, each candidate document vector is a node, and the nodes with higher document vector similarity are connected to each other through edges (the document vector similarity between two candidate document vectors is usually set to be greater than a preset similarity threshold of 60%). When searching, exploration is carried out from one or more starting nodes. Each time a node is explored, the similarity of all neighboring nodes of the node with the second vector of the current search vector is calculated, and based on the results of the current exploration, the node that is most similar to the current search vector and has not been explored is selected as the node to be explored next time and the next exploration is started. The above process ends when no new exploration node can be found, and the k nodes that are most similar to the current search vector among all the nodes visited during the exploration process are used as search results.

[0087] Graph-based methods usually have high search efficiency and accuracy, but a large number of vector distance calculations are required in the process of building the search graph, which leads to huge computational overhead. In addition, when adding a new vector to the vector set, the search graph usually needs to be rebuilt, which seriously affects the efficiency of vector insertion.

[0088] Combined method: The two search methods can be run in parallel with extremely high operating efficiency, and the first document set and the second document set returned at the same time are taken as the union.

[0089] Embodiment 3

[0090] Figure 3 This is a schematic diagram of the structure of a document search device based on a document knowledge base provided in Embodiment 3 of the present invention. This embodiment is applicable to the case of searching for documents in a document knowledge base, and the method can be executed by a document search device based on a document knowledge base, which can be implemented in software and / or hardware, and can be configured in an electronic device that carries a document search function based on a document knowledge base.

[0091] like Figure 3 As shown, the device includes: a search vector determination module 310, a first set determination module 320, a second set determination module 330 and a search result determination module 340. Among them,

[0092] A search vector determination module 310, configured to obtain document search data and determine a current search vector corresponding to the document search data;

[0093] A first set determination module 320 is used to determine a current hash value of the current search vector based on a locality sensitive hash function, and determine a first document set according to the current hash value and a preset target hash interval; wherein the target hash interval is constructed based on the locality sensitive hash function and candidate document vectors corresponding to each candidate document in a preset document knowledge base;

[0094] A second set determination module 330 is used to determine a second document set according to the current search vector and a preset document knowledge graph; wherein the document knowledge graph is constructed based on the document vector similarity between the candidate document vectors corresponding to each candidate document in the document knowledge base;

[0095] The search result determination module 340 is used to determine the search results corresponding to the document search data according to the first document set and the second document set.

[0096] The embodiment of the present invention provides a document search solution based on a document knowledge base, which obtains document search data and determines the current search vector corresponding to the document search data; determines the current hash value of the current search vector based on a local sensitive hash function, and determines the first document set according to the current hash value and a preset target hash interval; wherein the target hash interval is constructed based on the local sensitive hash function and the candidate document vectors corresponding to each candidate document in the preset document knowledge base; determines the second document set according to the current search vector and a preset document knowledge graph; wherein the document knowledge graph is constructed based on the document vector similarity between the candidate document vectors corresponding to each candidate document in the document knowledge base; determines the search results corresponding to the document search data according to the first document set and the second document set. The above scheme, by introducing the target hash interval and the document knowledge graph, determines the first document set and the second document set respectively, and then determines the search results, thereby avoiding determining the similarity between the current search vector and each candidate document vector in the document knowledge base one by one, and improving the efficiency and accuracy of document search.

[0097] The first set determination module 320 includes:

[0098] a sub-interval determining unit, configured to use a sub-interval of the current hash value in the target hash interval as a current sub-interval;

[0099] The first set determination unit is used to determine a first vector similarity between the current search vector and each candidate document vector corresponding to the current sub-interval, and determine a first document set according to the first vector similarity.

[0100] Optionally, after the sub-interval determination unit, the device further includes:

[0101] A reference sub-interval determining unit, configured to determine a reference sub-interval according to the current sub-interval; wherein the reference sub-interval is a sub-interval adjacent to the current sub-interval in the target hash interval;

[0102] Accordingly, the first set determination unit includes:

[0103] The first similarity determination subunit is used to determine the first vector similarities between the current search vector and each candidate document vector corresponding to the current subinterval and the reference subinterval.

[0104] Optionally, the target hash interval is determined based on the following means:

[0105] A document hash value determination module, used to determine a candidate document vector of each candidate document in a preset document knowledge base, and determine a document hash value corresponding to each candidate document vector based on the local sensitive hash function;

[0106] An initial hash interval determination module, used to determine candidate hash values ​​from the document hash values, and determine an initial hash interval based on the candidate hash values;

[0107] The target hash interval determination module is used to determine the local distribution of the candidate documents corresponding to the hash values ​​of the documents in each sub-interval of the initial hash interval, and adjust the initial hash interval according to the local distribution to obtain the target hash interval.

[0108] Optionally, the second set determination module 330 includes:

[0109] A reference document vector determination unit, used to determine a candidate document link in the document knowledge graph, and select any candidate document vector from each candidate document link as a reference document vector;

[0110] A reference document link determination unit, configured to determine reference similarities between the current search vector and the reference document vector, and select a reference document link from the candidate document links according to the reference similarities;

[0111] The second set determination unit is used to determine the second vector similarities between the current search vector and each candidate document vector in the reference document link, and determine the second document set according to the second vector similarities.

[0112] Optionally, the document knowledge graph is determined based on the following means:

[0113] The document knowledge graph construction module is used to determine the document vector similarity between candidate document vectors corresponding to any two candidate documents in the document knowledge base, and to construct the document knowledge graph based on the document vector similarity and a preset similarity threshold.

[0114] The document search device based on the document knowledge base provided in the embodiment of the present invention can execute the document search method based on the document knowledge base provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing each document search method based on the document knowledge base.

[0115] In the technical solution of the present invention, the collection, storage, use, processing, transmission, provision and disclosure of document search data and document knowledge base involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0116] Embodiment 4

[0117] Figure 4 It is a structural diagram of an electronic device for implementing a document search method based on a document knowledge base provided by Embodiment 4 of the present invention. Electronic device 410 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0118] like Figure 4 As shown, the electronic device 410 includes at least one processor 411, and a memory connected to the at least one processor 411 in communication, such as a read-only memory (ROM) 412, a random access memory (RAM) 413, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 411 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 412 or the computer program loaded from the storage unit 418 to the random access memory (RAM) 413. In the RAM 413, various programs and data required for the operation of the electronic device 410 can also be stored. The processor 411, the ROM 412, and the RAM 413 are connected to each other via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.

[0119] Multiple components in the electronic device 410 are connected to the I / O interface 415, including: an input unit 416, such as a keyboard, a mouse, etc.; an output unit 417, such as various types of displays, speakers, etc.; a storage unit 418, such as a disk, an optical disk, etc.; and a communication unit 419, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 419 allows the electronic device 410 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0120] The processor 411 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 411 executes the various methods and processes described above, such as a document search method based on a document knowledge base.

[0121] In some embodiments, the document search method based on the document knowledge base may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 418. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 410 via the ROM 412 and / or the communication unit 419. When the computer program is loaded into the RAM 413 and executed by the processor 411, one or more steps of the document search method based on the document knowledge base described above may be performed. Alternatively, in other embodiments, the processor 411 may be configured to execute the document search method based on the document knowledge base in any other appropriate manner (e.g., by means of firmware).

[0122] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0123] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0124] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0125] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0126] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0127] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0128] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0129] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A document search method based on a document knowledge base, characterized in that: include: Acquire document search data, and determine a current search vector corresponding to the document search data; Based on a locality-sensitive hash function, determine a current hash value of the current search vector, and determine a first document set according to the current hash value and a preset target hash interval; wherein the target hash interval is constructed based on the locality-sensitive hash function and candidate document vectors corresponding to each candidate document in a preset document knowledge base; Determine a second document set according to the current search vector and a preset document knowledge graph; wherein the document knowledge graph is constructed based on the document vector similarity between the candidate document vectors corresponding to each candidate document in the document knowledge base; The search results corresponding to the document search data are determined according to the first document set and the second document set.

2. The method according to claim 1, characterized in that: The determining of the first document set according to the current hash value and a preset target hash interval includes: Taking the subinterval of the current hash value in the target hash interval as the current subinterval; A first vector similarity between the current search vector and each candidate document vector corresponding to the current subinterval is determined, and a first document set is determined according to the first vector similarity.

3. The method according to claim 2, characterized in that After taking the subinterval of the current hash value in the target hash interval as the current subinterval, the method further includes: Determine a reference subinterval according to the current subinterval; wherein the reference subinterval is a subinterval adjacent to the current subinterval in the target hash interval; Accordingly, determining the first vector similarity between the current search vector and each candidate document vector corresponding to the current subinterval includes: Determine first vector similarities between the current search vector and candidate document vectors corresponding to the current subinterval and the reference subinterval, respectively.

4. The method according to claim 1, characterized in that: The target hash interval is determined based on the following method: Determine a candidate document vector of each of the candidate documents in a preset document knowledge base, and determine a document hash value corresponding to each of the candidate document vectors based on the local sensitive hash function; Determine a candidate hash value from the document hash value, and determine an initial hash interval based on the candidate hash value; The local distribution of the candidate documents corresponding to the hash values ​​of the documents in each sub-interval in the initial hash interval is determined, and the initial hash interval is adjusted according to the local distribution to obtain a target hash interval.

5. The method according to claim 1, characterized in that The determining a second document set according to the current search vector and a preset document knowledge graph includes: Determine candidate document links in the document knowledge graph, and select any candidate document vector from each candidate document link as a reference document vector; Determining reference similarities between the current search vector and the reference document vector respectively, and selecting a reference document link from the candidate document links according to the reference similarities; Determine second vector similarities between the current search vector and each candidate document vector in the reference document link, and determine the second document set according to the second vector similarities.

6. The method according to claim 1, characterized in that The document knowledge graph is determined based on the following method: Determine the document vector similarity between candidate document vectors corresponding to any two candidate documents in the document knowledge base, and construct the document knowledge graph based on the document vector similarity and a preset similarity threshold.

7. A document search device based on a document knowledge base, characterized in that: include: A search vector determination module, used to obtain document search data and determine a current search vector corresponding to the document search data; A first set determination module is used to determine a current hash value of the current search vector based on a local sensitive hash function, and determine a first document set according to the current hash value and a preset target hash interval; wherein the target hash interval is constructed based on the local sensitive hash function and candidate document vectors corresponding to each candidate document in a preset document knowledge base; A second set determination module, configured to determine a second document set according to the current search vector and a preset document knowledge graph; wherein the document knowledge graph is constructed based on the document vector similarity between the candidate document vectors corresponding to each candidate document in the document knowledge base; The search result determination module is used to determine the search results corresponding to the document search data according to the first document set and the second document set.

8. The device according to claim 7, characterized in that The first set determination module includes: a sub-interval determining unit, configured to use a sub-interval of the current hash value in the target hash interval as a current sub-interval; The first set determination unit is used to determine a first vector similarity between the current search vector and each candidate document vector corresponding to the current sub-interval, and determine a first document set according to the first vector similarity.

9. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement a document search method based on a document knowledge base as described in any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a document search method based on a document knowledge base as described in any one of claims 1 to 6 is implemented.

Citation Information

Cited By

  • Data processing method and device

    CN121412362A