Method and device for generating hierarchical clustering tree based on document set

Abstract

The invention relates to a method and device for generating a hierarchical clustering tree based on a document set, and the method comprises the steps: carrying out the clustering operation of documents in the document set, and obtaining a plurality of document clusters; performing keyword extraction on the target cluster to obtain a target keyword node consisting of a plurality of keywords, and taking the target keyword node as a father node of leaf nodes corresponding to each document in the target cluster; performing multiple rounds of target operation on each keyword node until clustering operation in the target operation cannot be performed; any round of target operation comprises the steps of performing clustering operation on each keyword node to obtain a plurality of first clustering clusters; carrying out keyword extraction on any first target cluster to obtain a first node consisting of a plurality of keywords, and taking the first node as a father node of each keyword node in the first target cluster; taking the plurality of first nodes as keyword nodes input by the next round of target operation; and taking a root node of the hierarchical clustering tree as a father node of a plurality of first nodes obtained by the last round of target operation.

Description

Technical Field

[0001] The present invention relates to the field of information retrieval, and in particular to a method and device for generating a hierarchical clustering tree based on a document set. Background Art

[0002] Large Language Models (LLMs) have demonstrated powerful capabilities in many fields and tasks. As the size of a large language model continues to grow, it can serve as an independent and effective knowledge repository, encoding facts into its parameters. However, even a large language model cannot contain enough domain-specific knowledge to cope with specific tasks. At the same time, the world is constantly changing, and new knowledge generated every day may invalidate the original facts learned by the large language model. Due to cost and efficiency issues, developers cannot update the large language model in real time.

[0003] At present, a mainstream method in the industry to solve the above problems is Retrieval-Augmented Generation (RAG). RAG combines information retrieval and natural language generation, providing the large language model with an external knowledge base as the basis for generating answers. In RAG, the knowledge base is first searched based on the prompt to be input into the large language model, and then the search results and the prompt are input into the large language model to generate the answer.

[0004] However, when the number of documents in the external knowledge base continues to increase and the documents cover more and more fields, the existing retrieval methods may find it difficult to efficiently index the knowledge base and to quickly and accurately retrieve the documents most relevant to the prompt words, thereby reducing the accuracy of the answers generated by the large language model. Therefore, a method is needed to better index and retrieve document collections to improve the efficiency and accuracy of RAG. Summary of the invention

[0005] The present invention describes a method and device for generating a hierarchical clustering tree based on a document set. The method constructs a hierarchical clustering tree for a large-scale document set and performs information retrieval based on the hierarchical clustering tree to improve the efficiency and accuracy of retrieval of the large-scale document set.

[0006] In a first aspect, a method for generating a hierarchical clustering tree based on a document set is provided, comprising:

[0007] Obtain a document set containing multiple documents, and use each document as a leaf node of a hierarchical clustering tree;

[0008] Performing a clustering operation on the documents in the document set to obtain a plurality of document clusters;

[0009] Perform keyword extraction on any target cluster in each document cluster to obtain a target keyword node composed of a number of keywords as a parent node of a leaf node corresponding to each document in the target cluster;

[0010] Perform multiple rounds of target operations on each keyword node until the clustering operation in the target operation cannot be performed; wherein any round of target operations includes: performing clustering operations on each keyword node to obtain multiple first clustering clusters; performing keyword extraction on any first target clustering cluster in each first clustering cluster to obtain a first node composed of a plurality of keywords as a parent node of each keyword node in the first target clustering cluster; using the multiple first nodes as keyword nodes input for the next round of target operations;

[0011] The root node of the hierarchical clustering tree is used as the parent node of the first nodes obtained by the last round of target operation.

[0012] In a possible implementation, it further includes:

[0013] Searching the query text input by the user in the hierarchical clustering tree to obtain multiple query result nodes;

[0014] The contents in the multiple query result nodes are determined as query results corresponding to the query text.

[0015] In a possible implementation, the query text is a prompt word to be input into a large language model; the method further includes:

[0016] The query result and the prompt word are input into the large language model together to obtain the answer text of the large language model for the prompt word.

[0017] In a possible implementation, it further includes:

[0018] receiving an instruction to add a first document to the document set, determining a first document cluster to which the first document belongs, and updating the first document cluster;

[0019] Generate a first leaf node corresponding to the first document, use a first keyword node corresponding to the first document cluster as a parent node of the first leaf node, and add the first leaf node to the hierarchical clustering tree;

[0020] Update the content of each ancestor node of the first leaf node.

[0021] In a possible implementation, it further includes:

[0022] When the number of documents in any document cluster exceeds a preset first threshold, the tree structure of the hierarchical clustering tree is updated.

[0023] In a possible implementation, it further includes:

[0024] receiving a deletion instruction for a second document in the document set, determining a second document cluster to which the second document belongs, and updating the second document cluster;

[0025] The second leaf node corresponding to the second document is deleted from the hierarchical clustering tree, and the content in each ancestor node of the second leaf node is updated.

[0026] In a possible implementation, it further includes:

[0027] When the number of documents in any document cluster is less than a preset second threshold, the tree structure of the hierarchical clustering tree is updated.

[0028] In a possible implementation, the query text input by the user is searched in the hierarchical clustering tree to obtain multiple query result nodes, including:

[0029] Perform multiple rounds of target matching starting from the root node of the hierarchical clustering tree until reaching the leaf node layer of the hierarchical clustering tree; wherein any round of target matching includes: determining the similarity between the child nodes of each node input in the current round and the query text, selecting the top k nodes with the highest similarity ranking as the matching result nodes of the current round and as the input nodes of the next round;

[0030] The matching result nodes of each round of target matching are determined as query result nodes.

[0031] In a possible implementation, the query text input by the user is searched in the hierarchical clustering tree to obtain multiple query result nodes, including:

[0032] Each node in the hierarchical clustering tree is matched with the query text in terms of similarity, and the top m nodes with the highest similarity ranking are determined as query result nodes.

[0033] In a second aspect, a device for generating a hierarchical clustering tree based on a document set is provided, comprising:

[0034] An acquisition unit is configured to acquire a document set including a plurality of documents, and use each document therein as a leaf node of a hierarchical clustering tree;

[0035] A first clustering unit is configured to perform a clustering operation on the documents in the document set to obtain a plurality of document clusters;

[0036] A first keyword extraction unit is configured to perform keyword extraction on any target cluster in each document cluster to obtain a target keyword node composed of a plurality of keywords as a parent node of a leaf node corresponding to each document in the target cluster;

[0037] The first tree construction unit is configured to perform multiple rounds of target operations on each keyword node until the clustering operation in the target operation cannot be performed; wherein any round of target operations includes: performing clustering operations on each keyword node to obtain multiple first clustering clusters; performing keyword extraction on any first target clustering cluster in each first clustering cluster to obtain a first node composed of a plurality of keywords as a parent node of each keyword node in the first target clustering cluster; using the multiple first nodes as keyword nodes input for the next round of target operations;

[0038] The second tree construction unit is configured to use the root node of the hierarchical clustering tree as the parent node of the first nodes obtained by the last round of target operation.

[0039] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method of the first aspect.

[0040] In a fourth aspect, a computing device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method of the first aspect is implemented.

[0041] The present invention proposes a method and device for generating a hierarchical clustering tree based on a document set. The method uses documents in the document set as leaf nodes of the hierarchical clustering tree, then clusters the documents in the document set, extracts keywords in clusters, and uses keyword nodes as parent nodes of document nodes. Next, the operation of clustering-extracting keywords-building parent nodes is repeated for each keyword node, and the hierarchical clustering tree is built layer by layer from bottom to top until clustering can no longer be performed, thereby completing the construction of the hierarchical clustering tree. When searching, you can search from top to bottom level by level to obtain multi-level search results with different levels of detail. You can also flatten the hierarchical clustering tree into a one-dimensional structure and then perform a single search to quickly obtain the search results. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the multiple embodiments disclosed in this specification, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only the multiple embodiments disclosed in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 A schematic diagram of an implementation scenario of a method for generating a hierarchical clustering tree based on a document set disclosed in an embodiment of the present invention;

[0044] Figure 2 A schematic diagram of an implementation scenario of querying based on a hierarchical clustering tree disclosed in an embodiment of the present invention;

[0045] Figure 3 A flowchart of a method for generating a hierarchical clustering tree based on a document set disclosed in an embodiment of the present invention;

[0046] Figure 4 A schematic block diagram of a device for generating a hierarchical clustering tree based on a document set disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] Figure 1 Schematic diagram of an implementation scenario of a method for generating a hierarchical clustering tree based on a document set disclosed in an embodiment of the present invention. Figure 1 As shown in , the method of generating a hierarchical clustering tree starts from the document and builds the hierarchical clustering tree layer by layer from bottom to top. The documents in the document set can be used as leaf nodes of the hierarchical clustering tree, such as Figure 1 As shown on the far left. Using a clustering algorithm, cluster each document based on its content to obtain multiple document clusters. Take the document cluster as a unit, extract keywords from the documents in each document cluster, and obtain the keywords of each document cluster. Use the keywords as the second-level nodes from bottom to top, and as the parent nodes of the leaf nodes corresponding to each document in the corresponding document cluster, as shown in Figure 1 Shown second from the left.

[0049] It should be noted that due to the differences in clustering algorithms, some clustering algorithms generate hard clustering, that is, an element can only belong to one cluster, corresponding to any node in the hierarchical clustering tree has only one parent node, such as the K-means clustering algorithm; other clustering algorithms generate soft clustering, that is, an element can belong to multiple clusters, corresponding to any node in the hierarchical clustering tree can have multiple parent nodes, such as the Gaussian Mixture Models (GMM) clustering algorithm. Figure 1In a node, there can be one or more parent nodes.

[0050] For example, if the content of a document involves multiple fields, such as field A, field B and field C at the same time, then the document may be assigned to three clusters related to field A, field B and field C by the soft clustering algorithm.

[0051] Next, for each keyword node in the second layer, clustering algorithm is used again to cluster and obtain multiple keyword clusters. Keywords are extracted again from the content in each keyword cluster to obtain the keywords of each keyword cluster as the third layer node from bottom to top, corresponding to the parent node of the keyword node, such as Figure 1 The third part from the left.

[0052] Then, repeat the aforementioned clustering-keyword extraction-generation of parent nodes process, continuously extract keywords for keyword clustering and generate nodes at higher levels, until clustering cannot be continued at a certain level. At this point, generate a root node as the parent node of all nodes at this level, and complete the construction of the hierarchical clustering tree. Figure 1 The rightmost part is shown.

[0053] After the hierarchical clustering tree is generated, queries can be performed based on the hierarchical clustering tree. Figure 2 Schematic diagram of the implementation scenario of query based on hierarchical clustering tree disclosed in the embodiment of the present invention. The query method can be divided into two types, namely Figure 2 The tree traversal method shown in (a) and Figure 2 The folding tree method shown in (b).

[0054] like Figure 2 As shown in (a), in the tree traversal method, the query text is first encoded into an embedded representation using the same word embedding method as when building a hierarchical clustering tree. Then, the hierarchical clustering tree is queried layer by layer from top to bottom. Starting from the root node, all child nodes of the root node are traversed, and the contents of these child nodes are compared with the query text for similarity, and the top k nodes are selected as the matching result nodes of the current round, as shown in Figure 2 Then, in the next round of matching, the contents of the child nodes of the matching result nodes of the previous round are compared with the query text for similarity, and the top k nodes are selected as the matching result nodes of the current round. Repeat the above operation until the leaf node layer, and then output the matching result nodes of each layer as the final matching result. The matching results obtained by the tree traversal method can generate different degrees of generalization and specific query results, which are convenient for subsequent use as needed.

[0055] like Figure 2As shown in (b), in the folding tree method, all nodes in the hierarchical clustering tree are directly matched with the query text at one time, and the top m nodes with the highest similarity are determined as query result nodes. This is equivalent to directly converting the hierarchical clustering tree into a one-dimensional structure. Since the folding tree method only has one round of matching, the matching results can be obtained quickly.

[0056] It should be noted that Figure 1 and Figure 2 It does not constitute a limitation to the technical solution of the present invention.

[0057] The specific implementation steps of the method for generating a hierarchical clustering tree based on a document set are described below in conjunction with a specific embodiment.

[0058] Figure 3 The flowchart of a method for generating a hierarchical clustering tree based on a document set disclosed in an embodiment of the present invention is as follows. Figure 3 As shown, the method at least includes: step 301, obtaining a document set containing multiple documents, and taking each document therein as a leaf node of a hierarchical clustering tree; step 302, performing a clustering operation on the documents in the document set to obtain multiple document clustering clusters; step 303, performing keyword extraction on any target clustering cluster in each document clustering cluster, and obtaining a target keyword node composed of several keywords as the parent node of the leaf node corresponding to each document in the target clustering cluster; step 304, performing multiple rounds of target operation on each keyword node until the clustering operation in the target operation cannot be performed; wherein any round of target operation includes: performing a clustering operation on each keyword node to obtain multiple first clustering clusters; performing keyword extraction on any first target clustering cluster in each first clustering cluster, and obtaining a first node composed of several keywords as the parent node of each keyword node in the first target clustering cluster; taking multiple first nodes as keyword nodes input for the next round of target operation; step 305, taking the root node of the hierarchical clustering tree as the parent node of several first nodes obtained by the last round of target operation.

[0059] The specific execution process of each of the above steps is described below.

[0060] First, in step 301, a document set including multiple documents is obtained, and each document in the document is used as a leaf node of a hierarchical clustering tree.

[0061] Documents in a document set can belong to multiple technical fields, and the same document can also belong to multiple technical fields.

[0062] Generating corresponding leaf nodes for each document can form the first layer of nodes from bottom to top in the hierarchical clustering tree.

[0063] Then, in step 302, a clustering operation is performed on the documents in the document set to obtain a plurality of document clusters.

[0064] Before clustering the documents, the documents can be embedded according to their contents to obtain the embedding representations corresponding to each document. The embedding representations can be in vector form. A variety of embedding processing methods can be used, such as word2vec, BERT (Bidirectional Encoder Representations from Transformers), etc., which are not limited here.

[0065] Then, a clustering algorithm is used to cluster each document according to its embedded representation to obtain multiple document clusters.

[0066] A variety of clustering algorithms can be used, such as the K-means algorithm and the DBSCAN algorithm for generating hard clusters, and the Gaussian mixture model algorithm for generating soft clusters, which are not limited here.

[0067] Next, in step 303, keyword extraction is performed on any target cluster in each document cluster to obtain a target keyword node composed of a number of keywords as a parent node of a leaf node corresponding to each document in the target cluster.

[0068] The keyword extraction used in step 303 may be a keyword extraction algorithm, such as TF-IDF algorithm (termfrequency-inverse document frequency), BM25 algorithm (Best Matching25), keyBERT algorithm, etc., which are not limited here.

[0069] One or more keywords can be obtained by extracting keywords from the target cluster. These keywords are regarded as a document, and target keyword nodes corresponding to several keywords are generated. The target keyword node is the parent node of the leaf node corresponding to each document in the target cluster.

[0070] In one embodiment, the clustering algorithm used in the clustering operation of step 302 is a Gaussian mixture model algorithm. In this embodiment, any document may belong to one or more document clusters. Correspondingly, any leaf node of step 303 may have one or more parent nodes.

[0071] By using the process in step 303 for each document clustering cluster respectively, the keyword nodes corresponding to each document clustering cluster and the parent-child node relationship between the keyword nodes and the leaf nodes corresponding to the documents in the document clustering cluster can be obtained, forming the second layer of nodes from bottom to top in the hierarchical clustering tree.

[0072] Then, in step 304, multiple rounds of target operations are performed on each keyword node until the clustering operation in the target operation cannot be performed; wherein, any round of target operation includes: performing clustering operations on each keyword node to obtain multiple first clustering clusters; performing keyword extraction on any first target clustering cluster in each first clustering cluster to obtain a first node composed of several keywords as the parent node of each keyword node in the first target clustering cluster; and using the multiple first nodes as keyword nodes input for the next round of target operation.

[0073] The clustering operation for each keyword node can use a process similar to the process for clustering the documents in step 302, which will not be described in detail here.

[0074] Since several keywords in the keyword node are also regarded as documents, keyword extraction for any first target cluster in each first cluster can use a process similar to that for clustering documents in step 303, which will not be repeated here.

[0075] The first node obtained can be regarded as a keyword extracted from multiple keyword nodes, which is the parent node of these keyword nodes.

[0076] By using the process in step 304 for each first cluster respectively, the first node corresponding to each first cluster and the parent-child node relationship between each first node and the node corresponding to the keyword in the corresponding first cluster can be obtained, forming the third layer to the second to last layer nodes from bottom to top in the hierarchical clustering tree.

[0077] In one embodiment, the clustering algorithm used in the clustering operation of step 304 is a Gaussian mixture model algorithm. In this embodiment, any keyword node may have one or more parent nodes.

[0078] After performing several rounds of target operations, if clustering operations can no longer be performed on nodes at a certain level, or in other words, each node belongs to a separate cluster, the target operation is stopped and the process goes to step 305 .

[0079] Finally, in step 305, the root node of the hierarchical clustering tree is used as the parent node of the first nodes obtained by the last round of target operation.

[0080] Step 305 generates a root node without specific content as the common parent node of each node in the penultimate layer of the hierarchical clustering tree, thereby completing the first construction of the hierarchical clustering tree based on the document set.

[0081] After the initial construction of the hierarchical clustering tree is completed according to the process from step 301 to step 305 , a query may be performed based on the hierarchical clustering tree.

[0082] In some possible implementations, after step 305 , the method further includes steps 306 and 307 .

[0083] In step 306, the query text input by the user is searched in the hierarchical clustering tree to obtain a plurality of query result nodes.

[0084] The query text may be processed by the same embedding method as in step 302 to obtain a query text representation corresponding to the query text. Then, a query is performed in the hierarchical clustering tree based on the query text representation to obtain a plurality of query result nodes.

[0085] Then, in step 307, the contents in the plurality of query result nodes are determined as query results corresponding to the query text.

[0086] In some more specific implementations, step 306 uses a tree traversal method, specifically including step 3061 and step 3062:

[0087] In step 3061, multiple rounds of target matching are performed starting from the root node of the hierarchical clustering tree until the leaf node layer of the hierarchical clustering tree is reached; wherein, any round of target matching includes: determining the similarity between the child nodes of each node input in the current round and the query text, selecting the top k nodes with the highest similarity ranking as the matching result nodes of the current round, and as the input nodes of the next round.

[0088] In the first round of target matching, all child nodes of the root node are matched with the query text by similarity, and the top k nodes with the highest similarity ranking are selected as the matching result nodes of the current round, and these nodes are recorded as set S1.

[0089] In the i-th round, the target matching receives the set S determined in the i-1th round. i-1 , then S i-1 The child nodes of each node in the query text are matched with similarity, and the top k nodes with similarity ranking are selected as the matching result nodes of the current round. These nodes are recorded as a set S i .

[0090] In step 3062, the matching result nodes of each round of target matching are determined as query result nodes.

[0091] Assuming that the hierarchical clustering tree has n+1 layers, step 3062 sets S1, S2, ..., S n Each node in is determined as a query result node.

[0092] In some other more specific implementations, step 306 uses a folding tree method, which specifically includes:

[0093] Step 3063: perform similarity matching between each node in the hierarchical clustering tree and the query text, and determine the top m nodes with the highest similarity ranking as query result nodes.

[0094] After the query result is obtained, the search enhancement generation may be performed based on the query result. In some possible implementations, after step 307 , the method further includes step 308 .

[0095] In step 308, the query result and the prompt word are input into the large language model together to obtain the answer text of the large language model for the prompt word.

[0096] After the hierarchical clustering tree is constructed in step 301 to step 305, the documents in the document set may be updated, for example, a new document is added to the document set, or an existing document is deleted from the document set. At this time, the hierarchical clustering tree needs to be updated accordingly.

[0097] In some possible implementations, after step 305 , the method further includes steps 309 to 311 .

[0098] In step 309, an instruction to add a first document to the document set is received, a first document cluster to which the first document belongs is determined, and the first document cluster is updated.

[0099] The first document cluster may be updated by re-executing a clustering algorithm on the first document cluster to which the first document is added.

[0100] In step 310, a first leaf node corresponding to the first document is generated, a first keyword node corresponding to the first document cluster is used as a parent node of the first leaf node, and the first leaf node is added to the hierarchical clustering tree.

[0101] In step 311, the content in each ancestor node of the first leaf node is updated.

[0102] Each ancestor node of the first leaf node is each node on the path from the first leaf node to the root node.

[0103] In some possible implementations, after step 311, the method further includes:

[0104] Step 312: when the number of documents in any document cluster exceeds a preset first threshold, the tree structure of the hierarchical clustering tree is updated.

[0105] Updating the tree structure of the hierarchical clustering tree may be achieved by re-executing the process from step 301 to step 305 .

[0106] In some possible implementations, after step 305 , the method further includes steps 313 and 314 .

[0107] In step 313, a deletion instruction for a second document in the document set is received, a second document cluster to which the second document belongs is determined, and the second document cluster is updated.

[0108] The second document cluster may be updated by re-executing a clustering algorithm on the second document cluster from which the second document has been deleted.

[0109] In step 314, the second leaf node corresponding to the second document is deleted from the hierarchical clustering tree, and the content in each ancestor node of the second leaf node is updated.

[0110] Each ancestor node of the second leaf node is each node on the path from the second leaf node to the root node.

[0111] In some possible implementations, after step 314, the method further includes:

[0112] Step 315: When the number of documents in any document cluster is less than a preset second threshold, the tree structure of the hierarchical clustering tree is updated.

[0113] Updating the tree structure of the hierarchical clustering tree may be achieved by re-executing the process from step 301 to step 305 .

[0114] The present invention proposes a method that takes the documents in the document set as leaf nodes of a hierarchical clustering tree, then clusters the documents in the document set, extracts keywords in clusters, and takes keyword nodes as parent nodes of document nodes. Next, the operation of clustering-extracting keywords-building parent nodes is repeated for each keyword node, and a hierarchical clustering tree is built layer by layer from bottom to top until clustering can no longer be performed, thereby completing the construction of the hierarchical clustering tree.

[0115] When searching, you can enhance the generation of specific needs based on the search. For example, when you need accurate large language model inference results, you can use the tree traversal method to search from top to bottom, obtain multi-level search results with different levels of detail, and assist the large language model in reasoning. When you need to quickly obtain the inference results of the large language model, you can use the folding tree method to flatten the hierarchical clustering tree into a one-dimensional structure and then perform a single search to quickly obtain the search results.

[0116] According to an embodiment of another aspect, a device for generating a hierarchical clustering tree based on a document set is also provided. Figure 4 This is a schematic block diagram of a device for generating a hierarchical clustering tree based on a document set disclosed in an embodiment of the present invention. The device can be deployed in any device, platform or device cluster with computing and processing capabilities. Figure 4 As shown, the device 400 includes:

[0117] The acquisition unit 401 is configured to acquire a document set including a plurality of documents, and use each document therein as a leaf node of a hierarchical clustering tree;

[0118] A first clustering unit 402 is configured to perform a clustering operation on the documents in the document set to obtain a plurality of document clusters;

[0119] The first keyword extraction unit 403 is configured to perform keyword extraction on any target cluster in each document cluster, and obtain a target keyword node composed of a plurality of keywords as a parent node of a leaf node corresponding to each document in the target cluster;

[0120] The first tree construction unit 404 is configured to perform multiple rounds of target operations on each keyword node until the clustering operation in the target operation cannot be performed; wherein any round of target operations includes: performing clustering operations on each keyword node to obtain multiple first clustering clusters; performing keyword extraction on any first target clustering cluster in each first clustering cluster to obtain a first node composed of a plurality of keywords as a parent node of each keyword node in the first target clustering cluster; using the multiple first nodes as keyword nodes input for the next round of target operations;

[0121] The second tree construction unit 405 is configured to use the root node of the hierarchical clustering tree as the parent node of the first nodes obtained by the last round of target operation.

[0122] In a possible implementation manner, the device 400 further includes:

[0123] The query unit 406 is configured to query the query text input by the user in the hierarchical clustering tree to obtain multiple query result nodes.

[0124] The query result determination unit 407 is configured to determine the contents in the plurality of query result nodes as the query results corresponding to the query text.

[0125] In a possible implementation manner, the device 400 further includes:

[0126] The model calling unit 408 is configured to input the query result and the prompt word into the large language model to obtain the answer text of the large language model for the prompt word.

[0127] In a possible implementation manner, the device 400 further includes:

[0128] The first adding unit 409 is configured to receive an instruction to add a first document to the document set, determine a first document cluster to which the first document belongs, and update the first document cluster.

[0129] The second adding unit 410 is configured to generate a first leaf node corresponding to the first document, use the first keyword node corresponding to the first document cluster as the parent node of the first leaf node, and add the first leaf node to the hierarchical clustering tree.

[0130] The first updating unit 411 is configured to update the content in each ancestor node of the first leaf node.

[0131] In a possible implementation manner, the device 400 further includes:

[0132] The second updating unit 412 is configured to update the tree structure of the hierarchical clustering tree when the number of documents in any document cluster exceeds a preset first threshold.

[0133] In a possible implementation manner, the device 400 further includes:

[0134] The first deleting unit 413 is configured to receive a deletion instruction for a second document in the document set, determine a second document cluster to which the second document belongs, and update the second document cluster.

[0135] The second deleting unit 414 is configured to delete the second leaf node corresponding to the second document from the hierarchical clustering tree, and update the content in each ancestor node of the second leaf node.

[0136] In a possible implementation manner, the device 400 further includes:

[0137] The third updating unit 415 is configured to update the tree structure of the hierarchical clustering tree when the number of documents in any document cluster is less than a preset second threshold.

[0138] According to another aspect of the embodiment, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method described in any of the above embodiments.

[0139] According to yet another embodiment, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in any one of the above embodiments is implemented.

[0140] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0141] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0142] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0143] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0144] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for generating a hierarchical clustering tree based on a document set, comprising: Obtain a document set containing multiple documents, and use each document as a leaf node of a hierarchical clustering tree; Performing a clustering operation on the documents in the document set to obtain a plurality of document clusters; Perform keyword extraction on any target cluster in each document cluster to obtain a target keyword node composed of a number of keywords as a parent node of a leaf node corresponding to each document in the target cluster; Perform multiple rounds of target operations on each keyword node until the clustering operation in the target operation cannot be performed; wherein any round of target operations includes: performing clustering operations on each keyword node to obtain multiple first clustering clusters; performing keyword extraction on any first target clustering cluster in each first clustering cluster to obtain a first node composed of a plurality of keywords as a parent node of each keyword node in the first target clustering cluster; using the multiple first nodes as keyword nodes input for the next round of target operations; The root node of the hierarchical clustering tree is used as the parent node of the first nodes obtained by the last round of target operation.

2. The method according to claim 1, further comprising: Searching the query text input by the user in the hierarchical clustering tree to obtain multiple query result nodes; The contents in the multiple query result nodes are determined as query results corresponding to the query text.

3. The method according to claim 2, characterized in that The query text is a prompt word to be input into the large language model; the method further includes: The query result and the prompt word are input into the large language model together to obtain the answer text of the large language model for the prompt word.

4. The method according to claim 1, further comprising: receiving an instruction to add a first document to the document set, determining a first document cluster to which the first document belongs, and updating the first document cluster; Generate a first leaf node corresponding to the first document, use a first keyword node corresponding to the first document cluster as a parent node of the first leaf node, and add the first leaf node to the hierarchical clustering tree; Update the content of each ancestor node of the first leaf node.

5. The method according to claim 4, further comprising: When the number of documents in any document cluster exceeds a preset first threshold, the tree structure of the hierarchical clustering tree is updated.

6. The method according to claim 1, further comprising: receiving a deletion instruction for a second document in the document set, determining a second document cluster to which the second document belongs, and updating the second document cluster; The second leaf node corresponding to the second document is deleted from the hierarchical clustering tree, and the content in each ancestor node of the second leaf node is updated.

7. The method according to claim 6, further comprising: When the number of documents in any document cluster is less than a preset second threshold, the tree structure of the hierarchical clustering tree is updated.

8. The method according to claim 2, wherein the query text input by the user is queried in the hierarchical clustering tree to obtain a plurality of query result nodes, including: Perform multiple rounds of target matching starting from the root node of the hierarchical clustering tree until reaching the leaf node layer of the hierarchical clustering tree; wherein any round of target matching includes: determining the similarity between the child nodes of each node input in the current round and the query text, selecting the top k nodes with the highest similarity ranking as the matching result nodes of the current round and as the input nodes of the next round; The matching result nodes of each round of target matching are determined as query result nodes.

9. The method according to claim 2, wherein the query text input by the user is queried in the hierarchical clustering tree to obtain a plurality of query result nodes, including: Each node in the hierarchical clustering tree is matched with the query text in terms of similarity, and the top m nodes with the highest similarity ranking are determined as query result nodes.

10. A device for generating a hierarchical clustering tree based on a document set, comprising: An acquisition unit is configured to acquire a document set including a plurality of documents, and use each document therein as a leaf node of a hierarchical clustering tree; A first clustering unit is configured to perform a clustering operation on the documents in the document set to obtain a plurality of document clusters; A first keyword extraction unit is configured to perform keyword extraction on any target cluster in each document cluster to obtain a target keyword node composed of a plurality of keywords as a parent node of a leaf node corresponding to each document in the target cluster; The first tree construction unit is configured to perform multiple rounds of target operations on each keyword node until the clustering operation in the target operation cannot be performed; wherein any round of target operations includes: performing clustering operations on each keyword node to obtain multiple first clustering clusters; performing keyword extraction on any first target clustering cluster in each first clustering cluster to obtain a first node composed of a plurality of keywords as a parent node of each keyword node in the first target clustering cluster; using the multiple first nodes as keyword nodes input for the next round of target operations; The second tree construction unit is configured to use the root node of the hierarchical clustering tree as the parent node of the first nodes obtained by the last round of target operation.

Images