A method for real-time information aggregation to form topics

Abstract

This invention relates to a method for real-time topic generation through information aggregation, belonging to the field of big data technology. The method includes: obtaining key sentences from input text; using the key sentences as input documents, obtaining N candidate documents similar to the input document; constructing N <input document, candidate document> pairs, obtaining the topic corresponding to the input document or determining that there is no corresponding topic for the input document in the k-d tree; and storing the content feature vector of the input document and the topic corresponding to the input document in the k-d tree. The method provided in this application avoids redundant text interference by obtaining key sentences from the input text; simultaneously, by storing the content feature vectors of all documents to be used based on the k-d tree and obtaining N candidate documents similar to the input document in the k-d tree, all features can be fully utilized; furthermore, by leveraging the powerful semantic extraction capabilities of deep learning, the topic to which the document belongs is selected, thereby achieving real-time and effective automatic document topic generation capabilities.

Description

Technical Field

[0001] This invention relates to the field of big data technology, and in particular to a method for real-time information aggregation to form topics. Background Technology

[0002] Currently, techniques for text topic analysis and clustering can be broadly categorized into two types: partitioning clustering and hierarchical clustering. Partitioning clustering involves creating k partitions from a set of documents containing n terms or keywords, each partition representing a cluster, thus generating k topic clusters. Concepts within the same cluster are similar, while concepts in different clusters are dissimilar. Hierarchical clustering, on the other hand, constructs a tree of clusters based on whether the decomposition is bottom-up or top-down. Hierarchical clustering methods can be further divided into clustering methods and splitting clustering methods. Agglomerative clustering treats each concept as a small topic cluster and gradually converges them, while splitting clustering treats all concepts as a large topic cluster and then splits them according to certain criteria.

[0003] However, both partitioning and hierarchical clustering methods rely on manually given cluster boundaries and association coefficients when generating topics, and these parameters are often difficult to define. Furthermore, the topic discovery process of these methods only uses the compositional information of the text, without utilizing deeper semantic information. In business systems, data also contains other descriptive information, such as source and publication time, which these methods do not consider. Additionally, in the general text processing of topics, including main idea information and other relevant information, these methods do not discard irrelevant text when associating topics. Summary of the Invention

[0004] The present invention aims to provide a method for real-time information aggregation to form topics, in order to overcome the shortcomings of the existing technology. The technical problem to be solved by the present invention is achieved through the following technical solution.

[0005] The method for real-time information aggregation and topic formation provided by this invention includes:

[0006] Calculate the similarity between sentences in the input text, calculate the first score for each sentence in the input text based on the similarity between sentences, obtain the second score for each sentence in the input text based on whether each sentence in the input text is the main idea paragraph, and obtain the key sentences in the input text based on the second score;

[0007] Based on a kd-tree, the content feature vectors of all documents to be used are stored. The key sentences in the input text are used as the input document. The content feature vectors of the input document are obtained. The kd-tree query algorithm is used to obtain N candidate documents in the kd-tree that are similar to the input document.

[0008] Construct N pairs of <input documents, candidate documents>, obtain the input string of the BERT pre-trained model based on the N pairs of <input documents, candidate documents>, input the input string into the BERT pre-trained model to obtain a long vector, and obtain the topic corresponding to the input document based on the long vector or determine that there is no topic corresponding to the input document in the kd tree;

[0009] The content feature vector of the input document and the corresponding topic of the input document are stored in a kd-tree.

[0010] In the above scheme, the steps of calculating the similarity between sentences in the input text, calculating a first score for each sentence in the input text based on the similarity between sentences, obtaining a second score for each sentence in the input text based on whether each sentence in the input text is a main idea paragraph, and obtaining key sentences in the input text based on the second score include:

[0011] One sentence in the input text is treated as the query statement, and the remaining sentences are treated as the documents to be queried. The similarity between the sentences in the input text is calculated using the similarity calculation formula in the BM25 algorithm.

[0012] Each sentence in the input text is used as a graph node, and the obtained similarity is used as the edge weight. The first score of each sentence in the input text is calculated using the TextRank scoring formula.

[0013] Based on whether each sentence in the input text is a main idea paragraph, the first score of each sentence in the input text is weighted and calculated to obtain the second score of each sentence;

[0014] The sentences in the input text are sorted from highest to lowest based on the second score, and the top K sentences are selected as the key sentences of the input text.

[0015] In the above scheme, obtaining the content feature vector of the input document includes:

[0016] Obtain the bag-of-words vector of the input document;

[0017] The bag-of-words vector of the input document is calculated using the doc2vec model;

[0018] Obtain a vector of a preset length as the content feature vector of the input document.

[0019] In the above scheme, the steps of constructing N <input document, candidate document> pairs, obtaining the input string of the BERT pre-trained model based on the N <input document, candidate document> pairs, inputting the input string into the BERT pre-trained model to obtain a long vector, and obtaining the topic corresponding to the input document or determining that there is no topic corresponding to the input document in the kd-tree based on the long vector include:

[0020] Construct N pairs of <input documents, candidate documents>, and calculate the feature vector of the context of the input document and the feature vector of the description of similar documents in each of the N pairs of <input documents, candidate documents>.

[0021] The feature vector of the context of the input document is directly concatenated with the feature vector of similar document descriptions to obtain the input string of the BERT pre-trained model;

[0022] The input string is fed into the BERT pre-trained model, which then transforms, modifies, and concatenates the input string to obtain a long vector.

[0023] The long vector is input into the MLP multilayer perceptron, which adds the classification label that cannot be linked at the prediction result label, to obtain the topic corresponding to the input document or to determine that there is no topic corresponding to the input document in the kd tree.

[0024] In the above scheme, the step of inputting the input string into the BERT pre-trained model and using the BERT pre-trained model to transform, convert, and concatenate the input string to obtain a long vector includes:

[0025] The input string is fed into the BERT pre-trained model, which transforms the input string into a text vector token_ids_i and a position vector segment_ids_i, where i is the index of the candidate document;

[0026] Perform a functional transformation on each pair of text vectors token_ids_i and position vectors segment_ids_i;

[0027] The Functional transformation result is input into the concatenate layer for concatenation to obtain a long vector.

[0028] In the above scheme, the step of adding an unlinkable classification label to the prediction result label in the MLP multilayer perceptron by inputting a long vector includes obtaining the topic corresponding to the input document or determining that there is no topic corresponding to the input document in the kd-tree:

[0029] The long vector is input into the Dence layer of the MLP multilayer perceptron, which adds the classification label that cannot be linked at the prediction result label. The Dence layer performs a Dense linear transformation on the input long vector.

[0030] The result of the Dense linear transformation is input into the Softmax layer of the MLP multilayer perceptron, which adds the unlinkable classification label at the prediction result label. The probability distribution of each prediction result is obtained. The prediction result with the highest probability is taken as the topic corresponding to the input document, or it is determined that there is no topic corresponding to the input document in the kd tree.

[0031] In the above scheme, the similarity calculation formula in the BM25 algorithm is:

[0032]

[0033] Among them, V i For the i-th sentence in the input text, V j For the j-th sentence in the input text, v jh To process the j-th sentence V in the input text j The set V obtained by word segmentation j (v j1 v j2 , ..., v jn Terms in ) IDF(v jh (V) is a sentence j Chinese term v jh Inverse document frequency, TF(v) jh V i ) represents the j-th sentence V in the input text. j Chinese term v jh The i-th sentence V in the input text i The frequency of occurrence in the text, avgdL is the average length of all documents in the document to be queried, and k1 and b are both adjustment factors.

[0034] In the above scheme, the TextRank score calculation formula is:

[0035]

[0036] Among them, WS(V j Let V be the j-th sentence in the input text as the graph node. j The obtained TextRank score, BM 25 (V i V j ) represents the i-th sentence V in the input text. i With the j-th sentence V in the input text j Similarity, In(V i ) refers to all pointers to graph node V i The set, Out(V) j ) represents the node V in the graph j The starting point is a set of nodes that point to other nodes.

[0037] In the above scheme, the length of the preset length vector is 128 dimensions.

[0038] In the above scheme, obtaining N candidate documents similar to the input document in the kd-tree using the kd-tree query algorithm includes:

[0039] Starting from the root node of the kd-tree, a depth-first search is used to search until the leaf node is reached. At the same time, the visited nodes are stored sequentially in the kd-tree stack.

[0040] If a leaf node is found, the current leaf node is set as the nearest neighbor node;

[0041] If the current point is closer than the nearest neighbor, update the nearest neighbor node;

[0042] Determine whether the circle with the nearest radius intersects the hyperplane of the parent node;

[0043] When the circle with the nearest distance as its radius intersects the hyperplane of the parent node, then go to the other side of the parent node and use the depth-first search method to search for the nearest neighbor node.

[0044] When the circle with the nearest distance as its radius does not intersect the hyperplane of the parent node, a stack backtracking is performed, and the leaf nodes on the other side of the parent node are eliminated.

[0045] When the search returns to the root node, the search is complete, and N nearest neighbor nodes are obtained;

[0046] The N documents corresponding to the N nearest neighbor nodes are used as N candidate documents similar to the input document.

[0047] The embodiments of the present invention have the following advantages:

[0048] The method for real-time topic generation based on information aggregation provided in this invention avoids redundant text interference by acquiring key sentences from the input text. Simultaneously, it stores the content feature vectors of all documents to be used based on a kd-tree, and obtains N candidate documents similar to the input document in the kd-tree through a kd-tree query algorithm, fully utilizing all features. Furthermore, by constructing N <input document, candidate document> pairs, the method obtains the topic corresponding to the input document or determines that there is no corresponding topic in the kd-tree based on the N <input document, candidate document> pairs. Leveraging the powerful semantic extraction capabilities of deep learning, it selects the topic to which the document belongs, thereby achieving real-time and effective automatic document topic generation. Attached Figure Description

[0049] Figure 1This is a flowchart illustrating the steps of an embodiment of a method for real-time information aggregation and topic formation according to the present invention.

[0050] Figure 2 This is a flowchart of the steps for obtaining key sentences from input text in this invention.

[0051] Figure 3 This is a flowchart of the steps for obtaining the topic corresponding to the input document in this invention.

[0052] Figure 4 This is a flowchart of the steps for obtaining a long vector in this invention.

[0053] Figure 5 This is a flowchart illustrating the steps of obtaining the topic corresponding to the input document based on a long vector in this invention. Detailed Implementation

[0054] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0055] like Figure 1 As shown, the present invention provides a method for real-time information aggregation to form topics, comprising:

[0056] Step S1: Calculate the similarity between sentences in the input text, calculate the first score of each sentence in the input text based on the similarity between sentences, obtain the second score of each sentence in the input text based on whether each sentence in the input text is the main idea paragraph, and obtain the key sentences in the input text based on the second score.

[0057] like Figure 2 As shown, step S1 includes:

[0058] Step S11: Treat one sentence in the input text as a query statement and the remaining sentences as documents to be queried. Calculate the similarity between the sentences in the input text using the similarity calculation formula in the BM25 algorithm.

[0059] Step S12: Using each sentence in the input text as a graph node, the obtained similarity is used as the edge weight, and the first score of each sentence in the input text is calculated using the TextRank score calculation formula;

[0060] Step S13: Based on whether each sentence in the input text is a main idea paragraph, the first score of each sentence in the input text is weighted and calculated to obtain the second score of each sentence;

[0061] Step S14: Sort the sentences in the input text from high to low according to the second score, and select the top K sentences as the key sentences of the input text.

[0062] In this embodiment, the similarity calculation formula in the BM25 algorithm is as follows:

[0063]

[0064] Among them, V i For the i-th sentence in the input text, V j For the j-th sentence in the input text, v jh To process the j-th sentence V in the input text j The set V obtained by word segmentation j (v j1 v j2 , ..., v jn Terms in ) IDF(v jh (V) j Chinese term v jh Inverse document frequency, TF(v) jh V i ) represents the j-th sentence V in the input text. j Chinese term v jh The i-th sentence V in the input text i The frequency of occurrence in the text, avgdL is the average length of all documents in the document to be queried, and k1 and b are both adjustment factors.

[0065] In this embodiment, the TextRank score calculation formula is as follows:

[0066]

[0067] Among them, WS(V j Let V be the j-th sentence in the input text as the graph node. j The obtained TextRank score, BM 25 (V i V j ) represents the i-th sentence V in the input text. i With the j-th sentence V in the input text j Similarity, In(V i ) refers to all pointers to graph node V i The set, Out(V) j ) represents the node V in the graph j The starting point is a set of nodes that point to other nodes.

[0068] Step S2: Store the content feature vectors of all documents to be used based on the kd-tree. Take the key sentences in the input text as the input document, obtain the content feature vector of the input document, and obtain N candidate documents similar to the input document in the kd-tree through the kd-tree query algorithm.

[0069] In this embodiment, the kd-tree is a binary tree, where each node represents a spatial range. The data type of each node in the kd-tree is shown in the table below:

[0070] Table 1. Data type of each node in the kd-tree

[0071]

[0072]

[0073] From the description of the data type of each node in kd above, it can be seen that constructing the kd tree is a recursive process of gradual expansion. The pseudocode for constructing the kd tree is shown in the table below:

[0074]

[0075] By following the above steps to construct the kd-tree, data can be inserted into it, and the calculated document vectors of all documents to be used can be stored in the database.

[0076] In this embodiment, obtaining the content feature vector of the input document includes:

[0077] Obtain the bag-of-words vector of the input document;

[0078] The bag-of-words vector of the input document is calculated using the doc2vec model;

[0079] Obtain a vector of a preset length as the content feature vector of the input document.

[0080] In this embodiment, the length of the preset length vector is 128 dimensions.

[0081] In this embodiment, obtaining N candidate documents similar to the input document in the kd-tree using the kd-tree query algorithm includes:

[0082] Starting from the root node of the kd-tree, a depth-first search is used to search until the leaf node is reached. At the same time, the visited nodes are stored sequentially in the kd-tree stack.

[0083] If a leaf node is found, the current leaf node is set as the nearest neighbor node;

[0084] If the current point is closer than the nearest neighbor, update the nearest neighbor node;

[0085] Determine whether the circle with the nearest radius intersects the hyperplane of the parent node;

[0086] When the circle with the nearest distance as its radius intersects the hyperplane of the parent node, then go to the other side of the parent node and use the depth-first search method to search for the nearest neighbor node.

[0087] When the circle with the nearest distance as its radius does not intersect the hyperplane of the parent node, a stack backtracking is performed, and the leaf nodes on the other side of the parent node are eliminated.

[0088] When the search returns to the root node, the search is complete, and N nearest neighbor nodes are obtained;

[0089] The N documents corresponding to the N nearest neighbor nodes are used as N candidate documents similar to the input document.

[0090] Step S3: Construct N <input document, candidate document> pairs. Obtain the input string of the BERT pre-trained model based on the N <input document, candidate document> pairs. Input the input string into the BERT pre-trained model to obtain a long vector. Obtain the topic corresponding to the input document based on the long vector or determine that there is no topic corresponding to the input document in the kd-tree.

[0091] like Figure 3 As shown, step S3 includes:

[0092] Step S31: Construct N pairs of <input documents, candidate documents>, and calculate the feature vector of the context of the input document and the feature vector of the description of similar documents in each of the N pairs of <input documents, candidate documents>.

[0093] Step S32: Concatenate the feature vector of the context of the input document with the feature vector of similar document descriptions to obtain the input string of the BERT pre-trained model;

[0094] Step S33: Input the input string into the BERT pre-trained model, and use the BERT pre-trained model to transform, convert, and concatenate the input string to obtain a long vector;

[0095] Step S34: Input the long vector into the MLP multilayer perceptron with the unlinkable classification label added at the prediction result mark, and obtain the topic corresponding to the input document or determine that there is no topic corresponding to the input document in the kd tree.

[0096] like Figure 4 As shown, step S33 includes:

[0097] Step S331: Input the input string into the BERT pre-trained model, and use the BERT pre-trained model to convert the input string into a text vector token_ids_i and a position vector segment_ids_i, where i is the index of the candidate document;

[0098] Step S332: Perform a functional transformation on each pair of text vectors token_ids_i and position vectors segment_ids_i;

[0099] Step S333: Input the Functional transformation result into the concatenate layer for concatenation to obtain a long vector.

[0100] like Figure 5 As shown, step S34 includes:

[0101] Step S341: Input the long vector into the Dence layer of the MLP multilayer perceptron model where the classification label that cannot be linked is added at the prediction result label. The Dence layer performs a Dense linear transformation on the input long vector.

[0102] Step S342: Input the Dense linear transformation result into the Softmax layer of the MLP multilayer perceptron model with the unlinkable classification label added at the prediction result mark to obtain the probability distribution of each prediction result. Take the prediction result with the highest probability as the topic corresponding to the input document, or determine that there is no topic corresponding to the input document in the kd tree.

[0103] Step S4: Store the content feature vector of the input document and the corresponding topic of the input document in a kd-tree.

[0104] In one embodiment of the present invention, the present invention also provides a system for real-time topic formation through information aggregation, which uses the above-described method for real-time topic formation through information aggregation to perform information aggregation. The system includes:

[0105] The key sentence acquisition module is used to calculate the similarity between sentences in the input text, calculate the first score of each sentence in the input text based on the similarity between sentences, obtain the second score of each sentence in the input text based on whether each sentence in the input text is a main idea paragraph, and obtain the key sentences in the input text based on the second score;

[0106] The candidate document acquisition module is used to store the content feature vectors of all documents to be used based on a kd-tree. It takes the key sentences in the input text as the input document, obtains the content feature vector of the input document, and obtains N candidate documents similar to the input document in the kd-tree through the kd-tree query algorithm.

[0107] The input document topic acquisition module is used to construct N <input document, candidate document> pairs, obtain the input string of the BERT pre-trained model based on the N <input document, candidate document> pairs, input the input string into the BERT pre-trained model to obtain a long vector, and obtain the topic corresponding to the input document based on the long vector or determine that there is no topic corresponding to the input document in the kd tree;

[0108] The storage module is used to store the content feature vector of the input document and the corresponding topic of the input document into a kd-tree.

[0109] In another embodiment of the present invention, the present invention also provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method for real-time information aggregation to form a topic as described above.

[0110] In another embodiment of the present invention, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of the method for real-time information aggregation to form a topic as described above.

[0111] It should be noted that the above detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0112] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0113] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein.

[0114] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or apparatus.

[0115] For ease of description, spatial relative terms such as "above," "on top of," "on the upper surface of," "above," etc., are used herein to describe the spatial positional relationship of a device or feature as shown in the figures to other devices or features. It should be understood that spatial relative terms are intended to encompass different orientations in use or operation beyond the orientation of the device as described in the figures. For example, if the device in the figures were inverted, a device described as "above" or "on top of" other devices or structures would subsequently be positioned as "below" or "under" other devices or structures. Thus, the exemplary term "above" can include both "above" and "below." The device may also be positioned in other different ways, such as rotated 90 degrees or in other orientations, and the spatial relative descriptions used herein will be interpreted accordingly.

[0116] In the detailed description above, reference has been made to the accompanying drawings, which form part of this document. In the drawings, similar symbols typically identify similar parts unless the context otherwise indicates otherwise. The illustrated embodiments described in the detailed specification, drawings, and claims are not intended to be limiting. Other embodiments may be used and other changes may be made without departing from the spirit or scope of the subject matter presented herein.

[0117] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for real-time information aggregation to form topics, characterized in that, The method includes: Calculate the similarity between sentences in the input text, calculate the first score of each sentence in the input text based on the similarity between sentences, obtain the second score of each sentence in the input text based on whether each sentence in the input text is the main idea paragraph, and obtain the key sentences in the input text based on the second score; The content feature vectors of all documents to be used are stored using a kd-tree. The key sentences in the input text are used as the input documents. The content feature vectors of the input documents are obtained. The kd-tree query algorithm is used to obtain N candidate documents in the kd-tree that are similar to the input documents. Construct N pairs of <input document, candidate document>. Based on these N pairs, obtain the input string for the BERT pre-trained model. Input the input string into the BERT pre-trained model to obtain a long vector. Then, based on the long vector, determine the topic corresponding to the input document or identify a topic for which there is no corresponding topic in the kd-tree. This includes: Construct N pairs of <input documents, candidate documents>, and calculate the feature vector of the context of the input document and the feature vector of the description of similar documents in each of the N pairs of <input documents, candidate documents>. The feature vector of the context of the input document is directly concatenated with the feature vector of similar document descriptions to obtain the input string of the BERT pre-trained model; The input string is fed into the BERT pre-trained model, which then transforms, modifies, and concatenates the input string to obtain a long vector, including: The input string is fed into the BERT pre-trained model, which transforms the input string into a text vector token_ids_i and a position vector segment_ids_i, where i is the index of the candidate document; Perform a functional transformation on each pair of text vectors token_ids_i and position vectors segment_ids_i; The functional transformation result is input into the concatenate layer for concatenation to obtain a long vector. The long vector is input into the MLP multilayer perceptron that adds the unlinkable classification label at the prediction result label, to obtain the topic corresponding to the input document or determine that there is no topic corresponding to the input document in the kd-tree, including: The long vector is input into the Dence layer of the MLP multilayer perceptron, which adds the classification label that cannot be linked at the prediction result label. The Dence layer performs a Dense linear transformation on the input long vector. The Dense linear transformation result is input into the Softmax layer of the MLP multilayer perceptron, which adds the classification label that cannot be linked at the prediction result label. The probability distribution of each prediction result is obtained. The prediction result with the highest probability is taken as the topic corresponding to the input document, or it is determined that there is no topic corresponding to the input document in the kd tree. The content feature vector of the input document and the corresponding topic of the input document are stored in a kd-tree.

2. The method for real-time information aggregation and topic formation according to claim 1, characterized in that, The process involves calculating the similarity between sentences in the input text, calculating a first score for each sentence based on the similarity, obtaining a second score for each sentence based on whether it is a main idea paragraph, and identifying key sentences in the input text based on the second score. One sentence in the input text is treated as the query statement, and the remaining sentences are treated as the documents to be queried. The similarity between the sentences in the input text is calculated using the similarity calculation formula in the BM25 algorithm. Each sentence in the input text is used as a graph node, and the obtained similarity is used as the edge weight. The first score of each sentence in the input text is calculated using the TextRank scoring formula. Based on whether each sentence in the input text is a main idea paragraph, the first score of each sentence in the input text is weighted and calculated to obtain the second score of each sentence; The sentences in the input text are sorted from highest to lowest based on the second score, and the top K sentences are selected as the key sentences of the input text.

3. The method for real-time information aggregation and topic formation according to claim 1, characterized in that, The process of obtaining the content feature vector of the input document includes: Obtain the bag-of-words vector of the input document; The bag-of-words vector of the input document is calculated using the doc2vec model; Obtain a vector of a preset length as the content feature vector of the input document.

4. The method for real-time information aggregation and topic formation according to claim 2, characterized in that, The similarity calculation formula in the BM25 algorithm is as follows: Among them, V i For the i-th sentence in the input text, V j For the j-th sentence in the input text, v jh To process the j-th sentence V in the input text j The set V obtained by word segmentation j (v) j1 v j2 , ..., v jn Terms in ) IDF(v jh ( ) is sentence V j Chinese term v jh Inverse document frequency, TF(v) jh V i ) represents the j-th sentence V in the input text. j Chinese term v jh The i-th sentence V in the input text i The frequency of occurrence in the text, avgdL is the average length of all documents in the document to be queried, and k1 and b are both adjustment factors.

5. The method for real-time information aggregation and topic formation according to claim 2, characterized in that, The formula for calculating the TextRank score is as follows: Among them, WS(V j Let V be the j-th sentence in the input text as the graph node. j The obtained TextRank score, BM 25 (V) i V j (V) represents the i-th sentence in the input text. i With the j-th sentence V in the input text j Similarity, In(V i ) refers to all pointers to graph node V i The set, Out(V) j ) represents the node V in the graph j The starting point is a set of nodes that point to other nodes.

6. The method for real-time information aggregation and topic formation according to claim 3, characterized in that, The preset length vector has 128 dimensions.

7. The method for real-time information aggregation and topic formation according to claim 1, characterized in that, The step of obtaining N candidate documents similar to the input document in the kd-tree using the kd-tree query algorithm includes: Starting from the root node of the kd-tree, a depth-first search is used to search until the leaf node is reached. At the same time, the visited nodes are stored sequentially in the kd-tree stack. If a leaf node is found, the current leaf node is set as the nearest neighbor node; If the current point is closer than the nearest neighbor, update the nearest neighbor node; Determine whether the circle with the nearest radius intersects the hyperplane of the parent node; When the circle with the nearest distance as its radius intersects the hyperplane of the parent node, then go to the other side of the parent node and use the depth-first search method to search for the nearest neighbor node. When the circle with the nearest distance as its radius does not intersect the hyperplane of the parent node, a stack backtracking is performed, and the leaf nodes on the other side of the parent node are eliminated. When the search returns to the root node, the search is complete, and N nearest neighbor nodes are obtained; The N documents corresponding to the N nearest neighbor nodes are used as N candidate documents similar to the input document.

Images