Two-stage clustering-based document intelligent processing method, device, equipment and medium

Abstract

The application relates to the technical field of two-stage clustering-based document intelligent processing, and discloses a two-stage clustering-based document intelligent processing method, device, equipment and medium, wherein the method comprises the following steps: through density clustering on chapter titles, a macro theme structure of a document set can be automatically and efficiently found, and the number of first clustering centers is dynamically determined, so that the blindness of a traditional K-Means method and the like in artificially presetting a K value is fundamentally overcome; secondly, the number of second clustering centers is determined according to the number of first clustering centers, and content clustering is guided. The application has the beneficial effects that: the huge computing cost of directly performing density clustering on high-dimensional full texts is significantly reduced, hierarchical document understanding is realized through semantic complementation of title themes and content details, and finally, a vector index constructed based on content clustering lays a solid foundation for accurate and efficient semantic retrieval and analysis of large-scale documents.

Description

Technical Field

[0001] This invention relates to the field of document intelligent processing technology based on two-level clustering, and in particular to a document intelligent processing method, apparatus, device and medium based on two-level clustering. Background Technology

[0002] With the deepening of digital transformation, enterprises and institutions have accumulated massive amounts of electronic documents (such as regulations, technical standards, contracts, and agreements). Traditional keyword matching or simple vectorized retrieval methods struggle to understand the deep semantics of these documents, resulting in coarse classification, inaccurate retrieval, and weak ability to discover correlations. Existing document clustering methods, such as K-Means, typically require pre-specifying the number of clusters (K value), which can be somewhat blind when dealing with document sets with unknown structures. While density-based clustering methods (such as DBSCAN and HDBSCAN) can automatically determine the number of clusters, they are computationally expensive and parameter-sensitive when processing high-dimensional, large-scale full-text content. Furthermore, a single clustering strategy struggles to simultaneously consider both the macro-level thematic structure of a document (usually reflected in the title) and the micro-level content details. Summary of the Invention

[0003] Therefore, it is necessary to propose a document intelligent processing method, device, equipment and medium based on two-level clustering to address the existing document intelligent processing problem based on two-level clustering.

[0004] A document intelligent processing method based on two-level clustering, the method comprising:

[0005] Obtain the chapter titles of each document in the document set to be processed; wherein, the document set to be processed includes multiple documents to be processed;

[0006] Chapter density clustering is performed based on the chapter titles of each of the documents to be processed to obtain the chapter density clustering results;

[0007] The number of first cluster centers of the document set to be processed is determined based on the chapter density clustering results.

[0008] The number of second cluster centers is determined based on the number of the first cluster centers;

[0009] Obtain the document content of each document to be processed, and perform content clustering based on the number of the second cluster centers to obtain the content clustering result;

[0010] Based on the content clustering results, a vector similarity index is constructed for the set of documents to be processed, and a retrieval database is obtained.

[0011] Further, after the step of constructing a vector similarity index for the set of documents to be processed based on the content clustering results to obtain the retrieval database, the method further includes:

[0012] Get the new input text;

[0013] The new input text is vectorized to obtain the target text vector;

[0014] Based on the similarity between the target text vector and the center vectors of each content cluster in the retrieval database, the main cluster to which the target text vector belongs is determined;

[0015] Based on the primary affiliation cluster, a multi-level similar document retrieval is performed to obtain the retrieval results;

[0016] Based on the search results, determine whether the new input text conflicts with the document to be processed in the search database.

[0017] Further, the step of performing multi-level similar document retrieval based on the primary affiliation cluster to obtain retrieval results includes:

[0018] Calculate the first similarity between the target text vector and the document vectors corresponding to each document to be processed within the main cluster, and collect the documents to be processed with the first similarity greater than the first similarity threshold to obtain the first similar document set;

[0019] Calculate the second similarity between the target text vector and the corresponding center vectors of other non-primary cluster cluster centers, and sort the non-primary clusters according to the second similarity to obtain the sorting result;

[0020] Based on the sorting results, the third similarity between the target text vector and the document vector within each non-primary cluster is calculated sequentially. Documents with a third similarity greater than the second similarity threshold are then grouped together to obtain a second similar document set.

[0021] Based on the first set of similar documents and the second set of similar documents, the final similar document retrieval results are determined;

[0022] Based on the similarity between each document to be processed and the target text vector in the final similar document retrieval results, hierarchical semantic analysis is performed.

[0023] Further, the step of performing hierarchical semantic analysis based on the similarity between each document to be processed and the target text vector in the final similar document retrieval results includes:

[0024] Based on multiple preset similarity threshold intervals, each document to be processed in the final similar document retrieval results is classified; wherein, the multiple similarity threshold intervals include specified threshold intervals and unspecified threshold intervals;

[0025] The documents to be processed within a specified threshold range are input into a preset large language model for deep semantic conflict detection along with the target text vector.

[0026] Perform semantic analysis based on the detection results.

[0027] Further, the step of performing chapter density clustering based on the chapter titles of each of the documents to be processed to obtain the chapter density clustering results includes:

[0028] Preprocess the chapter titles of each of the documents to be processed; wherein, the preprocessing includes removing format characters, truncating the valid text after numbering, filtering invalid titles, and limiting the number of duplicate titles;

[0029] The preprocessed chapter title text is converted into a semantic vector, and the semantic vector is then normalized.

[0030] Hierarchical density clustering algorithm is used to cluster the normalized semantic vectors, and the confidence of outliers in the clustering results is determined and redistributed to obtain the chapter density clustering results.

[0031] Further, the step of obtaining the document content of each of the documents to be processed and performing content clustering based on the number of the second cluster centers to obtain the content clustering result includes:

[0032] Obtain the document content of each document in the set of documents to be processed;

[0033] The content of each document is vectorized to obtain the semantic vector of the document content.

[0034] The K-Means algorithm is used to cluster the semantic vectors of each document content with the second number of cluster centers as the number of cluster centers, thereby obtaining the content clustering result.

[0035] Furthermore, before the step of vectorizing the content of each document to obtain the semantic vector of the document content, the method further includes:

[0036] Calculate the document hash value for each document's content;

[0037] Determine if there are duplicate document contents based on the hash values ​​of each document;

[0038] Duplicate document content is deduplicated, and the deduplicated document content is then vectorized.

[0039] A document intelligent processing device based on two-level clustering, the device comprising:

[0040] The first acquisition module is used to acquire the chapter titles of each document to be processed in the document to be processed set; wherein, the document to be processed set includes multiple documents to be processed;

[0041] The clustering module is used to perform chapter density clustering based on the chapter titles of each document to be processed, and obtain the chapter density clustering results;

[0042] The first determining module is used to determine the number of first cluster centers of the document set to be processed based on the chapter density clustering results;

[0043] The second determining module is used to determine the number of second cluster centers based on the number of the first cluster centers;

[0044] The second acquisition module is used to acquire the document content of each document to be processed, and perform content clustering based on the number of the second cluster centers to obtain the content clustering result;

[0045] The construction module is used to construct a vector similarity index for the set of documents to be processed based on the content clustering results, thereby obtaining a retrieval database.

[0046] An electronic device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the following steps:

[0047] Obtain the chapter titles of each document in the document set to be processed; wherein, the document set to be processed includes multiple documents to be processed;

[0048] Chapter density clustering is performed based on the chapter titles of each of the documents to be processed to obtain the chapter density clustering results;

[0049] The number of first cluster centers of the document set to be processed is determined based on the chapter density clustering results.

[0050] The number of second cluster centers is determined based on the number of the first cluster centers;

[0051] Obtain the document content of each document to be processed, and perform content clustering based on the number of the second cluster centers to obtain the content clustering result;

[0052] Based on the content clustering results, a vector similarity index is constructed for the set of documents to be processed, and a retrieval database is obtained.

[0053] A computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the following steps:

[0054] Obtain the chapter titles of each document in the document set to be processed; wherein, the document set to be processed includes multiple documents to be processed;

[0055] Chapter density clustering is performed based on the chapter titles of each of the documents to be processed to obtain the chapter density clustering results;

[0056] The number of first cluster centers of the document set to be processed is determined based on the chapter density clustering results.

[0057] The number of second cluster centers is determined based on the number of the first cluster centers;

[0058] Obtain the document content of each document to be processed, and perform content clustering based on the number of the second cluster centers to obtain the content clustering result;

[0059] Based on the content clustering results, a vector similarity index is constructed for the set of documents to be processed, and a retrieval database is obtained.

[0060] The beneficial effects of this invention are as follows: By performing density clustering on chapter titles, the macro-thematic structure of a document set can be automatically and efficiently discovered, and the number of first cluster centers can be dynamically determined, fundamentally overcoming the blindness of traditional methods such as K-Means that require manual preset of K values. Secondly, the number of second cluster centers is determined by using the number of first cluster centers to guide content clustering, which significantly reduces the huge computational overhead of directly performing density clustering on high-dimensional full texts. Furthermore, hierarchical document understanding is achieved through the semantic complementarity between title themes and content details. Finally, the vector index built based on content clustering lays a solid foundation for accurate and efficient semantic retrieval and analysis of large-scale documents. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] in:

[0063] Figure 1 This is an application environment diagram of a document intelligent processing method based on two-level clustering in one embodiment;

[0064] Figure 2 This is a flowchart of a document intelligent processing method based on two-level clustering in one embodiment;

[0065] Figure 3 This is a structural block diagram of a document intelligent processing device based on two-level clustering in one embodiment;

[0066] Figure 4 This is a structural block diagram of an electronic device in one embodiment. Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] Figure 1 This is a diagram illustrating an application environment for intelligent document processing based on two-level clustering in one embodiment. (Refer to...) Figure 1 This document intelligent processing method based on two-level clustering is applied to a document intelligent processing system based on two-level clustering. The system includes a terminal 110 and a server 120. The terminal 110 and server 120 are connected via a network. The terminal 110 can be a desktop terminal or a mobile terminal; a mobile terminal can be at least one of a mobile phone, tablet, or laptop. The server 120 can be a standalone server or a server cluster consisting of multiple servers. The terminal 110 is used to retrieve chapter titles, and the server 120 is used to generate the retrieval database.

[0069] like Figure 2 As shown, in one embodiment, a document intelligent processing method based on two-level clustering is provided. This method can be applied to both terminals and servers; this embodiment illustrates its application to a terminal. The document intelligent processing method based on two-level clustering specifically includes the following steps:

[0070] S1: Obtain the chapter titles of each document in the document set to be processed; wherein, the document set to be processed includes multiple documents to be processed;

[0071] S2: Perform chapter density clustering based on the chapter titles of each document to be processed to obtain the chapter density clustering results;

[0072] S3: Determine the number of first cluster centers of the document set to be processed based on the chapter density clustering results;

[0073] S4: Determine the number of second cluster centers based on the number of the first cluster centers;

[0074] S5: Obtain the document content of each document to be processed, and perform content clustering based on the number of the second cluster centers to obtain the content clustering result;

[0075] S6: Construct a vector similarity index for the set of documents to be processed based on the content clustering results to obtain the retrieval database.

[0076] As described in step S1 above, the chapter titles of each document in the document set to be processed are obtained. This document set consists of documents from an enterprise or organization, such as documents used in email communication or for handling matters. These can be recorded in a database, and the information can be retrieved directly from the corresponding database. Then, the chapter titles of each document are extracted from the document set. The chapter titles contain the core thematic information of the document, representing its main content and direction. If a document lacks a title, the first paragraph or summary can be used as a substitute. The document set may include various document types, such as regulations, technical standards, contracts, etc. Effective extraction of chapter titles lays the foundation for subsequent cluster analysis. This process requires document parsing to identify and extract formatted chapter titles, involving preprocessing the text to remove invalid characters and formatting. Furthermore, the system must ensure that each document has at least one non-empty, valid chapter title to avoid invalid data affecting subsequent analysis. Obtaining the title information will further provide necessary data support for chapter clustering, enabling subsequent processing to accurately reflect the overall structure and hierarchy of the document set.

[0077] As described in step S2 above, chapter density clustering is performed based on the chapter titles of each document to be processed to obtain chapter density clustering results. Specifically, the extracted chapter titles are subjected to chapter density clustering. This can be achieved using density clustering algorithms such as HDBSCAN to process the chapter titles and identify their similarities and relationships. Through this clustering method, the system can identify documents with similar themes, thus forming a hierarchical topic structure. Unlike the traditional K-Means algorithm, density clustering is more flexible; it does not require a preset number of clusters and can automatically detect clusters of arbitrary shapes. Therefore, this method is very suitable for large-scale document collections with diverse document types and complex content structures. The clustering results output a set of cluster labels representing the topic cluster to which each chapter title belongs, further providing a basis for determining the number of first and second cluster centers. However, density clustering may also generate noise points, requiring subsequent processing to improve the quality of the clustering results.

[0078] As described in step S3 above, the number of first cluster centers for the document set to be processed is determined based on the chapter density clustering results. After completing the chapter density clustering, the clustering results are analyzed to determine the number of first cluster centers. By statistically analyzing the clustering results, the number and distribution of chapter titles in each cluster are analyzed, and the most representative cluster centers are finally selected. Typically, the statistics of cluster centers are not simply a matter of counting; the selection process is optimized by considering factors such as cluster density and dispersion. Choosing an appropriate number of first cluster centers is crucial; too few or too many cluster centers can lead to information loss or confusion, affecting the overall accuracy of the analysis. Specifically, the number of first cluster centers is determined through comprehensive evaluation. For example, the elbow point of the Elbow method can be used as an initial candidate K value, and the silhouette coefficient and Davies-Bouldin index can be used to verify its rationality. In most cases, this multivariate analysis method helps the system select a suitable number of cluster centers for the dataset, thereby improving the accuracy of subsequent clustering analysis and information retrieval. Through this optimized selection process, the uncertainty in document clustering can be better overcome, ensuring that the final result not only improves the clustering effect but also provides more accurate information retrieval capabilities.

[0079] As described in step S4 above, the number of second cluster centers is determined based on the number of the first cluster centers. The number of second cluster centers is further calculated based on the previously determined number of the first cluster centers. This process is based on an in-depth analysis of the chapter clustering results and aims to help process the content of the document to be processed more precisely. The calculation of the number of second cluster centers typically uses a simple addition method to reserve space for processing small clusters or outliers, for example, setting K2 = K1 + 1, where K1 is the number of the first cluster centers and K2 is the number of the second cluster centers. This design is mainly to reserve space to process smaller clusters or outliers, ensuring that important information or topics are not missed during content clustering. Through this method, the system can improve the coverage and granularity of subsequent content clustering while retaining the main clusters. This reflects a hierarchical understanding of the document set's themes and content, ensuring that not only the overall structure of the document is reflected, but also detailed differences are taken into account, laying a solid foundation for the next step of content clustering.

[0080] As described in step S5 above, the document content of each document to be processed is obtained, and content clustering is performed based on the number of the second cluster centers to obtain the content clustering result. Text content is extracted from the set of documents to be processed, and content clustering is performed according to the previously set number of the second cluster centers. Based on the number of the second cluster centers, the system will use an appropriate clustering algorithm (such as K-Means) to vectorize the text content to obtain semantic vectors of the document content. The vectorization process includes steps such as word segmentation and encoding, which can convert the text content into a set of vectors in a high-dimensional feature space. These vectors numerically reflect the similarity between documents. Subsequently, the system will use clustering algorithms such as K-Means, with the number of the second cluster centers as the key parameter, to cluster the document content. This process allows the system to automatically identify documents with similar semantic features and classify them into the same content cluster. Finally, this step will output the content clustering result, reflecting the topic association between document content, providing detailed information for subsequent construction of a vector similarity index and retrieval database.

[0081] As described in step S6 above, a vector similarity index for the document set to be processed is constructed based on the content clustering results to obtain the retrieval database. The retrieval database is generated by constructing a vector similarity index based on the content clustering results. This involves vectorizing the clustered document content and constructing an index to enable fast and efficient retrieval of similar documents. The system creates a vector representation for each document, reflecting its position and similarity within the cluster. Indexing methods based on inner product or cosine similarity are used, allowing the database to quickly find similar documents with extremely high efficiency when querying new documents. Furthermore, to support large-scale indexing, efficient retrieval libraries such as FAISS can be used to manage and query these vectors. Ultimately, fast retrieval and correlation analysis are achieved, enabling users to quickly find content relevant to their needs from a massive amount of documents. FAISS (Facebook AI Similarity Search) is a similarity retrieval library.

[0082] This method combines multiple techniques, including density clustering, vectorization, and semantic analysis. In the cluster center selection process, density clustering (such as HDBSCAN) first ensures the flexibility and accuracy of document processing, adaptively identifying the natural distribution characteristics of data without a pre-defined number of clusters. By performing density clustering analysis on document chapter titles, the system can dynamically determine the number of cluster centers, thereby optimizing subsequent processing steps. Next, vectorization technology is used to process the document content, transforming textual information into numerical forms easily understood by machines. This ensures the accuracy of subsequent similarity calculations while fully utilizing the rich semantic information of the documents. The embedding models used in the vectorization process (such as BERT and Word2Vec) effectively capture the semantic relationships between texts, improving the clustering and retrieval results. Finally, combined with semantic analysis, the system can deeply understand and analyze the clustering and retrieval results, helping users better identify relevant information and potential semantic conflicts. This combination of methods enables the technology to not only achieve intelligent document processing but also possess high operability and a good user experience in practical applications, fully demonstrating its application potential in document management and intelligent retrieval.

[0083] In one embodiment, after step S6, which involves constructing a vector similarity index for the set of documents to be processed based on the content clustering results to obtain the database retrieval result, the method further includes:

[0084] S701: Get new input text;

[0085] S702: The new input text is vectorized to obtain the target text vector;

[0086] S703: Based on the similarity between the target text vector and the center vectors of each content cluster in the retrieval database, determine the main cluster to which the target text vector belongs;

[0087] S704: Perform multi-level similar document retrieval based on the primary affiliation cluster to obtain retrieval results;

[0088] S705: Determine whether the new input text conflicts with the document to be processed in the search database based on the search results.

[0089] As described in step S701 above, new input text is received from the user or data source. This can be a complete document, a short sentence, or any form of processable text content. Users can upload new text through the interface or API. Depending on the application scenario, the new input text can be content entered by the user when searching for specific documents or information, or it may be new documents that are automatically collected and processed periodically. By acquiring new text, the system can use the previously constructed similarity index to compare the new information with existing documents, thereby identifying relevance or similarity. This process requires the system to effectively identify and parse text in different formats to ensure data consistency and validity. Furthermore, the new input text can be preprocessed, such as through noise reduction, word segmentation, and standardization, which helps with subsequent text vectorization processing, ensuring that the target text accurately reflects its content characteristics.

[0090] As described in step S702 above, the new input text is vectorized to obtain the target text vector. Converting the text into a numerical format that can be understood by a computer, i.e., the target text vector, typically involves several processes, such as word segmentation, word embedding, and normalization. First, the system segments the new text into individual words or phrases. Then, a word embedding model (such as Word2Vec, GloVe, or BERT (Bidirectional Encoder Representations from Transformers)) is used to convert each word into a vector representation, thereby capturing the semantic features of the words. The vectorization of the entire text can be achieved by averaging or weighting the vectors of each word to generate a vector representing the entire text. Furthermore, to improve model performance and facilitate subsequent processing, the vector may also undergo L2 normalization to scale it to the same range, ensuring that the direction (meaning) of the vector is more important than its size when calculating similarity.

[0091] As described in step S703 above, the primary cluster to which the target text vector belongs is determined based on the similarity between the target text vector and the center vectors of each content cluster in the retrieval database. The similarity between the target text vector of the new input text and the center vectors of each content cluster in the retrieval database is calculated. Similarity calculation typically uses indices such as cosine similarity or inner product, which can effectively measure the similarity between two vectors. The system compares the target text vector with each of the center vectors of all content clusters in the retrieval database, and the calculated similarity score helps identify the most relevant content cluster for the new text. Through this process, the system can determine which topic cluster the target text belongs to. The selected "primary cluster" not only reflects the main topic direction of the target text but also greatly simplifies the scope of subsequent searches, ensuring that the search results are concentrated on documents most closely related to the target topic. This approach effectively improves search efficiency, avoids the necessity of performing a complete search of all documents, and makes the entire system faster and more accurate in processing and responding to user requests.

[0092] As described in step S704 above, multi-level similar document retrieval is performed based on the primary belonging cluster to obtain retrieval results. Multi-level similar document retrieval is performed according to the previously determined primary belonging cluster to find the document most similar to the new input text. In the primary belonging cluster, the system calculates the similarity between the target text vector and each document vector within that cluster. This process determines the document most similar to the target text and forms a "first similar document set." Next, the system calculates the similarity between the target text vector and each cluster center in other non-primary belonging clusters, and sorts the non-primary belonging clusters according to these similarities to identify the most relevant candidate active document clusters. Subsequently, the system sequentially retrieves documents within these sorted clusters and calculates the similarity between the target text vector and the document vectors in each non-primary belonging cluster, forming a "second similar document set." Finally, by combining the "first similar document set" and the "second similar document set," the system outputs the final retrieval results, ensuring that users can quickly obtain documents highly relevant to the query text content, improving the accuracy and efficiency of information retrieval, and meeting the user's retrieval needs. It should be noted that the aforementioned content clustering is a document structure that has undergone semantic induction and organization. Using the K-Means algorithm, it divides the massive, unordered full-text content of documents into K2 ordered "content clusters" based on their semantic similarity. Each cluster has a representative "center vector," and documents with high semantic similarity are gathered within the cluster. This process essentially constructs a multi-level, semantic index structure for the entire document library. "Multi-level similar document retrieval" is an intelligent, hierarchical search strategy designed by fully utilizing the above clustering structure. Its "multi-level" nature directly stems from the hierarchical and clustering structure generated by "content clustering." The first level (coarse-grained positioning: determining the primary cluster): The retrieval first uses the "center vectors" of each cluster generated by content clustering to quickly locate the query text to the most relevant "primary cluster." This is equivalent to using the clustering structure at the very beginning of the retrieval to drastically narrow the search scope from the "global document library" to "a most relevant semantic subset," achieving a significant leap in retrieval efficiency. The second level (fine-grained search: intra-cluster refinement and cross-cluster expansion): After determining the primary cluster, the system performs refined similarity calculations within that cluster to obtain the most core similar documents (the first set of similar documents). Next, based on the similarity of the query text to other cluster center vectors, the system prioritizes non-primary clusters and systematically expands the search within these potentially related clusters (obtaining the second set of similar documents). This strategy ensures a balance between recall and precision: it guarantees the accuracy of core results within the primary cluster while avoiding the omission of potentially related documents through orderly cross-cluster searches.

[0093] As described in step S705 above, the system determines whether the new input text conflicts with the documents to be processed in the retrieval database based on the search results. Based on the search results obtained from multi-level similarity document retrieval (i.e., several documents most similar to the target text vector), the similarity between the target text vector and each search result document vector is calculated. If at least one document vector has a similarity to the target text vector exceeding a preset conflict threshold, the new input text is determined to conflict with the documents to be processed in the retrieval database (i.e., semantically identical); otherwise, it is determined not to conflict. The conflict threshold is usually set very high, for example, 0.85, to ensure that only highly similar semantically texts are considered to conflict. Alternatively, completely identical text can be directly considered to conflict (through string matching). If the search results are empty, there is obviously no conflict. If the search results are not empty, but the similarity does not reach the conflict threshold, it is determined not to conflict. Finally, based on the determination result, corresponding operations can be performed, such as refusing to add new text if there is a conflict, or recording the conflicting documents and prompting the user.

[0094] In one embodiment, step S704, which involves performing multi-level similar document retrieval based on the primary affiliation cluster to obtain retrieval results, includes:

[0095] S7041: Calculate the first similarity between the target text vector and the document vectors corresponding to each document to be processed within the main cluster, and collect the documents to be processed with the first similarity greater than the first similarity threshold to obtain the first similar document set;

[0096] S7042: Calculate the second similarity between the target text vector and the corresponding center vectors of other non-primary cluster cluster centers, and sort the non-primary clusters according to the second similarity to obtain the sorting result;

[0097] S7043: Based on the sorting results, calculate the third similarity between the target text vector and the document vector within each non-primary cluster in sequence, and collect the documents to be processed with a third similarity greater than the second similarity threshold to obtain the second similar document set.

[0098] S7044: Based on the first similar document set and the second similar document set, determine the final similar document retrieval result;

[0099] S7045: Based on the similarity between each document to be processed and the target text vector in the final similar document retrieval results, perform hierarchical semantic analysis.

[0100] As described in step S7041 above, the first similarity between the target text vector and the document vectors corresponding to each document to be processed within the main cluster is calculated. Documents with a first similarity greater than a first similarity threshold are then grouped together to obtain a first similar document set. Similarity calculation typically uses methods such as cosine similarity or inner product to obtain a quantitative similarity score. First, the system compares the vector of each document within the main cluster with the target text vector to obtain a first similarity score. Next, the system applies a preset first similarity threshold to filter out documents with high similarity and good relevance to the target text. The first similarity threshold can be set to 0.7. The filtered documents are collected into a new set, forming the first similar document set. In this way, the system can efficiently focus on the most relevant documents, avoiding data redundancy and improving retrieval efficiency and accuracy.

[0101] As described in step S7042 above, the second similarity between the target text vector and the corresponding center vectors of each non-primary cluster is calculated. The non-primary clusters are then ranked according to these second similarities to obtain a ranking result. The second similarity between the target text vector and the corresponding center vectors of each cluster center in the non-primary clusters is calculated. This process aims to evaluate the relevance of the target text to other topic clusters (i.e., clusters different from the primary cluster). Through this calculation, the system can identify potentially relevant topics, even if these topics are not directly classified into the primary cluster of the target text. The similarity calculation method is the same, using methods such as cosine similarity or inner product. After the calculation, the system compares the obtained similarity scores and ranks the non-primary clusters accordingly, forming a ranking result from high to low. The ranking result will indicate other topics most similar to the target text, allowing the system to perform cross-topic searches more flexibly. The goal of this step is to expand the sources of similar documents, especially when the user's intent is ambiguous or involves multiple topics, ensuring the comprehensiveness of relevant information. This method not only enhances the breadth of document retrieval but also provides users with a rich selection of documents.

[0102] As described in step S7043 above, based on the sorting results, the third similarity between the target text vector and the document vectors within each non-primary cluster is calculated sequentially. Documents with a third similarity greater than the second similarity threshold are grouped together to obtain a second similar document set. Further similarity calculations are then performed on the sorted non-primary clusters. First, the system iterates through the documents in each non-primary cluster, calculating the third similarity between these document vectors and the target text vector. The similarity calculation method is the same as in the previous steps, using cosine similarity or inner product methods. Next, the system compares these calculated similarity values ​​with the obtained second similarity threshold. The second similarity threshold can be the same as or different from the first similarity threshold; for example, it can be set to 0.6. Only when the third similarity of a document exceeds this threshold will that document be included in the set, forming the second similar document set. The documents in this set are those that have a certain similarity to the target text, but may not be directly classified into the primary cluster; rather, they may be cross-topic related content.

[0103] As described in step S7044 above, the final similar document retrieval results are determined based on the first similar document set and the second similar document set. The final similar document retrieval results are integrated and determined based on the previously generated first and second similar document sets, as this directly affects the quality and validity of the retrieval results received by the user. Documents in these two sets are merged to ensure that each document is considered and to further avoid duplicate documents. After merging, the system sorts the integrated documents according to their similarity to the target text vector, prioritizing the most relevant documents. This typically involves evaluating the similarity score of each document. Through this sorting, users can quickly locate documents highly relevant to their query, thereby improving retrieval efficiency and accuracy. The final generated retrieval results are available for users to view, allowing them to quickly find the information they need based on their own needs and query objectives. Furthermore, the system can perform further analysis, such as providing document recommendations and classifications, resulting in a more optimized user experience.

[0104] In one embodiment, step S7045, which involves performing hierarchical semantic analysis based on the similarity between each document to be processed and the target text vector in the final similar document retrieval results, includes:

[0105] S70541: Classify each document to be processed in the final similar document retrieval result according to multiple preset similarity threshold intervals; wherein, the multiple similarity threshold intervals include specified threshold intervals and unspecified threshold intervals;

[0106] S70542: Input the document to be processed within the specified threshold range and the target text vector into a preset large language model for deep semantic conflict detection;

[0107] S70543: Perform semantic analysis based on the detection results.

[0108] As described in step S70541 above, the documents to be processed in the final similar document retrieval results are classified according to multiple preset similarity threshold intervals. The preset similarity threshold intervals mainly include two parts: specified threshold intervals and unspecified threshold intervals. Specified threshold intervals generally refer to documents that are highly similar to the target text, with similarity scores within a certain range (e.g., above 0.9), while unspecified threshold intervals refer to documents with lower similarity scores, such as between 0.6 and 0.9. This classification helps the system to rationally organize the documents to be processed according to importance and relevance, ensuring the flexibility and efficiency of subsequent processing. During the classification process, the system compares the similarity score of each document one by one to determine its corresponding threshold interval. The output results provide a basic data structure for subsequent semantic analysis, enabling targeted analysis and detection of documents of different categories. Through reasonable classification, the system can intelligently distinguish the relevance of documents in subsequent processing stages, thereby improving the accuracy and effectiveness of the information obtained by the end user.

[0109] As described in step S70542 above, the documents to be processed within the specified threshold range and the target text vector are input into a preset large language model for deep semantic conflict detection. Documents to be processed that remain within the specified threshold range are then analyzed in depth. The content of these documents, along with the previously extracted target text vector, is input into a preset large language model (such as the GPT (Generative Pre-trained Transformer) series or BERT) for deep semantic conflict detection. The core objective of this process is to assess whether there are semantic conflicts or inconsistencies between the target text and these highly related documents. Through the powerful capabilities of the large language model, a deep understanding and contextual analysis of the document content can be performed, identifying potential semantic contradictions, opposing viewpoints, or pragmatic differences. This deep semantic detection significantly improves the intelligence of document analysis, revealing deep-seated problems that are difficult to discover using traditional methods alone. Based on the detection results output by the model, the system marks potentially conflicting documents for further processing. This step not only reveals the true connections between documents but also provides users with more detailed and accurate information, helping them make better decisions in complex information environments. It should be noted that the large language model is used for "deep semantic analysis," while vector retrieval is used for "preliminary similarity screening," and there is no overlap between the two.

[0110] As described in step S70543 above, semantic analysis is performed based on the detection results. Subsequent semantic analysis is then performed based on the obtained deep semantic conflict detection results. This process focuses on the analytical information output from the large language model, including the relationships between the documents to be processed and the target text, viewpoint consistency, and potential semantic conflicts. First, the system organizes the conflict results given by the language model to identify the most important semantic differences and whether there are any inconsistencies in information. This is crucial because understanding these differences helps users discover potential problems and avoid decision-making errors caused by information misunderstandings. Next, the system integrates the analysis results from different documents to form an overall semantic analysis report. The report will include documents related to the target text, key semantic consistency or conflict analysis, and possible content recommendations. In this way, users can not only gain a clear understanding of the target text but also establish extensive semantic connections between documents, providing a basis and guidance for future document retrieval and content management.

[0111] In one embodiment, step S2, which involves performing chapter density clustering based on the chapter titles of each of the documents to be processed to obtain the chapter density clustering results, includes:

[0112] S201: Preprocess the chapter titles of each of the documents to be processed; wherein, the preprocessing includes removing format characters, truncating the valid text after numbering, filtering invalid titles, and limiting the number of duplicate titles;

[0113] S202: Convert the preprocessed chapter title text into a semantic vector, and normalize the semantic vector;

[0114] S203: Use the hierarchical density clustering algorithm to cluster the normalized semantic vectors, and determine and redistribute the confidence of outliers in the clustering results to obtain the chapter density clustering results.

[0115] As described in step S201 above, the chapter titles of the document to be processed are preprocessed to ensure the accuracy and effectiveness of subsequent clustering operations. The preprocessing process typically includes several sub-steps: First, formatting characters are removed. Special characters or redundant spaces that may exist in the chapter titles are removed to ensure the accuracy of text comparison and analysis. Next, the system extracts the valid text after the number. This is necessary because many chapter titles may contain numerical numbers, and the core information that needs attention is usually after the number, which helps to extract meaningful title content. Furthermore, the system filters invalid titles, such as titles shorter than a certain threshold (e.g., less than 3 characters) or completely blank titles, to avoid contaminating the clustering of invalid data. Finally, to further improve the clustering effect, the system can also limit the number of duplicate titles to ensure that each title is representative in the dataset and avoid distortion of clustering results due to redundant data.

[0116] As described in step S202 above, the preprocessed chapter title text is converted into semantic vectors, and these semantic vectors are then normalized. Vectorization of each chapter title typically uses pre-trained embedding models such as Word2Vec, GloVe, or BERT. These models consider the contextual relationships between words, mapping the title text to vectors in a high-dimensional space. This makes the vector connections between titles with similar meanings more approximate. After vectorization, the system normalizes the generated semantic vectors. Normalization adjusts the vectors to unit length, meaning that regardless of the original size of the vectors, they are ultimately comparable in direction, avoiding unnecessary bias caused by differences in vector size. This helps improve the effectiveness of subsequent clustering and provides a fair basis for calculating document similarity (such as cosine similarity). The semantic vectors formed by the vectorized and normalized chapter titles provide data support for subsequent clustering analysis, ensuring that the clustering results accurately reflect the thematic connections between documents.

[0117] As described in step S203 above, a hierarchical density clustering algorithm is used to cluster the normalized semantic vectors, and outliers in the clustering results are assessed for confidence and reassigned. The confidence assessment is based on a distance threshold between the vector distance and the cluster center, thus obtaining the chapter density clustering results. A hierarchical density clustering algorithm (such as HDBSCAN) is used to perform cluster analysis on the previously obtained normalized semantic vectors to identify the document's topic structure. Hierarchical density clustering can automatically identify and adjust the number of clusters based on the natural distribution of the data without pre-specifying the number of clusters. During entity clustering, the system references the density information surrounding each data point, thus better identifying related headings and grouping them into the same cluster. This method can not only handle complex data clusters but also effectively identify outliers, i.e., chapter headings with weak relationships to other documents. For identified outliers, the system assesses confidence to determine whether they should belong to a main cluster. If an outlier has a high confidence level, the system reassigns it to the most suitable cluster. Finally, after the above operations, the system will output stable chapter density clustering results. This result not only refers to the number of clusters and the center index, but also provides in-depth structured information for subsequent analysis and decision-making. HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise) is a hierarchical density clustering algorithm.

[0118] In one embodiment, step S5, which involves obtaining the document content of each of the documents to be processed and performing content clustering based on the number of the second cluster centers to obtain the content clustering result, includes:

[0119] S501: Obtain the document content of each document to be processed in the set of documents to be processed;

[0120] S502: Vectorize the content of each document to obtain a semantic vector of the document content;

[0121] S503: Using the K-Means algorithm with the second number of cluster centers as the number of cluster centers, the semantic vectors of each document content are clustered to obtain the content clusters with the second number of cluster centers and the center vector of each content cluster, thereby obtaining the content clustering result.

[0122] As described in step S501 above, the document content of each document in the document set to be processed is obtained. The complete content of each document is extracted from the document set. Document content typically includes core information such as the main text, figures, appendices, and other textual sections. This information is crucial for understanding the document's theme and background. To achieve this, the system needs to ensure that all documents can be effectively parsed, regardless of their format (e.g., PDF, Word document, or plain text). During this process, the system will use appropriate parsing tools and techniques for different document formats to extract valid text content. In the extraction process, the system should also consider the text's layout and structure to avoid information loss or the parsed content losing its context. After obtaining the document content, the system will store and manage this content for subsequent processing and analysis.

[0123] As described in step S502 above, the document content is vectorized to obtain semantic vectors. The system vectorizes the acquired document content, converting it into a numerical form that can be used for computation. By mapping the text to a high-dimensional space, a foundation is provided for the feature representation of the text. First, the system typically segments the document content into words, identifying words and phrases in the text. Next, the system uses advanced word embedding models (such as Word2Vec, GloVe, or BERT) to convert each word or phrase into a corresponding semantic vector. This mapping not only preserves the semantic information of words but also considers the contextual relationships between words. To further improve the model's performance, the vector representation of the entire document is usually derived from the sum of individual word vectors, which can be achieved through simple averaging or weighted summation. Simultaneously, to ensure that the generated document content vectors are not affected by vector size during subsequent similarity calculations, the system can normalize them.

[0124] As described in step S503 above, the K-Means algorithm is used to cluster the semantic vectors of each document content using the second number of cluster centers as the number of cluster centers, resulting in content clusters with the second number of cluster centers and the center vector of each content cluster, thus obtaining the content clustering result. The K-Means clustering algorithm is used to perform cluster analysis on the previously generated semantic vectors of document content, using the second number of cluster centers (K2) as the number of clusters to implement a systematic classification of the document content. First, the system randomly selects K2 document content vectors as initial cluster centers based on the value of K2. Subsequently, the remaining document content vectors are assigned to the nearest cluster based on their distance from the current cluster center (usually using Euclidean distance or cosine similarity). Based on this, the system recalculates the center of each cluster, typically by taking the average of all document vectors in the cluster. This process iterates until the cluster centers no longer change, or the change is less than a preset threshold. Finally, the step outputs the content clustering result, including the determined content clusters and the center vector corresponding to each content cluster. These content clusters will represent different topics or information categories within the document collection, making subsequent information retrieval and semantic analysis more efficient and accurate.

[0125] In one embodiment, before step S502 of vectorizing the content of each document to obtain the semantic vector of the document content, the method further includes:

[0126] S5011: Calculate the document hash value for each document's content;

[0127] S5012: Determine whether there is duplicate document content based on the hash value of each document;

[0128] S5013: Perform deduplication on duplicate document content, and then vectorize the deduplicated document content.

[0129] As described in step S5011 above, a document hash value is calculated for each document content. The document hash value is obtained by converting the document content into a fixed-length string representation using a hash algorithm (such as SHA-256 or MD5). The core of this process is compressing the document content to enable efficient comparison and identification. Hash values ​​are unique and efficient, characterized by the fact that the same input will always produce the same hash value, while different inputs will generate different hash values ​​as much as possible. This means that even a small change to the document content (such as changing a single letter) will result in a completely new hash value. Therefore, by calculating the hash value of the document content, the system can quickly determine the uniqueness of the document. This also provides the foundational data for duplicate document detection in subsequent steps. As a digital fingerprint of the document content, the document hash value not only facilitates subsequent comparisons but also reduces the data overhead when storing and processing similar documents, providing the intelligent processing system with high efficiency and scalability.

[0130] As described in step S5012 above, the system determines whether duplicate document content exists based on the hash values ​​of each document. This is achieved by comparing the hash values ​​of all documents. The system first establishes a hash value storage structure, typically using a set or dictionary for quick lookup and judgment. It iterates through the hash values ​​of each document. If a document's hash value already exists in the hash storage structure, it indicates that the document content already exists in the set, meaning it is a duplicate document. Conversely, if the hash value is not in the storage structure, it indicates that the document content is unique and its hash value can be stored for further processing. This method allows the system to efficiently identify duplicate documents without directly comparing content, improving efficiency and reducing computational burden.

[0131] As described in step S5013 above, duplicate document content is deduplicated to allow for vectorization. Based on the previous hash value detection results, the identified duplicate document content is deduplicated. The main goal of deduplication is to retain a copy of each unique document and delete all other duplicate documents to prevent interference from redundant data in subsequent vectorization and clustering processes. Specifically, the system first iterates through the document set to be processed. For each document, its hash value is used to determine if it is a duplicate. If a duplicate is detected, the system will not vectorize it but will exclude it from subsequent processing. For unique documents, the system ensures that all document content can effectively enter the vectorization stage. This deduplication process also enhances the accuracy of the final clustering results, avoids artificially high similarity between different document contents, and ensures the representativeness of the target vectors. The clean, deduplicated document set provides a direct and accurate data source for subsequent content vectorization, improving the processing efficiency of the entire intelligent document processing system and ensuring the quality and reliability of the final output. This process plays a crucial role in information retrieval and knowledge management, providing users with high-quality information support.

[0132] Reference Figure 3 The present invention also provides a document intelligent processing device based on two-level clustering, the device comprising:

[0133] The first acquisition module 902 is used to acquire the chapter titles of each document to be processed in the document to be processed set; wherein, the document to be processed set includes multiple documents to be processed;

[0134] Clustering module 904 is used to perform chapter density clustering based on the chapter titles of each of the documents to be processed, and obtain chapter density clustering results;

[0135] The first determining module 906 is used to determine the number of first cluster centers of the document set to be processed based on the chapter density clustering results.

[0136] The second determining module 908 is used to determine the number of second cluster centers based on the number of the first cluster centers;

[0137] The second acquisition module 910 is used to acquire the document content of each of the documents to be processed, and perform content clustering based on the number of the second cluster centers to obtain the content clustering result;

[0138] The construction module 912 is used to construct a vector similarity index of the document set to be processed based on the content clustering results, so as to obtain the retrieval database.

[0139] In one embodiment, the document intelligent processing device based on two-level clustering further includes:

[0140] The new input text acquisition module is used to acquire new input text.

[0141] The vectorization module is used to vectorize the new input text to obtain the target text vector.

[0142] The primary affiliation cluster determination module is used to determine the primary affiliation cluster of the target text vector based on the similarity between the target text vector and the center vectors of each content cluster in the retrieval database;

[0143] The multi-level similar document retrieval module is used to perform multi-level similar document retrieval based on the main belonging cluster and obtain retrieval results.

[0144] In one embodiment, the multi-level similar document retrieval module includes:

[0145] The first similarity calculation submodule is used to calculate the first similarity between the target text vector and the document vectors corresponding to each document to be processed in the main cluster, and to collect the documents to be processed with the first similarity greater than the first similarity threshold to obtain the first similar document set.

[0146] The second similarity calculation submodule is used to calculate the second similarity between the target text vector and the corresponding center vectors of other non-primary cluster cluster centers, and sort the non-primary clusters according to the second similarity to obtain the sorting result;

[0147] The third similarity calculation submodule is used to calculate the third similarity between the target text vector and the document vector in each non-primary cluster according to the sorting result, and to collect the documents to be processed with the third similarity greater than the second similarity threshold to obtain the second similar document set.

[0148] The final similar document retrieval result determination submodule is used to determine the final similar document retrieval result based on the first similar document set and the second similar document set;

[0149] The hierarchical semantic analysis submodule is used to perform hierarchical semantic analysis based on the similarity between each document to be processed and the target text vector in the final similar document retrieval results.

[0150] In one embodiment, the hierarchical semantic analysis submodule includes:

[0151] A classification unit is used to classify each document to be processed in the final similar document retrieval result according to a plurality of preset similarity threshold intervals; wherein, the plurality of similarity threshold intervals include specified threshold intervals and unspecified threshold intervals;

[0152] The deep semantic conflict detection unit is used to input the document to be processed within a specified threshold range and the target text vector into a preset large language model for deep semantic conflict detection.

[0153] The semantic analysis unit is used to perform semantic analysis based on the detection results.

[0154] In one embodiment, the clustering module 904 includes:

[0155] The preprocessing submodule is used to preprocess the chapter titles of each of the documents to be processed; wherein, the preprocessing includes removing format characters, truncating the valid text after numbering, filtering invalid titles, and limiting the number of duplicate titles;

[0156] The text conversion submodule is used to convert the preprocessed chapter title text into semantic vectors and to normalize the semantic vectors.

[0157] The clustering submodule is used to cluster the normalized semantic vectors using the hierarchical density clustering algorithm, and to determine the confidence level and redistribute the outliers in the clustering results to obtain the chapter density clustering results.

[0158] In one embodiment, the second acquisition module 910 includes:

[0159] The document content acquisition submodule is used to acquire the document content of each document to be processed in the document collection to be processed;

[0160] The document content vectorization processing submodule is used to vectorize the content of each document to obtain the semantic vector of the document content;

[0161] The document content semantic vector clustering submodule is used to cluster each document content semantic vector using the K-Means algorithm with the second number of cluster centers as the number of cluster centers, to obtain the content clusters with the second number of cluster centers and the center vector of each content cluster, thereby obtaining the content clustering result.

[0162] In one embodiment, the second acquisition module 910 further includes:

[0163] The document hash value calculation submodule is used to calculate the document hash value of each document's content;

[0164] The duplicate document content detection submodule is used to determine whether duplicate document content exists based on the hash value of each document.

[0165] The deduplication submodule is used to deduplicatize duplicate document content, and then vectorize the deduplicated document content.

[0166] Figure 4An internal structural diagram of an electronic device in one embodiment is shown. This electronic device can specifically be a terminal or a server, and more specifically, a computer device. Figure 4 As shown, the electronic device includes a processor, a memory, and a network interface connected via a system bus. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and may also store a computer program. When executed by the processor, this computer program enables the processor to implement a document intelligent processing method based on two-level clustering. The internal memory may also store a computer program, which, when executed by the processor, enables the processor to implement the document intelligent processing method based on two-level clustering. Those skilled in the art will understand that... Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0167] In one embodiment, an electronic device is provided, including a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the following steps:

[0168] Obtain the chapter titles of each document in the document set to be processed; wherein, the document set to be processed includes multiple documents to be processed;

[0169] Chapter density clustering is performed based on the chapter titles of each of the documents to be processed to obtain the chapter density clustering results;

[0170] The number of first cluster centers of the document set to be processed is determined based on the chapter density clustering results.

[0171] The number of second cluster centers is determined based on the number of the first cluster centers;

[0172] Obtain the document content of each document to be processed, and perform content clustering based on the number of the second cluster centers to obtain the content clustering result;

[0173] Based on the content clustering results, a vector similarity index is constructed for the set of documents to be processed, and a retrieval database is obtained.

[0174] By performing density clustering on chapter titles, the macro-thematic structure of a document set can be automatically and efficiently discovered, and the number of first cluster centers can be dynamically determined. This fundamentally overcomes the blindness of traditional methods such as K-Means, which require manual pre-setting of K values. Secondly, the number of second cluster centers is determined by the number of first cluster centers to guide content clustering. This significantly reduces the huge computational overhead of directly performing density clustering on high-dimensional full texts. Furthermore, hierarchical document understanding is achieved through the semantic complementarity between title themes and content details. Finally, the vector index built based on content clustering lays a solid foundation for accurate and efficient semantic retrieval and analysis of large-scale documents.

[0175] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, causes the processor to perform the following steps:

[0176] Obtain the chapter titles of each document in the document set to be processed; wherein, the document set to be processed includes multiple documents to be processed;

[0177] Chapter density clustering is performed based on the chapter titles of each of the documents to be processed to obtain the chapter density clustering results;

[0178] The number of first cluster centers of the document set to be processed is determined based on the chapter density clustering results.

[0179] The number of second cluster centers is determined based on the number of the first cluster centers;

[0180] Obtain the document content of each document to be processed, and perform content clustering based on the number of the second cluster centers to obtain the content clustering result;

[0181] Based on the content clustering results, a vector similarity index is constructed for the set of documents to be processed, and a retrieval database is obtained.

[0182] By performing density clustering on chapter titles, the macro-thematic structure of a document set can be automatically and efficiently discovered, and the number of first cluster centers can be dynamically determined. This fundamentally overcomes the blindness of traditional methods such as K-Means, which require manual pre-setting of K values. Secondly, the number of second cluster centers is determined by the number of first cluster centers to guide content clustering. This significantly reduces the huge computational overhead of directly performing density clustering on high-dimensional full texts. Furthermore, hierarchical document understanding is achieved through the semantic complementarity between title themes and content details. Finally, the vector index built based on content clustering lays a solid foundation for accurate and efficient semantic retrieval and analysis of large-scale documents.

[0183] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0184] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0185] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A document intelligent processing method based on two-level clustering, characterized in that, The method includes: Obtain the chapter titles of each document in the document set to be processed; wherein, the document set to be processed includes multiple documents to be processed; Chapter density clustering is performed based on the chapter titles of each of the documents to be processed to obtain the chapter density clustering results; The number of first cluster centers of the document set to be processed is determined based on the chapter density clustering results; wherein, the number of first cluster centers is determined by using the elbow point of the Elbow method as the initial candidate K value based on the chapter density clustering results, and verifying it using the silhouette coefficient and Davies-Bouldin index to obtain the number of first cluster centers. The number of second cluster centers is determined based on the number of the first cluster centers; Obtain the document content of each document to be processed, and perform content clustering based on the number of the second cluster centers to obtain the content clustering result; Based on the content clustering results, a vector similarity index is constructed for the set of documents to be processed, and a retrieval database is obtained; The step of performing chapter density clustering based on the chapter titles of each of the documents to be processed to obtain the chapter density clustering results includes: Preprocess the chapter titles of each of the documents to be processed; wherein, the preprocessing includes removing format characters, truncating the valid text after numbering, filtering invalid titles, and limiting the number of duplicate titles; The pre-trained embedding model is used to convert the pre-processed chapter title text into semantic vectors, and the semantic vectors are then normalized. Hierarchical density clustering algorithm is used to cluster the normalized semantic vectors, and the confidence of outliers in the clustering results is determined and redistributed to obtain the chapter density clustering results. After the step of constructing a vector similarity index for the set of documents to be processed based on the content clustering results to obtain the retrieval database, the method further includes: Get the new input text; The new input text is vectorized to obtain the target text vector; Based on the similarity between the target text vector and the center vectors of each content cluster in the retrieval database, the main cluster to which the target text vector belongs is determined; Based on the primary affiliation cluster, a multi-level similar document retrieval is performed to obtain the retrieval results; Based on the search results, determine whether the new input text conflicts with the document to be processed in the search database; if at least one document vector and the target text vector have a similarity exceeding a preset conflict threshold, then determine that the new input text conflicts with the document to be processed in the search database; otherwise, determine that there is no conflict, and if there is a conflict, refuse to add new text. The step of performing multi-level similar document retrieval based on the primary affiliation cluster to obtain retrieval results includes: Calculate the first similarity between the target text vector and the document vectors corresponding to each document to be processed within the main cluster, and collect the documents to be processed with the first similarity greater than the first similarity threshold to obtain the first similar document set; Calculate the second similarity between the target text vector and the corresponding center vectors of other non-primary cluster cluster centers, and sort the non-primary clusters according to the second similarity to obtain the sorting result; Based on the sorting results, the third similarity between the target text vector and the document vector within each non-primary cluster is calculated sequentially. Documents with a third similarity greater than the second similarity threshold are then grouped together to obtain a second similar document set. Based on the first set of similar documents and the second set of similar documents, the final similar document retrieval results are determined; Based on the similarity between each document to be processed and the target text vector in the final similar document retrieval results, hierarchical semantic analysis is performed; The step of performing hierarchical semantic analysis based on the similarity between each document to be processed and the target text vector in the final similar document retrieval results includes: Based on multiple preset similarity threshold intervals, each document to be processed in the final similar document retrieval results is classified; wherein, the multiple similarity threshold intervals include specified threshold intervals and unspecified threshold intervals; The documents to be processed within a specified threshold range are input into a preset large language model for deep semantic conflict detection along with the target text vector. Perform semantic analysis based on the detection results.

2. The document intelligent processing method based on two-level clustering according to claim 1, characterized in that, The step of obtaining the document content of each of the documents to be processed, and performing content clustering based on the number of the second cluster centers to obtain the content clustering result includes: Obtain the document content of each document in the set of documents to be processed; The content of each document is vectorized to obtain the semantic vector of the document content. The K-Means algorithm is used to cluster the semantic vectors of each document content with the second number of cluster centers as the number of cluster centers, thereby obtaining the content clustering result.

3. The document intelligent processing method based on two-level clustering according to claim 2, characterized in that, Before the step of vectorizing the content of each document to obtain the semantic vector of the document content, the method further includes: Calculate the document hash value for each document's content; Determine if there are duplicate document contents based on the hash values ​​of each document; Duplicate document content is deduplicated, and the deduplicated document content is then vectorized.

4. A document intelligent processing device based on two-level clustering, characterized in that, The device includes: The first acquisition module is used to acquire the chapter titles of each document to be processed in the document to be processed set; wherein, the document to be processed set includes multiple documents to be processed; The clustering module is used to perform chapter density clustering based on the chapter titles of each document to be processed, and obtain the chapter density clustering results; The first determining module is used to determine the number of first cluster centers of the document set to be processed based on the chapter density clustering results; wherein, the number of first cluster centers is determined by using the elbow point of the Elbow method as the initial candidate K value based on the chapter density clustering results, and verifying it using the silhouette coefficient and the Davies-Bouldin index to obtain the number of first cluster centers. The second determining module is used to determine the number of second cluster centers based on the number of the first cluster centers; The second acquisition module is used to acquire the document content of each document to be processed, and perform content clustering based on the number of the second cluster centers to obtain the content clustering result; The construction module is used to construct a vector similarity index of the document set to be processed based on the content clustering results, so as to obtain the retrieval database; The clustering module includes: The preprocessing submodule is used to preprocess the chapter titles of each of the documents to be processed; wherein, the preprocessing includes removing format characters, truncating the valid text after numbering, filtering invalid titles, and limiting the number of duplicate titles; The text conversion submodule is used to convert the pre-processed chapter title text into semantic vectors using a pre-trained embedding model, and to normalize the semantic vectors. The clustering submodule is used to cluster the normalized semantic vectors using the hierarchical density clustering algorithm, and to determine the confidence level and redistribute the outliers in the clustering results to obtain the chapter density clustering results. The new input text acquisition module is used to acquire new input text. The vectorization module is used to vectorize the new input text to obtain the target text vector. The primary affiliation cluster determination module is used to determine the primary affiliation cluster of the target text vector based on the similarity between the target text vector and the center vectors of each content cluster in the retrieval database; A multi-level similar document retrieval module is used to perform multi-level similar document retrieval based on the primary belonging cluster and obtain retrieval results; The conflict determination module is used to determine whether the new input text conflicts with the document to be processed in the retrieval database based on the retrieval results; if at least one document vector and the target text vector have a similarity exceeding a preset conflict threshold, the new input text is determined to conflict with the document to be processed in the retrieval database; otherwise, it is determined not to conflict, and if there is a conflict, the addition of new text is rejected. The multi-level similar document retrieval module includes: The first similarity calculation submodule is used to calculate the first similarity between the target text vector and the document vectors corresponding to each document to be processed in the main cluster, and to collect the documents to be processed with the first similarity greater than the first similarity threshold to obtain the first similar document set. The second similarity calculation submodule is used to calculate the second similarity between the target text vector and the corresponding center vectors of other non-primary cluster cluster centers, and sort the non-primary clusters according to the second similarity to obtain the sorting result; The third similarity calculation submodule is used to calculate the third similarity between the target text vector and the document vector in each non-primary cluster according to the sorting result, and to collect the documents to be processed with the third similarity greater than the second similarity threshold to obtain the second similar document set. The final similar document retrieval result determination submodule is used to determine the final similar document retrieval result based on the first similar document set and the second similar document set; The hierarchical semantic analysis submodule is used to perform hierarchical semantic analysis based on the similarity between each document to be processed and the target text vector in the final similar document retrieval results; The hierarchical semantic analysis submodule includes: A classification unit is used to classify each document to be processed in the final similar document retrieval result according to a plurality of preset similarity threshold intervals; wherein, the plurality of similarity threshold intervals include specified threshold intervals and unspecified threshold intervals; The deep semantic conflict detection unit is used to input the document to be processed within a specified threshold range and the target text vector into a preset large language model for deep semantic conflict detection. The semantic analysis unit is used to perform semantic analysis based on the detection results.

5. A computer-readable storage medium, characterized in that, The document contains a computer program that, when executed by a processor, causes the processor to perform the steps of the document intelligent processing method based on two-level clustering as described in any one of claims 1 to 3.

6. An electronic device, characterized in that, The device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the document intelligent processing method based on two-level clustering as described in any one of claims 1 to 3.

Images