Multi-source document clustering method, system and equipment based on machine learning
By using machine learning-based methods to cluster and generate labels for multi-source documents, the problem of declining clustering performance for long documents from multiple domains is solved, achieving efficient and interpretable document organization and display.
Patent Information
- Application Number
- CN202511342140.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-19
- Publication Date
- 2025-12-23
AI Technical Summary
Existing technologies suffer from decreased clustering performance when processing multi-domain, long document collections, making it difficult to generate representative category labels. Furthermore, the cross-domain semantic span is large, resulting in weak interpretability of clustering results.
Using a machine learning-based approach, multiple documents are acquired and document summaries that meet the target word count are generated. Semantic analysis is then performed and the summaries are converted into embedding vectors. The Transformer architecture and a specified clustering algorithm are used to divide the documents into clusters, generate cluster labels that represent the semantic topics of the document clusters, and display them in a structured view.
It enables automated and intelligent deep semantic organization and clustering of multi-source, multi-modal, and long documents, improving the efficiency and accuracy of cross-domain document processing, and the clustering results have high interpretability.
Smart Images
Figure CN121188198A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of natural language processing technology, and specifically relates to a multi-source document clustering method, system and device based on machine learning. Background Technology
[0002] With the widespread application of Natural Language Processing (NLP) technology, word embedding, as one of the fundamental methods, plays a crucial role in tasks such as text classification, semantic matching, and information retrieval. The core of word embedding technology lies in converting words or phrases into multi-dimensional vector representations that can be processed by machine learning algorithms, thereby capturing the semantic relationships between words in the vector space. Typically, the semantic similarity between words can be measured by the distance or angle between their embedding vectors.
[0003] Existing word embedding techniques can be broadly categorized into three types: traditional word embedding, static word embedding, and contextualized word embedding. Traditional word embedding methods, such as Term Frequency-Inverse Document Frequency (TF-IDF), construct vectors based on the frequency of word occurrences, but fail to capture the contextual semantic information between words. Static word embedding methods (such as Word2Vec and GloVe) map semantically similar words to similar vectors through unsupervised learning, but cannot distinguish the different meanings of the same word in different contexts. To address these issues, contextualized word embedding techniques have emerged. Representative methods include Language Model Embedding (ELMo), the Transformer-based BERT model, and the generative pre-trained model GPT-2. These models, pre-trained on large-scale corpora, can dynamically generate context-dependent word representations, significantly improving semantic understanding capabilities.
[0004] Furthermore, the aforementioned embedding methods have been gradually extended to the sentence and paragraph levels, forming sentence embeddings and paragraph embeddings for higher-level text representation and clustering analysis. However, existing technologies still face many challenges when dealing with multi-domain, long document collections. For example, the large semantic span between different document lengths and domains leads to a decline in the clustering performance of existing embedding models and makes it difficult to generate representative category labels.
[0005] Therefore, there is an urgent need for a comprehensive processing system or method that can effectively perform content normalization, clustering, and tag generation on long documents from multiple domains, so as to improve the automation level of semantic organization and analysis of long documents. Summary of the Invention
[0006] To address the aforementioned issues, this invention provides a multi-source document clustering method, system, and device based on machine learning. This method solves the problems of poor processing performance for long and multimodal documents and weak interpretability of clustering results in existing technologies. It enables automated, intelligent, and deep semantic organization and insight into complex document sets, and is applicable to large-scale information retrieval, knowledge management, investment analysis, and other fields.
[0007] To achieve the above objectives, the present invention provides the following technical solution:
[0008] In one aspect, this invention proposes a multi-source document clustering method based on machine learning, the method comprising:
[0009] Obtain multiple documents to be clustered; use the document and the target document summary word count as input data for a first language model, and generate a document summary that matches the target document summary word count through the first language model;
[0010] Perform semantic analysis on the document summary, and convert the document summary into an embedding vector based on the semantic analysis results;
[0011] Based on the similarity between the embedding vector of any document and the embedding vectors of other documents, the documents are divided into several document clusters; clustering labels representing the semantic topics of document clusters are generated.
[0012] Create a collection of documents under the same document cluster based on the clustering labels, and display the document collection in a structured view.
[0013] Optionally, obtaining multiple documents to be clustered includes: converting the received multimodal documents into a unified format to obtain a multi-source document set; standardizing the character encoding of each document in the multi-source document set, and then segmenting each document based on text paragraph markers or sentence boundaries in the document; storing the segmented documents to obtain multiple documents to be clustered.
[0014] Optionally, the pre-establishment of the first language model includes: reading historical multi-source document data and document summary data from a pre-established knowledge base, and modeling based on the relationship between the historical multi-source document data and the document summary data to obtain an initial model of the Transformer architecture;
[0015] The controllable digest length is determined by setting the number of tokens output by the model or a custom digest word count; the controllable digest length is used as a training sample to train the initial model based on the Transformer architecture to obtain the first language model.
[0016] Optionally, the step of performing semantic analysis on the document summary and converting the document summary into an embedding vector based on the semantic analysis results includes:
[0017] A word segmenter is used to convert the format of the document summary to obtain the document summary of the input encoder.
[0018] The encoder analyzes the semantic information of individual sentences based on the context of each input document summary, and converts the semantic information of individual sentences in the document summary into a fixed-dimensional semantic vector;
[0019] All semantic vectors are mapped sequentially to obtain the embedding vector.
[0020] Optionally, the step of dividing documents into several document clusters based on the similarity between the embedding vectors of each document and the embedding vectors of other documents includes: using a specified clustering algorithm to calculate the cosine similarity between any document embedding vector and the embedding vectors of each other document pairwise, constructing a similarity matrix, and performing clustering operations on the embedding vectors of all documents by aggregation or splitting according to a set similarity threshold or number of clusters to generate a tree-like clustering structure; and obtaining several semantically similar document clusters by cutting the tree-like clustering structure.
[0021] The specified clustering algorithm includes at least one of K-means clustering, DBSCAN, and hierarchical clustering.
[0022] Optionally, generating clustering labels representing the semantic topics of document clustering includes: selecting at least two document summaries from the document cluster as summary samples, and combining them with a predefined prompt template to form an input sequence; wherein the prompt template includes semantic topic generation control instructions;
[0023] The input sequence is input into a locally deployed second language model. The second language model parses the semantic features of the summary samples in the input sequence and calls a pre-built long text summary dataset to retrieve historical information associated with the semantic features of the summary samples. Based on the associated historical information, the document clusters of the input sequence are summarized into semantic topics, and the clustering labels of the document clusters are output.
[0024] Optionally, selecting at least two document summaries from each document cluster as summary samples includes:
[0025] Calculate the cosine similarity between the embedding vector of each document summary in the document cluster and the embedding vectors of other document summaries in the same cluster. Select the top K document summaries with the highest similarity from the similarity ranking results in descending order as the summary samples; where K≥2.
[0026] Optionally, displaying the document collection in a structured view includes: presenting all document clusters in an interactive tree structure; the root node of the tree structure represents all document collections, the child nodes represent each document cluster, and each document cluster node displays its corresponding cluster label;
[0027] After a user queries a cluster node, a list of titles or document summaries of all documents in that cluster is displayed.
[0028] In another aspect, the present invention proposes a multi-source document clustering system based on machine learning, the system comprising:
[0029] The summary generation module is used to obtain multiple documents to be clustered; the document and the target document summary word count are used as input data for the first language model, and the first language model generates a document summary that conforms to the target document summary word count;
[0030] The vector transformation module performs semantic analysis on the document summary and converts the document summary into an embedded vector based on the semantic analysis results.
[0031] The document clustering module is used to divide documents into several document clusters based on the similarity between the embedding vector of any document and the embedding vectors of other documents; and to generate clustering labels that represent the semantic topics of document clusters.
[0032] The presentation module is used to create a collection of documents under the same document cluster based on the clustering labels, and to display the document collection in a structured view.
[0033] A third aspect of the present invention provides an electronic device including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of a machine learning-based multi-source document clustering method as described in any one of the first aspects.
[0034] The beneficial effects of this invention are:
[0035] Compared with existing technologies, this invention provides a machine learning-based multi-source document clustering method, system, and device that can effectively handle multi-source, multimodal, and long document collections, and effectively normalize and cluster cross-domain document collections. Through automated preprocessing and summary generation, it overcomes the obstacles posed by format and length. Utilizing modern language models based on Transformer and sentence embedding technology, it performs deep semantic understanding and representation of document content, achieving high clustering accuracy and strong cross-domain generalization ability.
[0036] This invention innovatively introduces a language model to generate semantic labels for clustering, thereby clearly identifying the topics of document collections. This makes machine clustering results human-understandable, significantly reducing the user's comprehension cost.
[0037] This invention provides an end-to-end intelligent document organization solution that eliminates the need for manual intervention throughout the entire process of preprocessing, summarizing, clustering, and tag generation of multi-source documents. The machine learning-based multi-source document clustering scheme effectively organizes and displays long document collections of different domains and lengths, making it suitable for large-scale document classification, information retrieval, knowledge management, and other fields. It provides a clear and structured document content display solution, significantly improving the efficiency and accuracy of cross-domain document processing. Attached Figure Description
[0038] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0039] Figure 1 The flowchart of the multi-source document clustering method based on machine learning provided by this invention;
[0040] Figure 2 A schematic diagram of the structure of a machine learning-based multi-source document clustering system provided by this invention;
[0041] Figure 3 This is a schematic diagram of the structure of an electronic device used to implement the methods and system embodiments of this application. Detailed Implementation
[0042] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.
[0043] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0044] The first embodiment of the present invention provides a multi-source document clustering method, system and device based on machine learning, specifically relating to a method and system for automatically clustering, summarizing and generating semantic tags for multi-source, multimodal and long documents, and in particular a multi-source document clustering method, system and electronic device based on machine learning.
[0045] To more clearly illustrate the first embodiment of the present invention, a multi-source document clustering method based on machine learning, the following will be combined with... Figure 1 The following is a detailed description of each part in the first embodiment 1 of the present invention.
[0046] like Figure 1The method shown is a multi-source document clustering method based on machine learning. The specific steps of the method include:
[0047] Step S101: Obtain multiple documents to be clustered; use the document and the target document summary word count as input data for the first language model, and generate a document summary that matches the target document summary word count through the first language model;
[0048] Step S102 performs semantic analysis on the document summary and converts the document summary into an embedding vector based on the semantic analysis results;
[0049] Step S103: Based on the similarity between the embedding vector of any document and the embedding vectors of other documents, divide the document into several document clusters; generate clustering labels that represent the semantic topics of the document clusters;
[0050] Step S104: Establish a collection of documents under the same document cluster based on the clustering labels, and display the document collection in a structured view.
[0051] In step S101, obtaining multiple documents to be clustered includes: converting the received multimodal documents into a unified format to obtain a multi-source document set; standardizing the character encoding of each document in the multi-source document set, and then segmenting each document based on text paragraph markers or sentence boundaries in the document; storing the segmented documents to obtain multiple documents to be clustered.
[0052] In the embodiment of step S101 above, the received multimodal documents (such as PDF and Word) are specifically multiple long text documents of varying lengths covering multiple fields. After preprocessing, a first language model trained on cross-domain text summarization is invoked to automatically generate a summary for each document in order to compress and standardize the original document content. The first language model supports controllable summary length settings.
[0053] In step S101, the pre-establishment of the first language model includes: reading historical multi-source document data and document summary data from a pre-established knowledge base, and modeling based on the relationship between the historical multi-source document data and the document summary data to obtain an initial model of the Transformer architecture.
[0054] The controllable digest length is determined by setting the number of tokens output by the model or a custom digest word count; the controllable digest length is used as a training sample to train the initial model based on the Transformer architecture to obtain the first language model.
[0055] Optionally, in actual use of the model, normalized text data and the user-specified target document summary word count are input to the first language model. This first language model is based on the Transformer architecture and has been fine-tuned on a large-scale, multi-domain long text summarization dataset, enabling it to generate document summaries that meet the target word count and accurately summarize the core information of the original text.
[0056] In step S102, the step of performing semantic analysis on the document summary and converting the document summary into an embedding vector based on the semantic analysis results includes:
[0057] A word segmenter is used to convert the format of the document summary to obtain the document summary of the input encoder.
[0058] The encoder analyzes the semantic information of individual sentences based on the context of each input document summary, and converts the semantic information of individual sentences in the document summary into a fixed-dimensional semantic vector;
[0059] All semantic vectors are mapped sequentially to obtain the embedding vector.
[0060] The embedding vector transformation in step S102 above can also be achieved using a sentence embedding model (such as Sentence-BERT) to convert each document summary into an embedding vector. This vector numerically represents the deep semantic features of the summary, and summaries with similar semantics have vectors that are closer together in space.
[0061] In step S103, the document is divided into several document clusters based on the similarity between the embedding vectors of each document and the embedding vectors of other documents. This includes: using a specified clustering algorithm, calculating the cosine similarity between any document embedding vector and the embedding vectors of all other documents pairwise, constructing a similarity matrix, and performing clustering operations on the embedding vectors of all documents by aggregation or splitting according to a set similarity threshold or number of clusters to generate a tree-like clustering structure; and obtaining several semantically similar document clusters by cutting the tree-like clustering structure.
[0062] The specified clustering algorithm includes at least one of the following algorithms: K-means clustering, DBSCAN, and hierarchical clustering.
[0063] In step S103, generating clustering labels that represent the semantic topics of document clustering includes: selecting at least two document summaries from the document clusters as summary samples, and combining them with a predefined prompt template to form an input sequence; wherein, the prompt template includes semantic topic generation control instructions;
[0064] The input sequence is input into a locally deployed second language model. The second language model parses the semantic features of the summary samples in the input sequence and calls a pre-built long text summary dataset to retrieve historical information associated with the semantic features of the summary samples. Based on the associated historical information, the document clustering of the input sequence is summarized into semantic topics, and the clustering labels of the document clusters are output, so that the clustering results have semantic interpretability.
[0065] In the above embodiments, the second language model is trained based on a large language model architecture and supports the generation of general summaries, titles or topic tags based on multiple text inputs. Preferably, the Transformer language model fine-tuned for the summary and title generation task is selected.
[0066] In the above embodiments, selecting at least two document summaries from each document cluster as summary samples includes: Calculate the cosine similarity between the embedding vector of each document summary in the document cluster and the embedding vectors of other document summaries in the same cluster. Select the top K document summaries with the highest similarity from the similarity ranking results in descending order as the summary samples; where K≥2.
[0067] In step S104, displaying the document collection in a structured view includes: presenting all document clusters in an interactive tree structure; the root node of the tree structure represents all document collections, the child nodes represent each document cluster, and each document cluster node displays its corresponding cluster label;
[0068] After a user queries a cluster node, a list of titles or document summaries of all documents in that cluster is displayed.
[0069] Although the steps in the above embodiments are described in the above order, those skilled in the art will understand that in order to achieve the effect of this embodiment, different steps do not need to be executed in such an order. They can be executed simultaneously (in parallel) or in a reverse order. These simple variations are all within the protection scope of this invention.
[0070] like Figure 2 As shown, the second embodiment of the present invention, based on the above-described technical concept, proposes a virtual system corresponding to the method of the first embodiment, namely, a multi-source document clustering system based on machine learning. The following is a detailed explanation... Figure 2 The various parts of the second embodiment of the present invention will be described in detail below.
[0071] like Figure 2 The system shown is a multi-source document clustering system based on machine learning, the system comprising:
[0072] The summary generation module 210 is used to acquire multiple documents to be clustered; and to use the word count of the document and the target document summary as input data of the first language model, and to generate a document summary that conforms to the word count of the target document summary through the first language model.
[0073] The vector conversion module 220 performs semantic analysis on the document summary and converts the document summary into an embedded vector based on the semantic analysis results.
[0074] The document clustering module 230 is used to divide documents into several document clusters based on the similarity between the embedding vector of any document and the embedding vectors of other documents; and to generate clustering labels that represent the semantic topics of document clustering.
[0075] The display module 240 is used to create a collection of documents under the same document cluster based on the clustering labels, and to display the document collection in a structured view.
[0076] The above modules can be integrated through a distributed machine learning platform, or encapsulated and deployed in a single-machine environment using languages such as Python or Java.
[0077] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process and related descriptions of the system described above can be found in the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0078] To more clearly illustrate the multi-source document clustering method based on machine learning in the first embodiment of this invention, the third embodiment uses PDF-format scientific research, legal, and news documents as processing objects; it provides a complete workflow for unsupervised clustering and semantic annotation of long document collections from heterogeneous sources across multiple domains. The specific workflow is described from six aspects: document preprocessing, document summary generation, summary embedding and extraction, summary clustering, cluster label generation, and structured result output, including:
[0079] 1. Document preprocessing;
[0080] First, import a collection of PDF documents containing research reports, legal contracts, and press releases from multiple data sources. Extract the text using OCR (such as Tesseract) and perform the following processing: unify the encoding to UTF-8; remove HTML tags and special characters; segment and remove stop words; optionally, perform language detection, retaining only Chinese and English documents. The processed text is then input into the next stage.
[0081] 2. Document summary generation;
[0082] This function invokes a T5 or BART model fine-tuned from a large-scale cross-domain corpus (such as arXiv, PubMed, LexisNexis, etc.) to automatically generate a summary. The summary generation supports setting a maximum character limit (e.g., 200 characters) to control the semantic compression ratio. Example input text (excerpt from a research document): "This study proposes a novel graph neural network algorithm for handling multi-hop reasoning problems, and experiments show superior performance on three public datasets..." Output summary: "A graph neural network model is proposed for multi-hop reasoning, and experimental results outperform existing methods."
[0083] 3. Abstract embedding and extraction;
[0084] The abstract text is generated into a fixed-length vector representation using a pre-trained embedding model (such as Sentence-BERT or Universal SentenceEncoder). Each abstract embedding is a 768-dimensional floating-point vector, which facilitates subsequent clustering.
[0085] 4. Abstract clustering;
[0086] Unsupervised clustering methods (such as K-means or HDBSCAN) are used to cluster all embedding vectors to obtain several semantically similar document clusters. The K value is automatically optimized based on the silhouette coefficient or manually set by the user; each document cluster contains multiple documents from different sources but with consistent semantics. The specified clustering algorithm includes at least one of the following: K-means clustering, mean-shift clustering, density-based clustering, expectation-maximum clustering using a Gaussian mixture model, and agglomerative hierarchical clustering.
[0087] 5. Cluster label generation;
[0088] For each cluster, several summary samples are extracted; for example, 3-5 samples. These are then fed into a second language model fine-tuned for topic generation (e.g., T5). The model outputs labels or phrase titles that describe the semantics of the clusters.
[0089] Example input (summary set): "Proposing a graph neural network model for multi-hop reasoning", "Research on multi-hop reasoning based on Transformer", "Optimizing multi-hop question-answering paths using reinforcement learning methods". Output label: "Research on multi-hop reasoning models".
[0090] 6. Structured result output;
[0091] The system organizes each cluster and its labels into a tree-like or card-like structure, including: cluster labels; a list of document summaries for each cluster; and links to the original document or a visual embedding for each summary. The system can be displayed using front-end frameworks such as React, or exported as JSON for use in knowledge graphs or information retrieval systems.
[0092] Optional extended functions of this embodiment 3 include: support for abstract style control (such as news style, academic paper style); support for interactive parameter tuning, such as adjusting abstract length and embedding model type; and the ability to manually correct clustering labels through the interface to enhance human-computer collaboration.
[0093] To more clearly illustrate the first embodiment of the multi-source document clustering method based on machine learning of the present invention, the present invention proposes a fourth embodiment of applying multi-source document clustering based on machine learning, combined with a specific application scenario:
[0094] Suppose an investment firm needs to analyze hundreds of industry reports, company announcements, and news articles from the internet daily to identify investment opportunities. The system described in this invention can be used to:
[0095] The system automatically crawls and collects these PDF and HTML documents (multi-source, multi-modal).
[0096] The preprocessing module converts them into plain text and cleans and segments them.
[0097] The summary generation module condenses each long report into a 100-word core summary.
[0098] The vector transformation and clustering modules clustered these summaries and found that these documents were mainly divided into three categories: "New Energy Vehicle Battery Technology", "Artificial Intelligence Chip Market", and "Consumer Medical Electronics".
[0099] The tag generation module generated the three clear tags mentioned above for these three classes.
[0100] Analysts can quickly browse all relevant report summaries simply by clicking the "AI Chip Market" tab in the system interface, greatly improving research efficiency.
[0101] Example 5: This embodiment of the invention also provides an electronic device corresponding to Examples 1 and 2.
[0102] One electronic device includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor performs the steps of any one of S101-S104 of a multi-source document clustering method based on machine learning.
[0103] like Figure 3As shown, the electronic device may include: at least one processor 31, at least one network interface 35, user interface 34, memory 36, and at least one communication bus 32.
[0104] The communication bus 32 is used to enable communication between these components.
[0105] The user interface 34 may include a display screen and a camera. Optionally, the user interface 34 may also include a standard wired interface and a wireless interface.
[0106] The network interface 35 may optionally include a standard wired interface or a wireless interface (such as a WIFI interface).
[0107] The processor 31 may include one or more processing cores. The processor 31 connects to various parts of the server using various interfaces and lines, and performs various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 36, and by calling data stored in the memory 36. Optionally, the processor 31 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 31 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor 31.
[0108] The memory 36 may include random access memory (RAM) or read-only memory. Optionally, the memory 36 may include a non-transitory computer-readable storage medium. The memory 36 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 36 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 36 may also be at least one storage device located remotely from the aforementioned processor 31. Figure 3 As shown, the memory 36, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application program for a multi-source document clustering method, system, and electronic device based on machine learning.
[0109] exist Figure 3 In the electronic device shown, the user interface 34 is mainly used to provide an interface for users to input data and obtain user input data; while the processor 31 can be used to call an application stored in the memory 36 that is a multi-source document clustering method based on machine learning. When executed by one or more processors 31, the electronic device performs one or more of the methods described in steps S101-S104 of the above embodiment.
[0110] Those skilled in the art will clearly understand that the technical solutions of this application can be implemented using software and / or hardware. In this specification, "unit" and "module" refer to software and / or hardware capable of independently or in conjunction with other components to perform a specific function. Hardware may include, for example, a Field-Programmable Gate Array (FPGA), an Integrated Circuit (IC), etc.
[0111] Specifically, the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.
[0112] The code of the computer program can be in the form of source code, object code, executable file, or some intermediate form.
[0113] Computer-readable storage media may include cache, high-speed random access memory (RAM), such as the common double data rate synchronous dynamic random access memory (DDR SDRAM), and may also include non-volatile memory (NVRAM), such as one or more read-only memories (ROM), disk storage devices, flash memory devices, or other non-volatile solid-state storage devices such as optical discs (CD-ROM, DVD-ROM), floppy disks, or data tapes.
[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.
Claims
1. A multi-source document clustering method based on machine learning, characterized in that, The method includes: Obtain multiple documents to be clustered; use the document and the target document summary word count as input data for a first language model, and generate a document summary that matches the target document summary word count through the first language model; Perform semantic analysis on the document summary, and convert the document summary into an embedding vector based on the semantic analysis results; Based on the similarity between the embedding vector of any document and the embedding vectors of other documents, the documents are divided into several document clusters; clustering labels representing the semantic topics of document clusters are generated. Create a collection of documents under the same document cluster based on the clustering labels, and display the document collection in a structured view.
2. The method according to claim 1, characterized in that, The process of obtaining multiple documents to be clustered includes: converting the received multimodal documents into a unified format to obtain a multi-source document set; standardizing the character encoding of each document in the multi-source document set, and then segmenting each document based on text paragraph markers or sentence boundaries; and storing the segmented documents to obtain multiple documents to be clustered.
3. The method according to claim 1, characterized in that, The pre-establishment of the first language model includes: reading historical multi-source document data and document summary data from a pre-established knowledge base, modeling based on the relationship between historical multi-source document data and document summary data, and obtaining an initial model of the Transformer architecture; The controllable digest length is determined by setting the number of tokens output by the model or a custom digest word count; the controllable digest length is used as a training sample to train the initial model based on the Transformer architecture to obtain the first language model.
4. The method according to claim 1, characterized in that, The step of performing semantic analysis on the document summary and converting the document summary into an embedding vector based on the semantic analysis results includes: A word segmenter is used to convert the format of the document summary to obtain the document summary of the input encoder. The encoder analyzes the semantic information of individual sentences based on the context of each input document summary, and converts the semantic information of individual sentences in the document summary into a fixed-dimensional semantic vector; All semantic vectors are mapped sequentially to obtain the embedding vector.
5. The method according to claim 1, characterized in that, The method of dividing documents into several document clusters based on the similarity between the embedding vectors of each document and the embedding vectors of other documents includes: using a specified clustering algorithm, calculating the cosine similarity between any document embedding vector and the embedding vectors of all other documents pairwise, constructing a similarity matrix, and performing clustering operations on the embedding vectors of all documents by aggregation or splitting according to a set similarity threshold or number of clusters to generate a tree-like clustering structure; and obtaining several semantically similar document clusters by cutting the tree-like clustering structure. The specified clustering algorithm includes at least one of the following algorithms: K-means clustering, DBSCAN, and hierarchical clustering.
6. The method according to claim 1, characterized in that, The generation of clustering labels representing the semantic topics of document clustering includes: selecting at least two document summaries from the document clusters as summary samples, and combining them with a predefined prompt template to form an input sequence; wherein, the prompt template includes semantic topic generation control instructions; The input sequence is input into a locally deployed second language model. The second language model parses the semantic features of the summary samples in the input sequence and calls a pre-built long text summary dataset to retrieve historical information associated with the semantic features of the summary samples. Based on the associated historical information, the document clusters of the input sequence are summarized into semantic topics, and the clustering labels of the document clusters are output.
7. The method according to claim 6, characterized in that, The step of selecting at least two document summaries from each document cluster as summary samples includes: Calculate the cosine similarity between the embedding vector of each document summary in the document cluster and the embedding vectors of other document summaries in the same cluster. Select the top K document summaries with the highest similarity from the similarity ranking results in descending order as the summary samples; where K≥2.
8. The method according to claim 1, characterized in that, The step of displaying the document collection in a structured view includes: presenting all document clusters in an interactive tree structure; the root node of the tree structure represents all document collections, the child nodes represent each document cluster, and each document cluster node displays its corresponding cluster label; After a user queries a cluster node, a list of titles or document summaries of all documents in that cluster is displayed.
9. A multi-source document clustering system based on machine learning, characterized in that, The system includes: The summary generation module is used to obtain multiple documents to be clustered; the document and the target document summary word count are used as input data for the first language model, and the first language model generates a document summary that conforms to the target document summary word count; The vector transformation module performs semantic analysis on the document summary and converts the document summary into an embedded vector based on the semantic analysis results. The document clustering module is used to divide documents into several document clusters based on the similarity between the embedding vector of any document and the embedding vectors of other documents; and to generate clustering labels that represent the semantic topics of document clusters. The presentation module is used to create a collection of documents under the same document cluster based on the clustering labels, and to display the document collection in a structured view.
10. An electronic device, comprising a memory and a processor, characterized in that, The memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of a machine learning-based multi-source document clustering method as described in any one of claims 1 to 7.