Intelligent content collection and knowledge storage system and method based on large model driving

Abstract

The invention discloses an intelligent content collection and knowledge storage system and method based on large model driving. The system comprises a configuration management module, a site detection and change monitoring module, a content extraction and processing module, an intelligent analysis and storage module and a knowledge retrieval and question and answer module. The method comprises the following steps: discovering an acquirable URL list by automatically analyzing input site information, and extracting core text content from HTML (Hypertext Markup Language) by utilizing network analysis to carry out correlation judgment and Markdown conversion; based on an LLM large language model and a pre-trained Embedding model, text information of the content is converted into high-dimensional vector representation, and knowledge storage is carried out; and configuring an RAG architecture, integrating a vector database and an LLM large language model, and generating high-quality and accurate answers. According to the scheme, the problems of low content collection efficiency, insufficient intelligence, poor robustness, knowledge updating lagging and knowledge application limitation in the prior art are solved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an intelligent content acquisition and knowledge storage system and method based on large model-driven approaches. Background Technology

[0002] Currently, many organizations and individuals use manual or semi-automated methods to collect and organize information when conducting industry intelligence monitoring, competitor analysis, and literature review. Common methods and their inherent problems include: Manual browsing and copy-pasting: inefficient, error-prone, and cannot be scaled. Rule-based crawlers can only crawl structured or semi-structured data. They are ineffective at extracting unstructured content (such as news reports, technical documents, blog posts, etc.) and are prone to failure due to changes in website structure, requiring frequent maintenance. Simple keyword matching: It is difficult to understand the deeper meaning and context of the content, and it is easy to misjudge relevance, resulting in the collection of a large amount of invalid information; Discrete knowledge management tools lack deep integration with the content acquisition process, making the import, updating, association, and retrieval of knowledge cumbersome and difficult to form a dynamically evolving knowledge system.

[0003] Based on the commonly used methods described above, the following problems can be identified in the existing technology: Inefficiency and high cost: Traditional methods rely heavily on manual labor, resulting in low efficiency and high labor costs; Poor robustness: Rule-based crawlers are sensitive to changes in website structure and have high maintenance costs; Insufficient intelligence: It is difficult to deeply understand the semantics of the content, effectively determine the relevance and value of the content, and accurately classify and tag it; Lagging knowledge updates: The incremental update and error correction mechanisms of the knowledge base are imperfect, making it difficult to maintain the timeliness and accuracy of knowledge. Limited retrieval and question-answering capabilities: Traditional databases struggle to support complex semantic queries and knowledge-based question answering using natural language; Lack of closed-loop learning: The processes of data collection, storage, use, and feedback fail to form an effective closed loop, thus failing to drive the system's self-evolution and improvement. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides an intelligent content acquisition and knowledge storage system and method based on a large model. By constructing a fully intelligent content acquisition and knowledge storage system that is "configurable and ready to use, unattended," it achieves automated content acquisition, deep understanding, structured storage, intelligent retrieval, and question answering. Furthermore, it supports continuous learning and evolution of the system based on feedback from the knowledge base to form a professional and comprehensive knowledge system.

[0005] The application is implemented by adopting the following technical solutions: In a first aspect, the intelligent content collection and knowledge storage system based on a large model driver includes the following modules: Configuration management module: configure user input ports for collecting target and theme keywords; Site detection and change monitoring module: automatically parse the input site information, identify the website structure, find the list of collectable URLs, and monitor the changes in the target site content; Content extraction and processing module: extract core text content from HTML using network analysis and remove irrelevant information, perform relevance determination and Markdown conversion; Intelligent analysis and storage module: classify content by theme based on LLM large language model, extract key tags and abstract, convert text information of content into high-dimensional vector representation using pre-trained Embedding model, and store knowledge; Knowledge retrieval and question answering module: configure RAG architecture, integrate vector database and LLM large language model, when a user initiates a natural language query, retrieve the most relevant knowledge fragments in the vector library, input the query and the retrieved knowledge fragments into the LLM large language model together, and generate high-quality and accurate answers.

[0006] Specifically, it also includes a learning and evolution module: Configure incremental update and backfill error correction mechanism: according to the existing knowledge points of the knowledge base, actively find the omissions or errors in the content, and perform backfill or correction; at the same time, collect user feedback, fine-tune or update the rules of the LLM large language model; Configure knowledge point perfecting mechanism: LLM large language model actively identifies knowledge blind spots according to the structure and content of the existing knowledge base, and guides the configuration management module to discover or generate related content.

[0007] Specifically, the relevance determination in the content extraction and processing module includes: using LLM large language model to perform semantic level relevance judgment and filtering on the extracted content according to user set theme keywords and positive and negative rules; the Markdown conversion includes: converting the extracted structured or semi-structured content into a unified Markdown format.

[0008] Specifically, the knowledge storage in the intelligent analysis and storage module includes: File library: persistently store the converted Markdown format content, extracted metadata, and vector data; Vector library: store content vectors for similarity search and semantic retrieval.

[0009] In another aspect, the method for intelligent content collection and knowledge storage based on large model driving comprises the following steps in sequence: user configuration, detection and monitoring, content extraction and processing, intelligent analysis and storage, knowledge retrieval and question answering, and system learning and evolution. Step S01: The user provides a collection task to the configuration management module through a Web interface or API, and the configuration management module receives and analyzes the information and prepares to start the collection process. The detection and monitoring step specifically includes: Step S02: The site detection and change monitoring module initiates an HTTP request according to the configured URL, parses the HTML structure of the target page, discovers all valid links, and constructs a crawlable URL queue. Step S03: The system periodically or in real time monitors the update of the target site content, and when new content or content update is found, triggers the content extraction and processing module.

[0010] Specifically, the content extraction and processing step specifically includes: Step S04: For the URL identified as a content page, the content extraction and processing module uses an advanced web parsing library to extract pure text content from the HTML while retaining necessary structural information. Step S05: The extracted text content is sent to a pre-tuned LLM large language model, and the model scores and determines the semantic relevance of the content according to the user-set theme keywords and positive and negative rules. Step S06: The original HTML fragments and pure text are uniformly converted to Markdown format through the relevance determination content.

[0011] Specifically, the intelligent analysis and storage step specifically includes: Step S07: The converted Markdown content is sent to another LLM large language model, or another function of the LLM large language model is called to automatically extract key information, and the content is classified by theme, extract core keywords as tags, and generate a refined content summary. Step S08: An Embedding model is used to encode the text information of the Markdown content into a high-dimensional vector. Step S09: The processed content is stored persistently, the Markdown file and extracted metadata are stored in a file library, and the generated vector data is sent to a vector library.

[0012] Specifically, the knowledge retrieval and question answering step specifically includes: Step S10: When the user initiates a natural language question to the system through the user interaction interface, the knowledge retrieval and question answering module is started; first, the user's question is vectorized, and then a similarity search is performed in the vector library to find several knowledge fragments closest to the question vector; Step S11: The retrieved relevant knowledge fragments are sent to the LLM large language model for generating answers together with the user's original question, and the model generates an accurate, fluent and context-related answer.

[0013] Specifically, the learning and evolution steps of the system collect feedback provided by the user during interaction with the system and optimize through the learning and evolution module, specifically including: According to the existing knowledge points of the knowledge base, the supplementary information or errors that may exist in the collected content are found; Using the LLM large language model to analyze the structure and content of the knowledge base, identify knowledge blind spots or topics that need to be further explored, and guide the site detection and change monitoring module to find related information, and instruct the content extraction and processing module to focus on specific types of content; The user feedback is used as a supervised learning signal to fine-tune the LLM large language model.

[0014] The beneficial effects of the present application are that: through the intelligent content collection and knowledge storage system based on large model driving proposed in the present application, the user only needs to simply configure the collection target and theme, and the full-process automation can be realized, greatly reducing the manpower investment and maintenance cost. With the help of the large language model, the system can deeply understand the content semantics, realize accurate relevance determination, automatic classification, label extraction and knowledge question answering, far exceeding traditional keyword matching and rule crawler. Even if the structure of the target website changes, the system can still maintain a high content extraction and determination accuracy through semantic understanding and model adjustment. Through the incremental update, backfill error correction and knowledge point improvement mechanism, the knowledge base can continuously learn, self-evolve, and maintain professionalism and integrity. The RAG architecture supports complex queries in natural language, and users can interact with the knowledge base like talking to an expert, quickly obtaining the required information. It is suitable for industry intelligence, competitive product analysis, technical literature, policies and regulations, enterprise internal knowledge management and other scenarios, improving information utilization efficiency and decision-making quality. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the drawings shown.

[0016] Fig. 1 This is a system architecture diagram of intelligent content acquisition and knowledge storage based on a large model in an embodiment of the present invention; Fig. 2 This is a schematic diagram of content acquisition in this embodiment; Fig. 3 This is a schematic diagram of the content extraction and semantic determination process in this embodiment; Fig. 4 This is a schematic diagram of the intelligent analysis vectorization data entry process in this embodiment; Fig. 5 This is a schematic diagram of the RAG retrieval and question-answering interaction process in this embodiment; Fig. 6 This is a schematic diagram of the system learning and evolution closed loop in this embodiment; Fig. 7 This is a schematic diagram of the underlying storage and index design in this embodiment. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0018] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0019] The following is in conjunction with the appendix Figs. 1-7 The following describes some embodiments of the present invention in detail. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0020] This invention proposes an intelligent content acquisition and knowledge storage system and method based on a large model-driven approach. In a preferred embodiment, such as... Fig. 1 As shown, the system mainly includes the following modules: Configuration Management Module: Users can input the collection target (such as target site domain name, column link, RSS subscription source, site map URL, etc.) and topic keywords (which can include positive and negative filtering rules) through this module.

[0021] Site detection and change monitoring module: Site detection: Automatically parses the input site information, identifies the website structure, and discovers a list of URLs that can be collected (such as links, news lists, article detail pages, etc.); Change monitoring: Regularly or in real time monitor changes to the target site's content (such as adding articles or updating content) to trigger subsequent data collection processes.

[0022] Content extraction and processing module: Content extraction: Utilize advanced web page parsing technologies (such as Boilerpipe, Readability.js, etc.) to extract core text content from HTML and remove irrelevant information such as advertisements and navigation; Relevance assessment: Using a Large Language Model (LLM), the extracted content is semantically relevance assessed and filtered based on user-defined topic keywords and positive / negative rules. Markdown conversion: Converts extracted structured or semi-structured content (such as HTML and plain text) into a unified Markdown format for easier subsequent processing and storage.

[0023] Intelligent analysis and data entry module: Automatic categorization and tagging: LLM is used to categorize content by topic and extract key tags and summaries; Vectorization: Using a pre-trained embedding model to convert the textual information of the content into a high-dimensional vector representation; Knowledge entry into the database: File repository: Persistently stores the converted Markdown content, extracted metadata (such as title, author, publication time, URL, category, tags), and vector data; Vector Library: Specifically stores content vectors for efficient similarity search and semantic retrieval.

[0024] Knowledge retrieval and question answering module: RAG architecture: Integrating a vector database and LLM, when a user initiates a natural language query, the system first retrieves the most relevant knowledge fragments (Retrieval) from the vector database, and then inputs the query and the retrieved knowledge fragments into the LLM, which generates a high-quality and accurate answer (Generation).

[0025] System Learning and Evolution Module: Incremental updates and backfilling / correction: Based on the existing knowledge points in the knowledge base, the system proactively identifies omissions or errors in the content and backfills or corrects them. User feedback (such as evaluations of answers and suggestions for content revision) is also collected and used to fine-tune the LLM or update rules, enabling the system to continuously learn and evolve. Knowledge point improvement: Based on the existing knowledge base structure and content, LLM proactively identifies knowledge blind spots and guides the data collection module to discover or generate relevant content, continuously improving the professionalism and completeness of the knowledge base.

[0026] The application also correspondingly proposes an intelligent content collection and knowledge storage process based on large model driving, as shown in Figs. 2-7 , which includes the following in sequence: Configuration phase (steps S100-S101): The user provides the collection task to the configuration management module through the Web interface or API. For example, the user specifies "target site: `www.example-tech.com / blog`, topic keywords: `artificial intelligence`, `large model`, `natural language processing`; exclude keywords: `recruitment`, `advertising`". The configuration management module receives and parses these information, and prepares to start the collection process.

[0027] Detection and monitoring phase (steps S102-S103): Step S102: The site detection and change monitoring module initiates an HTTP request according to the configured URL, parses the HTML structure of the target page, discovers all valid links (such as article list pages, article detail pages, column pages, etc.), and constructs a crawlable URL queue; Step S103: The system will periodically (such as every hour / day) or through RSSFeed, site map, etc. mechanism, monitor the update of the target site content. Once new content or content update is found, the content extraction and processing module is triggered.

[0028] Content extraction and processing phase (steps S104-S106): Step S104: For the URL identified as a content page, the content extraction and processing module will use advanced web parsing libraries (such as BeautifulSoup combined with custom rules or more advanced AI-driven parsers) to extract pure text content from HTML, while retaining necessary structure information (such as title, paragraph, list, code block, etc.); Step S105: The extracted text content is sent to a pre-tuned large language model (LLM), which scores and judges the semantic relevance of the content according to the user-set topic keywords and positive and negative rules. For example, the model will judge whether the article mainly discusses "artificial intelligence", "large model", etc., and exclude content containing "recruitment", "advertising", etc. Only the content that passes the relevance judgment will enter the next step; Step S106: The content that passes the relevance judgment, whether it is the original HTML fragment or the pure text, will be uniformly converted to Markdown format by the content extraction and processing module. This ensures the consistency of data format, facilitating storage and subsequent processing.

[0029] Intelligent analysis and storage phase (steps S107-S111): Step S107: The converted Markdown content is fed into another (or different function call of the same LLM) LLM for automatic extraction of key information. This includes: topic classification for the content (e.g., "Technology Frontier", "Industry Analysis"), extraction of core keywords as tags, and generation of a refined content summary; Step S108: An efficient Embedding model (e.g., Sentence-BERT, OpenAI Embeddings, etc.) is used to encode the text information (or its summary, key paragraphs) of the Markdown content into high-dimensional vectors; Steps S109-S111: Finally, the processed content is stored persistently. The file repository (e.g., object storage, distributed file system) is responsible for storing the Markdown files, extracted metadata (title, URL, author, publication time, LLM-generated classification and tags). At the same time, the generated vector data is sent to the vector repository (e.g., Milvus, Pinecone, ChromaDB, etc.) for efficient similarity search.

[0030] Knowledge Retrieval and Question Answering Phase (Steps S112-S113): Step S112: When the user initiates a natural language question to the system through the user interaction interface, the knowledge retrieval and question answering module starts. It first vectorizes the user's question, then performs a similarity search in the vector repository to find a number of knowledge pieces (i.e., the most relevant text content or its vector representation) closest to the question vector; Step S113: The retrieved relevant knowledge pieces (as context) are sent to a powerful LLM (usually a model for generating answers) together with the user's original question. The LLM generates an accurate, fluent, and contextually relevant answer based on this information.

[0031] System Learning and Evolution Phase (Steps S114-S115): Step S115: During the interaction with the system, the user may provide feedback (such as satisfaction evaluation of the answer, pointing out errors in the answer, suggesting additional content, etc.); Step S114: The system learning and evolution module is responsible for collecting these feedbacks and performing: Incremental updates and backfill error correction: The system can find possible supplementary information or errors in the collected content based on the existing knowledge points in the knowledge base. For example, if the knowledge base already has an introduction to "Model A", but the newly collected "Model B" is linked to "Model A", the system may actively retrieve the latest information about "Model A", or apply the user's feedback correction information to the existing data; Knowledge refinement: LLM can analyze the structure and content of the knowledge base, identify knowledge "blind spots" or topics that need to be further explored, and guide the site detection and change monitoring module to find relevant information, or instruct the content extraction and processing module to focus on specific types of content. Model tuning: User feedback (such as low satisfaction answers) can be used as a signal for supervised learning to fine-tune the LLM or use RLHF (Reinforcement Learning from Human Feedback) technology to continuously improve the LLM's ability to understand, determine relevance, generate answers, and other aspects.

[0032] These learning and evolution processes will feedback to various modules of the system (such as updating collection strategies, adjusting classification models, optimizing retrieval algorithms, etc.), so as to realize the self-improvement and intelligent upgrading of the system.

[0033] In this embodiment, the present scheme has the following technical points: (1) The core role of LLM in content relevance determination, automatic classification, label extraction, knowledge question answering, and system evolution; (2) The RAG architecture realizes retrieval-based enhanced generation, ensuring the accuracy and timeliness of the question and answer; (3) Dual-track storage of file library and vector library, taking into account the original format of the content, structured metadata, and efficient semantic retrieval capabilities; (4) Closed-loop learning and self-evolution mechanism, driven by user feedback and knowledge base analysis to continuously optimize and improve the system; (5) "Configure and use" automatic process, minimizing human intervention and improving efficiency.

[0034] In one embodiment, the selection of LLM model can be based on actual demand and cost, selecting open source LLM (such as Llama, Mistral) or closed source LLM (such as GPT series, Claude). The Embedding model can be replaced with other excellent Embedding models to adapt to different languages or specific field requirements. For vector database, different vector database products can be selected according to data volume, query performance requirements, deployment convenience, etc. For content extraction tools, different web parsing libraries or custom extraction logic can be used according to the complexity and content type of the website. For file storage, cloud object storage (such as AWS S3, Azure Blob Storage) or distributed file system (such as HDFS) can be used according to the scale and cost considerations.

[0035] In a preferred embodiment, the present scheme can also use: Hybrid LLM strategy: use small and medium-sized, high-performance LLM for content extraction, classification, and summary, and use large and more powerful LLM for complex question and answer and knowledge point mining; Knowledge graph combined with vector library: based on the vector library, construct a knowledge graph to represent the explicit relationship between knowledge points, further enhance the structured and reasoning ability of knowledge; Active learning strategy: introduce an active learning mechanism to let the system actively identify the unlabeled data that can best help it learn and evolve, and prioritize processing; Multi-modal support: future scalable support for image, video and other content collection, analysis and vectorization, to build a more comprehensive knowledge system.

[0036] For the foregoing embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the order of the described actions, because according to the present application, some steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily required by the present application.

[0037] In the above embodiments, the basic principles and main features of the present application and the advantages of the present application are described. Those skilled in the art should understand that the present application is not limited by the above embodiments, and the above embodiments and the description in the specification are only to illustrate the principles of the present application. Any modification and change made by those skilled in the art without departing from the spirit and scope of the present application shall be within the protection scope of the claims of the present application.

Claims

1. An intelligent content collection and knowledge storage system based on large model driving, characterized in that, Comprise the following modules: Configuration management module: configure user input port for target acquisition and theme keyword; Site detection and change monitoring module: automatically parse the input site information, identify the website structure, find the list of collectable URLs, and monitor the changes in the content of the target site; Content extraction and processing module: extract core text content from HTML using network analysis and remove irrelevant information, perform relevance determination and Markdown conversion; Intelligent analysis and warehousing module: based on LLM large language model, classify the content by theme, extract key tags and abstract, use pre-trained Embedding model to convert text information into high-dimensional vector representation and store knowledge; Knowledge retrieval and question answering module: configure RAG architecture, integrate vector database and LLM large language model, when the user initiates a natural language query, retrieve the most relevant knowledge fragments in the vector library, input the query and the retrieved knowledge fragments into the LLM large language model, and generate high-quality and accurate answers. 2.The large model driven intelligent content collection and knowledge storage system according to claim 1, wherein, Also includes learning and evolution module: Configure incremental update and backfill error correction mechanism: according to the existing knowledge points in the knowledge base, actively find the omissions or errors in the content and perform backfill or correction; at the same time, collect user feedback and fine-tune or update the rules of the LLM large language model; Configure knowledge point perfecting mechanism: LLM large language model actively identifies knowledge blind spots based on the structure and content of the existing knowledge base, and guides the configuration management module to discover or generate related content. 3.The large model driven intelligent content collection and knowledge storage system of claim 1, wherein, The relevance determination in the content extraction and processing module includes: using LLM large language model to perform semantic level relevance determination and filtering on the extracted content according to the user set theme keywords and positive and negative rules; the Markdown conversion includes: converting the extracted structured or semi-structured content into a unified Markdown format. 4.The large model driven intelligent content collection and knowledge storage system of claim 3, wherein, The knowledge warehousing in the intelligent analysis and warehousing module includes: File library: persistently store the converted Markdown format content, extracted metadata and vector data; Vector library: store content vectors for similarity search and semantic retrieval.

5. The method for intelligent content collection and knowledge storage based on large model driving, according to any one of claims 1-4, wherein the system for intelligent content collection and knowledge storage based on large model driving is implemented. In turn includes user configuration, detection and monitoring, content extraction and processing, intelligent analysis and warehousing, knowledge retrieval and question answering and system learning and evolution steps; the user configuration step specifically includes: Step S01: the user provides the acquisition task to the configuration management module through the Web interface or API, and the configuration management module receives and parses these information, and prepares to start the acquisition process; The detection and monitoring step specifically includes: Step S02: the site detection and change monitoring module initiates HTTP request according to the configured URL, parses the HTML structure of the target page, finds all valid links, and constructs a crawlable URL queue; Step S03: the system periodically or in real time monitors the update of the target site content, and when new content or content update is found, triggers the content extraction and processing module. 6.The method of claim 5, wherein, The content extraction and processing step specifically includes: Step S04: For the URL identified as a content page, the content extraction and processing module uses an advanced web parsing library to extract the plain text content from the HTML while preserving the necessary structural information; Step S05: The extracted text content is sent to a pre-tuned LLM large language model, which scores and determines the semantic relevance of the content based on user-set topic keywords and positive and negative rules; Step S06: The original HTML fragments and plain text are converted into Markdown format through the relevance-determined content.

7. The large model driven intelligent content collection and knowledge warehousing method of claim 6, wherein, The intelligent analysis and warehousing steps include: Step S07: The converted Markdown content is sent to another LLM large language model or another function of the LLM large language model is called to automatically extract key information and classify the content by topic, extract core keywords as tags, and generate a refined content summary; Step S08: Use the Embedding model to encode the text information of the Markdown content into a high-dimensional vector; Step S09: The processed content is stored persistently, and the Markdown file and extracted metadata are stored in the file library, while the generated vector data is sent to the vector library. 8.The method of claim 7, wherein, The knowledge retrieval and question answering steps include: Step S10: When the user initiates a natural language question to the system through the user interaction interface, the knowledge retrieval and question answering module starts; first, the user's question is vectorized, then a similarity search is performed in the vector library to find several knowledge fragments closest to the question vector; Step S11: The retrieved relevant knowledge fragments and the user's original question are sent to the LLM large language model for generating answers, which generates an accurate, smooth, and contextually related answer. 9.The method of claim 8, wherein, The system learning and evolution steps collect user feedback during interaction with the system and optimize it through the learning and evolution module, including: According to the existing knowledge points in the knowledge base, find the possible supplementary information or errors in the collected content; Use the LLM large language model to analyze the structure and content of the knowledge base, identify knowledge gaps or topics that need to be further explored, and guide the site detection and change monitoring module to find related information and instruct the content extraction and processing module to focus on specific types of content; Use user feedback as a supervised learning signal to fine-tune the LLM large language model.

Images