Systematic literature retrieval enhancement and knowledge mining method based on citation network

Abstract

The present invention belongs to the field of data mining, and discloses a method for systematic retrieval enhancement and knowledge mining of documents based on citation networks, including the following steps: obtaining citation network data, the citation network data including citation relationships between documents; systematic retrieval enhancement of documents based on citation networks, further expanding and pruning the citation network to ensure the quality and relevance of input data; injecting pruned knowledge into a large model, and the large model learns knowledge and its complete citation network to generate initial new knowledge; different intelligent agent roles examine the initial new knowledge from multiple perspectives, and increase the diversity and depth of new knowledge through feedback; real-time introduction of prior knowledge and professional knowledge to guide the new knowledge generation process of the model, and in-depth mining of new knowledge; outputting the mined new knowledge. The present invention imitates the divergent thinking and convergent thinking process of human beings to effectively generate and refine new scientific knowledge.

Description

Technical Field

[0001] The present invention belongs to the technical field of knowledge mining, and in particular to a method for systematically enhancing document retrieval and knowledge mining based on a citation network. Background Art

[0002] With the explosive growth of scientific literature and the deepening of research fields, researchers face huge information processing challenges when proposing innovative new scientific knowledge. Although large language models (LLMs) have shown great potential in data processing and knowledge integration, they still have many shortcomings in generating innovative and in-depth new scientific knowledge. Current research mainly focuses on how to use large models to achieve rapid advancement and improvement of existing theories and technologies, while ignoring the initial stage of scientific research from scratch, which involves the mining of new knowledge and is the key to scientific progress. Citation networks include scientific journals, patent documents, conference proceedings, scientific reports, and dissertations, which better demonstrate the depth and breadth of academic research in a certain field over time. Proficient use of citation networks can save users a lot of time in aimlessly browsing and screening documents, and focus on reading and understanding high-quality documents. With the development of modern information technology, the number of literature works has increased rapidly, and the citation network has formed a large-scale complex network system. Any deviation in the network will lead to inaccurate literature search results.

[0003] With the rapid increase in the amount of scientific literature and the deepening of research fields, researchers are facing huge challenges. On the one hand, researchers need to extract valuable information from a large amount of literature, which is a time-consuming and labor-intensive process; on the other hand, it is difficult to quickly mine new knowledge through traditional methods.

[0004] In this context, artificial intelligence (AI), especially large language models (LLMs), is regarded as an important tool in scientific research due to its powerful data processing and information integration capabilities. Large language models can understand and generate complex text information by training on a large amount of text data, and show the ability to process and generate knowledge on different tasks. For example, models such as GPT-4 not only perform well in text generation, but also show their application potential in professional fields such as programming, law, and biomedicine. In the field of scientific research, large models have been used for various tasks, including literature retrieval, information extraction, data analysis, and reasoning evaluation, providing strong support for researchers, especially showing their unique advantages in quickly browsing and organizing a large amount of scientific literature. For example, LLMs can automatically identify key concepts and associations in the literature, thereby accelerating the process of writing literature reviews.

[0005] In current scientific research and engineering practice, most studies focus on how to use LLMs to quickly advance existing theories or technologies from the preliminary stage to the mature stage, so as to improve the efficiency of scientific research, that is, the process from 1 to 100. This includes some tasks that are complicated but require less innovation, such as automated literature summarization and code implementation. Although research at this stage is crucial to technological development and popularization, it relies on existing frameworks and theoretical foundations. However, the initial stage of scientific exploration from nothing to something, that is, the process from 0 to 1, is overlooked in many cases. This stage involves the excavation of new knowledge. It is these activities that define the frontier and innovation of scientific research. The proposal of good new scientific knowledge is the initiator and cornerstone of scientific progress.

[0006] Although large language models provide strong support in scientific research tasks, they still have obvious deficiencies and limitations in their application in generating new scientific knowledge. First, large models rely on a large amount of pre-training data, which cannot reflect the latest progress in the scientific frontier in a timely manner, resulting in them relying on outdated or surpassed knowledge when generating new knowledge or providing information. The knowledge update in the academic community is very rapid, especially in rapidly developing fields such as computers, biology, and materials. The training of large models is usually not updated in real time, resulting in the model relying on outdated or incomplete information when generating new knowledge, and unable to innovate on the "shoulders of giants" at the forefront of science. Second, although large models can generate seemingly complex scientific content, these contents are insufficient in terms of accuracy and depth of scientific content, often lacking real innovation and depth, and tend to repeat patterns in training data rather than propose new knowledge. In addition, large models also show limitations when performing complex scientific reasoning. They lack the intuition and insights of human experts who have been deeply involved in specific fields for many years. These intuitions and insights are often the key to promoting scientific breakthroughs. The lack of expert guidance makes the knowledge generated by the model often remain on the surface and difficult to touch deep scientific problems, making these new knowledge may not have practical application value or scientific significance. For example, the model may ignore factors such as the feasibility of experimental operations. These difficulties mainly stem from the inherent limitations of large models and the complex needs of scientific research. These difficulties also show that although large models provide new possibilities for the generation of new scientific knowledge, in-depth exploration and improvement are still needed to achieve their effective application in scientific research.

[0007] Innovation is generated from the divergent and convergent process of thinking. Divergent thinking refers to the ability to generate multiple possible answers or solutions from a problem or topic. This mode of thinking emphasizes imagination, creativity and quantity generation. Convergent thinking refers to the ability to find specific and correct answers to specific problems. This mode of thinking focuses more on logical reasoning, accuracy and efficiency. Innovative ideas in scientific research come from divergent thinking and convergent materialization of research problems. For large models, the problems they have can be summarized as the divergent angle of knowledge mining is not broad enough and the convergent direction is not accurate and reliable enough. Summary of the invention

[0008] In view of this, based on the divergent thinking and convergent thinking in the theory of structural intelligence, this paper proposes an innovative human-in-loop Multi-agent Framework (HMA) to achieve reliable initial generation of new scientific knowledge. This framework combines a real-time systematic knowledge retrieval enhancement mechanism, dynamically integrates the latest scientific research progress, constructs a citation network subgraph, and provides a cutting-edge and complete scientific research knowledge review for the large model. At the same time, the scientific peer review process is simulated through a multi-agent mining method, and the generated new knowledge is further optimized and refined by combining prior knowledge, thereby enhancing the diversity and argumentation depth of new knowledge. A series of human-machine evaluations show that compared with existing baselines, this method shows significant advantages in generating high-quality new scientific knowledge and is expected to become an important means to promote scientific and technological innovation.

[0009] The present application discloses a method for systematically enhancing document retrieval and knowledge mining based on a citation network, comprising the following steps:

[0010] Acquiring citation network data, wherein the citation network data includes documents and citation relationships between documents;

[0011] Enhanced systematic retrieval of literature based on citation networks, further expansion and pruning of citation networks to ensure the quality and relevance of input data;

[0012] Inject pruned knowledge into the large model, and the large model learns the knowledge and its complete citation network to generate initial new knowledge;

[0013] Different agent roles examine the initial new knowledge from multiple perspectives and increase the diversity and depth of the new knowledge through feedback;

[0014] Introduce prior knowledge and professional knowledge in real time to guide the model's new knowledge generation process and conduct in-depth mining of new knowledge;

[0015] Output new knowledge after mining.

[0016] Furthermore, the systematic retrieval of documents based on citation networks is enhanced, including:

[0017] Top-down subgraph citation network construction: starting from relevant research keywords, find relevant core literature, and conduct in-depth mining based on the core literature to build a subgraph citation network;

[0018] The bottom-up literature network review generation starts from each subgraph citation network, summarizes the research context and status of each subgraph network, and forms a high-level subgraph research review.

[0019] Furthermore, the top-down subgraph citation network construction includes:

[0020] First, use Semantic Scholar API to search for core keywords to obtain the digital object unique identifier DOI of the literature related to the research content;

[0021] Screen the literature and retain the core literature with importance and influence: evaluate and sort the literature based on the number of citations, literature type and literature source; specifically, define the core literature set as ,in Indicates the number of central documents, C m This is the mth central document;

[0022] Based on the determined central document, the subgraph network will be expanded and constructed from three dimensions: citation, cited, and related documents. Specifically, the subgraph network based on the specific central document will be constructed through the citation relationship, cited relationship, and related relationship of the document. The subgraph citation network of ,in represents the set of nodes in the network, represents the edge set; citation network E cite 、Citation Network E cited 、Related Network E related The construction process is expressed by the following formula:

[0023] ;

[0024] ;

[0025] ;

[0026] .

[0027] Furthermore, the bottom-up literature network review generation includes:

[0028] For the ith subgraph citation network , its literature collection List of references included ,in Indicates the number of documents;

[0029] The big model will use the text information of the papers in the literature list and combine it with its in-depth understanding of the research field to write a research review. In the process of writing the review, the big model takes into account the interrelationships between the papers, as well as their importance and influence in the research field.

[0030] In this way, each subgraph citation network Have a research review , Sn is a summary of the nth subgraph network.

[0031] Furthermore, the large model learns knowledge and its complete citation network to generate initial new knowledge, including:

[0032] First, a top-down subgraph citation network construction method is used to obtain highly structured document subsets organized around specific central documents. These subsets form a network of multi-dimensional information flows, providing background knowledge and context for LLMs. are all built around their core literature. Indicates the number of subgraphs;

[0033] By analyzing each sub-graph, the research , LLMs are scored from the perspective of potential research space and future development trends, so as to select the subgraphs with the most research potential and value. This selection process is formally expressed as:

[0034] ;

[0035] After selecting the research target, the corresponding target subgraph The literature in the literature is further structured and organized, so that the large model can obtain the development context and research status of the field; on this basis, LLMs will integrate the knowledge in these literatures and its own language generation ability to generate initial new knowledge. The initial new knowledge generation process is expressed by the following formula:

[0036] ;

[0037] in represents the initial generation of new scientific knowledge, For instruction prompts, is the selected target citation subgraph, for A review of research on .

[0038] Furthermore, the different agent roles described above examine the initial new knowledge from multiple perspectives and increase the diversity and depth of the new knowledge through critical feedback, including:

[0039] Each agent is assigned a specific role, including new knowledge proposer, reviewer, and neutral analyst, and each agent examines new knowledge from a different perspective;

[0040] Based on the output of the knowledge-enhanced large model, a set of initial new knowledge is formed;

[0041] Each agent proposes arguments for or against new knowledge according to its role. Prior knowledge participates in this process and guides the direction of knowledge mining among agents.

[0042] Valid arguments raised in knowledge mining are used to modify or strengthen new knowledge. This process includes requesting more citation subgraphs, reanalyzing existing data, or introducing new scientific theories. Prior knowledge guides the depth and breadth of mining in this process. At the same time, based on the key findings in the knowledge mining process, the knowledge-enhanced LLMs are asked to query data or adjust the model to obtain more evidence to support or refute new knowledge.

[0043] Iterative cycle: This process is repeated multiple times, and the new knowledge is evaluated and adjusted after each knowledge mining until it reaches the predetermined scientific rigor and innovation standards; after each round of knowledge mining, the integrator agent summarizes the mining points and jointly evaluates the improvement space of the new knowledge with the prior knowledge, thereby iteratively generating new knowledge versions.

[0044] Furthermore, the large model includes at least one of the following: ChatGPT, CoT, ICL, RAG, Multi-Agent.

[0045] The main technical effects of the present invention are as follows:

[0046] This paper systematically analyzes the limitations of large language models in dealing with scientific creativity problems, and points out the shortcomings of large models in terms of divergent thinking and convergent thinking;

[0047] This invention proposes a structured organization query method for scientific literature. By adaptively organizing and constructing the literature citation network, it effectively integrates the latest and most extensive scientific research results, ensuring that the knowledge provided for the large model is comprehensive and cutting-edge.

[0048] Based on the theory of structural intelligence, a multi-agent mining and discussion framework for human-machine collaboration is proposed. Under the collaboration of multiple agents and prior knowledge, it can imitate the divergent and convergent thinking processes of humans, effectively generate and refine new scientific knowledge, and improve the quality and innovation of new knowledge. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Schematic diagram of the enhanced process of systematic literature retrieval based on citation network;

[0050] Figure 2 An example of a citation subgraph using SubGTR as an example;

[0051] Figure 3 Comparison between initial new knowledge and new knowledge after human-machine collaboration;

[0052] Figure 4 The impact of introducing subgraph citation networks on generation quality; DETAILED DESCRIPTION

[0053] The present invention is further described below in conjunction with the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention belong to the protection scope of the present invention.

[0054] Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0055] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0056] The technical solutions provided in the embodiments of the present application involve technologies such as machine learning and natural language processing of artificial intelligence, which are specifically introduced and explained through the following embodiments.

[0057] Before introducing the embodiments of the present application, some terms involved in the present application are first explained.

[0058] 1. Generation of new scientific knowledge:

[0059] The generation of new scientific knowledge is a core issue in the field of AI4Science, which involves how to use artificial intelligence to assist or automate the initial stages of knowledge mining. In this field, research mainly focuses on developing algorithms and models that can extract new knowledge from existing data or predict new scientific phenomena. In recent years, with the advancement of machine learning technology, especially the successful application of deep learning in a variety of data-intensive tasks, the research on scientific knowledge mining has been significantly strengthened and gradually expanded to more complex scientific problem solutions. The ability of large models to use their existing knowledge to propose reasonable new scientific knowledge has become a topic of increasing interest. For example, Shojaee et al. used the programming ability of large models to effectively discover scientific equations. However, it is worth noting that despite the increasing exploration of LLMs in scientific settings and question answering, they still have many shortcomings in knowledge mining.

[0060] 2. Knowledge-enhanced large model:

[0061] Knowledge-enhanced large models combine traditional large-scale language models with external knowledge bases to improve the model's knowledge understanding and generation capabilities. These models are considered to be an effective way to overcome the generation limitations of large-scale language models, including issues such as factuality and long-term memory. Typically, such models first use external retrievers to retrieve relevant structured and unstructured knowledge from specific knowledge sources (such as Wikipedia, graphs, and databases), and then use this knowledge as external context for large-scale language models to generate knowledge-based answers.

[0062] Most previous works attempt to optimize the retriever and reader independently. In the initial stage, some researchers used sparse retrievers, such as BM25 and TF-IDF, to make relevance decisions and retrieve evidence. However, sparse methods are insufficient in extracting semantic features of text content. To address this problem, some researchers proposed dense retrieval methods based on language models, which effectively represent the semantic features of text content by encoding documents and queries into dense vectors. However, these complex architectures and interactions are not feasible and inefficient in the era of large models.

[0063] Recently, there have been a number of studies exploring the performance of LLMs as retrievers. For example, Shen et al. first demonstrated that large models can be used as zero-shot retrievers on multiple benchmark datasets, while Ma et al. proposed a Listwise Reranker that leverages large language models and achieves strong reranking effects without using task-specific training data. Sun et al. investigated the relevance ranking of generative large models such as ChatGPT and GPT-4 and found that properly guided LLMs can output results that are competitive or even better than state-of-the-art supervised methods. In addition, they explored the potential of refining ChatGPT's ranking capabilities into small specialized models to improve efficiency in practical applications. It is worth noting that this way of being able to more accurately capture and apply expertise is particularly important in scientific research that requires deep domain knowledge. For example, Jeong et al. introduced knowledge-enhanced large models in the biomedical field for generating explanations, retrieving domain-specific documents, and self-reflection-generated answers, which can more accurately support the development of the biomedical field.

[0064] In addition, some studies have attempted to update the model’s knowledge base in real time to ensure the timeliness and accuracy of its output. In this way, knowledge-enhanced large models not only improve their performance in specific fields, but also adapt to the rapidly changing scientific research environment and generate content that is aligned with the current research frontier.

[0065] 3. Intelligent system of human-machine collaboration:

[0066] Human-machine collaborative intelligent systems are designed to accomplish specific tasks together with human prior knowledge, emphasizing the complementarity of artificial intelligence and prior knowledge, combining prior knowledge with the computing power and data processing speed of machines. Traditional research has focused on improving the naturalness and efficiency of human-robot and artificial intelligence system interactions to effectively meet human needs. The rise of big models marks a major shift in the field, emphasizing the role of prior knowledge in enhancing the capabilities of intelligent agents. This approach significantly improves the performance of intelligent agents by leveraging prior knowledge to improve performance and decision-making processes. Recent studies have adopted heuristic rules or learnable algorithms to guide these agents to seek help from prior knowledge. In addition, there is increasing attention to developing specialized prompts to motivate big model-based agents to actively seek input from prior knowledge, thereby cultivating more interactive and collaborative dynamics in these partnerships. Feng et al. designed a general and learnable method to coordinate prior knowledge with big model-based agents for efficient collaboration in the form of direct planning.

[0067] In the field of scientific research, human-computer collaborative systems are also of great significance, because the complexity and innovation of many scientific problems require both deep expertise and efficient information processing capabilities. Through human-computer collaboration, the quality and innovation of knowledge mining can be improved, especially when dealing with complex data sets, solving unknown problems, or making innovative designs. The design of such systems usually involves the optimization of user interfaces, the design of interactive processes, and the construction of feedback mechanisms to ensure that prior knowledge and artificial intelligence systems can collaborate effectively.

[0068] In order to solve the problems existing in the generation of new scientific knowledge by large models, this paper proposes a human-in-loop multi-agent framework (HMA) based on the theory of structural intelligence. First, in order to solve the problem of the lack of dynamic knowledge update of large models, this paper injects the latest scientific research-related knowledge into the large models through the method of retrieval enhancement. Without the need for expensive retraining or complex fine-tuning of the model, low-cost contextual prompts are used to make up for the lack of dynamic knowledge update of large models. The model can access and learn the latest scientific research papers and their complete citation network, so that it has the latest and complete scientific research progress when generating new scientific knowledge. Secondly, by introducing a new knowledge iterative enhancement method based on multi-agent mining, the divergent thinking of humans is imitated in the form of mining, the ability of large models is stimulated, and divergent thinking is used to generate as much preliminary knowledge as possible. In this framework, different agent roles are responsible for examining the initial knowledge from multiple perspectives, and through critical feedback, the diversity and depth of new knowledge are increased. This method simulates the peer review process in the scientific community and improves the scientificity and practicality of new knowledge. Finally, in order to further improve the professionalism and pertinence of large models, the HMA framework combines prior knowledge and the data processing capabilities of the model. Prior knowledge and expertise are introduced in real time during the mining process to guide the model's new knowledge generation process, and the most promising solutions are selected through convergent thinking for deepening and refinement. This collaboration aims to maximize the potential of large models and prior knowledge to generate new scientific knowledge that better meets actual research needs. The implementation of this comprehensive strategy can significantly improve the application effect of large models in the field of scientific research, so that the generated new knowledge meets the requirements of scientific research in terms of quality and innovation.

[0069] This invention proposes a new scientific new knowledge generation framework HMA, which aims to extract and generate valuable new scientific knowledge from massive scientific literature through human-computer collaboration and multi-agent technology. The HMA framework includes three modules: systematic literature retrieval enhancement based on citation network, initial knowledge generation of knowledge-enhanced large model, and multi-agent mining iteration of human-computer collaboration. First, the literature related to the research topic will be searched through keywords, and further expanded and pruned based on the network of literature citations to ensure the quality and relevance of the input data; then, a large model will be used to conduct in-depth exploration and analysis based on systematic literature information to generate preliminary knowledge content; finally, the human-computer collaborative multi-agent system will be used to iterate new knowledge and simulate the divergent and convergent process of thinking. Ultimately, the accuracy and innovation of new knowledge generation are achieved.

[0070] Enhanced systematic retrieval of literature based on citation network

[0071] In order to provide LLMs with systematic scientific knowledge, the present invention proposes a method for enhancing systematic literature retrieval based on citation networks. Through the adaptive organization and construction of literature citation networks, the latest and extensive scientific research results are effectively integrated, and the research context of different research points is systematically sorted out to ensure that the knowledge provided for the large model is cutting-edge and systematic. The method is mainly divided into two parts: the top-down subgraph citation network construction starts from relevant research keywords, searches for relevant central documents, and conducts in-depth mining based on the central documents to construct a subgraph citation network; the bottom-up literature network review generation starts from each subgraph citation network, summarizes the research context and status of each subgraph network, and forms a high-order subgraph research review. The combination of the two can achieve in-depth mining and sorting of literature in a specific field, and provide a systematic and reliable source of literature knowledge for the large model. Figure 1 The enhanced process of systematic literature retrieval based on citation network is demonstrated.

[0072] Top-down subgraph citation network construction

[0073] In order to conduct systematic retrieval enhancement in the field of scientific literature, this paper adopts a citation network-based method to build a structured document organization and query system. The following is a detailed description of the top-down subgraph citation network construction process, including keyword retrieval, document screening, and network expansion steps.

[0074] First, use the Semantic Scholar API to search for core keywords to obtain the digital object unique identifier (DOI) of documents related to the research content. The key to this step is to select keywords that can accurately represent the core of the research field to ensure the accuracy and comprehensiveness of the search results. Due to the large number and complexity of the search results, it is necessary to screen the documents and retain the central documents with importance and influence. This step uses the number of citations, document type and document source to evaluate and sort the documents. Specifically, the central document set is defined as ,in Indicates the number of central documents.

[0075] Based on the determined central document, the subgraph network will be expanded and constructed from three dimensions: citation, cited, and related documents. Specifically, the present invention constructs a subgraph network based on a specific central document through the citation relationship, cited relationship, and related relationship of the document. The subgraph citation network of ,in represents the set of nodes in the network, Represents the edge set. The construction process of citation network, cited network and related network can be expressed by the following formula:

[0076] ;

[0077] ;

[0078] ;

[0079] ;

[0080] Through the above steps, a structured, multi-dimensional literature network can be constructed from a central document. The network takes the central document as the core and includes documents closely related to the research content, as well as the development context and trend of the research. This provides an in-depth information foundation and theoretical support for further research.

[0081] Bottom-up literature network review generation:

[0082] Despite the screening, each subgraph citation network constructed is still relatively large (dozens of articles). Considering the length of the context window of the large language model and the computational efficiency, the present invention adopts a bottom-up literature network review generation method to summarize the literature in each subgraph citation network and let the large model generate a review of the literature.

[0083] For each subgraph citation network , its literature collection List of references included ,in Represents the number of documents. The large model will use the text information of the papers in the literature list and combine it with its in-depth understanding of the research field to write a research review. These reviews will cover the research topics, research methods, research results, etc. involved in the subgraph citation network. In the process of writing the review, the large model will take into account the relationship between the documents, as well as their importance and influence in the research field.

[0084] In this way, each subgraph citation network Have a research review These reviews will help the big model quickly select the literature it wants to investigate in the subsequent literature search and research process, avoiding unnecessary calculations and queries. At the same time, these reviews can also provide users with a comprehensive and systematic research overview to promote their understanding of subsequent new knowledge generation.

[0085] Figure 2 An example of a citation subgraph network using SubGTR is shown. Through the top-down construction of the subgraph citation network, a structured and systematic knowledge base is established, which includes both comprehensive and complete specific literature and complete and systematic literature reviews in different subfields, thus providing a solid foundation for the subsequent generation and verification of new knowledge. The design of this system fully reflects the balance between the pursuit of new knowledge and the respect for existing knowledge in scientific research, laying the foundation for the generation of high-quality and innovative new scientific knowledge.

[0086] Knowledge-enhanced research idea generation

[0087] After building a systematic literature retrieval enhancement mechanism based on citation networks, we will now explore how to apply knowledge-enhanced large-scale language models to generate in-depth and innovative research ideas. This process aims to integrate structured and systematic literature knowledge directly into the model's reasoning and generation capabilities to overcome the limitations of traditional large models in generating innovative new scientific knowledge.

[0088] First, a top-down subgraph citation network construction method is used to obtain highly structured document subsets organized around specific central documents. These subsets form a network of multi-dimensional information flows, providing rich background knowledge and context for LLMs. are all built around their core literature. Indicates the number of subgraphs.

[0089] By analyzing each sub-graph, the research , LLMs can score from the perspectives of research significance, potential research space, future development trends, etc., so as to select the subgraphs with the most research potential and value. This selection process can be formally expressed as:

[0090] ;

[0091] After selecting the research target, the corresponding target subgraph The literature in the literature is further structured and organized, so that the large model can fully capture the development context and research status of the field. On this basis, LLMs will combine the knowledge in these literatures and its own language generation ability to generate comprehensive and effective research ideas. This idea generation process can be expressed by the following formula:

[0092] ;

[0093] in represents the initial generation of new scientific knowledge, For the command prompt, is the selected target citation subgraph, for A review of research on .

[0094] Through this knowledge-enhanced approach, research proposals generated by LLMs are not only more credible and innovative, but also improve the overall quality of new knowledge. Such knowledge-enhanced LLMs can accelerate the process of generating new knowledge.

[0095] Human-machine collaborative multi-agent mining iteration

[0096] In the exploration of rigorous scientific research, the incubation of innovative new knowledge is a dynamic and iterative process that requires constant questioning, revision and deepening. In one embodiment, an innovative human-machine collaborative strategy is introduced to achieve deep refinement and optimization of new scientific knowledge through a multi-agent mining iteration mechanism. The core of this method is to use the diverse perspectives of agents from different backgrounds and expertise (including LLMs and prior knowledge) to iteratively refine and enhance the initially generated new knowledge through mining and critical thinking, ensuring that each new knowledge is fully considered and verified.

[0097] The multi-agent mining iteration process is designed as follows:

[0098] 1. Role assignment: Each agent is assigned a specific role, including new knowledge proposer, reviewer, and neutral analyst. These roles help structure mining, allowing each agent to look at the problem from different perspectives;

[0099] 2. Initial new knowledge generation: Based on the output of the knowledge-enhanced large model, a set of preliminary new scientific knowledge is first formed;

[0100] 3. Open Mining: Agents conduct open mining, and each agent proposes arguments for or against the new knowledge according to its role. This stage aims to reveal potential weaknesses and unconsidered variables of the new knowledge. Prior knowledge participates in this process and guides the mining direction of the agents;

[0101] 4. Evidence integration: The valid points raised in the mining process will be used to modify or strengthen the new knowledge. This process includes requesting more citation subgraphs, reanalyzing existing data, or introducing new scientific theories. Prior knowledge guides the depth and breadth of mining in a timely manner during this process. At the same time, based on the key findings in the mining process, the knowledge-enhanced LLMs are asked to query data or adjust the model to obtain more evidence to support or refute the new knowledge.

[0102] 5. Iterative loop: This process can be repeated multiple times, and the new knowledge is evaluated and adjusted after each mining until it reaches the predetermined scientific rigor and innovation standards. After each round of mining, the integrator agent summarizes the mining points and evaluates the improvement space of the new knowledge together with the prior knowledge, so as to iteratively generate a more mature and refined new knowledge version.

[0103] The main purpose of multi-agent mining iterations is to use collective intelligence to improve the quality and reliability of new knowledge. Each new knowledge is exhaustively challenged and defended to ensure its logical soundness and solid empirical foundation, helping to reveal new research directions or methodological issues that may not have been noticed by a single agent.

[0104] In the framework of the present invention, human-computer collaboration is a key step, which combines prior knowledge with the efficient data processing and integration capabilities of large models. The core purpose of this link is to make full use of the respective advantages of prior knowledge and large model agents to jointly create new scientific knowledge that meets scientific standards and is highly innovative. In this process, prior knowledge is responsible for providing directional guidance, in-depth analysis, and complex decision-making, which includes pointing out undiscovered new knowledge gaps, providing additional expertise, or redefining the boundaries of the problem, while multi-agents play a role in big data processing, new knowledge iteration, and pattern extraction. The human-computer collaboration model not only accelerates the generation process of new scientific knowledge, but also enhances the applicability and accuracy of new knowledge, ensuring that the generated new scientific knowledge has both depth and breadth, and can adapt to the complex and changing scientific research environment.

[0105] The present invention obtains real-time academic literature information through the API interface of Semantic Scholar. In the HMA framework, Qwen-Max is used as the base model. Taking the material discipline as an example, the present invention asks 20 graduate students majoring in materials to propose five real research questions in their respective research fields, and obtains a total of 100 different research topics.

[0106] Based on 100 materials science research questions, the baseline model is used to generate corresponding new scientific knowledge. Each baseline obtains a total of 100 new scientific knowledge for evaluation, and a total of 600 new scientific knowledge is generated for evaluation.

[0107] In this work, since the present invention targets the new task of scientific knowledge generation, there is no directly available baseline for comparison. Therefore, the present invention compares the full HMA framework with the following general baselines and ablation variants:

[0108] ChatGPT: ChatGPT is a large-scale language model developed by OpenAI that can generate coherent and creative texts. It is widely used in dialogue systems, content creation, and text generation tasks. This baseline uses ChatGPT to directly generate new scientific knowledge;

[0109] CoT (Chain-of-Thought) is a large model improvement engineering method that aims to guide the large model to think step by step to generate more reliable answers. In the experiment, this baseline guides the large model to generate new scientific knowledge by adding step-by-step thinking instructions to the prompt words;

[0110] ICL (In-Context Learning) is a technique for learning and generating through contextual cues, relying on the input context provided to the model in order to generate relevant outputs. In the experiment, we used manually written examples of new scientific knowledge as contextual cues, allowing the model to refer to these examples to generate new scientific knowledge;

[0111] RAG (Retrieval-Augmented Generation) is a technology that combines retrieval and generation. It enhances the generation capability of the model by dynamically retrieving relevant literature during the generation process. We use subject keywords to retrieve relevant scientific research paper abstracts in real time, and select the top 10 to provide to the large model as auxiliary reference. This method can alleviate the problem of untimely knowledge update of the large model to a certain extent, and provide more novel and cutting-edge scientific research information;

[0112] Multi-Agent: A multi-agent system achieves complex tasks through the collaborative work of multiple agents with different roles and functions. This application designs a multi-agent framework in which different agents are responsible for different tasks, including new knowledge proposers, critics, or neutral analysts. Through automated collaboration and mining between agents, new scientific knowledge is iteratively generated. This method aims to maximize the use of collective intelligence and improve the scientificity and innovation of new knowledge.

[0113] Given that scientific knowledge generation is a new task, there is no known benchmark to measure the generation quality. Therefore, we use model-based automatic evaluation and prior knowledge evaluation to validate different models on our experimental benchmark.

[0114] Model-based evaluation: Following the recent paradigm of using LLM to judge the quality of output text, we use GPT-4 to judge the quality of generated scientific new knowledge. We designed five different criteria to measure the quality of scientific new knowledge, and then asked the evaluation model to rate the generation ideas of each criterion on a 5-point Likert scale. We provide detailed expert-developed criteria and prompts used to guide the evaluation in Table 1.

[0115] Prior knowledge-based evaluation: Similar to model-based evaluation, it allows prior knowledge to be compared in pairs between two answers without model information, and selects new scientific knowledge with higher quality.

[0116] Table 1 5-point Likert rating scale used in model evaluation:

[0117] ;

[0118] In order to compare the effectiveness of our model with the baseline method, extensive comparative tests were conducted on the generated scientific new knowledge data.

[0119] Table 2 shows the GPT-4 scoring results of the Likert scale for different baseline models;

[0120] Table 2 Model evaluation results of different baselines:

[0121] ;

[0122] First, judging from the overall score of the baseline model, the new scientific knowledge generated by the large model achieved relatively high scores in terms of practicality and theoretical basis, which is mainly attributed to the pre-training of the large model on massive texts, which can master a large amount of basic theory and common sense knowledge. However, it did not perform well in the three scoring dimensions of innovation, feasibility, and data support. This is because the large model is limited to repetitive knowledge and patterns that have been seen, lacks innovative thinking, and is prone to put forward unrealistic ideas that lack feasibility and theoretical support.

[0123] Secondly, by comparing the performance of CoT, ICL and ChatGPT, we can find that by guiding the big model to think gradually and provide specific examples of new scientific knowledge, it can effectively improve the quality of generating new knowledge, which shows that the general big model capability improvement method is also applicable to new scientific knowledge tasks; by retrieving the latest relevant literature, the RAG method can significantly improve the quality of generated new scientific knowledge, which is due to the model's ability to receive the latest scientific research progress, effectively improving the innovation and data support scores, making the generated new scientific knowledge cutting-edge and effective. In addition, thanks to the iterative discussion of multiple agents, Multi-Agent achieved a better model score, indicating that by maximizing the use of collective wisdom, the scientificity and innovation of new knowledge can be improved.

[0124] The HMA framework of the present invention significantly outperforms all baseline models, with an innovation score of 4.6 points. The total score is the only model among all baselines that exceeds 4 points, reaching the measurement standard of "having high scientific value and practical significance". The experimental results show that by making full use of the respective advantages of prior knowledge and large model agents, new scientific knowledge that meets scientific standards and is highly innovative can be jointly created. The experimental results highlight the importance of human-computer collaboration. In this process, prior knowledge is responsible for providing directional guidance, in-depth analysis and complex decision-making, while the large model continuously optimizes new knowledge through citation network construction, multi-agent iteration, etc., enhancing the applicability and accuracy of new knowledge, and ensuring that the generated new scientific knowledge has both depth and breadth.

[0125] To further verify the importance of human-machine collaboration, we first used the HMA framework to generate initial new knowledge that had not been discussed, and then iterated the human-machine collaboration to generate the final scientific new knowledge based on this, in order to compare the score differences between the two. Figure 3 As shown in the figure, the experimental results prove that human-machine collaborative iteration can significantly improve the generation effect of the model in all indicators. This is because prior knowledge can provide directional guidance, help LLMs make complex decisions and inspire new ideas. In the process of human-machine iteration, some detailed errors and unreasonable new knowledge can also be discovered and corrected in time, further improving the overall quality of new knowledge.

[0126] Figure 4 The radar chart comparison between the new scientific knowledge generated based on the subgraph citation network and the new scientific knowledge based on the large model itself is shown. It can be found that the innovation and overall evaluation of the new scientific knowledge generated by the large model can be significantly enhanced after the subgraph citation network, which also reflects that the subgraph citation network can help LLMs explore innovative ideas and cutting-edge methods from the latest research through the latest scientific literature and research progress for the large model.

[0127] Table 3 Examples of new scientific knowledge generation with "silicon nitride ceramics" as the research topic:

[0128] ;

[0129] Table 3 shows the new scientific knowledge generated by ChatGPT, RAG, Multi-Agent, and HMA with "silicon nitride ceramics" as the keyword. Among them, the new knowledge of ChatGPT proposed a design idea that combines materials science and electronic engineering, but lacks specific details on how to achieve this combination. RAG proposed the concept of microsystems through retrieval enhancement, but also lacked specific implementation steps and technical details, making it difficult to transform these new knowledge into research plans.

[0130] In contrast, the new scientific knowledge generated by Multi-Agent and HMA is more specific and operational. Multi-Agent describes in detail experimental methods such as chemical synthesis, electrochemical technology, and quantum chemical calculations, providing a clear research path. The new knowledge of HMA is more comprehensive and in-depth. It proposes a multi-level structured silicon nitride ceramic composite material design, elaborates on the design methods of each level of materials, and gives a theoretical explanation of the expected microstructure and performance improvement. These detailed methods and steps make this new knowledge easier to transform into actual scientific research practice, with high operability and feasibility, and more in line with user expectations.

[0131] As scientific research continues to develop, the generation of innovative new knowledge has become increasingly important. However, faced with massive scientific literature and complex interdisciplinary knowledge systems, traditional scientific research methods face huge challenges in information processing and knowledge integration. This paper proposes a multi-agent scientific new knowledge generation framework based on human-computer collaboration, which effectively improves the generation quality and innovation of new scientific knowledge by combining divergent thinking and convergent thinking in structural intelligence theory. A series of experimental results show that compared with the existing baseline model, the HMA framework shows significant advantages in generating high-quality new scientific knowledge. This not only theoretically verifies the application value of structural intelligence theory in scientific research, but also proves the potential of human-computer collaboration in scientific innovation in practice. This framework combines multi-agent and human-computer collaboration and has wide scalability.

[0132] The main technical effects of the present invention are as follows:

[0133] This paper systematically analyzes the limitations of large language models in dealing with scientific creativity problems, and points out the shortcomings of large models in terms of divergent thinking and convergent thinking;

[0134] The present invention proposes a structured organization query method for scientific literature. By adaptively organizing and constructing the literature citation network, it effectively integrates the latest and most extensive scientific research results, ensuring that the knowledge provided for the large model is comprehensive and cutting-edge.

[0135] Based on the theory of structural intelligence, a multi-agent mining and discussion framework for human-machine collaboration is proposed. Under the collaboration of multiple agents and prior knowledge, it can imitate the divergent and convergent thinking processes of humans, effectively generate and refine new scientific knowledge, and improve the quality and innovation of new knowledge.

[0136] In the evaluation of a series of models, the HMA framework showed significant advantages. Compared with the existing LLMs-driven baseline model, the new scientific knowledge generated was significantly improved in terms of innovation, practicality, and feasibility.

[0137] As used herein, the word "preferred" is intended to be used as an example, instance, or illustration. Any aspect or design described herein as "preferred" is not necessarily to be construed as being more advantageous than other aspects or designs. On the contrary, the use of the word "preferred" is intended to present concepts in a specific way. The term "or" as used in this application is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X uses A or B" means any one of the naturally included permutations. That is, if X uses A; X uses B; or X uses both A and B, then "X uses A or B" is satisfied in any of the foregoing examples.

[0138] Moreover, although the present disclosure has been shown and described with respect to one or implementations, those skilled in the art will think of equivalent variations and modifications based on the reading and understanding of this specification and the accompanying drawings. The present disclosure includes all such modifications and variations, and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the above-mentioned components (such as elements, etc.), the terms used to describe such components are intended to correspond to any component (unless otherwise indicated) that performs the specified function of the component (such as it is functionally equivalent), even if the structure is not equivalent to the disclosed structure of the function in the exemplary implementation of the present disclosure shown herein. In addition, although the specific features of the present disclosure have been disclosed with respect to only one of several implementations, such features can be combined with one or other features of other implementations that may be desired and advantageous for a given or specific application. Moreover, insofar as the terms "including", "having", "containing" or their variations are used in specific embodiments or claims, such terms are intended to be included in a manner similar to the term "comprising".

[0139] The functional units in the embodiments of the present invention may be integrated into a processing module, or each unit may exist physically separately, or multiple or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc. The above-mentioned devices or systems may execute the storage method in the corresponding method embodiment.

[0140] To sum up, the above embodiment is an implementation mode of the present invention, but the implementation mode of the present invention is not limited by the embodiment. Any other changes, modifications, substitutions, combinations, and simplifications that deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A method for systematically enhancing document retrieval and knowledge mining based on citation networks, characterized in that: The following steps are involved: Acquiring citation network data, wherein the citation network data includes documents and citation relationships between documents; Enhanced systematic retrieval of literature based on citation networks, further expansion and pruning of citation networks to ensure the quality and relevance of input data; Inject pruned knowledge into the large model, and the large model learns the knowledge and its complete citation network to generate initial new knowledge; Different agent roles examine the initial new knowledge from multiple perspectives and increase the diversity and depth of the new knowledge through feedback; Introduce prior knowledge and professional knowledge in real time to guide the model's new knowledge generation process and conduct in-depth mining of new knowledge; Output the new knowledge after mining; The enhancement of systematic literature retrieval based on citation network includes: Top-down subgraph citation network construction: starting from relevant research keywords, find relevant core literature, and conduct in-depth mining based on the core literature to build a subgraph citation network; The bottom-up literature network review generation starts from each subgraph citation network, summarizes the research context and status of each subgraph network, and forms a high-level subgraph research review; The different agent roles described above examine the initial new knowledge from multiple perspectives and increase the diversity and depth of the new knowledge through critical feedback, including: Each agent is assigned a specific role, including new knowledge proposer, reviewer, and neutral analyst, and each agent examines new knowledge from a different perspective; Based on the output of the knowledge-enhanced large model, a set of initial new knowledge is formed; Each agent proposes arguments for or against new knowledge according to its role. Prior knowledge participates in this process and guides the direction of knowledge mining among agents. Valid arguments raised in knowledge mining are used to modify or strengthen new knowledge. This process includes requesting more citation subgraphs, reanalyzing existing data, or introducing new scientific theories. Prior knowledge guides the depth and breadth of mining in this process. At the same time, based on the key findings in the knowledge mining process, the knowledge-enhanced LLMs are asked to query data or adjust the model to obtain more evidence to support or refute new knowledge. Iterative cycle: This process is repeated multiple times, and the new knowledge is evaluated and adjusted after each knowledge mining until it reaches the predetermined scientific rigor and innovation standards; after each round of knowledge mining, the integrator agent summarizes the mining points and jointly evaluates the improvement space of the new knowledge with the prior knowledge, thereby iteratively generating new knowledge versions.

2. The method for systematically enhancing document retrieval and knowledge mining based on citation networks according to claim 1, characterized in that: The top-down subgraph citation network construction includes: First, use Semantic Scholar API to search for core keywords to obtain the digital object unique identifier DOI of the literature related to the research content; Screen the literature and retain the core literature with importance and influence: evaluate and sort the literature based on the number of citations, literature type and literature source; specifically, define the core literature set as ,in represents the number of central documents, C m For the m Central document; Based on the determined central document, the subgraph network will be expanded and constructed from three dimensions: citation, cited, and related documents. Specifically, the subgraph network based on the specific central document will be constructed through the citation relationship, cited relationship, and related relationship of the document. The subgraph citation network of ,in represents the set of nodes in the network, Represents a set of edges; citation network E cite , Cited Network E cited , related networks E related The construction process is expressed by the following formula: ； ； ； 。 3. The method for systematically enhancing document retrieval and knowledge mining based on citation network according to claim 2, characterized in that: The bottom-up literature network review generation includes: For the i Subgraph Citation Network , its literature collection List of references included ,in Indicates the number of documents; The big model will use the text information of the papers in the literature list and combine it with its in-depth understanding of the research field to write a research review. In the process of writing the review, the big model takes into account the interrelationships between the papers, as well as their importance and influence in the research field. In this way, each subgraph citation network Have a research review , S n It is n A review of subgraph networks.

4. The method for systematically enhancing document retrieval and knowledge mining based on citation network according to claim 3, characterized in that: The large model learns knowledge and its complete citation network to generate initial new knowledge, including: First, a top-down subgraph citation network construction method is used to obtain highly structured document subsets organized around specific central documents. These subsets form a network of multi-dimensional information flows, providing background knowledge and context for LLMs. are all built around their core literature. Indicates the number of subgraphs; By analyzing each sub-graph , LLMs are scored from the perspective of potential research space and future development trends, so as to select the subgraphs with the most research potential and value. This selection process is formally expressed as: ； After selecting the research target, the corresponding target subgraph The literature in the literature is further structured and organized, so that the large model can obtain the development context and research status of the field; on this basis, LLMs will integrate the knowledge in these literatures and its own language generation ability to generate initial new knowledge. The initial new knowledge generation process is expressed by the following formula: ； in represents the initial generation of new scientific knowledge, For instruction prompts, is the selected target citation subgraph, for A review of research on .

5. The method for systematically enhancing document retrieval and knowledge mining based on citation network according to claim 4, characterized in that: The large model includes at least one of the following: ChatGPT, CoT, ICL, RAG, Multi-Agent.

Images