Large-model-driven intelligent recommendation method and system for large-scale scientific data sharing platform

Abstract

The invention belongs to the technical field of artificial intelligence and information retrieval, and relates to a large-model-driven intelligent recommendation method and system for a large-scale scientific data sharing platform. The method comprises the following steps: analyzing a scientific research problem expressed by a natural language input by a user, and generating structured scientific research intention information; multiple rounds of dialogues in the scientific research query process are compressed, and important context information is reserved; and performing retrieval based on the structured scientific research intention information to obtain a candidate data set, screening the candidate data set by using context information, and taking the screened data set as a recommendation result. The system comprises a scientific research intention perception module, a structured memory compression module and a trusted data set retrieval and recommendation module. According to the method, deep semantic perception, multi-round context continuous tracking and traceable credible recommendation of results for complex scientific research requirements are realized, and the applicability and user experience of a scientific data set recommendation system in an actual scientific research environment are remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and information retrieval technology, specifically to a dataset recommendation method and system for scientific research scenarios, and in particular to an intelligent recommendation method and system that combines scientific research intent parsing, multi-round memory compression, and trusted dataset retrieval and reordering. Background Technology

[0002] With the widespread application of artificial intelligence in scientific research (AI for Science, or AI4S for short), the value of high-quality scientific data in various fields such as life sciences, physics, chemistry, and earth sciences is becoming increasingly prominent. To promote the open sharing and reuse of scientific data, governments and research institutions around the world have successively established national-level scientific data centers and dataset sharing platforms, such as NCBI, OpenAIRE, and ScienceDB. These platforms effectively improve the accessibility and reusability of scientific data by providing unified data storage and access interfaces, thereby promoting the collaborative development of interdisciplinary research.

[0003] However, with the explosive growth of scientific data, it has become increasingly difficult for researchers to quickly locate data resources that are "task-relevant, experimentally matched, and credible and citationable" within massive datasets. To improve research efficiency, existing research has attempted to build automated dataset recommendation systems, mainly divided into the following two categories: 1) Behavior-driven recommendation systems, such as collaborative filtering and graph representation learning, which rely on users' historical interaction behavior for recommendations. 2) Content-driven recommendation systems, which match the query content itself (such as keywords and semantic vectors) with the dataset, and are often used in platform search engines or semantic retrieval systems.

[0004] While existing methods have achieved good results in industrial recommendation systems, they still suffer from significant limitations in scientific research data recommendation scenarios. Firstly, scientific research data requirements are highly task-relevant and exhibit cross-domain variability, making accurate modeling based on user history impossible. For example, researchers' needs regarding research objects, species, and data modalities often change continuously with research tasks, and traditional behavioral recommendation methods lack generalization ability. Secondly, data requirements in scientific research tasks are often presented in experimental-level natural language, exhibiting high complexity and strong domain context, frequently involving multiple constraints such as data modalities, processing flows, experimental conditions, and evaluation metrics. Traditional keyword-based or embedding-based recommendation methods struggle to fully understand these complex intentions, leading to deviations between recommendation results and actual research needs.

[0005] Meanwhile, although Large Language Models (LLMs) and intelligent agents have shown great potential in multi-round recommendation tasks in recent years, they suffer from significant **"forgetting" and "illusion" problems**. That is, they may miss key information expressed by the user in the context, or generate fictitious or inaccessible recommendation results. This is unacceptable in a research environment that prioritizes data accessibility, credibility, and citationability.

[0006] Therefore, existing technologies still lack an intelligent scientific research dataset recommendation system that can deeply understand research intentions, support multi-turn dialogues, and possess reliable traceability capabilities. Summary of the Invention

[0007] To address the problems of existing scientific research dataset recommendation systems failing to effectively understand the complex intentions of scientific research tasks, forgetting key information in multi-turn dialogues, and lacking reliable traceability of recommendation results, this invention proposes a large-model-driven intelligent recommendation method and system for large-scale scientific data sharing platforms, called ScienceDB.AI.

[0008] The technical solution adopted in this invention is as follows:

[0009] A large-model-driven intelligent recommendation method for large-scale scientific data sharing platforms includes the following steps:

[0010] The system parses the research questions expressed in natural language input by users and generates structured research intent information.

[0011] The multi-turn dialogues during the scientific research inquiry process are compressed while retaining important contextual information;

[0012] The candidate datasets are obtained by searching based on the structured research intent information, and the candidate datasets are filtered using the context information. The filtered datasets are then used as the recommended results.

[0013] Furthermore, the process of parsing the research question expressed in natural language input by the user to generate structured research intent information includes:

[0014] Based on six predefined intent elements in the research process, including: research topic, research task, data modality, experimental setup / constraints, evaluation metrics, and other specific needs;

[0015] The large language model is invoked to parse the user input intent, extract the six types of intent elements, form a standardized intent template, and generate the parsing result for the current round.

[0016] Furthermore, the multi-turn dialogue in the scientific research query process is compressed while retaining important contextual information, including:

[0017] Let Θ 1:T This indicates the dialogue history up to round T. in τ represents the user query after intent parsing of the user input in the t-th round. t This represents the tool call and execution log for round t, r t This represents the response in round t;

[0018] Θ 1:t Compressed to structured memory In the middle, memory The information from the first t-1 rounds is retained:

[0019]

[0020] Where, Θ 1:1 This indicates the first round of dialogue records, S t-1 This represents a summary of historical information up to round t-1. This indicates a summary of information derived from combining historical information from the previous t-1 rounds and the current round's dialogue. This represents an intelligent agent based on a large language model, used to summarize historical dialogue logs into structured information.

[0021] Furthermore, the process of retrieving candidate datasets based on the structured research intent information, filtering the candidate datasets using the context information, and using the filtered datasets as recommendation results includes:

[0022] A combination of vector matching and structured attribute filtering was used to screen the initial candidate set;

[0023] A deep interaction model is used to refine the initial candidate set to obtain the final candidate set;

[0024] The final candidate set, compressed multi-turn context information, and structured user intent information are input into the large language model agent to obtain the final recommendation dataset;

[0025] Assign a unique identifier to the recommended dataset.

[0026] Furthermore, the preliminary candidate set is screened by combining vector matching and structured attribute filtering, which includes: based on explicit user input and constraints extracted by the language model, multi-dimensional screening is performed by combining vector matching and structured attribute filtering to obtain the preliminary candidate set; the multi-dimensional factors include publication time, affiliated organization, and data type.

[0027] Furthermore, the step of using a deep interaction model to refine the preliminary candidate set includes: inputting explicit user input and constraints extracted by the language model, as well as the preliminary candidate set; obtaining similarity scores between the preliminary candidate set and the query through the deep interaction model, and ranking and filtering based on the similarity scores.

[0028] Furthermore, assigning a unique identifier to the recommended dataset includes: forcibly attaching a unique CSTR identifier link to each recommendation result according to the system prompt strategy; the system prompt strategy refers to using prompt words to force the large language model to output a CSTR representation.

[0029] A large-model-driven intelligent recommendation system for large-scale scientific data sharing platforms, comprising:

[0030] The research intent perception module is used to parse the research questions expressed in natural language by the user and generate structured research intent information.

[0031] The structured memory compression module is used to compress multi-turn dialogues during scientific research queries while retaining important contextual information;

[0032] The trusted dataset retrieval and recommendation module is used to retrieve candidate datasets based on the structured research intent information, and to filter the candidate datasets using the context information, and to use the filtered datasets as the recommendation results.

[0033] Through the above technical solutions, this invention achieves deep semantic perception of complex scientific research needs, continuous multi-turn context tracking, and traceable and reliable recommendation of results, significantly improving the applicability and user experience of scientific dataset recommendation systems in real-world research environments. This invention supports multi-turn conversational queries, can achieve high-precision dataset recommendations based on complex user descriptions of scientific research, can be integrated into scientific data platforms, supports real-time retrieval and recommendation of datasets with tens of millions of records, and has good scalability and response efficiency. Attached Figure Description

[0034] Figure 1 This is a flowchart of the key steps of the large-model-driven intelligent recommendation method for large-scale scientific data sharing platforms of the present invention.

[0035] Figure 2 This is a block diagram of the module composition of the large-model-driven intelligent recommendation system for a large-scale scientific data sharing platform according to the present invention.

[0036] Figure 3 This is a comparison of the evaluation metrics between the present invention and existing methods. Detailed Implementation

[0037] The present invention will now be described in further detail with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0038] The present invention provides a large-model-driven intelligent recommendation method for large-scale scientific data sharing platforms, such as... Figure 1 As shown, the core technical solution includes the following three key steps:

[0039] 1) Research Intent Perception Step: Analyze the research questions expressed in natural language by the user and generate structured research intent information.

[0040] This step involves designing structured intent templates and parsing and abstracting researchers' multi-turn natural language queries based on a large language model. This transforms complex research task requirements into a machine-understandable structured representation, including elements such as research topic, research task, data modality, experimental setup / constraints, and evaluation metrics. Compared to traditional keyword matching and simple embedding, this approach enables a deep understanding of the fine-grained semantics at the research task level.

[0041] 2) Structured memory compression steps: Compress the multi-turn dialogues in the scientific research query process and retain important contextual information.

[0042] This step addresses the challenge of long and easily forgotten multi-turn dialogue history. It designs a dialogue history compression mechanism based on a large language model, which can dynamically summarize key information in multi-turn conversations and generate structured, time-related compressed memory representations to avoid the loss of key information. At the same time, it supports context-based conflict identification and clarification, improving the continuity of dialogue and recommendation accuracy.

[0043] 3) Trusted dataset retrieval and recommendation steps: Retrieve candidate datasets based on structured research intent information, and use contextual information to filter the candidate datasets, and use the filtered datasets as the recommendation results.

[0044] This step proposes a two-stage retrieval architecture that combines semantic vector retrieval with a ColBERT-based re-ranking method to balance retrieval efficiency and matching accuracy in large-scale data. Simultaneously, to address the issues of traceability and citation of recommendation results, a "Citationable Research Task Record (CSTR)" mechanism is introduced. This mechanism assigns a unique identifier to each recommendation dataset and mandates its inclusion in the output, ensuring the authority and credibility of the recommendation results.

[0045] Corresponding to the three steps in the above method, this invention proposes a large-model-driven intelligent recommendation system for large-scale scientific data sharing platforms, such as... Figure 2As shown, the system comprises three main modules: (1) a research intent perception module; (2) a structured memory compression module; and (3) a trusted dataset retrieval and recommendation module. This system supports multi-turn conversational queries and can achieve high-precision dataset recommendations based on the user's complex research descriptions. The relationship between these three modules is as follows: the research intent perception module first parses the research question input by the user to generate structured research intent information; the dataset retrieval and recommendation module uses the structured information output by the research intent perception module as the retrieval basis when performing retrieval and re-ranking, thereby ensuring that the retrieval results match the user's needs; in the recommendation stage, the dataset retrieval and recommendation module further combines the multi-turn conversational context information output by the structured memory compression module to filter candidate datasets, ensuring that the recommendation results are consistent with the user's overall research context. Through the cooperation between these modules, the system achieves a complete closed loop from research intent parsing to context-enhanced retrieval to trusted recommendation. This system can be integrated into a research data platform, supports real-time retrieval and recommendation of datasets at the tens of millions level, and has good scalability and response efficiency.

[0046] The specific implementation of the above three steps / modules of the present invention will be described in detail below.

[0047] 1. Problem Definition

[0048] Let Q = {q1,q2,…,q} T} represents a multi-turn researcher query, where q t Let D represent the input query in round t. This query uses a technical description, including research objectives, methodological descriptions, experimental constraints, etc. N} represents a large-scale candidate dataset set, where d1, d2, ..., d N Let d represent N datasets, where N is greater than 10 million in this example. Each dataset d... i Each dataset has corresponding metadata and text descriptions. This invention aims to design a dataset recommender F that recommends the most suitable K (K << N) datasets to researchers using as few dialogues as possible, thus minimizing the number of query rounds T.

[0049] 2. Research Intent Perception Steps (Research Intent Perception Module)

[0050] like Figure 1 As shown, experimental-level input can be extremely complex. This step / module is used to convert the user's research needs expressed in natural language into a structured intent template for use by the subsequent retrieval module. Specifically, it includes:

[0051] Intent template construction sub-step (intent template construction unit): Based on the general information structure in the scientific research process, six types of intent elements are predefined, including: research topic, scientific research task, data modality, experimental setup / constraints, evaluation indicators and other specific needs.

[0052] Language model parsing sub-step (language model parsing unit): The large language model is invoked to parse the user input intent, extracting the aforementioned structured elements to form a standardized intent template I, and generating the parsing result for the current round.

[0053] For example, the research topic is identified as human cross-organ cell differentiation, and the research task is identified as single-cell fate trajectory inference, taking into account cell evolution over time. Cross-organ scope and human tissue context are interpreted as experimental constraints.

[0054] 3. Structured memory compression steps (structured memory compression module)

[0055] Given the lengthy and context-dependent nature of multi-turn dialogues in scientific research queries, this step / module aims to retain core information about historical query intents and system responses under limited memory constraints. It tracks all real-time dialogue states and historical records within the platform. Let Θ 1:T This represents the dialogue history up to round T.

[0056]

[0057] in, τ represents the user query processed by the research intent perception module in round t. t This represents the tool call and execution logs for round t. These tool logs help avoid redundant operations in subsequent dialogue rounds. t This represents the response of the method of this invention (ScienceDB.AI) in round t. The memory budget is limited to L. max A token (e.g., L) max =32K), where token represents the smallest text processing unit obtained by the word segmentation algorithm of the large language model, and therefore completely contains Θ. 1:T This is significant and challenging. Directly inputting extremely long dialogue records into an LLM can cause it to forget system prompts or the given candidate dataset, leading to misinterpretations of the response. This invention aims to address this issue. 1:t Compressed to structured memory In this memory, information from the previous t-1 rounds is retained.

[0058]

[0059] Where, Θ 1:1This indicates the first round of dialogue records, S t-1 This represents a summary of historical information up to round t-1. This indicates a summary of information based on historical information from the previous t-1 rounds and the current round's dialogue.

[0060] This invention uses explicit compression rather than implicit compression to maintain the structured intent template. This represents an agent based on a large language model, used to summarize historical dialogue logs into structured information. Then, the compressed structured memory... Used as the context for the final LLM response.

[0061] In one embodiment of the present invention, the structured memory compression step / module includes the following sub-steps:

[0062] (1) Multi-turn dialogue tracking sub-step (multi-turn dialogue tracking unit) is used to record all dialogue content between the user and the system and tool call logs.

[0063] (2) Memory compression sub-step (memory compression unit) is used to compress historical dialogues into structured memories and prioritize the retention of recently updated information.

[0064] (3) Semantic conflict detection sub-step (semantic conflict detection unit), used to generate clarifying questions when ambiguity is found.

[0065] 4. Trusted Dataset Retrieval and Recommendation Steps (Trusted Dataset Retrieval and Recommendation Module)

[0066] The steps / modules for trustworthy dataset retrieval and recommendation include:

[0067] 1) First-stage candidate set filtering step (first-stage candidate set filtering unit): The initial candidate set is screened by combining vector matching and structured attribute filtering.

[0068] Specifically, based on explicit user input (the user-input query) and constraints extracted by the language model, filtering is performed according to dimensions such as publication time, affiliated institution, and data type. The constraints extracted by the language model refer to the structured user query processed by the research intent awareness module. The database is then queried for the most relevant datasets to the user query using a combination of vector matching and structured filtering. Vector matching involves performing similarity matching between the user's explicit query vector and the dataset vector. Structured filtering compares the constraints extracted by the language model with the attributes of the datasets in the database, such as "publication time" and "affiliated institution."

[0069] 2) Second-stage re-ranking step (second-stage re-ranking unit): Using deep interaction models such as ColBERT, the preliminary candidate set is finely ranked to obtain the final candidate set.

[0070] Specifically, the method for refining the preliminary candidate set is as follows:

[0071] a) Input: Explicit user input and constraints extracted by the language model, as well as the initial candidate set for the first stage.

[0072] b) Obtain the similarity score between the preliminary candidate set and the query through a deep interaction model, and rank and filter based on the score to obtain the final candidate set.

[0073] 3) Final Recommendation Dataset Acquisition Step (Final Recommendation Dataset Acquisition Unit): Input the final candidate set, the context information compressed through multiple rounds, and the structured user intent information into the large language model agent to obtain the final recommendation dataset.

[0074] 4) Unique identifier appending step (unique identifier appending unit): Assign unique identifiers to the recommended dataset based on CSTR (Record of Referenced Research Tasks).

[0075] The unique identifier appending unit, based on the system prompt strategy, mandates that each recommendation result be accompanied by a unique CSTR identifier link. The "system prompt strategy" refers to using prompt words to force the large language model to output a CSTR representation, such as: "For each selected dataset, its CSTR identifier must be returned."

[0076] In one embodiment of the present invention, a user submits a data request through a natural language input box on the platform interface, for example: "Give me a dataset of biology data after 2022." This input, as the original query statement, is collected by the system front-end and transmitted to the back-end processing module. The system's built-in large language model (such as the "large model-driven module" described in the patent) performs semantic parsing on the user's natural language query, extracts key information (such as "after 2022," "biology," and "dataset"), and performs semantic enhancement and standardization processing in conjunction with platform metadata standards (such as subject classification, publication time, and keyword tags) to generate a structured query intent. Based on the parsed query intent, the system recalls relevant dataset candidate sets from the large-scale data warehouse of ScienceDB. Through time filtering (publication date > 2022), subject classification matching (biology field), and keyword relevance scoring, a list of datasets that meet the conditions is selected (for example, multiple dataset entries can be displayed on the right). Finally, the system utilizes the domain understanding capability of the large model to select one or more optimal datasets as the recommendation results.

[0077] like Figure 3As shown, the proposed ScienceDB.AI outperforms the comparative models on all evaluation metrics, validating the effectiveness of the proposed framework. Compared to the best-performing comparative model, InteRecAgent, ScienceDB.AI achieves a performance improvement of over 20%. Although InteRecAgent includes additional modules (such as complex planning and reflection modules), it still falls short of the compact and efficient design of this invention. Based on the average number of turns (AT) metric, ScienceDB.AI requires the fewest turns to find the correct dataset. Compared to the best-performing comparative model in AT, Qwen, ScienceDB.AI achieves improvements of approximately 8% and 10% on AT@3 and AT@5, respectively.

[0078] Evaluation metrics description:

[0079] 1) Top-K Recall (Recall@K): This measures whether the top K candidate results contain the dataset that the user truly needs. A higher recall indicates that the system can cover more target results.

[0080] 2) Normalized Discounted Cumulative Gain (NDCG@K): This metric measures the ranking quality of recommendation results. It considers not only whether the target result is recalled, but also its position in the result list; the higher the ranking, the higher the score.

[0081] 3) Mean Reciprocal Rank (MRR@K): This measures the position of the first correct result in the recommendation list. The earlier the position, the higher the MRR.

[0082] 4) Average Turns (AT@K): This measures the average number of conversations required for the system to make a successful recommendation across multiple rounds of interaction. If a recommendation fails within t rounds, it is recorded as t+1. The lower the AT value, the more efficient the system.

[0083] The method and system described above can be deployed on a Web platform, supporting users to submit complex queries in multiple rounds online and automatically recommending traceable dataset resources.

[0084] It should be understood that the methods and systems disclosed in the above embodiments of this invention can be implemented in other ways. For example, the above module division can be implemented in other ways, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. The various modules in this invention can be implemented as software functional units and can be stored in a computer-readable storage medium, including several instructions to cause a computer device to execute some or all of the steps of the method described in this invention. For example:

[0085] An embodiment of the present invention provides a computer device (computer, server, smartphone, etc.) including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the steps of the method of the present invention.

[0086] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk, etc.) storing a computer program, which, when executed by a computer, implements the various steps of the method of the present invention.

[0087] Another embodiment of the present invention provides a computer program product, the computer program product including a computer program, which, when executed by a computer, implements the steps of the method of the present invention.

[0088] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and to implement it accordingly. Those skilled in the art will understand that various substitutions, changes, and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the content disclosed in the embodiments of this specification; the scope of protection of the present invention is defined by the claims.

Claims

1. A large-model-driven intelligent recommendation method for large-scale scientific data sharing platforms, characterized in that: Includes the following steps: The system parses the research questions expressed in natural language input by users and generates structured research intent information. The multi-turn dialogues during the scientific research inquiry process are compressed while retaining important contextual information; The candidate datasets are obtained by searching based on the structured research intent information, and the candidate datasets are filtered using the context information. The filtered datasets are then used as the recommended results.

2. The method according to claim 1, characterized in that, The process of parsing the research questions expressed in natural language input by the user to generate structured research intent information includes: Based on six predefined intent elements in the research process, including: research topic, research task, data modality, experimental setup / constraints, evaluation metrics, and other specific needs; The large language model is invoked to parse the user input intent, extract the six types of intent elements, form a standardized intent template, and generate the parsing result for the current round.

3. The method according to claim 1, characterized in that, The process of compressing multiple rounds of dialogue during scientific research inquiry while retaining important contextual information includes: Let Θ 1:T This indicates the dialogue history up to round T. in τ represents the user query after intent parsing of the user input in the t-th round. t This represents the tool call and execution log for round t, r t This represents the response in round t; Θ 1:t Compressed to structured memory In the middle, memory The information from the first t-1 rounds is retained: Where, Θ 1:1 This indicates the first round of dialogue records, S t-1 This represents a summary of historical information up to round t-1. This indicates a summary of information derived from combining historical information from the previous t-1 rounds and the current round's dialogue. This represents an intelligent agent based on a large language model, used to summarize historical dialogue logs into structured information.

4. The method according to claim 1, characterized in that, The process of retrieving candidate datasets based on the structured research intent information, filtering the candidate datasets using the context information, and using the filtered datasets as recommendation results includes: A combination of vector matching and structured attribute filtering was used to screen the initial candidate set; A deep interaction model is used to refine the initial candidate set to obtain the final candidate set; The final candidate set, compressed multi-turn context information, and structured user intent information are input into the large language model agent to obtain the final recommendation dataset; Assign a unique identifier to the recommended dataset.

5. The method according to claim 4, characterized in that, The method of screening the preliminary candidate set by combining vector matching and structured attribute filtering includes: based on explicit user input and constraints extracted by the language model, multi-dimensional screening is performed by combining vector matching and structured attribute filtering to obtain the preliminary candidate set; the multi-dimensional factors include publication time, affiliated organization, and data type.

6. The method according to claim 4, characterized in that, The process of refining the preliminary candidate set using a deep interaction model includes: inputting explicit user input and constraints extracted by a language model, as well as the preliminary candidate set; obtaining similarity scores between the preliminary candidate set and the query through the deep interaction model; and ranking and filtering based on the similarity scores.

7. The method according to claim 4, characterized in that, Assigning a unique identifier to the recommended dataset includes: forcibly attaching a unique CSTR identifier link to each recommendation result according to the system prompt strategy; the system prompt strategy refers to using prompt words to force the large language model to output a CSTR representation.

8. A large-model-driven intelligent recommendation system for large-scale scientific data sharing platforms, characterized in that: include: The research intent perception module is used to parse the research questions expressed in natural language by the user and generate structured research intent information. The structured memory compression module is used to compress multi-turn dialogues during scientific research queries while retaining important contextual information; The trusted dataset retrieval and recommendation module is used to retrieve candidate datasets based on the structured research intent information, and to filter the candidate datasets using the context information, and to use the filtered datasets as the recommendation results.

9. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer, implements the method according to any one of claims 1 to 7.

Images