System and method for idealization of research plan using large-scale language model agent oriented architecture

Abstract

To provide an idealization system and method of a research plan using a large-scale language model (LLM) agent oriented architecture.SOLUTION: There is provided a method for executing motivation validity confirmation and method synthesis for emulating an idealization process by using an LLM agent-oriented architecture having a colleague persona and an instructor persona, and a user is engaged to prepare a research planning report in a dialogue format. The research planning report comprises a set of solutions coping with motivation with the confirmed validity and a research problem based on a plurality of tasks implemented by an agent of the LLM agent-oriented architecture.EFFECT: Hallucination of LLM is reduced, response impossibility is coped with, and a result with a practical value is assured by using retrieval based on two-stage aspects. In a first stage of the two stages, a higher recall factor for reducing false negative and correcting false positive is introduced, and in a second stage, retrieval based on a more accurate and finely adjusted aspect is provided.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] (Cross - reference to related applications) This application claims priority to Indian Patent Application Publication No. 202321087441 (Patent Document 1) filed on December 20, 2023.

[0002] The present disclosure of this specification generally relates to the field of ideation of research plans, and more particularly, to systems and methods for ideation of research plans using large - language - model - agent - oriented architectures.

Background Art

[0003] Research is being conducted at a rapid pace in all fields, and scientific and research papers are exponentially increasing on numerous websites. It is difficult for individual researchers or small research groups to keep up with this information explosion and the ever - increasing movement in the fields of interest. Ensuring novelty at various stages of the research life cycle by continuously evaluating using literature, starting from ideation to experimental methods and analysis of results, has itself become a laborious task. This requires tools to assist researchers in accelerating the research life cycle by supplementing appropriate inputs at various stages. Several tools have been proposed to assist researchers during various stages of the research life cycle. However, these tools mainly focus on tasks such as searching for and recommending relevant literature, peer - reviewing and critiquing drafts, and writing research manuscripts. There is a significant lack of availability of tools specifically designed to assist researchers during the laborious ideation stage of the research life cycle.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Embodiments of the present disclosure present technical improvements as solutions to one or more of the above-described technical problems recognized by the inventors in the conventional system.

Means for Solving the Problems

[0006] For example, in one embodiment, a processor implementation method is provided. The processor implementation method includes receiving, via an input / output interface, a research proposal as an input from a user, the research proposal comprising text representing a research problem and a high-level description of the motivation behind the research problem; inputting, via one or more hardware processors, the research proposal as a query to a large language model (LLM) agent-oriented architecture, the LLM agent-oriented architecture comprising a first agent and a second agent that interact with each other and with the user, a first data repository, and a second data repository; using, via one or more hardware processors, the LLM agent-oriented architecture to enable the first agent to perform a first type of task and the second agent to perform a second type of task in response to the query; and obtaining, via one or more hardware processors, a revised research proposal together with a validated motivation and a set of valid solutions to the research problem, based on the first type of task performed by the first agent and the second type of task performed by the second agent, the validated motivation being repeatedly updated based on a plurality of gaps identified in a plurality of preceding research documents addressing the motivation behind the research problem.

[0007] In another aspect, a system is provided. The system includes a memory for storing instructions, one or more communication interfaces, and one or more hardware processors coupled to the one or more communication interfaces. The one or more hardware processors receive, as input from a user, a research proposal comprising text representing a research problem and a high-level description of the motivation behind the research problem. The one or more hardware processors input the research proposal as a query to a large language model (LLM) agent-oriented architecture comprising a first agent and a second agent that interact with each other, as well as with the user, a first data repository, and a second data repository. Using the LLM agent-oriented architecture, for the query, enable the first agent to perform a first type of task and the second agent to perform a second type of task, and based on the first type of task performed by the first agent and the second type of task performed by the second agent, obtain a revised research proposal with a validated motivation and a set of valid solutions to address the research problem. The validated motivation is repeatedly updated based on multiple gaps identified in multiple prior research documents addressing the motivation behind the research problem.

[0008] In yet another aspect, a non-transitory computer-readable medium is provided. The non-transitory computer-readable medium receives, as input from a user, a research proposal comprising text that represents, by instructions, a high-level description of a research problem and the motivation behind the research problem, and inputs the research proposal as a query to a large language model (LLM) agent-oriented architecture comprising a first agent and a second agent that interact with each other and with the user, a first data repository, and a second data repository. Using the LLM agent-oriented architecture, for the query, the first agent is enabled to perform a first type of task and the second agent is enabled to perform a second type of task, and based on the first type of task performed by the first agent and the second type of task performed by the second agent, a revised research proposal is obtained with a validated motivation and a set of valid solutions to address the research problem, where the validated motivation is repeatedly updated based on a plurality of gaps identified in a plurality of preceding research documents that address the motivation behind the research problem.

[0009] According to an embodiment of the present disclosure, the first type of task performed by the first agent comprises at least one of: (i) extracting relevant information from the research proposal; (ii) generating a plurality of relevant questions from the relevant information; (iii) retrieving, from the first repository, a plurality of top-K research documents having similarity to the research proposal using a vector representation of the research proposal; and (iv) obtaining, from the second repository, a plurality of paragraph chunks of each of the plurality of top-K research documents created using a syntax analysis program and an indexer of the LLM agent-oriented architecture.

[0010] According to an embodiment of the present disclosure, the second type of task performed by the second agent includes at least one of: (i) identifying a plurality of gaps in a plurality of preceding research documents that address the motivation behind the research problem; (ii) identifying a set of reasonable solutions to address the research problem; and (iii) rewriting the research plan based on the plurality of gaps identified in the plurality of preceding research documents and a set of reasonable solutions to address the research problem.

[0011] According to an embodiment of the present disclosure, a research problem defined in a research plan is decomposed into a plurality of sub-problems, a subset of the sub-problems is identified from the plurality of sub-problems so as to exclude a hallucinated set of problems, a plurality of top-K research documents having similarity with each of the subset of sub-problems are retrieved from a first repository based on the vector representation of the subset of sub-problems, and one or more relevant texts are extracted from each of the plurality of top-K research documents retrieved from the second repository, thereby determining a set of reasonable solutions for addressing each sub-problem obtained from the subset of sub-problems, repeatedly performing the step of retrieving a plurality of top-K research documents and identifying a set of reasonable solutions to generate an integrated list consisting of similar subsets of sub-problems and corresponding sets of reasonable solutions, and using the integrated list consisting of similar subsets of sub-problems and corresponding sets of reasonable solutions to identify a set of reasonable solutions for addressing the research problem, thereby identifying a set of reasonable solutions for addressing the research problem.

[0012] It should be understood that the foregoing general description and the following detailed description are exemplary only and are for the purpose of explanation only, and do not limit the invention as claimed. The accompanying drawings incorporated in and constituting a part of the present disclosure illustrate exemplary embodiments and, together with the specification, serve to explain the principles disclosed herein.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

[0014] Describe exemplary embodiments with reference to the accompanying drawings. In the figures, the leftmost digit or digits of a reference number identify the figure in which the reference number first appears. Whenever convenient, the same reference number is used throughout the drawings to refer to the same or similar parts. Although examples and features of the disclosed principles are described herein, modifications, adaptations, and other implementations are possible without departing from the scope of the disclosed embodiments. The following detailed description is considered to be exemplary only, and the true scope is intended to be indicated by the following embodiments described herein.

[0015] There is a need for a system specifically designed to assist researchers during the ideation phase of the research life cycle, which can be a challenging process. The ideation phase of the research life cycle involves: (i) analyzing existing literature to critically evaluate the motivation behind the research problem the researcher is addressing, ensuring the existence of one or more identified research gaps; (ii) re-formulating the research problem and objectives based on the output of the validation phase and the re-identification of research gaps; (iii) identifying similar research problems or sub-problems addressed in the literature and deriving a set of methods using the solutions to the similar research problems or sub-problems available in the literature, or synthesizing a set of appropriate methods as solutions to the research problem; and (iv) designing an experimental strategy for a given research problem and selected methodology.

[0016] Embodiments of the present disclosure provide systems and methods for ideation of research plans using a large language model agent-oriented architecture that helps accelerate the research life cycle. By leveraging the inferences of agents based on large language models (LLMs) and domain-specific skills, the systems of the present disclosure automate research activities, thereby reducing the burden on researchers. The methods of the present disclosure assist researchers by providing relevant inputs interactively at various stages. This helps speed up the process of achieving research objectives. The systems of the present disclosure facilitate the ideation of research problems specified by researchers. The researcher needs to provide a short paragraph along with the title of the research problem and a description of the motivation to solve the problem. Using agents based on LLMs, the systems of the present disclosure help the researcher interactively build a research proposal consisting of a validated motivation, a clearly defined research problem focusing on research gaps in the literature, a selected solution from a set of appropriate synthesized methods, and a set of possible experiments to conduct to evaluate the selected solution for the research problem. Embodiments of the present disclosure use LLM agents to simulate the ideation process of the research life cycle that researchers follow.

[0017] In the present disclosure, a system designed particularly to assist in the ideation process accelerates different phases of the research life cycle. The system of the present disclosure guides researchers through the formulation of a comprehensive research plan that encompasses research questions. By identifying gaps in existing literature and suggesting a list of appropriate techniques to address the research questions, the validity of the motivation in the research plan is confirmed. In the present disclosure, an agent-oriented architecture that incorporates peer personas and mentor personas for the large language model (LLM) is created by leveraging the inference and unique techniques of the LLM. The LLM agent emulates the ideation process that researchers perform and helps the researchers engage interactively to develop a research plan. The present disclosure addresses challenges specific to the LLM, such as hallucinations, implements aspect-based retrieval based on a two-stage modality to manage the precision-recall trade-off, and addresses the issue of unanswerability. To introduce the ideation capabilities of the system of the present disclosure, the execution of the workflow of motivation verification and method synthesis for research plans is demonstrated. The system receives research plans provided as input by three separate researchers from distinct fields such as computer science, materials science, and life science, from the fields of machine learning and natural language processing. The observations and evaluations provided by the researchers assist the researchers in appropriately inputting at separate stages, resulting in an illustration of the effectiveness of the system of the present disclosure in terms of improved time efficiency.

[0018] Next, with reference to the drawings, and more particularly to FIGS. 1-6 in which like reference characters consistently denote corresponding features throughout the figures, preferred embodiments are shown and described in connection with the following representative systems and / or methods.

[0019] Figure 1 illustrates a representative system for the ideation of a research plan using a large language model agent-oriented architecture according to some embodiments of the present disclosure. In one embodiment, system 100 includes, or is otherwise in communication with, one or more hardware processors 104, one or more communication interface devices or one or more input / output (I / O) interfaces 106, and one or more data storage devices or memories 102 operably coupled to the one or more hardware processors 104. The one or more hardware processors 104, memory 102, and one or more I / O interfaces 106 may be coupled to a system bus 108 or similar mechanism.

[0020] The one or more I / O interfaces 106 may include various software and hardware interfaces, such as a web interface, a graphical user interface, etc. The one or more I / O interfaces 106 may include interfaces for various software and hardware interfaces, such as interfaces for one or more peripheral devices such as a keyboard, a mouse, an external memory, a plurality of sensor devices, a printer, etc. Further, the one or more I / O interfaces 106 may enable the system 100 to communicate with other devices such as a web server and an external database.

[0021] One or more I / O interfaces 106 can facilitate multiple communications within a wide range of network and protocol types, including wired networks such as local area networks (LANs), cables, and wireless networks such as Wireless LAN (WLAN), cellular phones, or satellites. For this purpose, one or more I / O interfaces 106 may include one or more ports for connecting several computing systems to each other or to other server computers. Further, one or more I / O interfaces 106 may include one or more ports for connecting several devices to each other or to other servers.

[0022] One or more hardware processors 104 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that operates signals based on operational instructions. One or more hardware processors 104 are configured to retrieve and execute computer-readable instructions stored in memory 102, among other capabilities. In the context of the present disclosure, the expressions processor and hardware processor may be used interchangeably. In some embodiments, system 100 can be implemented in the form of various computing systems such as laptop computers, portable computers, notebooks, handheld devices, workstations, mainframe computers, servers, network clouds, and the like.

[0023] Memory 102 may include any computer-readable medium known in the art, such as volatile memory such as static random access memory (SRAM) and dynamic random access memory (DRAM), and / or non-volatile memory such as read only memory (ROM), erasable programmable ROM, flash memory, hard disk, optical disk, and magnetic tape. In certain embodiments, memory 102 includes a plurality of modules 102a and a repository 102b for storing data processed, received, and generated by one or more of the plurality of modules 102a. Repository 102b further comprises a first repository and a second repository. The plurality of modules 102a may include routines, programs, objects, components, and data structures that perform a particular task or implement a particular abstract data type.

[0024] The plurality of modules 102a may include programs or computer-readable instructions or encoded instructions that supplement the applications or functions performed by system 100. The plurality of modules 102a may also be used as one or more signal processors, one or more state machines, logic circuits, and / or any other device or component that operates on signals based on operational instructions. Further, the plurality of modules 102a may be used by hardware, by computer-readable instructions executed by one or more hardware processors 104, or by a combination thereof. Further, memory 102 may include information related to one or more processors of system 100 and one or more inputs / one or more outputs of each step performed by the methods of the present disclosure.

[0025] Repository 102b may include a database or a data engine. Further, repository 102b may, among other things, serve as a database or include multiple databases for storing data processed, received, or generated as a result of the execution of multiple modules 102a. Although repository 102b is shown inside system 100, it is noted that in an alternative embodiment, repository 102b can also be implemented outside system 100, in which case repository 102b may be stored inside an external database (not shown in FIG. 1) communicatively coupled to system 100. The data included inside such an external database may be updated periodically. For example, new data may be added to the external database and / or existing data may be modified and / or data that is not useful may be deleted from the external database. In one example, data may be stored in an external system such as a Lightweight Directory Access Protocol (LDAP) directory and a Relational Database Management System (RDBMS). In another example, the data stored in repository 102b may be distributed between system 100 and an external database.

[0026] Embodiments of the present disclosure provide systems and methods for ideation of research plans using a large language model agent-oriented architecture. FIG. 2 illustrates a representative flowchart exemplifying a method for ideation of a research plan using a large language model agent-oriented architecture using the system of FIG. 1 according to some embodiments of the present disclosure.

[0027] Referring to FIG. 1, in one embodiment, one or more systems 100 include one or more data storage devices or memories 102 operably coupled to one or more hardware processors 104, and the one or more processors 104 are configured to store instructions for performing steps of a method. Next, with reference to the components of the system 100 of FIG. 1, the flow diagrams as depicted in FIG. 2, the high-level diagrams of FIG. 3, and one or more examples, the steps of the method 200 of the present disclosure will be described. The steps of the method 200, including processing steps, method steps, techniques, etc., may be described in a sequential order, but such processing, methods, and techniques may be configured to operate in an alternative order. In other words, any sequence or order of steps that may be described does not necessarily indicate a requirement to perform the steps in that order. The processing steps described herein may be performed in any practical order. Further, some steps may be performed simultaneously, or some steps may be performed individually or independently.

[0028] In one embodiment, at step 202 of the present disclosure, one or more input / output (I / O) interfaces are configured to receive a research proposal as an input from a user. The research proposal comprises text representing a research topic and a high-level description of the motivation behind the research topic. The user may include, but is not limited to, researchers associated with the ideation of the research plan, any other person. Further, input may also be received from an external system. FIG. 3 shows a web-based input / output (I / O) interface of the system of FIG. 1 for the ideation of a research plan using a large language model agent-oriented architecture, according to some embodiments of the present disclosure. In the context of the present disclosure, a system for the ideation of a research plan using a large language model agent-oriented architecture helps to accelerate the research life cycle.

[0029] Furthermore, in step 204 of the present disclosure, one or more hardware processors 104 are configured to input a research proposal as a query to a large language model (LLM) agent-oriented architecture. The LLM agent-oriented architecture includes a first agent and a second agent that interact with each other as well as with a user, a first data repository, and a second data repository. FIG. 4 illustrates a high-level functional structure overview of the system of FIG. 1 according to some embodiments of the present disclosure. As shown in FIG. 4, a web-based interface is provided for a user, such as a researcher, to interact with. In the LLM agent-oriented architecture, two separate types of profile or persona agents are provided. The first agent represents a "colleague" persona and performs a first type of task (e.g., also referred to as the first type of task or a task that is not substantially complex). The second agent represents a "mentor" persona and performs a second type of task (e.g., also referred to as the second type of task or a task that is substantially complex or more complex). The LLM agent-oriented architecture is flexible such that LLM agents, including the first agent and the second agent, can interact with an LLM using API calls or (ii) an open-source LLM resident on an internal hosting server. As shown in FIG. 4, the first repository is a global repository that is a vector store of region-specific scientific papers indexed by specter embedding created using the research proposal. The second repository is a user-specific language material comprising all paragraph chunks of the retrieved papers related to the current research proposal the researcher is working on. A syntax analysis program that treats paragraphs as semantic segments is used to create paragraph chunks of the retrieved papers. During chunking, if a paragraph does not fit within the maximum token length of the LLM agent, the paragraph is further split to fit within the maximum token length. The paragraph chunks of the retrieved papers are further converted to vector embeddings and indexed for efficient retrieval based on semantic similarity to the query. This user language material serves as a shared "memory" for the LLM agents.

[0030] In step 206 of the present disclosure, one or more hardware processors 104 are configured to use an LLM agent-oriented architecture to enable a first agent to perform a first type of task and a second type of agent to perform a second type of task in response to a query. The first type of task performed by the first agent includes at least one of: (i) extracting relevant information from a research proposal; (ii) generating a plurality of relevant questions from the relevant information; (iii) retrieving, from a first repository, a plurality of top-K research documents having similarity to the research proposal using a vector representation of the research proposal; and (iv) obtaining, from a second repository, a plurality of paragraph chunks of each of a plurality of top-K research documents created using the functionality of a syntax analysis program and an indexer of the LLM agent-oriented architecture. The second type of task performed by the second agent includes at least one of: (i) identifying a plurality of gaps in a plurality of prior research documents addressing the motivation behind the background of the research problem; (ii) identifying a set of reasonable solutions to address the research problem; and (iii) rewriting the research proposal based on the plurality of gaps identified in the plurality of prior research documents and the set of reasonable solutions to address the research problem.

[0031] In one embodiment, in step 208 of the present disclosure, one or more hardware processors 104 are configured to obtain a revised research plan along with a justified motivation for validation and a set of sound solutions to address the research problem, based on a first type of task performed by a first agent and a second type of task performed by a second agent. The justified motivation for validation is repeatedly updated based on a plurality of gaps identified in a plurality of prior research documents that address the motivation underlying the research problem. In one embodiment, a set of sound decision-making for addressing the research problem is identified by first decomposing the research problem defined in the research plan into a plurality of sub-problems and identifying a subset of the sub-problems from the plurality of sub-problems to eliminate a set of problems that are hallucinated. Further, based on the vector representation of the subset of the sub-problems, a plurality of top-K research documents having similarity with each of the subset of the sub-problems are retrieved from a first repository. Moreover, by storing and extracting one or more relevant texts from each of the plurality of top-K research documents retrieved from a second repository, a set of sound solutions for addressing each sub-problem obtained from the subset of the sub-problems is determined. The step of retrieving a plurality of top-K research documents and identifying a set of sound solutions is repeatedly performed to generate an integrated list consisting of similar subsets of sub-problems and corresponding sets of sound solutions. Further, an integrated list consisting of similar subsets of sub-problems and corresponding sets of sound solutions is used to identify a set of sound solutions for addressing the research problem.

[0032] In one embodiment, step 208 is better understood by the following description provided as a representative illustration.

[0033] The ideation process of a research plan involves a dialogue between the researcher and the first agent and the second agent of the LLM agent-oriented architecture, where the first agent and the second agent perform activities based on the feedback received by the researcher or another agent. The ideation process obtains, as input from the researcher, a research plan with a high-level specified research problem description, along with the motivation and the research problem. The output of the ideation process is (i) a motivated or updated research problem whose validity is confirmed by identifying gaps in a plurality of preceding works addressing the motivation, and (ii) a revised research plan with a set of valid solutions to the research problem. The ideation process is divided into two work streams, namely (i) the work stream of motivation validity confirmation and (ii) the work stream of method synthesis.

[0034] Figure 5 illustrates a representative flowchart exemplifying the workflow of a motivation validation task for the ideation of a research plan using a large language model agent-oriented architecture according to some embodiments of the present disclosure. As shown in Figure 5, in the first step, the researcher provides a title and abstract of a research proposal that details the motivation behind the research plan, and a high-level description of the research problem statement that the researcher wishes to solve. Further, the functionality of the retriever of the system 100 uses this title and abstract of the plan as a query to obtain a vector representation of this title and abstract of the plan. Using this vector representation, the top K papers similar to the content of the research proposal are retrieved from the first repository, which is a global language material of scientific papers. The top K papers are presented to the researcher along with a description of the relevance of each document to the research proposal. These papers can be edited by the researcher who can delete papers that are found to be irrelevant, or by adding relevant papers. The functionality of the syntax analysis program and indexer of the LLM agent-oriented architecture chunks the final set of papers into paragraphs (i.e., semantic segments). The paragraph chunks of the final set of papers are stored in a second repository, which is a user language material with an appropriate indexing mechanism. The first agent (also interchangeably referred to as a peer agent throughout this specification) extracts the title and abstract of the research proposal provided by the researcher. The first agent extracts the motivation from the research proposal (prompt 1) and generates a list of questions to pose to the scientific papers on the final candidate list to confirm the validity of the motivation of the plan (prompt 2). Tables 1 and 2 below show the prompts for motivation extraction and motivation question generation, respectively.

[0035]

Table 1

[0036]

Table 2

[0037] The generated questions are binary-choice. For scientific papers, when the answer to a question is "yes", the question is formulated such that the paper already addresses the motivation of the research proposal it mentions. For example, if a researcher proposes to develop a technique to solve a novel aspect of a problem, the generated question would be in the form of "Does this research paper address that specific aspect of the problem?" If the scientific paper answers "yes" to this question, it means the paper addresses that aspect of the problem, and thus the motivation in the research background is weak or invalid. Show the researcher the set of generated questions. The system allows the researcher to edit the generated questions by updating the question format, deleting questions that the researcher deems irrelevant, or adding missing relevant questions. For each question and the corresponding retrieved papers stored in the user language materials, the peer agent searches for multiple paragraph chunks of papers related to that question and attempts to answer the question using retrieval augmented generation (RAG). Table 3 below shows a prompt for asking questions and answering them using retrieval augmented generation (RAG).

[0038]

Table 3

[0039] As shown in Table 3, the responses can be "Yes" with an explanation, "No", or potentially unable to answer. If all papers answer "No" or "Unable to answer" to all questions generated to confirm the validity of the motivation, this indicates that the existing literature does not address the motivation underlying the research plan, and thus, this phase ends with the view that the researcher has confirmed the validity of the plan's motivation. Otherwise, show the researcher question-research paper pairs with only "Yes" as the answer along with an explanation. The system enables the researcher to edit this output in that it removes research papers for which the researcher does not agree, based on the provided explanation, that the question is being addressed. If the researcher feels hallucinations in the answers, this step enables the researcher to remove such research papers. (Throughout this specification, interchangeably referred to as the instructor agent) The second agent uses each paper listed in the final candidate list, the initial research plan, and the paragraph chunks of the preceding questions addressing the motivation. The instructor agent extracts the limitations or gaps of each of the papers listed in the final candidate list identified as addressing the motivation of the research plan such that the gaps may help redefine the problem in the research plan. Table 4 below shows a prompt for extracting the gaps of the papers listed in the final candidate list.

[0040]

Table 4

[0041] Show the extracted gaps to the researcher, and the researcher can ignore the gaps found to be irrelevant and select some of the gaps that address part of the research problem. If the researcher does not agree with any of the specified research gaps, the system enables the researcher to add a set of the researcher's own research gaps. The instructor agent uses these gaps along with the initial research plan to re-formulate the motivation of the research plan and the statement of the research problem to address the new research gaps. Table 5 below shows a prompt for re-formulating the motivation and the statement of the research problem of the research plan to obtain the first revised research plan.

[0042]

Table 5

[0043] The system enables researchers to edit the initial revision plan for finalization or reject the edit and return to the previous research plan. The workflow of the motivation validation task is repeatedly applied to the research plan. This means that by starting the same workflow, the validity of the motivation behind the resulting updated research plan can be reconfirmed. This can be repeatedly executed until there are no scientific papers retrieved as dealing with the motivation behind the plan, and the validity of the novelty of the research plan is confirmed.

[0044] Figure 6 illustrates a representative flowchart exemplifying the workflow of method synthesis for the ideation of a research plan using a large language model agent-oriented architecture according to some embodiments of the present disclosure. As shown in Figure 6, the workflow of method synthesis starts from the initially revised research plan that has been confirmed for validity of motivation based on a literature review and accepted by the researcher. The peer agent obtains the initially revised research plan as input. The peer agent extracts the research topics defined in the initially revised research plan. The following Table 6 shows the prompts for extracting the research topic statements.

[0045]

Table 6

[0046] The supervisor agent receives this research topic as input and uses the parametric knowledge of the research topic to generate a set of appropriate similar research topics. The following Table 7 shows the prompts for generating a set of appropriate similar research topics.

[0047]

Table 7

[0048] For example, when the task defined in the research plan is "to design an evaluation metric without using criteria for a question - answering task", a similar task could be "to come up with an evaluation metric for text summarization that may have multiple possible criteria - forming summaries". The instructor agent also uses its parametric knowledge to break down the research task defined in the research plan into subtasks or (if any) sub - sub - tasks. Table 8 below shows the prompts for generating sub - tasks.

[0049]

Table 8

[0050] For example, the research task of "question - answering about scientific papers" can be broken down into "text extraction from the PDF document of the scientific paper", "segmenting the paper and memorizing it for efficient retrieval", "paragraph search from the paper relevant to the question", "using the retrieved paragraph as context to answer the question", "evaluating the retrieved paragraph", and "evaluating the answer". Show the generated similar tasks and subtasks to the researcher. The system enables the researcher to edit the subtasks by removing the subtasks found to be irrelevant, updating the subtasks, or adding the missing subtasks. This helps the instructor agent to remove the tasks it hallucinated. Use each of the edited similar tasks or subtasks as a query to search for scientific papers in the large - language materials that address these corresponding subtasks. Parse the retrieved scientific papers, chunk them into paragraphs (i.e., relevant text), and further memorize the paragraphs in the user - language materials.

[0051]

Table 9

[0052] As shown in Table 9, for each retrieved paper, the peer agent extracts relevant texts or paragraphs discussing the methods or techniques adopted by the retrieved paper to solve the corresponding sub - tasks, and generates an integrated list consisting of pairs of similar tasks or sub - tasks and their solutions. The system introduces this integrated list to the researchers with the authority to edit the integrated list.

[0053]

Table 10

[0054] As shown in Tables 9 and 10, together with the integrated list consisting of pairs of similar tasks or sub - tasks and the corresponding solutions of similar tasks or sub - tasks in the literature, the extracted tasks of the research proposal are provided to the supervisor agent. The supervisor agent uses this information together with its parametric knowledge to synthesize a list of reasonable methods for solving the tasks defined in the research proposal. Table 11 below shows the prompts for method synthesis.

[0055]

Table 11

[0056] Show the list of reasonable methods to the researcher. The system enables the researcher to select a subset of these methods that the researcher deems to be the most reasonable, and these methods can be further edited if necessary. Provide the updated list to the supervisor agent together with the original research proposal. The supervisor agent rewrites the research proposal including these methods. This updated research proposal is the final revised research proposal shown to the researcher. The system enables the researcher to further edit the updated research proposal and finalize the research proposal.

[0057] In one embodiment, hallucinations are one of the main difficulties in using an LLM for knowledge-based tasks. In the present disclosure, a dual solution is used to eliminate the problem of causing a set of hallucinations. That is, (i) there is a search expansion component of the workflow, where the workflow of motivation validation generates questions to validate the motivation of the research plan regarding the papers memorized and searched in the user language materials, or raises the extraction limit of the papers addressing the motivation of the research plan, or the workflow of method synthesis extracts the techniques used to solve similar problems or sub-problems obtained from the searched papers. In these search expansion tasks by skillful processing with appropriate prompts, it is ensured that the answer is provided by restricting knowledge to only the searched context. This has been recognized to help reduce hallucinations. (ii) There are components of the workflow that rely on the parametric knowledge of the LLM. For example, motivation validation involves rewriting the research plan, and method synthesis involves generating and synthesizing methods for similar sub-problems related to the research problems defined in the research plan. In these tasks, the output cannot be restricted to the provided input. In such cases, the output is more likely to cause hallucinations. In such scenarios, the system enables user interaction at every step so that the output that causes hallucinations can be edited or deleted, ensuring the reduction of the output that causes hallucinations. Moreover, at every stage of the workflow, the LLM agent is required to justify the output of the LLM agent, and the fact that the justification has been provided is announced to the researcher through the interface. This applies a chain-of-thought to the system, enabling the researcher to check the validity of the output and whether the provided hallucinations are synchronized with the output. This helps mitigate the impact of hallucinations.

[0058] In one embodiment, the present disclosure uses a two-stage manner-based search to ensure practically valuable results. The large language materials include a number of scientific papers that store the spectra embedding the titles and abstracts of the papers. The titles and abstracts of the papers include information regarding the motivation and problem statement of the papers as well as high-level descriptions of the methodology and results. In the ideation, more detailed information obtained from the papers is required across various aspects such as methodology, limitations, etc. This is achieved by performing the search in two stages. In the workflow of the motivation validity verification, the research proposal is used as a query to search for the top K papers from the large language materials, and a large value of K is used to obtain a good recall rate. This helps to obtain a set of papers with motivations and problem statements similar to those of the motivation and problem statement of the research proposal. These research papers are chunked and stored in the user language materials for further manner-based search, such as papers with motivations similar to the motivation of the research proposal and paragraphs of papers that mention the research gaps of these papers. In the workflow of the method synthesis, the first top K papers obtained from the large language materials are searched using a similar sub-problem statement as a query, and a large value of K is used to obtain a good recall rate. This helps to obtain a set of papers with problems similar to the problems described in the research proposal, or problems similar to either the subtasks or sub-problems of the research problems described in the research proposal. These papers are chunked and stored in the user language materials for further manner-based search, such as the extraction of the methods of the papers. By using a high recall rate value, the application scope of the papers is ensured in the first stage of the search, and more accurate results are ensured in the manner-based search in the second stage.

[0059] In one embodiment, the present disclosure addresses the problem of unanswerability. The output of the aspect-based search is always the top K paragraphs obtained from the searched and chunked papers. The id value of K was kept low to obtain a more accurate search for a given aspect-based query. However, there is a possibility that the searched paragraphs do not have an answer to the query (i.e., the query is unanswerable). For example, in the workflow of motivation validity verification, the paragraphs searched from the papers do not answer the question of whether the paper addresses the specific motivation of the research proposal, and do not specify the limitations of the paper that are useful for refining the research questions defined in the research proposal. Similarly, in the workflow of method synthesis, the searched paragraphs may not have a method for solving similar problems. In such cases, the LLM-based agent checks the relevance of the paragraphs searched for a given query, and if all of the searched paragraphs are irrelevant, identifies the query as "unanswerable" and avoids irrelevant outputs. Furthermore, allowing unanswerability helps to reduce hallucinations. Qualitative analysis of the workflow

[0060] In the present disclosure, a qualitative analysis of the workflow is provided using a number of (e.g., three) research proposals received from separate researchers, specifically in the fields of Artificial Intelligence (AI), Machine Learning (ML), and Natural Language Processing (NLP). The topics of these research proposals are (i) datasets for peer-reviewed computer-aided research, (ii) citation searches based on topics for research planning, and (iii) an evaluation metric without a benchmark for retrieval-augmented question answering. These researchers have been engaged in separate research topics and aim to write research proposals regarding separate research topics. In the present disclosure, semantic academic data retrieved using the well-known Semantic Scholar Open Research Corpus (S2ORC), a large-scale repository with various papers in the fields of AI, ML, and NLP, is used. By leveraging the information-gathering functionality of the system of the present disclosure, the movement of the conversation between the researcher and the LLM agent was tracked, and the observations described in the following paragraphs were obtained.

[0061] The summary of the first research proposal titled "Dataset for Computer-Aided Research on Peer Review" is as follows: "Peer review constitutes a core component of academic publishing, requires significant expertise and training, and is vulnerable to errors and biases. Various applications of natural language processing (NLP) to assist peer review attempt to support reviewers in this complex process, but the lack of clearly identified datasets and multi-domain language materials has hindered systematic research on NLP for peer review. To address this, system 100 and method are configured to introduce ethically sourced multi-domain language materials of papers and review reports obtained from five different arguments." Using the theme and summary of the first research proposal as input, the workflow of the motivation validation task generates the question "Does this research paper address the lack of a clearly identified dataset for researching natural language processing for peer review?" As part of this workflow, to confirm the validity of the motivation of the first research proposal, a peer (LLM) agent posed this question to the top 50 paragraph chunks of scientific papers retrieved from the user's language materials and similar to the theme and summary of the research proposal. From these 50 scientific papers, 5 papers with an answer of "yes" were retrieved. Among them, the researcher agreed with the justifications provided for 4 and disagreed with the justification for 1. The following are the 4 research papers with valid justifications. (i) NL Equivalent (Peer): Integrated Resources for Computer-Aided Research on Peer Review (ii) Peer Review Dataset (PeerRead): Collection, Insights, and NLP Applications (iii) Scrutiny of Fairness Gender Differences in Peer Review: Language Model Enhancement Techniques (iv) MOPRD: Interdisciplinary Open Peer Review Dataset The paper titled "What can be done to improve peer review with NLP?" agrees with the motivation of the first research proposal, but nevertheless does not address the motivation of the first research proposal. As the next part of the workflow for motivation validation, the peer agent extracts the following research gaps from the four identified research papers. (i) NL Equivalent: Integrated resources for computer-based research in peer review, the identified research gaps are as follows. (a) This paper does not include blind review data, which is a standard practice in most research fields. (b) This paper does not perform extensive hyperparameter exploration and model tuning, which may limit the effectiveness of the model. (c) This paper acknowledges the risk of "casual reading" where reviewers may only read the paragraphs suggested by the model and does not provide a solution to prevent this. (ii) Peer Review Dataset (PeerRead): Collection, Insights, and NLP Applications, the identified research gaps are as follows. (a) The models used in the research are relatively simple, which may limit the effectiveness of the models in complex peer review scenarios. (b) This paper leaves room for further research in areas such as demographic bias in pass / fail decisions. (c) The researchers do not provide multi-domain language materials for the papers and do not review the reports from different arguments, limiting the scope of this application. (iii) Scrutiny of Fairness Gender Differences in Peer Review: Language Model Enhancement Techniques, the identified research gaps are as follows. (a) This research paper does not draw any causal conclusions from the fairness analysis, limiting the depth of understanding of the mechanisms underlying bias in peer review. (b) This research paper does not scrutinize the effect of rebuttals in the peer review process, which may be an important factor in the final decision made on the paper. (c) This research paper does not provide a thorough analysis of continuous learning in Pretrained Language Models (PLMs), which may be very important for improving automatic peer review generation. (iv) MOPRD: Interdisciplinary Open Peer Review Dataset, the identified research gaps are as follows. (a) This research paper recognizes that the interdisciplinary bias of this method may be a concern and indicates that a more balanced dataset across various academic fields is needed. (b) This paper points out that input limitations still exist when the maximum length of the input text reaches only 16,384 tokens, indicating that a model capable of handling larger input sizes is needed. (c) This paper mentions that as the input length increases, the performance of the attention mechanism deteriorates, indicating that a more efficient attention mechanism or an alternative model for handling long input sequences is needed. This introduces the quality of the output provided by the motivation workflow in terms of identifying research gaps in existing papers, finding gaps without partially eliminating the need for a detailed literature review of these papers, and thus reducing research effort.

[0062] In this disclosure, it is further recognized that a researcher selects a subset of identified research gaps that are known to be relevant. The instructor agent further uses these selected gaps for the first research proposal. The updated research proposal provided as a result of the motivation workflow considering the selected research gaps is as follows. "This document presents several gaps that motivate the need for a more comprehensive approach to applying natural language processing (NLP) in peer review. The lack of blinded data, which is a standard practice in most research fields, is a major limitation. The lack of multi-domain language materials and review reports from different sources in the papers limits the scope of this application. The unexpected effect of counterarguments in peer review, which is the difficulty of studying the review process compared to the decision-making process, indicates that more sophisticated methods or tools are needed. The interdisciplinary bias of methods, the input limitations of language models, and the degradation of the attention mechanism's performance as the input length increases all suggest the need for an improved NLP model for peer review, a more balanced dataset across different academic fields, and a model that can handle larger input sizes and long input sequences. To address these gaps, it is proposed to introduce ethically sourced multi-domain language materials of papers and review reports obtained from different sources." From the above research proposal, the instructor agent takes into account the following selected research gaps and introduces the following selected research gaps in the updated version of the research proposal. (i) Availability of blinded language materials across many academic fields and many sources, (ii) Lack of research on the impact of counterarguments, (iii) Contextual limitations of language models (LMs) that handle long reviews of existing works in this area. The researcher was found to make little editing to the summary of the updated research proposal and accept the summary of the updated research proposal. The total time required to confirm the validity of the motivation and update the summary of the research proposal accordingly was substantially reduced (to about one-fifth in the above research proposal) by this motivation workflow.

[0063] In one embodiment, a second research proposal titled "Citation Search Based on the Theme for the Research Plan" and a corresponding abstract "Searching for Research Papers Related to a Given Query represents thoroughly vetted research challenges. Typically, the query takes the form of the title and abstract of a research paper, or a specific sentence or paragraph obtained from an existing research paper that requires citation. However, existing methods assume the availability of well-structured manuscripts, which are inappropriate assumptions during the stage of writing the initial research plan. During this first phase, researchers often search for relevant literature to cite in their plans, focusing on specific topics or intentions, and further construct the plan. In this work, we attempt to address the problem of citation search based on the theme for the research plan. The researcher is expected to provide the title and abstract of their research plan, covering elements such as research gaps, problem statements, and high-level overviews of the proposed methodology and experiments. Additionally, the researcher provides a list of topics related to the scientific papers that need to be searched. The proposed algorithm aims not only to retrieve research papers related to a given plan from the language materials but also to establish a very important many-to-many correspondence between these papers and the specified topics." is received from another researcher. The peer agent generates the following questions for motivation validation. 1. Does this research paper specifically address the search for research papers related to the theme of the research plan? And 2. Does this research paper develop techniques for matching research papers to the themes specified in the research plan?

[0064] By raising the above questions, among the top 50 research papers used to validate the motivation of the second research proposal, the following four research papers were retrieved as answering "yes" to at least one of the above questions, thus invalidating the motivation behind the second research proposal. 1. Citation Recommendation: Methods and Datasets 2. CitationIE: Leveraging Citation Graphs for Scientific Information Extraction 3. Content-Based Citation Recommendation, and 4. unarXive 2022: All arXiv publications preprocessed for NLP, including structured full texts and citation networks.

[0065] However, the justifications provided for these research papers emphasize that the first and third research papers introduce methods for citation recommendations during the phase of writing the target manuscript rather than during the writing of the research plan. Also, the second research paper utilizes the content of the target paper and the citation graph for extracting scientific information. The results of the fourth scientific paper may be useful for the second research plan but are datasets that do not address the task of "topic-based citation search for the research plan". Thus, after evaluating the retrieved research papers, which claim to invalidate the second research plan, it is recognized that researchers may object to the justifications provided for each retrieved paper to address the motivation behind the second research plan. Therefore, the novelty of the second research plan has been confirmed for validity. This illustrates not only the necessity but also the effectiveness of the user interface functions provided by the system for the workflow of motivation. Generally, researchers manually scrutinize a large number of relevant research papers retrieved by general research engines or academic research engines to ensure that the literature does not have solutions to the specific problems the researchers are trying to address. This leads to a time-consuming process. This example illustrates accelerating the motivation validity confirmation stage of the research lifecycle (by ~8 times in the second research plan) by eliminating the need for researchers to manually scrutinize a large number of relevant research papers retrieved by general research engines or academic research engines.

[0066] In one embodiment, an input with the summary "In a third research proposal titled 'Evaluation Metrics without Using Criteria for Retrieval-Augmented Question Answering Tasks', it is recognized that questions with long answers for long documents do not have specific reference evidence (relevant paragraphs obtained from the documents) and answers. Rather, there is a distribution regarding the reference answers, making the expert-based evaluation costly and inappropriate for existing evaluation metrics based on specific criteria. Also, no evaluation metrics without using criteria designed for evaluating retrieval-enhanced question answering tasks have been obtained. Therefore, this work proposes to define this metric" was received from another researcher. The peer agent generated the following questions to confirm the validity of the motivation of the third research proposal. Does this research paper propose an evaluation metric without using criteria designed for evaluating retrieval-enhanced question answering tasks? Among the top 50 scientific papers retrieved related to the third research proposal, none of the scientific papers provide an answer of "yes" to the question generated above, and no research papers invalidating the motivation of the third research plan are retrieved at all. The result is demonstrated not only by the manual analysis of the top 50 retrieved research papers but also by the manual analysis of other related research papers conducted by the researcher to evaluate the results of the motivation process.

[0067] In the process flow of the method synthesis related to the third research proposal, the leader agent generates the following set of research questions similar to the research questions defined in the third research proposal. 1. Evaluate complex tasks with no specific correct answer or criteria. 2. Design evaluation metrics for tasks involving retrieval and interpretation of large amounts of data. 3. Create evaluation metrics without using criteria for tasks where criteria-based metrics are inappropriate or not practical. 4. Appraise the quality of answers in tasks where the answers may be long and may be drawn from a wide range of documents. The instructor agent also generates the following subtasks or sub - problems regarding the research problems defined in the third research proposal. 1. Define new metrics that can effectively evaluate the retrieval - augmented question - answering task, and 2. Overcome the inadequacies of the existing proprietary - based evaluation metrics for questions with long answers for long documents. Using the above sub - problems as queries, since the colleague agent may search for the same research papers for a number of queries, with some duplication, the colleague agent first searches for the top 10 similar scientific papers per citation, which means a total of 40 papers. Further, the colleague agent poses the question "Does the paper provide a methodology or approach to solve the set of sub - problems or subtasks defined above?". The researcher receives a total of 17 scientific papers that answer "yes" to the question, along with a description of the methodology implemented for each set of sub - problems or subtasks defined above. From these scientific papers, the researcher accepts the following 11 scientific papers and finds that the following 11 scientific papers are most relevant to the research problem. 1. AVA: An Automatic Evaluation Method for Question - Answering Systems 2. Evaluation: From Precision, Recall, and F - measure to ROC, Informedness, Markedness, and Correlation 3. Rethinking Automatic Topic Model Evaluation Using Large - Scale Language Models 4. SacreROUGE: An Open - Source Library for Using and Developing Summary Evaluation Metrics 5. Quantifying the Reproducibility of NLP Results 6. Rethinking the Criteria of Value: Grounded in Robust Human Evaluation for Summary Evaluation 7. Extremely Important Evaluations for Long - Form Question - Answering 8. Do you consider direct answers to question - answering to be solved? ARC - DA attempts, direct answers to the AI2 Reasoning Challenge 9. Beyond Reading Comprehension: An Investigation into Datasets and Metrics for Text - Based Question - Answering 10. Obstacles Hindering the Improvement of Long - Form Question - Answering 11. Critical evaluation of evaluation for long-form question answering. Based on the approaches discussed in the above scientific paper to solve the sub-problems and / or sub-tasks listed above, the instructor agent suggests the following 10 valid methodologies for the third research proposal. These 10 valid methodologies suggested, with the exact word-for-word explanations provided by the instructor agent, are as follows. 1. AVA (Automatic Evaluation Approach): This approach uses a Transformer-based language model to encode questions, answers, and reference texts. By measuring the similarity between the reference and the automatic answer, which is biased towards question semantics, AVA can evaluate system accuracy. Using this approach, the search-enhanced question answering task and the inappropriateness of the results of the evaluation metrics based on specific criteria can be evaluated. 2. Evaluation criteria for information availability and markedness: These new probabilistic information-theoretic evaluation criteria can be used to evaluate complex tasks where there are no specific correct answers or references. Furthermore, these new probabilistic information-theoretic evaluation criteria can be used to design evaluation metrics for tasks involving the search and interpretation of large amounts of data. 3. Large Language Model (LLM): The LLM can be used as an automated metric for evaluating tasks involving the search and interpretation of large amounts of data. Furthermore, the LLM can be used to assess the quality of answers in tasks where the answers may be long and drawn from a wide range of documents. 4. SacreROUGE: Using this open-source library dedicated to the development of summarization evaluation metrics, an evaluation metric without using criteria can be developed for tasks where the criteria-based metrics are inappropriate or not practical. 5. Quantified Reproducibility Assessment (QRA): Based on the regulations obtained from the concepts and methodologies, this method can be used to develop an evaluation metric without using criteria for tasks where the criteria-based metrics are inappropriate or not practical. 6. RoSE Benchmark: This benchmark can be used to provide a more robust evaluation across different tasks. Additionally, this benchmark can be used to develop a criterion-free evaluation metric for tasks where criterion-based metrics are inappropriate or not practical. 7. GENIE6: This human-in-the-loop scoreboard framework can be used to score responses in tasks where the responses may be long and drawn from a wide range of documents. 8. Sparse Attention and Contrastive Retriever Learning: This system uses a high-density retriever trained by expanding a remote teacher algorithm conditioned on answer generation for the identified documents. This system can be used to overcome the inadequacies of existing proprietary criterion-based evaluation metrics for questions with long answers regarding long documents. 9. Integrated Evaluation Benchmark for Long-Form Answers: This approach involves conducting a thorough study of evaluation, including both human evaluation protocols and automated evaluation protocols. This system can be used to overcome the inadequacies of existing proprietary criterion-based evaluation metrics for questions with long answers regarding long documents. 10. Training of Long-Form Question Answering (LFQA) Evaluation Metrics Directly Based on Human-Annotated Preference Judgments: This approach involves fine-tuning a pre-trained language model based on human judgment scores for the task. This output describes the quality of the method recommendations provided by the system of the present disclosure for a given research proposal. While mentioned at a high level, the researchers agree that most of the methods described above are optimal as a sound approach for the third research proposal. While further work is needed to finalize the optimal and sound methods for the third research proposal, the researchers have found that this initial cut of the output provided by the system of the present disclosure has practical value and that the entire process is ~10 times more efficient than the normal process that researchers follow to construct a sound set of methods for a given research topic by searching the relevant literature from the start.

[0068] These examples, which illustrate the motivation validation and method synthesis phases of the workflow of the ideation of the system of the present disclosure, demonstrate the effectiveness of the system of the present disclosure in that they provide relevant output at each stage of the workflow. The observations made regarding the time saved by the researchers with respect to the system usage for each task demonstrate the ability of the system of the present disclosure with respect to improving time efficiency.

[0069] The written description describes the subject matter of this specification such that any person skilled in the art can make and use the embodiments. The scope of the embodiments of the subject matter is defined in this specification and may include other modifications that occur to those skilled in the art. Such other modifications are intended to fall within the scope of the present disclosure if they have similar elements that do not differ from the language of the embodiments, or if they include equivalent elements with slight differences from the language of the embodiments described in this specification.

[0070] Embodiments of the present disclosure provide a system developed to accelerate the ideation phase of the research life cycle. To emulate the ideation process, a large language model (LLM) agent-oriented architecture is used with colleague personas and mentor personas that perform motivation validation and method synthesis, thereby engaging the user in an interactive manner to build a research proposal. The present disclosure (i) reduces LLM hallucinations, (ii) uses a two-stage modality-based search where the first stage introduces a higher recall rate that reduces false negatives, false positives are corrected by user interaction, and the second stage provides a more accurate and fine-tuned modality-based search to ensure valuable results, and (iii) introduces unanswerability. The present disclosure demonstrates accurate results with a ~7.5x improvement in time efficiency at various stages of the ideation phase.

[0071] The scope of protection is extended to such a program and further to computer-readable means having a message therein, and it should be understood that such computer-readable storage means includes program code means for implementing one or more steps of the method when the program is executed on a server or a mobile device, or any suitable programmable device. The hardware device may be any type of device that can be programmed, including, for example, any type of computer such as a server or a personal computer, or any combination thereof. The device may also include means that may be a combination of hardware means such as, for example, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or a combination of hardware means and software means such as ASIC and FPGA, or at least one microprocessor and at least one memory in which software processing components are arranged. Thus, the means may include both hardware means and software means. The embodiments of the method described herein can be implemented in the form of hardware and software. The device may also include software means. Alternatively, for example, multiple CPUs may be used to implement the embodiments on different hardware devices.

[0072] Embodiments herein may comprise hardware elements and software elements. Embodiments implemented in software may include, but are not limited to, firmware, resident software, microcode, etc. The functions performed by the various components described herein may be implemented by other components or in combination with other components. As intended herein, a computer-usable medium or computer-readable medium may be any device that can store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0073] The steps illustrated are presented to describe representative embodiments shown, and it should be expected that ways of performing specific functions may change with ongoing technological development. These examples are presented herein for purposes of illustration and not of limitation. Further, for convenience of description, the boundaries of functional building blocks have been arbitrarily defined herein. Alternative boundaries can be defined as long as the specified functions and their relationships are properly performed. Alternative forms (including equivalent, extended, variant, deviated forms, etc. of the embodiments described herein) will be apparent to those of ordinary skill in one or more relevant technical fields based on the teachings included herein. Such alternative forms fall within the scope of the disclosed embodiments. Also, the terms "comprising," "having," "containing," and "including," and other similar forms are intended to be equivalent in meaning, and do not mean that the one or more items following any one of these terms are an exhaustive listing of such one or more items, nor are they meant to be limited to the one or more items listed. In this regard, they are intended to be in open-ended form. Also, when used herein, the singular forms "a," "an," and "the" are to be noted as including plural references unless the context clearly dictates otherwise.

[0074] In addition, when implementing embodiments consistent with the present disclosure, one or more computer-readable storage media may be utilized. A computer-readable storage media refers to any type of physical memory that may store information or data readable by a processor. Thus, a computer-readable storage media may store instructions for execution by one or more processors, including instructions for causing one or more processors to perform steps or stages not inconsistent with the embodiments described herein. It should be understood that the term "computer-readable media" includes tangible items and excludes carrier waves and transient signals, i.e., is non-transitory. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard drives, CD ROMs, DVDs, flash drives, disks, and any other known physical storage media.

[0075] The present disclosure and the examples are considered to be merely representative, and it is intended that the true scope of the disclosed embodiments be indicated herein by the following claims.

Claims

1. A processor implementation method (200), comprising: receiving (202) a research proposal as input from a user via an input / output interface, said research proposal comprising text expressing a high level description of a research problem and the motivation behind said research problem; inputting (204) the research protocol as a query to a large scale language model (LLM) agent-oriented architecture via one or more hardware processors, the LLM agent-oriented architecture comprising a first agent and a second agent that interact with each other, the user, a first data repository, and a second data repository; using the LLM agent-oriented architecture via the one or more hardware processors to enable the first agent to perform a first type of task and the second agent to perform a second type of task in response to the query (206); obtaining, via the one or more hardware processors, a revised research proposal along with validated motivations and a set of plausible solutions addressing the research problem based on the first type of task performed by the first agent and the second type of task performed by the second agent (208), wherein the validated motivations may be iteratively updated based on gaps identified in prior research documents addressing the motivations behind the research problem; A processor-implemented method (200) comprising:

2. The first type of task performed by the first agent includes: (i) extracting relevant information from the research protocol; (ii) generating a number of relevant questions from the relevant information; (iii) searching the first data repository for a number of top-K research documents having similarity to the research protocol using a vector representation of the research protocol; and (iv) obtaining from the second data repository a number of paragraph chunks for each of the top-K research documents created using parser and indexer functionality of the LLM agent-oriented architecture. The processor-implemented method (200) of claim 1, comprising at least one of:

3. 2. The processor-implemented method (200) of claim 1, wherein the second type of task performed by the second agent comprises at least one of: (i) identifying gaps in the plurality of prior research documents that address the motivation behind the research problem; (ii) identifying the set of plausible solutions that address the research problem; and (iii) rewriting the research proposal based on the gaps identified in a plurality of prior research documents and the set of plausible solutions that address the research problem.

4. The set of reasonable solutions that address the research problem includes: Decomposing the research topic defined in the research plan into a number of sub-topics; identifying a subset of sub-tasks from the plurality of sub-tasks such that a set of hallucinated tasks is excluded; retrieving, from the first data repository, a number of top-K research documents having similarity to each of the subset of sub-tasks based on the vector representation of the subset of sub-tasks; determining a set of plausible solutions for addressing each sub-problem from the subset of sub-problems by extracting one or more relevant texts from each of the plurality of top-K research documents stored and retrieved from the second data repository; performing iteratively searching the plurality of top-K research documents to identify the set of plausible solutions to generate a consolidated list of subsets of similar sub-problems and corresponding sets of plausible solutions; using the consolidated list of subsets of similar sub-problems and the corresponding set of plausible solutions to identify the set of plausible solutions that address the research problem. The processor-implemented method (200) of claim 1 , wherein the processor-implemented method (200) is identified by:

5. A system (100), comprising: A memory (102) for storing instructions; one or more communication interfaces (106); one or more hardware processors (104) coupled to the memory (102) via the one or more communication interfaces (106); wherein the one or more hardware processors (104) are configured to: receiving as input from a user a research proposal comprising text representing a high-level description of a research problem and the motivation behind said research problem; inputting said research proposal as a query into a large scale language model (LLM) agent oriented architecture comprising a first agent and a second agent that interact with each other, and with said user, a first data repository, and a second data repository; using the LLM agent oriented architecture to enable the first agent to perform a first type of task and the second type of agent to perform a second type of task in response to the query; obtaining an iteratively updated revised research plan based on the validated motivations and a set of plausible solutions that address the research problem, along with a plurality of gaps identified in the plurality of prior research documents that address the motivations behind the research problem, based on the first type of task performed by the first agent and the second type of task performed by the second agent; The system (100) is configured as follows.

6. 6. The system of claim 5, wherein the first type of task performed by the first agent comprises at least one of: (i) extracting relevant information from the research protocol; (ii) generating relevant questions from the relevant information; (iii) using a vector representation of the research protocol to search the first data repository for top-K research documents having similarity to the research protocol; and (iv) obtaining from the second data repository a number of paragraph chunks for each of the top-K research documents created using parser and indexer workflow functionality of the LLM agent-oriented architecture.

7. 6. The system (100) of claim 5, wherein the second type of task performed by the second agent comprises at least one of the steps of: (i) identifying gaps in the plurality of prior research documents that address the motivation behind the research problem; (ii) identifying the set of plausible solutions that address the research problem; and (iii) rewriting the research proposal based on the gaps identified in the plurality of prior research documents and the set of plausible solutions that address the research problem.

8. The set of reasonable solutions that address the research problem includes: Decomposing the research topic defined in the research plan into a number of sub-topics; identifying a subset of sub-tasks from the plurality of sub-tasks such that a set of hallucinated tasks is excluded; retrieving from the first data repository a number of top-K research documents having similarity to each of the subset of sub-tasks based on the vector representation of the subset of sub-tasks; determining a set of plausible solutions for addressing each sub-problem from the subset of sub-problems by extracting one or more relevant texts from each of the plurality of top-K research documents stored and retrieved from the second data repository; performing iteratively searching the plurality of top-K research documents to identify the set of plausible solutions to generate a consolidated list of subsets of similar sub-problems and corresponding sets of plausible solutions; using the consolidated list of subsets of similar sub-problems and the corresponding set of plausible solutions to identify the set of plausible solutions that address the research problem. The system (100) of claim 5, identified by:

9. One or more non-transitory machine-readable information storage media comprising one or more instructions that, when executed by one or more hardware processors, receiving as input from a user a research proposal comprising text expressing a high-level description of a research problem and the motivation behind said research problem; inputting said research proposal as a query into a large scale language model (LLM) agent oriented architecture comprising a first agent and a second agent interacting with each other, with said user, a first data repository, and a second data repository; using the LLM agent oriented architecture to enable the first agent to perform a first type of task and the second agent to perform a second type of task in response to the query; obtaining a revised research proposal along with validated motivations and solutions that address the research problem based on a first type of task performed by the first agent and a second type of task performed by the second agent, the validated motivations being iteratively updated based on identified gaps in research documents that address the motivations behind the research problem; One or more non-transitory machine-readable information storage media that produce the

10. 10. The one or more non-transitory machine-readable information storage media of claim 9, wherein the first type of task performed by the first agent comprises at least one of: (i) extracting relevant information from the research protocol; (ii) generating a plurality of relevant questions from the relevant information; (iii) searching from the first data repository for a plurality of top-K research documents having similarity to the research protocol using a vector representation of the research protocol; and (iv) obtaining from the second data repository a plurality of paragraph chunks of each of the plurality of top-K research documents created using parser and indexer functionality of the LLM agent-oriented architecture.

11. 10. The one or more non-transitory machine-readable information storage media of claim 9, wherein the second type of task performed by the second agent comprises at least one of the steps of: (i) identifying gaps in the plurality of research documents that address the motivation behind the research problem; (ii) identifying the set of plausible solutions that address the research problem; and (iii) rewriting the research proposal based on the gaps identified in the plurality of prior research documents and the set of plausible solutions that address the research problem.

12. The set of reasonable solutions that address the research problem includes: Decomposing the research topic defined in the research plan into a number of sub-topics; identifying a subset of sub-tasks from the plurality of sub-tasks such that a set of hallucinated tasks is excluded; retrieving, from a first data repository, a number of top-K research documents having similarity to each of the subset of sub-tasks based on the vector representation of the subset of sub-tasks; determining a set of plausible solutions for addressing each of the sub-problems from the subset of sub-problems by extracting one or more relevant texts from each of the plurality of top-K research documents stored and retrieved from the second data repository; performing iteratively searching the top-K research documents to identify a set of plausible solutions to generate a consolidated list of subsets of similar sub-problems and corresponding sets of plausible solutions; using the consolidated list of the subsets of similar sub-problems and the corresponding set of plausible solutions to identify a set of plausible solutions that address the research problem.

10. One or more non-transitory machine-readable information storage media as recited in claim 9, identified by:

Images