A scientific research assistance method based on large language model

Abstract

The present invention discloses a scientific research assistance method based on a large language model, which belongs to the field of artificial intelligence. Specifically, the method comprises the following steps: first, building a scientific research assistance system including a core control module, a tool calling module and a multimodal input module; the user logs in to set initial parameters; then, the core control module receives an input instruction Y I , start the first round of task execution and get the task plan Y T1 ; and further obtain the task execution record Y * , combined with historical information Y H , generate decision action Y A1 , after being passed to the tool calling module, the action execution result Y is obtained O1 Finally, judge the input command Y I Is the first round of task execution completed? If so, generate the final output Y F , waiting for new user input; otherwise, call the large language model F θ The task plan for the second round is generated, and the next round of task execution cycle is started until the end of round n. The present invention provides more convenient support for scientific researchers to read documents, search documents, and obtain scientific research dynamics.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and relates to a scientific research assistance method based on a large language model. Background Art

[0002] Large Language Model (LLM) is a large-scale artificial intelligence computing model that has become popular in recent years. It can understand and generate language by learning large amounts of text data, and perform various language-related tasks such as text generation, language understanding, translation, and summary writing.

[0003] An artificial intelligence agent (AI Agent) is a software or hardware decision-making system that can automatically perform tasks or achieve goals in a specific environment. Its architecture mainly consists of three modules: a brain module, a perception module, and an execution module. The AI ​​agent makes decisions based on the perceived environmental information and takes actions to achieve predetermined goals or tasks.

[0004] The artificial intelligence agent based on the large language model uses the large language model as its brain, and uses the natural language understanding ability of the large language model to process natural language and generate tool call instructions. It can automatically make decisions in complex environments and use tools to perform tasks.

[0005] Traditional scientific research assistance systems generally include two main functions: literature retrieval and literature reading assistance. For literature retrieval, traditional methods rely on manually setting keywords and search conditions, and require researchers to filter out useful information that meets their needs from noisy noise information. For literature reading assistance, traditional methods can often only provide basic functions, such as word translation, and cannot achieve advanced functions such as content analysis, summarizing the main points, or answering questions in papers.

[0006] In short, traditional scientific research support systems require deep participation of scientific researchers and consume a lot of their time and energy. Summary of the Invention

[0007] The present invention discloses a scientific research assistance method based on a large language model, which aims to solve the problem of literature research difficulties existing in the work process of existing scientific researchers. By constructing an artificial intelligence agent based on a large language model, it provides more convenient support for scientific researchers to read literature, search literature, and obtain scientific research dynamics.

[0008] The specific steps of the scientific research assistance method based on the large language model are as follows:

[0009] Step 1: Build a scientific research support system including a core control module, a tool calling module, and a multimodal input module;

[0010] The core control module is the processing center, with a large language model as the main body. Its main function is to understand user input and generate tool call instructions.

[0011] The tool invocation module is responsible for executing specific tasks and includes three tools: paper parsing, paper retrieval, and search engine retrieval. The paper parsing tool leverages a large language model to answer user questions based on the paper content; the paper retrieval tool utilizes the API provided by the paper collection platform to obtain paper metadata and automatically downloads the paper to the local computer when returning search results; and the search engine retrieval tool invokes external search engines to answer user questions based on the search results.

[0012] In addition to traditional text input, the multimodal input module can also realize voice and image input.

[0013] Step 2: After logging into the system, the user initially sets the parameters of the large language model and the tool calling module.

[0014] The parameters of the large language model include model ID, model API-KEY, model BaseURL, and model temperature;

[0015] The parameters of the tool module include the available tools and the initialization parameters of each tool

[0016] Among them, ∑ is the set of all initialization parameters of the tool, σ i Represents different initialization parameters, N ∑ is the total number of initialization parameters required by the tool.

[0017] For the paper parsing tool, the initialization parameters include: the path of the paper to be parsed (string type), the length of the text segment (ChunkLength) (integer type), and whether to use the retrieval enhancement technology (Boolean type);

[0018] For the paper retrieval tool, the initialization parameters include: the paper collection platform Platform (string type) and whether to download the retrieved papers Download (Boolean type);

[0019] For the search engine retrieval tool, the initialization parameters include: the name of the search engine Name (string type) and the number of search results finally returned SearchNum (integer type).

[0020] Step 3: The core control module receives the user's input command Y I , start the first round of task execution and get the task plan Y T1 ;

[0021] 1) When the input command is natural language, the core control module calls the large language model F θ , based on historical information Y H And the user input command Y I Generate the current round 1 task plan Y T1 =F θ (Y I , Y H ).

[0022] Historical Information Y H Including the input instruction Y I All previous input instructions execute tasks, generate actions and loop results after action execution, and the historical information Y before the initial input instruction H Is an empty set.

[0023] The specific method is: given an input sequence text Y in , output text sequence Y out satisfy:

[0024]

[0025] The input sequence text Y in is (Y I , Y H ), output text sequence Y out Plan Y for task T1 , Y is a possible output text sequence, y i is the i-th word in the output text sequence Y, p(Y|Y in ) is given Y in The probability that the model generates the text sequence Y under the condition .

[0026] 2) If the input command is voice, the core control module calls the API to convert it into text and obtains the input sequence text Y in , and then use the case where the input instruction in this step is natural language to process. Output text sequence Y out satisfy:

[0027]

[0028] 3) When the input instruction is an image, the core control module calls the large language model F θ Generate mission plan Y T1 ; The specific method is: given an image input and text input sequence Vision encoder and connector P φ The image Aligned with the text modality, it outputs a text sequence Y outsatisfy:

[0029]

[0030] The user inputs Y I For images Historical Information Y H For text

[0031] Step 4: The core control module records the execution of the first round of tasks Y * , combined with historical information Y H , generate the decision action Y of the first round A1 , passing it to the specific tool calling module.

[0032] The core control module calls the large language model F θ Generate the decision action Y with the highest probability A1 :

[0033] Y A1 =F θ (Y * , Y H )

[0034] Execution Record Y * =(Y I , Y T1 ).

[0035] Step 5: The tool calls the module to execute decision action Y A1 , get the action execution result Y O1 Passed back to the core control module.

[0036] The tool call module of the task execution cycle contains initialization parameters and operating parameters Λ n Represents the set of all running parameters of the tool. Represents different operating parameters, Represents the total number of running parameters required by the tool, n represents the total number of rounds of task execution, and the initial value is 1.

[0037] The process of tool calling module generating tool calling result is as follows: tool calling module T An From decision action Y A1 Extract the operating parameters Λ n , generate tool call result Y O1 , that is, Y O1 =T An (∑ n , Λ n ) and returns to the core control module.

[0038] Specifically for the paper parsing tool, its workflow is as follows:

[0039] 1. Convert the documents uploaded by the user into plain text, or slice the obtained paper segments into a set T0. If Retrieval is set to true during initialization, the enhanced segment set T is obtained using retrieval enhancement technology, otherwise T is set to T0.

[0040] 2. The paper parsing module follows instruction Y A1 Extract the running parameters question and method.

[0041] 3. Embed the text set T into the set prompt words, use the large language model to generate the answer, and use it as the result Y of this round of action execution O1 , return to the core control module.

[0042] Specifically for the paper retrieval tool, its workflow is as follows:

[0043] 1. The paper retrieval module follows instruction Y A1 Extract the query, sort_by, and sort_order operation parameters.

[0044] 2. Call the API to search for papers and return the searched paper metadata in JSON format.

[0045] 3. The paper retrieval module extracts key information such as paper title, paper author, paper abstract, paper publication time, etc. from the paper metadata in JSON format and generates a structured action execution result Y O1 , and returns to the core control module.

[0046] Specifically for search engine retrieval tools, its workflow is as follows:

[0047] 1. Search engine retrieval module according to Y A1 Generates the query parameter.

[0048] 2. Call the API to search from the Internet, cut the retrieved web page results into content + link format, apply search enhancement technology to all the cut results, and obtain filtered search results. Splice the filtered search results as the action execution result Y O1 Return to the core control module.

[0049] Step 6. Execute the action result Y O1 Determine input command Y I Is the current round 1 task execution completed? If so, generate the final output Y F , wait for new user input and return to step 3; otherwise, call the large language model F θGenerate the next round of task planning and return to step 4 to start the next round of task execution cycle.

[0050] The next round of task planning is Y T2 =F θ (Y * , Y H ); and so on, when the n+1th round task plan Y is generated T(n+1) Before, the execution record of the task Y * Contains all the contents of the first n rounds of task execution cycles, that is,

[0051] Y * =(Y I , Y T1 , Y A1 , Y O1 , Y T2 , Y A2 , Y O2 ,...,Y Tn , Y An , Y On )

[0052] Let the task planning of round n be Y Tn , the decision action Y in round n An This is the tool call instruction, and the result of the action execution in round n is Y On ;

[0053] When the task has been completed, according to the execution record Y * and historical information Y H Generate the final output Y F , that is, Y F =F θ (Y * , Y H ), and feedback it to the user after sorting it out as the execution result; at the same time, use the execution record to update the historical information, that is, Y H ←(Y H , Y * ).

[0054] The advantages of the present invention are:

[0055] (1) The present invention provides a scientific research assistance method based on a large language model. The core control module uses the large language model to analyze semantics and, in combination with designed prompt words, generates tool call instructions in a specific format. With the large language model as the decision-making center, it can convert relatively vague natural language input into specific and clear tool call instructions, allowing the system to handle the user's vague needs and reducing the need for manual user intervention.

[0056] (2) The present invention provides a scientific research assistance method based on a large language model. Specifically, a tool call module is designed to execute tool call instructions for scientific research assistance tasks. Each tool focuses on a specific function, and the content and format of the input and output are relatively fixed. After the processing center defines the processing target, the target can be achieved in a relatively clear manner, ensuring the feasibility of achieving the target and the reliability of the processing results.

[0057] (3) The present invention provides a scientific research assistance method based on a large language model, which realizes a multimodal input function and can accept inputs in multiple different modes such as text, voice, and images. Different processing methods are used for different modules, which not only enriches the functions of the system but also can meet the diverse needs of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a module structure and workflow diagram of the scientific research assistance system (SciAgent) based on the large language model of the present invention.

[0059] Figure 2 This is a flow chart of a scientific research assistance method based on a large language model of the present invention; DETAILED DESCRIPTION

[0060] The present invention describes the system design and workflow of the scientific research assistance system SciAgent as a computer program.

[0061] This invention proposes a research assistance method based on a large language model, such as Figure 1 As shown, with a large language model as the decision-making center, the core control module can process relatively ambiguous natural language instructions and filter out noise information in search results. In addition, the system can provide advanced functions such as literature analysis and literature summarization based on the large language model, and utilize tool call modules for paper parsing, paper retrieval, and search engine retrieval. It can also handle multimodal input including voice, images, and text. This system can reduce the time and effort researchers spend on literature research, which is of great significance for improving their research efficiency.

[0062] The scientific research assistance method based on the large language model, such as Figure 2 The specific steps are as follows:

[0063] Step 1: Build a scientific research support system including a core control module, a tool calling module, and a multimodal input module;

[0064] The core control module is the processing center, with a large language model as the main body. Its main function is to understand user input and generate tool call instructions.

[0065] The tool invocation module is responsible for executing specific tasks and includes three tools: paper parsing, paper retrieval, and search engine retrieval. The paper parsing tool leverages a large language model to answer user questions based on the paper content; the paper retrieval tool utilizes the API provided by the paper collection platform to obtain paper metadata and automatically downloads the paper to the local computer when returning search results; and the search engine retrieval tool invokes external search engines to answer user questions based on the search results.

[0066] In addition to traditional text input, the multimodal input module can also realize voice and image input.

[0067] Step 2: After logging into the system, the user initially sets the parameters of the large language model and the tool calling module.

[0068] The parameters of the large language model include model ID, model API-KEY, model BaseURL, and model temperature;

[0069] The parameters of the tool module include the available tools and the initialization parameters of each tool

[0070] Among them, ∑ is the set of all initialization parameters of the tool, σ i Represents different initialization parameters, N ∑ is the total number of initialization parameters required by the tool.

[0071] For the paper parsing tool, the initialization parameters include: the path of the paper to be parsed (string type), the length of the text segment (ChunkLength) (integer type), and whether to use the retrieval enhancement technology (Boolean type);

[0072] For the paper retrieval tool, the initialization parameters include: the paper collection platform Platform (string type) and whether to download the retrieved papers Download (Boolean type);

[0073] For the search engine retrieval tool, the initialization parameters include: the name of the search engine Name (string type) and the number of search results finally returned SearchNum (integer type).

[0074] Step 3: The core control module receives the user's input command Y I , start the first round of task execution and get the task plan Y T1 ;

[0075] 1) When the input command is natural language, the core control module calls the large language model F θ , based on historical information Y H And the user input command YI Generate mission plan Y T1 =F θ (Y I , Y H ).

[0076] Historical Information Y H Including the input instruction Y I All previous input instructions execute tasks, generate actions and loop results after action execution, and the historical information Y before the initial input instruction H Is an empty set.

[0077] The specific method is: given an input sequence text Y in , output text sequence Y out satisfy:

[0078]

[0079] The input sequence text Y in is (Y I , Y H ), output text sequence Y out Plan Y for task Tn , Y is a possible output text sequence, y i is the i-th word in the output text sequence Y. p(Y|Y in ) is given Y in The probability that the model generates text Y under the condition .

[0080] 2) If the input command is voice, the core control module calls the API to convert it into text and obtains the input sequence text Y in , output text sequence Y out satisfy:

[0081]

[0082] 3) When the input instruction is an image, the core control module calls the large language model F θ Generate mission plan Y T1 ; The specific method is: given an image input and text input sequence Vision encoder and connector P φ The image Aligned with the text modality, it outputs a text sequence Y out satisfy:

[0083]

[0084] The user inputs Y I For images Historical Information Y H For text

[0085] Step 4: The core control module executes the record Y according to the task * , and combined with historical information Y H , generate the decision action Y of the first round A1 , passing it to the specific tool calling module.

[0086] Initial execution record Y * The value is Y * =(Y I , Y T1 ), according to the core control module, the large language model F θ Generate the decision action Y with the highest probability A1 :

[0087] Y A1 =F θ (Y * , Y H )

[0088] Step 5: The tool call module executes the tool call instruction Y A1 , get the action execution result Y O1 Passed back to the core control module.

[0089] Tool call module T for task execution cycle An Contains initialization parameters and operating parameters

[0090] Λ n Represents the set of all running parameters of the tool. Represents different operating parameters, Represents the total number of running parameters required by the tool, n represents the total number of rounds of task execution, and the initial value is 1.

[0091] The process of tool calling module generating tool calling result is as follows: tool calling module T A1 From decision action Y A1 Extract the running parameter Λ1 and generate the tool call result Y O1 , that is, Y O1 =T A1 (∑1, Λ1) and returns to the core control module.

[0092] Specifically for the paper parsing tool, its workflow is as follows:

[0093] 1. Convert the documents uploaded by the user into plain text, or slice the obtained paper segments into a set T0. If Retrieval is set to true during initialization, the enhanced segment set T is obtained using retrieval enhancement technology, otherwise T is set to T0.

[0094] 2. The paper parsing module follows instruction Y A1 Extract the running parameters question and method.

[0095] 3. Embed the text set T into the set prompt words, use the large language model to generate the answer, and use it as the result Y of this round of action execution O1 , return to the core control module.

[0096] Specifically for the paper retrieval tool, its workflow is as follows:

[0097] 1. The paper retrieval module follows instruction Y A1 Extract the query, sort_by, and sort_order operation parameters.

[0098] 2. Call the API to search for papers and return the searched paper metadata in JSON format.

[0099] 3. The paper retrieval module extracts key information such as paper title, paper author, paper abstract, paper publication time, etc. from the paper metadata in JSON format and generates a structured action execution result Y O1 , and returns to the core control module.

[0100] Specifically for search engine retrieval tools, its workflow is as follows:

[0101] 1. Search engine retrieval module according to Y A1 Generates the query parameter.

[0102] 2. Call the API to search from the Internet, cut the retrieved web page results into content + link format, apply search enhancement technology to all the cut results, and obtain filtered search results. Splice the filtered search results as the action execution result Y O1 Return to the core control module.

[0103] Step 6. Execute the action result Y O1 Determine input command Y I Is the current round 1 task execution completed? If so, generate the final output Y F , wait for new user input and return to step 3; otherwise, call the large language model F θ Generate the next round of task planning and return to step 4 to start the next round of task execution cycle.

[0104] The next round of task planning is Y T2 =F θ (Y * , Y H ); and so on, the core control module evaluates the execution record Y of the n+1th round of tasks * Contains all the contents of the first n rounds of task execution cycles, that is,

[0105] Y * =(Y I , Y T1 , Y A1 , Y O1 , Y T2 , Y A2 , Y O2 ,...,Y Tn , Y An , Y On )

[0106] The task planning for round n is Y Tn , the result of the action execution in round n is recorded as Y On , decision action Y An That is the tool call instruction;

[0107] When the task has been completed, according to the execution record Y * and historical information Y H Generate the final output Y F , that is, Y F =F θ (Y * , Y H ), and feedback it to the user after sorting it out as the execution result; at the same time, use the execution record to update the historical information, that is, Y H ←(Y H , Y * ).

[0108] Example:

[0109] The system workflow of SciAgent of the present invention specifically includes its system design and system implementation; it is divided into the following steps:

[0110] Step 1: After the user enters the system page, initialize an artificial intelligence agent instance according to the user settings.

[0111] The parameters that need to be set include those of the large language model and the tool module.

[0112] Step 2: Receive the user's natural language or other modal input instructions such as voice, image, etc. The core control module generates a tool call instruction based on the user input and passes it to the specific tool call module.

[0113] For natural language input Y I The core control module needs to first evaluate the task execution record Y * , call the large language model F θ Generate mission plan Y T :If the task has just started, then only Y I , that is, Y * =(Y I ); If several cycles have been executed, the execution record will also contain records of each cycle process, that is:

[0114] Y * =(Y I , Y A1 , Y O1 , Y T1 , Y A2 , Y O2 , Y T2 ,...,Y An , Y On ).

[0115] Afterwards, the core control module combines the current execution record Y * and historical information Y H Call the large language model F θ Generate the decision action Y with the highest probability A (i.e. tool call instructions):

[0116] Y A The generation probability can be calculated based on Calculate, where N A is the instruction length.

[0117] After getting the tool call instruction Y A After that, it is passed to the specific tool calling module T A .

[0118] Tool call module T A Contains initialization parameters and operating parameters For the input Y from the core control module A , the tool calling module needs to extract the operating parameters Λ from it, and then generate the tool output Y O , that is, Y O =T A (∑,Λ).

[0119] If the input contains voice, you need to call the API to convert it into text;

[0120] If the input contains an image X, it needs to pass through the visual encoder and connector P φ Align it with the text modal. A The probability of generation is based on The rest of the calculation is the same as above.

[0121] Since different tools have different computer program implementation methods under different initialization parameters ∑, the operating parameters Λ required to be provided by the upper-level module during actual operation are also different. Therefore, different tool instances need to provide different operating parameter descriptions.

[0122] In addition to relying on user input, the core control module also needs to refer to the operating parameter descriptions of different tool instances in the process of generating tool call instructions. In actual operation, the tool description needs to be organized into a specific format as part of the prompt words of the large language model.

[0123] For the paper parsing tool, there are two operating parameters: question and method. See Table 1 for detailed explanation.

[0124] For the paper retrieval tool, there are three operating parameters: query, sort_by, and sort_order. See Table 2 for detailed explanations.

[0125] For search engine retrieval tools, the only operating parameter is query. See Table 3 for detailed explanation.

[0126] Table 1

[0127]

[0128] Table 2

[0129]

[0130] Table 3

[0131]

[0132]

[0133] Step 3: The tool calling module executes the tool and passes the tool calling result back to the core control module;

[0134] The tool calling module includes paper parsing tools, paper retrieval tools and search engine retrieval tools;

[0135] The workflow of the paper parsing tool is:

[0136] (1) Convert the document uploaded by the user into plain text. If the text is too long, it needs to be sliced ​​to obtain the paper segments. The paper text or the original paper segments constitute the set T0. If Retrieval is set to true during initialization, the enhanced segment set T needs to be obtained using retrieval enhancement technology. Otherwise, T = T0.

[0137] (2) The core control module generates the paper parsing tool call instruction Y A , the paper analysis module is based on Y A Extract the running parameters question and method.

[0138] (3) Embed the text set T into the set prompt words and use the large model to generate answers. The generated answers are used as tool output Y O , return to the core control module.

[0139] The workflow of the paper retrieval tool is as follows:

[0140] (1) The core control module generates the paper retrieval tool call instruction Y A , the paper retrieval module is based on Y A Extract the query, sort_by, and sort_order operation parameters.

[0141] (2) Call the API to search for papers and return the searched paper metadata in JSON format.

[0142] (3) The paper retrieval module extracts key information such as paper title, paper author, paper abstract, and paper publication time from the paper metadata in JSON format and generates a structured tool output Y O , and returns to the core control module.

[0143] The workflow of the search engine retrieval tool is:

[0144] (1) The core control module generates the search engine retrieval tool call instruction Y A , the search engine retrieval module is based on Y A Generates the query parameter.

[0145] (2) Call the API to search from the Internet, cut the retrieved web page results into the form of content + link, use the search enhancement technology on all the cut results to obtain the filtered search results. The filtered search results are spliced ​​and output as tool Y O Return to the core control module.

[0146] During the invocation of the above modules, retrieval enhancement technology is used to complete the relevant tasks. Retrieval enhancement technology selects one or more passages with the highest relevance to the input passage from a number of candidate passages. This can filter out noisy passages and reduce the cost of invoking large language models. Details of retrieval enhancement technology are as follows:

[0147] The original text is segmented into an original text set c, which is then vectorized using the embedding model to form a vector set V = (V1, V2, ..., V n ), each paragraph corresponds to a set of vectors. The user input is also vectorized to obtain vector I0, which is traversed and compared in the vector database to obtain the correlation score R(V i , I0), the calculation method is cosine similarity, that is:

[0148]

[0149] The smaller the correlation score of two vectors, the higher the correlation between the texts corresponding to the two vectors. top and correlation threshold R s , filter out the ones that meet the relevance score R≤R S And the k with the smallest correlation top vectors, the set they form is V′, and the corresponding set of paragraphs is C′. Return C′.

[0150] Step 4: The core control module calls the large language model F θ , based on historical information Y H and the current execution record Y * =(Y I , Y A1 , Y O1 , Y T1 , Y A2 , Y O2 , Y T2 ,...,Y An , Y On ) Evaluate the current state and generate a task plan Y for the current state Tn ,Right now If the evaluation result is that the task is not completed, go to step 2 and perform the record Y according to the current execution * and historical information Y H Generate the next decision action Y A(n+1) If the evaluation result is that the task is completed, go to step 5 to generate the final output Y F .

[0151] Step 5: The core control module executes the record Y * and historical information Y HGenerate the final output Y F And it will be fed back to the user after sorting out the execution results. Use the current execution record to update the historical information, that is, Y H ←(Y H , Y * ).

[0152] The large language model-based research assistance system described in this specification can be implemented with tangible, concrete computer software or firmware, with computer hardware (including the structures disclosed in this specification and their structural equivalents), or with a combination of one or more of them. The large language model-based research assistance system described in this specification can be implemented as one or more computer programs. A computer program (which may also be referred to or described as a program, software, software application, module, software module, script, or code) can be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages, and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

Claims

1. A scientific research assistance method based on a large language model, characterized in that: The specific steps are as follows: Step 1: Build a scientific research support system including a core control module, a tool calling module, and a multimodal input module; Step 2: After logging into the system, the user initially sets the parameters of the large language model and the tool calling module. The parameters of the large language model include model ID, model API-KEY, model BaseURL, and model temperature; The parameters of the tool module include the available tools and the initialization parameters of each tool Among them, Σ is the set of all initialization parameters of the tool, σ i Represents different initialization parameters, N Σ is the total number of initialization parameters required by the tool; Step 3: The core control module receives the user's input command Y I , start the current first round of task execution and get the task plan Y T1 ; 1) When the input command is natural language, the core control module calls the large language model F θ , based on historical information Y H And the user input command Y I Generate the current round 1 task plan Y T1 =F θ (Y I ,Y H ); The specific method is: given an input sequence text Y in , output text sequence Y out satisfy: The input sequence text Y in is (Y I ,Y H ), output text sequence Y out Plan Y for task Tn , Y is a possible output text sequence, y i is the i-th word in the output text sequence Y; p(Y|Y in ) is given Y in The probability that the model generates the text sequence Y under the condition ; 2) If the input command is voice, the core control module calls the API to convert it into text and obtains the input sequence text Y in , which can be calculated using natural language output text sequence Y out ; 3) When the input instruction is an image, the core control module calls the large language model F θ Generate mission plan Y T1 ; The specific method is: given an image input and text input sequence Vision encoder and connector P φ The image Aligned with the text modality, it outputs a text sequence Y out satisfy: The user inputs Y I For images Historical Information Y H For text Step 4: The core control module uses the task planning Y T1 Get task execution record Y * , and combined with historical information Y H , generate the decision action Y of the first round A1 , pass it to the specific tool calling module; The core control module calls the large language model F θ Generate the decision action Y with the highest probability A1 : AND A1 =F θ (AND * ,AND H ) Initial execution record Y * The value is Y * =(Y I ,Y T1 ); Step 5: The tool call module executes the tool call instruction Y A1 , get the action execution result Y O1 Passed back to the core control module; Tool call module T for the first round of task execution cycle An Contains initialization parameters and operating parameters Λ n Represents the set of all running parameters of the tool. Represents different operating parameters, Represents the total number of parameters required for the tool to run; The process of tool calling module generating tool calling result is as follows: tool calling module T An From decision action Y A1 Extract the operating parameters Λ n , generate tool call result Y O1 , that is, Y O1 =T An (Σ n ,Λ n ) and returns to the core control module; Step 6. Execute the action result Y O1 Determine input command Y I Is the current round 1 task execution completed? If so, generate the final output Y F , wait for new user input and return to step 3; otherwise, call the large language model F θ Generate the second round of task planning and return to step 4 to start the next round of task execution cycle; The next round of task planning is Y T2 =F θ (Y * ,Y H ); and so on, when the execution record Y of the n+1th round of tasks is generated * Contains all the contents of the first n rounds of task execution cycles, that is, AND * =(And I ,AND T1 ,AND A1 ,AND O1 ,AND T2 ,AND A2 ,AND O2 ,…,AND Tn ,AND An ,AND On ) The task planning for round n is Y Tn , the result of the action execution in round n is recorded as Y On , decision action Y An That is the tool call instruction; When the task has been completed, according to the execution record Y * and historical information Y H Generate the final output Y F , that is, Y F =F θ (Y * ,Y H ), and feedback it to the user after sorting it out as the execution result; at the same time, use the execution record to update the historical information, that is, Y H ←(Y H ,Y * ).

2. A scientific research assistance method based on a large language model as claimed in claim 1, characterized in that: In step one, the core control module is the processing center, with a large language model as the main body. Its main function is to understand user input and generate tool call instructions; the tool call module is responsible for executing specific tasks, including three tools: paper parsing, paper retrieval, and search engine retrieval; in addition to traditional text input, the multimodal input module can also realize voice and image input.

3. A scientific research assistance method based on a large language model as claimed in claim 2, characterized in that: The paper parsing tool uses a large language model to answer user questions based on the content of the paper; the paper retrieval tool can use the API provided by the paper collection platform to obtain paper metadata, and automatically download the paper to the local computer while returning the retrieval results; the search engine retrieval tool calls an external search engine and answers user questions based on the search results.

4. A scientific research assistance method based on a large language model as claimed in claim 2, characterized in that: The paper parsing tool's initialization parameters include: the paper path to be parsed Path, the length of the text segment ChunkLength, and whether to use the retrieval enhancement technology Retrieval; For the paper retrieval tool, the initialization parameters include: the paper collection platform Platform and whether to download the retrieved papers Download; For search engine retrieval tools, the initialization parameters include: the name of the search engine Name, and the number of search results finally returned SearchNum.

5. A scientific research assistance method based on a large language model as claimed in claim 1, characterized in that: In step 3, historical information Y H Including the input instruction Y I All previous input instructions execute tasks, generate actions and loop results after action execution, and the historical information Y before the initial input instruction H Is an empty set.

6. A scientific research assistance method based on a large language model as claimed in claim 1, characterized in that: In step 5, the paper parsing tool obtains the action execution result Y O1 The workflow is: 1) Convert the documents uploaded by the user into plain text, or slice the obtained paper segments into a set T0. If Retrieval is set to true during initialization, the enhanced segment set T is obtained using retrieval enhancement technology. Otherwise, T is set to T0. 2) The paper parsing module follows instruction Y A1 Extract the running parameters question and method; 3) Embed the text set T into the set prompt words, use the large language model to generate the answer, and use it as the result Y of this round of action execution O1 , return to the core control module.

7. A scientific research assistance method based on a large language model as claimed in claim 1, characterized in that: In step 5, the paper retrieval tool obtains the action execution result Y O1 The workflow is: 1) The paper retrieval module follows instruction Y A1 Extract the running parameters query, sort_by, and sort_order; 2) Call the API to search for papers and return the searched paper metadata in JSON format; 3) The paper retrieval module extracts the paper title, paper author, paper abstract and paper publication time information from the paper metadata in JSON format and generates a structured action execution result Y O1 , and returns to the core control module.

8. A scientific research assistance method based on a large language model as claimed in claim 1, characterized in that: In step 5, the search engine retrieval tool obtains the action execution result Y O1 The workflow is: 1) Search engine retrieval module according to Y A1 Generate running parameter query; 2) Calling the API to search the Internet, cutting the retrieved web page results into the form of content + links, and applying search enhancement technology to all the cut results to obtain filtered search results; The filtered search results are spliced ​​together as the action execution result Y O1 Return to the core control module.

Images