Combined reasoning ability enhancing method based on large language model tool use

Abstract

The invention provides a combined reasoning ability enhancement method based on large language model tool use. The method comprises the steps of obtaining a reasoning task of a question to be answered; selecting and planning an enhancement tool corresponding to the reasoning task in a preset tool application set by utilizing a large language model according to the reasoning task; calling and executing an enhancement tool by utilizing the large language model, and generating candidate answers; performing self-evaluation on the candidate answers by using a preset external model to obtain a target answer corresponding to the reasoning task; wherein the tool application set at least comprises one or more of the following enhancement tools: a picture information processor, an equation extractor, a comparison sample constructor, a webpage knowledge extractor and a parameter knowledge generator; according to the method, through combination of adaptive tool selection and answer self-evaluation, the combined reasoning capability of a large language model is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for enhancing combinatorial reasoning capabilities based on the use of a large language model tool. Background Art

[0002] Currently, the rapid development of Large Language Models (LLMs) has transformed natural language processing, and large models have demonstrated reasoning capabilities. Reasoning tasks include common sense, mathematical, logical, causal, visual, audio, and agent-based reasoning. These tasks help individuals link information, evaluate connections, and draw sensible conclusions or solutions.

[0003] In the context of enhanced large language models, tools are broadly defined as external tools or resources used to enhance the functionality and performance of LLMs. Tool learning aims to enable LLMs to effectively interact with a variety of tools to complete complex tasks. This approach significantly improves the ability of LLMs to solve complex problems by leveraging external tools and resources. For example, LLMs can use search engines to acquire knowledge, online calculators to perform complex calculations, and symbolic modules or Python interpreters to perform complex algorithms and mathematical operations. Recent research has explored the use of combinatorial tools to represent a major innovation in this area.

[0004] Combinatorial reasoning in Natural Language Processing (NLP) enables large language models to handle complex tasks by breaking them down into simpler subtasks and solving them in a structured way. This is crucial for LLMs to integrate various tools and modules to enhance their reasoning capabilities. Recent research has highlighted some noteworthy work: for example, the ability to use external tools for question answering (QA) through LLM in the ToolQA testing project; for example, enhancing the compositional reasoning of LLMs with plug-and-play modules in the Chameleon project; for example, a multimodal collaborative reasoning framework was introduced in the Cantor project, and so on.

[0005] However, the aforementioned large language models have the following disadvantages: (1) they are rigid and inflexible when calling external tools; and (2) the widespread use of tool calls may result in the prompts passed to the LLM being too long, which may prevent the LLM from effectively distinguishing and extracting useful information from large amounts of data, thereby increasing the uncertainty of the generated answers and potentially increasing the error rate. Summary of the invention

[0006] In order to solve the problem that the existing large language model is rigid and inflexible when calling external tools, which makes it impossible to effectively distinguish and extract useful information during the reasoning process, increases the uncertainty of generating answers, and leads to a high reasoning error rate, the present invention proposes a combined reasoning ability enhancement method based on the use of large language model tools, including:

[0007] Obtain reasoning tasks for questions to be answered;

[0008] According to the reasoning task, using the large language model, selecting and planning an enhanced tool corresponding to the reasoning task from a preset tool application set;

[0009] Using the large language model to call and execute the enhancement tool to generate candidate answers;

[0010] Using a preset external model to self-evaluate the candidate answers to obtain a target answer corresponding to the reasoning task;

[0011] The tool application set includes at least one or more of the following enhanced tools: an image information processor, an equation extractor, a comparison sample constructor, a web page knowledge extractor, and a parameter knowledge generator.

[0012] Optionally, the selecting and planning of an enhancement tool corresponding to the reasoning task from a preset tool application set using a large language model according to the reasoning task includes:

[0013] According to the reasoning task, obtaining a question-answering scenario corresponding to the reasoning task;

[0014] Based on the question-answering scenario corresponding to the reasoning task, a large language model is utilized to select and plan enhanced tools for the reasoning task in different question-answering scenarios from a preset tool application set.

[0015] Optionally, the reasoning task includes: a multimodal reasoning task or a unimodal reasoning task; the multimodal reasoning task includes: question information and additional information; the additional information includes one or more of the following: image information, table information and prompt information;

[0016] The question-answering scenarios include: multimodal science question-answering scenarios, multimodal mathematics question-answering scenarios, and single-modal complex mathematics question-answering scenarios;

[0017] The enhancement tools of the multimodal reasoning task in the multimodal scientific question-answering scenario include one or more of the following: an image information processor, a parameter knowledge generator, a web page knowledge acquirer, and a comparison sample constructor;

[0018] The enhancement tool of the multimodal reasoning task in the multimodal math question-answering scenario includes one or more of the following: a parameter knowledge acquirer, a program generator, a program verifier, a program executor, and a data filter;

[0019] The enhancement tools for the unimodal reasoning task in the unimodal complex mathematical question-answering scenario include one or more of the following: a comparison sample constructor, a formula extractor, a parameter knowledge generator, and a web page knowledge acquirer.

[0020] Optionally, the using the large language model to call and execute the enhancement tool to generate candidate answers includes:

[0021] When the reasoning task includes image information, the image information processor is called and executed by using the large language model to obtain prompt information of the image information;

[0022] When the reasoning task includes a mathematical expression, the formula extractor is called and executed using the large language model to obtain a calculation result of the mathematical expression;

[0023] When the reasoning task includes question information, the comparison sample constructor is called and executed by the large language model to obtain the thinking process information of the question information; the web page knowledge extractor is called and executed by the large language model to obtain the web page knowledge information of the question information; the parameter knowledge generator is called and executed by the large language model to obtain the parameter knowledge information of the question information;

[0024] Candidate answers are generated according to the prompt information of the image information, the calculation result of the mathematical expression, the thinking process information of the question information, the web page knowledge information and the parameter knowledge information.

[0025] Optionally, the using the large language model to call and execute the image information processor to obtain prompt information of the image information includes:

[0026] Detecting object attribute information in the image information using a large language model;

[0027] Using an image processing expression to process the object attribute information, to obtain prompt information of the image information;

[0028] The object attribute information includes: an object and attributes corresponding to the object;

[0029] The image processing expression is as follows:

[0030] c i =T image (e i ,ppti );

[0031] Among them, e i represents the i-th image information; c i Indicates image information i Tips for: i Indicates prompt information c i Object attribute information of T image An expression function representing an image information processor.

[0032] Optionally, the using the large language model to call and execute the formula extractor to obtain the calculation result of the mathematical expression includes:

[0033] Using the large language model, generating a response value of the mathematical expression;

[0034] Based on the response value, the mathematical expression is calculated using an arithmetic expression to obtain a calculation result of the mathematical expression;

[0035] The arithmetic expression is as follows:

[0036] M j =T equ (r j );

[0037] Among them, M j represents the calculation result of the jth mathematical expression in the reasoning task; r j represents the response value of the large language model to the jth mathematical expression in the reasoning task; T equ Represents the expression function of the expression extractor.

[0038] Optionally, the using the large language model to call and execute the comparison sample constructor to obtain the thinking process information of the question information includes:

[0039] Calculating the similarity between the question information in the reasoning task and each question in the preset training sample set through the sentence converter;

[0040] Selecting the question with the highest similarity and the true answer corresponding to the question from the training set;

[0041] According to the question with the highest similarity and the real answer corresponding to the question, the positive thinking process information corresponding to the question information is obtained through a positive sample generator;

[0042] According to the question with the highest similarity and the positive thinking process information, the incorrect thinking process information corresponding to the question information is obtained through a negative sample generator;

[0043] Combining the positive thinking process information and the incorrect thinking process information into thinking process information;

[0044] The expression corresponding to the positive sample generator is as follows:

[0045] s pos =δ p (q sim ,a sim ,c sim );

[0046] Among them, s pos Represents the positive sequence thinking process information output by the positive sample generator; δ p Represents the expression function of the positive sample generator; q sim Indicates the question with the highest similarity; a sim Indicates the true answer corresponding to the question with the highest similarity; c sim Hint information indicating image information in reasoning tasks;

[0047] The expression corresponding to the negative sample generator is as follows:

[0048] s neg =δ n (q sim ,a sim ,c sim );

[0049] Among them, s neg Represents the wrong thought process information output by the negative sample generator; δ n Represents the expression function of the negative sample generator.

[0050] Optionally, the external model includes the following training process:

[0051] Randomly initializing parameters of the external model;

[0052] Using each question in the training set and the candidate answers corresponding to each question as input data of training data;

[0053] Using the true answers to the questions as output data of training data;

[0054] The external model is trained based on the input data and the output data.

[0055] Optionally, the self-evaluating the candidate answers using a preset external model to obtain a target answer corresponding to the reasoning task includes:

[0056] Inputting the candidate answer as an input item to the external model for self-evaluation multiple times, and outputting the best answer of the candidate answer under each self-evaluation;

[0057] According to the best answer of the candidate answer in each evaluation, the number of occurrences of each best answer is recorded;

[0058] The optimal answer with the highest number of occurrences is used as the target answer corresponding to the reasoning task.

[0059] Based on the same inventive concept, the present invention also provides a system for enhancing combinatorial reasoning capability based on the use of a large language model tool, comprising:

[0060] Task acquisition module, used to obtain reasoning tasks for questions to be answered;

[0061] A tool calling module, used to select and plan an enhanced tool corresponding to the reasoning task in a preset tool application set according to the reasoning task and using a large language model;

[0062] A candidate generation module, configured to use the large language model to call and execute the enhancement tool to generate candidate answers;

[0063] A self-evaluation module, used to use a preset external model to self-evaluate the candidate answers to obtain a target answer corresponding to the reasoning task;

[0064] The tool application set in the tool calling module includes at least one or more of the following enhanced tools: an image information processor, an equation extractor, a comparison sample constructor, a web page knowledge extractor, and a parameter knowledge generator.

[0065] Optionally, the tool calling module is specifically used to:

[0066] According to the reasoning task, obtaining a question-answering scenario corresponding to the reasoning task;

[0067] Based on the question-answering scenario corresponding to the reasoning task, a large language model is utilized to select and plan enhanced tools for the reasoning task in different question-answering scenarios from a preset tool application set.

[0068] Optionally, the reasoning task includes: a multimodal reasoning task or a unimodal reasoning task; the multimodal reasoning task includes: question information and additional information; the additional information includes one or more of the following: image information, table information and prompt information;

[0069] The question-answering scenarios include: multimodal science question-answering scenarios, multimodal mathematics question-answering scenarios, and single-modal complex mathematics question-answering scenarios;

[0070] The enhancement tools of the multimodal reasoning task in the multimodal scientific question-answering scenario include one or more of the following: an image information processor, a parameter knowledge generator, a web page knowledge acquirer, and a comparison sample constructor;

[0071] The enhancement tool of the multimodal reasoning task in the multimodal math question-answering scenario includes one or more of the following: a parameter knowledge acquirer, a program generator, a program verifier, a program executor, and a data filter;

[0072] The enhancement tools for the unimodal reasoning task in the unimodal complex mathematical question-answering scenario include one or more of the following: a comparison sample constructor, a formula extractor, a parameter knowledge generator, and a web page knowledge acquirer.

[0073] Optionally, the candidate generation module is specifically used to:

[0074] When the reasoning task includes image information, the image information processor is called and executed by using the large language model to obtain prompt information of the image information;

[0075] When the reasoning task includes a mathematical expression, the formula extractor is called and executed using the large language model to obtain a calculation result of the mathematical expression;

[0076] When the reasoning task includes question information, the comparison sample constructor is called and executed by the large language model to obtain the thinking process information of the question information; the web page knowledge extractor is called and executed by the large language model to obtain the web page knowledge information of the question information; the parameter knowledge generator is called and executed by the large language model to obtain the parameter knowledge information of the question information;

[0077] Candidate answers are generated according to the prompt information of the image information, the calculation result of the mathematical expression, the thinking process information of the question information, the web page knowledge information and the parameter knowledge information.

[0078] Optionally, the candidate generation module is specifically used to:

[0079] Detecting object attribute information in the image information using a large language model;

[0080] Using an image processing expression to process the object attribute information, to obtain prompt information of the image information;

[0081] The object attribute information includes: an object and attributes corresponding to the object;

[0082] The image processing expression is as follows:

[0083] c i =T image (e i ,ppt i );

[0084] Among them, e i represents the i-th image information; c i Indicates image information i Tips for: i Indicates prompt information c i Object attribute information of T image An expression function representing an image information processor.

[0085] Optionally, the candidate generation module is specifically used to:

[0086] Using the large language model, generating a response value of the mathematical expression;

[0087] Based on the response value, the mathematical expression is calculated using an arithmetic expression to obtain a calculation result of the mathematical expression;

[0088] The arithmetic expression is as follows:

[0089] M j =T equ (r j );

[0090] Among them, M j represents the calculation result of the jth mathematical expression in the reasoning task; r j represents the response value of the large language model to the jth mathematical expression in the reasoning task; T equ Represents the expression function of the expression extractor.

[0091] Optionally, the candidate generation module is specifically used to:

[0092] Calculating the similarity between the question information in the reasoning task and each question in the preset training sample set through the sentence converter;

[0093] Selecting the question with the highest similarity and the true answer corresponding to the question from the training set;

[0094] According to the question with the highest similarity and the real answer corresponding to the question, the positive thinking process information corresponding to the question information is obtained through a positive sample generator;

[0095] According to the question with the highest similarity and the positive thinking process information, the incorrect thinking process information corresponding to the question information is obtained through a negative sample generator;

[0096] Combining the positive thinking process information and the incorrect thinking process information into thinking process information;

[0097] The expression corresponding to the positive sample generator is as follows:

[0098] s pos =δ p (q sim ,a sim ,c sim );

[0099] Among them, s pos Represents the positive sequence thinking process information output by the positive sample generator; δ p Represents the expression function of the positive sample generator; q sim Indicates the question with the highest similarity; a sim Indicates the true answer corresponding to the question with the highest similarity; c sim Hint information indicating image information in reasoning tasks;

[0100] The expression corresponding to the negative sample generator is as follows:

[0101] s neg =δ n (q sim ,a sim ,c sim );

[0102] Among them, s neg Represents the wrong thought process information output by the negative sample generator; δ n Represents the expression function of the negative sample generator.

[0103] Optionally, a training module is also included to:

[0104] Randomly initializing parameters of the external model;

[0105] Using each question in the training set and the candidate answers corresponding to each question as input data of training data;

[0106] Using the true answers to the questions as output data of training data;

[0107] The external model is trained based on the input data and the output data.

[0108] Optional, self-assessment module specifically for:

[0109] Inputting the candidate answer as an input item to the external model for self-evaluation multiple times, and outputting the best answer of the candidate answer under each self-evaluation;

[0110] According to the best answer of the candidate answer in each evaluation, the number of occurrences of each best answer is recorded;

[0111] The optimal answer with the highest number of occurrences is used as the target answer corresponding to the reasoning task.

[0112] In another aspect, the present invention further provides an electronic device, comprising: at least one processor and a memory; the memory and the processor are connected via a bus;

[0113] The memory is used to store one or more programs;

[0114] When the one or more programs are executed by the at least one processor, a method for enhancing combinatorial reasoning capability based on the use of a large language model tool as described above is implemented.

[0115] On the other hand, the present invention also provides a computer-readable storage medium having an execution program stored thereon, which, when executed, implements the aforementioned method for enhancing combinatorial reasoning capabilities based on the use of a large language model tool.

[0116] Compared with the prior art, the present invention has the following beneficial effects:

[0117] The present invention provides a method for enhancing the combined reasoning ability based on the use of a large language model tool, comprising: obtaining a reasoning task for a question to be answered; according to the reasoning task, using a large language model, selecting and planning an enhanced tool corresponding to the reasoning task in a preset tool application set; using the large language model to call and execute the enhanced tool to generate a candidate answer; using a preset external model to self-evaluate the candidate answer to obtain a target answer corresponding to the reasoning task; wherein the tool application set includes at least one or more of the following enhanced tools: an image information processor, a formula extractor, a comparison sample constructor, a web page knowledge extractor, and a parameter knowledge generator; the present application enables the large language model to adaptively call different tools by adaptively selecting enhanced tools; through answer self-evaluation, the accuracy and reliability of the decision-making process of the large language model can be improved; the present invention is conducive to improving the combined reasoning ability of the large language model by combining adaptive tool selection and answer self-evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0118] Figure 1 A flow chart of a combined reasoning capability enhancement method based on the use of a large language model tool provided by the present invention;

[0119] Figure 2 A schematic diagram of the overall architecture of a combined reasoning capability enhancement method based on the use of a large language model tool provided by the present invention;

[0120] Figure 3 The number of enhanced tool calls in different question-answering scenarios by a combined reasoning capability enhancement method based on the use of a large language model tool provided by the present invention;

[0121] Figure 4A schematic diagram of the structure of a combined reasoning capability enhancement system based on a large language model tool provided by the present invention;

[0122] Figure 5 The present invention provides a schematic structural diagram of an electronic device. DETAILED DESCRIPTION

[0123] The present invention proposes a method and system for enhancing combinatorial reasoning capabilities based on the use of a large language model tool. The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0124] Embodiment 1:

[0125] A method for enhancing the combined reasoning capability based on the use of large language model tools. The flowchart is as follows: Figure 1 As shown, including:

[0126] Step 1: Obtain the reasoning task of the question to be answered;

[0127] Step 2: Based on the reasoning task, use the large language model to select and plan the enhancement tool corresponding to the reasoning task from the preset tool application set;

[0128] Step 3: Use the large language model to call and execute the enhancement tool to generate candidate answers;

[0129] Step 4: Use the preset external model to self-evaluate the candidate answers and obtain the target answer corresponding to the reasoning task;

[0130] Existing large language models are mainly optimized for unimodal tasks, that is, they mainly process text data, and these models are usually pre-trained on large-scale text data sets to obtain powerful text understanding and generation capabilities. For multimodal tasks (such as image understanding, video analysis, etc.), these models often require additional components or specific training to process non-text data; this also leads to the lack of a unified framework in existing technical solutions to comprehensively handle unimodal and multimodal reasoning tasks. For example, in the Chameleon research project, the focus is mainly on multimodal and scientific question answering tasks when combined with tools. These methods usually lack comprehensive coverage of other reasoning tasks. Therefore, in response to the above-mentioned problems, the present invention develops a method for diversified tool learning by combining the use of targeted tools, adopts a unified framework to handle unimodal reasoning tasks and multimodal reasoning tasks, alleviates some existing shortcomings of large language models, and improves combinatorial reasoning capabilities.

[0131] For example, the reasoning task in step 1 above may include: a multimodal reasoning task or a unimodal reasoning task;

[0132] For example, the above-mentioned multimodal reasoning task may include: question information and additional information;

[0133] For example, the additional information may include one or more of the following: image information, table information, and prompt information;

[0134] In one implementation, the process of selecting and planning an enhanced tool corresponding to the reasoning task in a preset tool application set based on the reasoning task and using a large language model in step 2 may include:

[0135] According to the reasoning task, the question-answering scenario corresponding to the reasoning task is obtained;

[0136] Based on the question-answering scenarios corresponding to the reasoning tasks, and using the large language model, we select and plan enhanced tools for reasoning tasks in different question-answering scenarios from the preset tool application set.

[0137] For example, the above tool application set includes at least one or more of the following enhanced tools: image information processor, formula extractor, comparison sample constructor, web page knowledge extractor and parameter knowledge generator;

[0138] For example, the above-mentioned question-answering scenarios may include: multimodal science question-answering scenarios, multimodal mathematics question-answering scenarios, and unimodal complex mathematics question-answering scenarios;

[0139] For example, the enhancement tools for the above-mentioned multimodal reasoning task in the multimodal scientific question-answering scenario may include one or more of the following: an image information processor, a parameter knowledge generator, a web page knowledge acquirer, and a comparison sample constructor;

[0140] For example, the enhancement tools for the above-mentioned multimodal reasoning task in the multimodal math question-answering scenario may include one or more of the following: a parameter knowledge acquirer, a program generator, a program verifier, a program executor, and a data filter;

[0141] For example, the enhancement tools for the above-mentioned unimodal reasoning task in the unimodal complex mathematical question-answering scenario may include one or more of the following: a comparison sample constructor, a formula extractor, a parameter knowledge generator, and a web page knowledge acquirer;

[0142] By adopting different processing strategies for different question-answering scenarios and selecting more targeted enhancement tools, the large language model can flexibly adjust its processing methods according to specific scenarios to cope with more diverse question types, and can significantly reduce unnecessary computing resource consumption and improve the overall performance of the model.

[0143] In one implementation, the process of calling and executing the enhancement tool using the large language model in step 3 to generate candidate answers may include:

[0144] When the reasoning task contains image information, the large language model is used to call and execute the image information processor to obtain prompt information of the image information;

[0145] When the reasoning task contains mathematical expressions, the large language model is used to call and execute the formula extractor to obtain the calculation result of the mathematical expression;

[0146] When the reasoning task contains question information, the large language model is used to call and execute the comparison sample constructor to obtain the thinking process information of the question information; the large language model is used to call and execute the web page knowledge extractor to obtain the web page knowledge information of the question information; the large language model is used to call and execute the parameter knowledge generator to obtain the parameter knowledge information of the question information;

[0147] Candidate answers are generated based on the prompt information of the image information, the calculation results of the mathematical expressions, the thinking process information of the question information, the web page knowledge information and the parameter knowledge information.

[0148] In this implementation, the present invention addresses the lack of targeted tool design in the prior art, such as using Python interpreters and integrated databases to acquire knowledge, and generally relies on direct off-the-shelf tools. These methods lack the specialized design required to address the specific weaknesses of large language models, thereby limiting the effectiveness of large language models in processing complex tasks. Therefore, in this implementation, the present invention is conducive to enhancing the reasoning ability of LLM when processing complex tasks through targeted designed tools.

[0149] In one implementation, the process of using the large language model to call and execute the image information processor to obtain prompt information of the image information may include:

[0150] Use large language models to detect object attribute information in image information;

[0151] Use image processing expressions to process object attribute information and obtain prompt information of image information;

[0152] For example, the above-mentioned object attribute information may include: an object and attributes corresponding to the object;

[0153] For example, the above image processing expression is as follows:

[0154] c=T image (e i ,ppt i );

[0155] Among them, e i represents the i-th image information; c i Indicates image information i Tips for:i Indicates prompt information c i Object attribute information of T image An expression function representing an image information processor.

[0156] In this implementation, for the image information processor, the execution flow mainly includes two steps. First, for the image to be detected, a prompt template containing task definition, guidelines and examples is designed; wherein the task definition specifies object attribute detection, which involves detecting and summarizing the attributes of the object; the guideline provides the model with directions to follow; and the example provides complete examples of objects and attributes to help the model better understand the task. For example, if the image contains a bear, then the bear is the object, and the attributes include a series of features such as color (e.g., black and brown), size, etc. Further, taking the large language model: GPT-4Vision as an example, the model is provided with hints through object attribute information, which is used as an image information processor to better identify key information in the image; in this way, the performance of the model in image understanding tasks can be improved, especially when it is necessary to understand the attributes of objects in the image. With the given prompt template, the model can better identify the key information in the image and perform corresponding operations according to the prompt, thereby achieving higher accuracy and practicality. This scheme can not only be used for image understanding, but also can be combined with other modalities (such as text) to provide support for more complex multimodal tasks.

[0157] In one implementation, the process of using the large language model to call and execute the formula extractor to obtain the calculation result of the mathematical expression may include:

[0158] Generate response values ​​of mathematical expressions using large language models;

[0159] Based on the response value, the mathematical expression is calculated using the arithmetic expression to obtain the calculation result of the mathematical expression;

[0160] For example, the above expression is as follows:

[0161] M j =T equ (r j );

[0162] Among them, M j represents the calculation result of the jth mathematical expression in the reasoning task; r j R represents the response value of the large language model to the jth mathematical expression in the reasoning task; equ Represents the expression function of the expression extractor.

[0163] In this implementation, the existing large language model is limited by the knowledge received during pre-training and lacks the ability to obtain updated information. In addition, since the training set of the large language model is often a general knowledge data set, it often shows deficiencies in professional fields. For example, LLM performs well in basic mathematical operations, but its processing ability decreases when processing more complex functions (such as division, power, logarithm and trigonometry). Therefore, the present invention uses a given question (for example, the i-th question information adopts q i ) and additional information (which may include images, prompts or other relevant data, such as the additional information of the i-th question information using e i ) to generate the LLM response (for example, the response of the j-th mathematical expression is represented by r j Next, the expression function T of the formula extractor equ From r j Extract all mathematical expressions and recalculate them using the calculator module to obtain the correct results. The calculation results are represented by M j , the above method can show more efficient performance when processing more complex functions.

[0164] In one implementation, the process of using the large language model to call and execute the comparison sample constructor to obtain the thinking process information of the question information may include:

[0165] Through the sentence converter, the similarity between the question information in the reasoning task and the questions in the preset training sample set is calculated;

[0166] Select the most similar questions and their corresponding true answers from the training set;

[0167] Based on the questions with the highest similarity and the real answers corresponding to the questions, the positive thinking process information corresponding to the question information is obtained through the positive sample generator;

[0168] According to the questions with the highest similarity and the positive thinking process information, the wrong thinking process information corresponding to the question information is obtained through the negative sample generator;

[0169] Combine the positive thinking process information and the incorrect thinking process information into thinking process information;

[0170] In this implementation, by using a comparison sample builder, for each question, first, the most similar question is retrieved, and then a thought chain is developed from the comparison sample pairs. Specifically, this process includes the following three steps:

[0171] Step 1: Retrieve similar questions: For each question q in the test set test, use the sentence transformer to calculate the difference between the given question and the training set Q train ={q1,q2,...,q n}; where {q1,q2,...,q n} represents the question set in the training set, and n represents the total number of questions in the training set; the sentence transformer is used to embed sentences into high-dimensional vectors, and cosine similarity is used to effectively compare semantic similarity.

[0172] v test =SenTrans(q test )

[0173] v i =SenTrans(q i )

[0174] Among them, SenTrans represents the conversion function of the sentence converter; q test represents the problem item in the test set; v test represents the question item after being converted by the sentence converter; q i Indicates the i-th question to be answered; v i represents the i-th question after being converted by the sentence converter;

[0175] Then, the most similar question is selected from the test set according to the highest similarity score, expressed as follows:

[0176]

[0177] Among them, sim represents the similarity function between the question items in the test set after being converted by the sentence converter and the questions in the questions to be answered after being converted by the sentence converter; q pos Indicates similarity;

[0178] Step 2: Positive sample generation: After obtaining similar questions (for example, using q sim After that, by providing a positive sample generator (for example, using δ p ), which contains the question, its corresponding answer (for example, using a sim ) and the captions generated for the related images generated by the first submodule (e.g., using c sim The model generates a positive thinking process information based on the correct answer (for example, using s pos express),

[0179] For example, the expression corresponding to the above positive sample generator is as follows:

[0180] s pos =δ p (q sim,a sim ,c sim );

[0181] Among them, s pos Represents the positive sequence thinking process information output by the positive sample generator; δ p Represents the expression function of the positive sample generator; q sim Indicates the question with the highest similarity; a sim Indicates the true answer corresponding to the question with the highest similarity; c sim Hint information indicating image information in reasoning tasks;

[0182] Step 3: Negative sample generation: For the generation of negative sample pairs, given similar questions and the positive thinking process information generated in step 2, the task of the negative sample generator is to generate an incorrect thinking process information. This incorrect thinking process information is used to prompt the model to learn from similar questions how to avoid the incorrect reasoning steps of each given question;

[0183] For example, the expression corresponding to the above negative sample generator is as follows:

[0184] s neg =δ n (q sim ,a sim ,c sim );

[0185] Among them, s neg Represents the wrong thought process information output by the negative sample generator; δ n Represents the expression function of the negative sample generator.

[0186] In this implementation, the present invention calls and executes the comparison sample constructor to help the model learn correct and incorrect reasoning patterns more quickly by providing positive samples and negative samples, thereby accelerating the learning process of the model, and can automatically generate high-quality training data (positive samples and negative samples), thereby simplifying the work in the data preparation stage and making the training process more efficient.

[0187] In one implementation, the above process of using the large language model to call and execute the parameter knowledge generator to obtain the parameter knowledge information of the question information can provide background knowledge reference for the basic model by utilizing the parameterized knowledge of the LLM. Specifically, it allows the LLM to capture the basic knowledge points related to each question, summarize them, and then provide this comprehensive knowledge as context information to the basic model. This method aims to enhance the model's ability to handle specific queries by effectively utilizing background knowledge. The formalization of these processes is as follows: K i About the question information i Background knowledge setting. knowledgeThe expression function representing the parameter knowledge generator is:

[0188] K i =T knowledge (q i )

[0189] In this implementation, the parameter knowledge generator can capture basic knowledge points related to the question and summarize them into contextual information; this helps the model better understand the context of the question, thereby improving the accuracy of the answer. When the model is processing a specific query, it can access richer background knowledge, which helps the model perform deeper logical reasoning and association analysis.

[0190] In one implementation, the web knowledge extractor is called and executed using a large language model to obtain web knowledge information of question information. By accessing a wide range of online resources, from academic articles and industry reports to real-time news updates, the model's understanding of rapidly developing fields can be enriched. This process can be formalized as follows:

[0191] W i =T web (q i )

[0192] Among them, W i The network knowledge set representing the i-th question information; q i Indicates the i-th question information; T web Represents the expression function of the network knowledge acquirer.

[0193] In view of the problems that the existing large language model is rigid, inflexible and lacks targeted tools when calling external tools, the present invention adopts adaptive tool selection. Specifically, given a set of tools S T ={T image ,T sample ,T equ ,T knowledge ,T web} and question information q i , you can get the corresponding tool chain S ST The above process can be formalized as follows, where ATS is the expression function of adaptive tool selection:

[0194] S ST =ATS(S T ,q i )

[0195] Among them, T image Represents the expression function of the image information processor; T equ represents the expression function of the formula extractor; T knowledgerepresents the expression function of the parameter knowledge generator; T web represents the expression function of the network knowledge acquirer; T sample represents the expression function of the comparison sample constructor; S ST Represents a toolchain consisting of enhancement tools.

[0196] In the process of reasoning question answering through a large language model, the lack of widespread use of self-evaluation process tool calls leads to too long prompts passed to the LLM, which may prevent the LLM from effectively distinguishing and extracting useful information from large amounts of data, thereby increasing the uncertainty of generated answers and possibly increasing the error rate. To address this problem, the present invention adopts a pre-built external model to evaluate candidate answers containing uncertainty and selects the best answer to the question by majority voting.

[0197] In one implementation, the process of using a preset external model to self-evaluate the candidate answers in step 4 to obtain the target answer corresponding to the reasoning task may include:

[0198] The candidate answers are input as input items to the external model for self-evaluation multiple times, and the optimal answer of the candidate answers under each self-evaluation is output;

[0199] According to the best answer of each candidate answer in each evaluation, record the number of occurrences of each best answer;

[0200] The optimal answer with the highest number of occurrences is taken as the target answer corresponding to the reasoning task.

[0201] In this implementation, the external model in step 4 above may include the following training process:

[0202] Randomly initialize the parameters of the external model;

[0203] Each question in the training set and the candidate answers corresponding to each question are used as input data of the training data;

[0204] The true answer to each question is used as the output data of the training data;

[0205] Based on the input data and output data, the external model is trained.

[0206] In the process of self-evaluating candidate answers using external models, given a question q i and the selected toolchain S ST , the present invention applies a solution generator Sample many candidate answers S A ={a1,…,a m}, these answers can be formalized as follows:

[0207]

[0208] Furthermore, a self-evaluation method based on majority voting is adopted to select the answer with the most votes, which can be formalized as follows:

[0209]

[0210] Among them, q i Indicates the i-th question information; S A ={a1,…,a m} represents a subset of candidate answers; m represents the number of answers in the candidate answers, {a1,…,a m} represents each candidate answer, for example, a1 represents the first candidate answer, a m represents the mth candidate answer; a best is the target answer; It is a pre-trained external model; the present invention designs a voting prompt during the external model training process: given a question information q i and multiple candidate answers S A ={a1,…,a m}, requiring the model to first analyze the problem and then select the target answer from the candidate answers. This process is repeated multiple times, and the answer with the most votes is selected as the final answer. In addition, during the training of the external model, by randomly initializing the model parameters, the model can be prevented from falling into the local optimal solution, improve the performance of the model on unseen data, enable the model to find an effective weight distribution more quickly, and accelerate the convergence speed during the training process, especially when the candidate answers are input into the external model multiple times, increasing the certainty of generating the target answer.

[0211] The present invention aims to address the problem that the existing large language models are rigid and inflexible when calling external tools, which makes it impossible to effectively distinguish and extract useful information during the reasoning process, increases the uncertainty of generating answers, and leads to a high reasoning error rate. The present invention proposes a combined reasoning capability enhancement method based on the use of large language model tools. By integrating a unified framework of single-modal and multi-modal tasks, it can be generalized and applied in different fields, which is conducive to alleviating the various defects of large language models. It is also task-independent and mode-independent in the process of reasoning question and answering, which is conducive to improving the overall performance of LLM across professional fields. The method provided by the present invention aims to enhance the reasoning ability of LLM by calling a series of tools specially designed for specific weaknesses. This method not only makes up for the shortcomings of LLM in reasoning tasks, but also improves the ability of LLM to solve problems in multiple fields, such as Figure 2As shown in the figure, it consists of three main parts: the upper part of the figure is 5 targeted enhancement tools, which enhance the reasoning ability of LLM by creating special tools to solve the specific weaknesses of LLM; the lower part of the figure contains two parts: an adaptive selection tool and self-evaluation; the adaptive selection tool uses LLM's contextual learning to enhance the ability to solve single-modal and multi-modal complex problems by autonomously forming a tool chain for various reasoning tasks; self-evaluation reduces the uncertainty of answer generation by enabling LLM to self-evaluate and compare the consistency and rationality of multiple inference answers. When dealing with a large amount of different background knowledge, LLM is more likely to make mistakes due to incorrect use or failure to comprehensively consider all information. Assuming that the correct reasoning process, even if the reasoning results are different, the results tend to converge more consistently to the final answer compared with the incorrect process. Based on this, the present invention designs and implements a self-evaluation method to identify inconsistencies and errors in its initial reasoning, thereby improving the overall quality and credibility of its conclusions, which can not only improve the accuracy and reliability of LLM responses, but also contribute to the overall robustness and transparency of the model decision-making process. Compared with previous tool learning methods, this invention innovatively targets the current weaknesses of the logical process involved in large model reasoning, and specifically designs a variety of tools. When faced with complex reasoning problems, these tools effectively address the shortcomings of large models. At the same time, unlike traditional methods that enhance reasoning capabilities through tool calls, our method integrates a self-evaluation mechanism. This new addition introduces an internal evaluation mechanism and an external model, which significantly reduces the uncertainty in model reasoning.

[0212] Embodiment 2:

[0213] A method for enhancing the combined reasoning ability based on the use of a large language model tool proposed by the present invention is described with a specific embodiment. Improving the combined reasoning ability of a large model through diversified tool learning can be specifically applied to multimodal question answering and unimodal mathematical reasoning in professional fields. In the specific implementation, the basic model selected is gpt-3.5-turbo, and the configuration temperature is 0.0°C to ensure the certainty of the output. In addition, GPT4-vision is used as an image information processor during the reasoning process, and llama-3-8b is used as a positive sample constructor and a negative sample constructor for the comparison sample constructor. For the formula extractor, Python code is used to extract mathematical expressions from the answers generated by GPT-3.5-turbo. These expressions are then calculated using a Python calculator, and the results are returned to GPT-3.5-turbo for rethinking. In addition, gpt-3.5-turbo is used as a parameter knowledge generator. For the self-evaluation stage, GPT-3.5-turbo is used as an LLM-based voter, and the candidate answers are set to 3, which means that three candidate answers are generated for each question for voting, and finally accuracy is used as an evaluation indicator of the experimental results, indicating the final accuracy. The following describes the process of the present invention for three problem scenarios:

[0214] 1. Multimodal scientific question answering

[0215] ScienceQA is the first multimodal science question-answering dataset, containing 21,208 questions and 10,332 images, with an average of 12.1 questions per image. Each image contains up to 141 questions, and the dataset covers grades 1 to 12. Compared with previous datasets, ScienceQA covers a variety of topics such as natural sciences, social sciences, and language sciences. In addition, most questions are accompanied by relevant lectures and detailed explanations to provide a more comprehensive learning experience.

[0216] In the adaptive toolchain construction reasoning part, the model autonomously selects submodules and toolchains suitable for the ScienceQA dataset. These tools include: image information processor, comparison sample constructor, web page knowledge acquirer, and parameter knowledge generator, as shown in Table 1:

[0217] Table 1 Adaptive selection tools

[0218]

[0219] (II) Multimodal Mathematics Question and Answer

[0220] MathQA is a large dataset designed to facilitate the study of math word problems. It consists of 37,200 math word problems, each with corresponding multiple-choice options and operations. This dataset spans a range of domains, including geometry, physics, probability, profit and loss, general mathematics, etc.

[0221] In the adaptive toolchain construction reasoning part, similar to the multimodal science question answering (ScienceQA dataset), the tools that can be selected by MathQA include: image information processor, comparison sample constructor, web page knowledge acquirer, and parameter knowledge generator, as shown in Table 1.

[0222] Furthermore, the call ratios of different tools in different scenarios are studied, as shown in Table 2. For the multimodal dataset MathVista, the image information processor is the most commonly used tool, selected by 92.9% of the questions. In contrast, the network knowledge extractor is the least used, selected by only 12.4% of the questions. In the ScienceQA dataset, the contrast sample constructor becomes the most popular tool, used in 98.3% of the questions. Interestingly, despite the multimodal nature of the task, only 36.3% of the questions selected the image information processor. This shows that how to better utilize image information when answering questions is a direction worth exploring in the future.

[0223] 3. Unimodal mathematical reasoning

[0224] MathQA is a large dataset consisting of 37,200 math word questions, each with corresponding multiple-choice options and operations. This dataset spans a range of fields, including geometry, physics, probability, profit and loss, general mathematics, etc.

[0225] As can be seen from Table 2, for the specialized math question answering dataset MathQA, all questions use the formula extractor, which emphasizes its importance in dealing with math problems. In addition, the comparison sample constructor is also frequently used (95.2%), while the network knowledge extractor is the least selected (7.2%).

[0226] Overall, web knowledge extractors are the least desired tool across all tasks, while contrast example constructors are consistently popular. These findings highlight the different requirements of different scenarios and the unique challenges that require a different set of tools to effectively solve each type of problem.

[0227] Table 2 Tool call percentage

[0228]

[0229] Furthermore, the number of tool choices in different scenarios was studied, e.g. Figure 3As shown in Figure 2, the distribution of tool counts on three different datasets is examined. Among all three types of tasks, the number of problems that require 4 tools or 0 tools is the least, almost 0. For the datasets Mathvista and ScienceQA, most of the problems are solved with two tools. In contrast, for the unimodal mathematical reasoning dataset MathQA, more than 70% of the problems can be solved using only one tool. From the analysis of the tool distribution part, we can infer that this tool is an equation extractor.

[0230] The present invention proposes a combined reasoning capability enhancement method based on the use of large language model tools, aiming to solve complex multimodal and unimodal problems in various fields. This method enhances the reasoning ability of LLM by calling a series of specially designed tools for specific weaknesses. This method not only makes up for the shortcomings of LLM in reasoning tasks, but also improves the ability of LLM to solve problems in multiple fields. In addition, a self-assessment mechanism is combined to significantly reduce the uncertainty of the reasoning results. At the same time, it covers three scenarios: multimodal scientific question-answering, multimodal mathematical problems, and unimodal complex mathematical reasoning, and can quickly match various simple or complex problems in practical applications. Experimental results show that the present invention is superior to competitive benchmarks and state-of-the-art technologies, and can also improve the reasoning ability of LLM to solve complex mathematical and scientific problems.

[0231] Embodiment 3:

[0232] The present invention based on the same inventive concept also provides a combined reasoning capability enhancement system based on the use of a large language model tool, the structural composition diagram is as follows Figure 4 As shown, including:

[0233] Task acquisition module, used to obtain reasoning tasks for questions to be answered;

[0234] The tool calling module is used to select and plan the enhanced tool corresponding to the reasoning task in the preset tool application set based on the reasoning task and using the large language model;

[0235] The candidate generation module is used to call and execute the enhancement tool using the large language model to generate candidate answers;

[0236] A self-evaluation module is used to self-evaluate candidate answers using a preset external model to obtain the target answer corresponding to the reasoning task;

[0237] Among them, the tool application set in the tool calling module includes at least one or more of the following enhanced tools: image information processor, formula extractor, comparison sample constructor, web page knowledge extractor and parameter knowledge generator.

[0238] In a possible implementation, the tool calling module is specifically used to:

[0239] According to the reasoning task, the question-answering scenario corresponding to the reasoning task is obtained;

[0240] Based on the question-answering scenarios corresponding to the reasoning tasks, and using the large language model, we select and plan enhanced tools for reasoning tasks in different question-answering scenarios from the preset tool application set.

[0241] In a possible implementation, the reasoning task includes: a multimodal reasoning task or a unimodal reasoning task; the multimodal reasoning task includes: question information and additional information; the additional information includes one or more of the following: image information, table information, and prompt information;

[0242] The question-answering scenarios include: multimodal science question-answering scenario, multimodal mathematics question-answering scenario, and single-modal complex mathematics question-answering scenario;

[0243] The enhancement tools for multimodal reasoning tasks in multimodal scientific question-answering scenarios include one or more of the following: image information processor, parameter knowledge generator, web page knowledge acquirer, and comparison sample constructor;

[0244] The enhancement tools for multimodal reasoning tasks in the multimodal math question answering scenario include one or more of the following: parameter knowledge acquirer, program generator, program verifier, program executor, and data filter;

[0245] The enhancement tools for unimodal reasoning tasks in unimodal complex mathematical question-answering scenarios include one or more of the following: a comparison sample constructor, a formula extractor, a parameter knowledge generator, and a web page knowledge acquirer.

[0246] In a possible implementation, the candidate generation module is specifically used to:

[0247] When the reasoning task contains image information, the large language model is used to call and execute the image information processor to obtain prompt information of the image information;

[0248] When the reasoning task contains mathematical expressions, the large language model is used to call and execute the formula extractor to obtain the calculation result of the mathematical expression;

[0249] When the reasoning task contains question information, the large language model is used to call and execute the comparison sample constructor to obtain the thinking process information of the question information; the large language model is used to call and execute the web page knowledge extractor to obtain the web page knowledge information of the question information; the large language model is used to call and execute the parameter knowledge generator to obtain the parameter knowledge information of the question information;

[0250] Candidate answers are generated based on the prompt information of the image information, the calculation results of the mathematical expressions, the thinking process information of the question information, the web page knowledge information and the parameter knowledge information.

[0251] In a possible implementation, the candidate generation module is specifically used to:

[0252] Use large language models to detect object attribute information in image information;

[0253] Use image processing expressions to process object attribute information and obtain prompt information of image information;

[0254] The object attribute information includes: the object and the attributes corresponding to the object;

[0255] The image processing expression is as follows:

[0256] c i =T image (e i ,ppt i );

[0257] Among them, e i represents the i-th image information; c i Indicates image information i Tips for: i Indicates prompt information c i Object attribute information of T image An expression function representing an image information processor.

[0258] In a possible implementation, the candidate generation module is specifically used to:

[0259] Generate response values ​​of mathematical expressions using large language models;

[0260] Based on the response value, the mathematical expression is calculated using the arithmetic expression to obtain the calculation result of the mathematical expression;

[0261] The calculation expression is as follows:

[0262] M j =T equ (r j );

[0263] Among them, M j represents the calculation result of the jth mathematical expression in the reasoning task; r j R represents the response value of the large language model to the jth mathematical expression in the reasoning task; equ Represents the expression function of the expression extractor.

[0264] In a possible implementation, the candidate generation module is specifically used to:

[0265] Through the sentence converter, the similarity between the question information in the reasoning task and the questions in the preset training sample set is calculated;

[0266] Select the most similar questions and their corresponding true answers from the training set;

[0267] Based on the questions with the highest similarity and the real answers corresponding to the questions, the positive thinking process information corresponding to the question information is obtained through the positive sample generator;

[0268] According to the questions with the highest similarity and the positive thinking process information, the wrong thinking process information corresponding to the question information is obtained through the negative sample generator;

[0269] Combine the positive thinking process information and the incorrect thinking process information into thinking process information;

[0270] The expression corresponding to the positive sample generator is as follows:

[0271] s pos =δ p (q sim ,a sim ,c sim );

[0272] Among them, s pos Represents the positive sequence thinking process information output by the positive sample generator; δ p Represents the expression function of the positive sample generator; q sim Indicates the question with the highest similarity; a sim Indicates the true answer corresponding to the question with the highest similarity; c sim Hint information indicating image information in reasoning tasks;

[0273] The expression corresponding to the negative sample generator is as follows:

[0274] s neg =δ n (q sim ,a sim ,c sim );

[0275] Among them, s neg Represents the wrong thought process information output by the negative sample generator; δ n Represents the expression function of the negative sample generator.

[0276] In a possible implementation, a training module is further included, which is used to:

[0277] Randomly initialize the parameters of the external model;

[0278] Each question in the training set and the candidate answers corresponding to each question are used as input data of the training data;

[0279] The true answer to each question is used as the output data of the training data;

[0280] Based on the input data and output data, the external model is trained.

[0281] In a possible implementation, the self-assessment module is specifically configured to:

[0282] The candidate answers are input as input items to the external model for self-evaluation multiple times, and the optimal answer of the candidate answers under each self-evaluation is output;

[0283] According to the best answer of each candidate answer in each evaluation, record the number of occurrences of each best answer;

[0284] The optimal answer with the highest number of occurrences is taken as the target answer corresponding to the reasoning task.

[0285] Embodiment 4:

[0286] like Figure 5 As shown, the present invention also provides an electronic device, which may be a computer device, a single-chip device, an intelligent mobile device, etc. The electronic device in this embodiment may include a processor, a memory, a transceiver component, etc. The memory, the processor, and the transceiver component are connected via a bus; the memory may be used to store an execution program, and an exemplary execution program may include instructions; the processor is used to execute the instructions stored in the memory. The memory may also be used to store data, which may be called and / or modified when the instructions are executed.

[0287] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in a storage medium to implement corresponding method flows or corresponding functions, so as to implement the steps of a method for enhancing combinatorial reasoning capabilities based on the use of a large language model tool in the above-mentioned embodiment.

[0288] Embodiment 4:

[0289] Based on the same inventive concept, the present invention also provides a readable storage medium, specifically an electronic device readable storage medium (Memory), which is a memory device in an electronic device for storing programs and data. It can be understood that the storage medium here can include both built-in storage media in electronic devices and, of course, extended storage media supported by electronic devices. The storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by a processor are also stored in the storage space, and these instructions can be one or more execution programs (including program codes). It should be noted that the storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage. The processor loads and executes one or more instructions stored in the storage medium, which can implement the steps of a method for enhancing the combinatorial reasoning capability based on the use of a large language model tool in the above embodiment.

[0290] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0291] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0292] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0293] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0294] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit its protection scope. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that after reading the present invention, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the application, but these changes, modifications or equivalent substitutions are all within the protection scope of the claims to be approved.

Claims

1. A method for enhancing combinatorial reasoning capability based on the use of a large language model tool, characterized in that: include: Obtain reasoning tasks for questions to be answered; According to the reasoning task, using the large language model, selecting and planning an enhanced tool corresponding to the reasoning task from a preset tool application set; Using the large language model to call and execute the enhancement tool to generate candidate answers; Using a preset external model to self-evaluate the candidate answers to obtain a target answer corresponding to the reasoning task; The tool application set includes at least one or more of the following enhanced tools: an image information processor, an equation extractor, a comparison sample constructor, a web page knowledge extractor, and a parameter knowledge generator.

2. The method according to claim 1, characterized in that According to the reasoning task, using the large language model, selecting and planning an enhancement tool corresponding to the reasoning task in a preset tool application set, including: According to the reasoning task, obtaining a question-answering scenario corresponding to the reasoning task; Based on the question-answering scenario corresponding to the reasoning task, a large language model is utilized to select and plan enhanced tools for the reasoning task in different question-answering scenarios from a preset tool application set.

3. The method according to claim 2, characterized in that The reasoning task includes: a multimodal reasoning task or a unimodal reasoning task; the multimodal reasoning task includes: question information and additional information; the additional information includes one or more of the following: image information, table information and prompt information; The question-answering scenarios include: multimodal science question-answering scenarios, multimodal mathematics question-answering scenarios, and single-modal complex mathematics question-answering scenarios; The enhancement tools of the multimodal reasoning task in the multimodal scientific question-answering scenario include one or more of the following: an image information processor, a parameter knowledge generator, a web page knowledge acquirer, and a comparison sample constructor; The enhancement tool of the multimodal reasoning task in the multimodal math question-answering scenario includes one or more of the following: a parameter knowledge acquirer, a program generator, a program verifier, a program executor, and a data filter; The enhancement tools for the unimodal reasoning task in the unimodal complex mathematical question-answering scenario include one or more of the following: a comparison sample constructor, a formula extractor, a parameter knowledge generator, and a web page knowledge acquirer.

4. The method according to claim 1, characterized in that The step of calling and executing the enhancement tool using the large language model to generate candidate answers includes: When the reasoning task includes image information, the image information processor is called and executed by using the large language model to obtain prompt information of the image information; When the reasoning task includes a mathematical expression, the formula extractor is called and executed using the large language model to obtain a calculation result of the mathematical expression; When the reasoning task includes question information, the comparison sample constructor is called and executed by the large language model to obtain the thinking process information of the question information; the web page knowledge extractor is called and executed by the large language model to obtain the web page knowledge information of the question information; the parameter knowledge generator is called and executed by the large language model to obtain the parameter knowledge information of the question information; Candidate answers are generated according to the prompt information of the image information, the calculation result of the mathematical expression, the thinking process information of the question information, the web page knowledge information and the parameter knowledge information.

5. The method according to claim 4, characterized in that The step of using the large language model to call and execute the image information processor to obtain prompt information of the image information includes: Detecting object attribute information in the image information using a large language model; Using an image processing expression to process the object attribute information, to obtain prompt information of the image information; The object attribute information includes: an object and attributes corresponding to the object; The image processing expression is as follows: c i =T image (e i ,ppt i ); Among them, e i represents the i-th image information; c i Indicates image information i Tips for: i Indicates prompt information c i Object attribute information of T image An expression function representing an image information processor.

6. The method according to claim 4, characterized in that The using the large language model to call and execute the formula extractor to obtain the calculation result of the mathematical expression includes: Using the large language model, generating a response value of the mathematical expression; Based on the response value, the mathematical expression is calculated using an arithmetic expression to obtain a calculation result of the mathematical expression; The arithmetic expression is as follows: M j =T equ (r j ); Among them, M j represents the calculation result of the jth mathematical expression in the reasoning task; r j represents the response value of the large language model to the jth mathematical expression in the reasoning task; T equ Represents the expression function of the expression extractor.

7. The method according to claim 4, characterized in that The step of using the large language model to call and execute the comparison sample constructor to obtain the thought process information of the question information includes: Calculating the similarity between the question information in the reasoning task and each question in the preset training sample set through the sentence converter; Selecting the question with the highest similarity and the true answer corresponding to the question from the training set; According to the question with the highest similarity and the true answer corresponding to the question, the positive thinking process information corresponding to the question information is obtained through a positive sample generator; According to the question with the highest similarity and the positive thinking process information, the incorrect thinking process information corresponding to the question information is obtained through a negative sample generator; Combining the positive thinking process information and the incorrect thinking process information into thinking process information; The expression corresponding to the positive sample generator is as follows: s pos =δ p (q sim ,a sim ,c sim ); Among them, s pos Represents the positive sequence thinking process information output by the positive sample generator; δ p Represents the expression function of the positive sample generator; q sim Indicates the question with the highest similarity; a sim Indicates the true answer corresponding to the question with the highest similarity; c sim Hint information indicating image information in reasoning tasks; The expression corresponding to the negative sample generator is as follows: s neg =δ n (q sim ,a sim ,c sim ); Among them, s neg Represents the wrong thought process information output by the negative sample generator; δ n Represents the expression function of the negative sample generator.

8. The method according to claim 7, characterized in that The external model includes the following training process: Randomly initializing parameters of the external model; Using each question in the training set and the candidate answers corresponding to each question as input data of training data; Using the true answers to the questions as output data of training data; The external model is trained based on the input data and the output data.

9. The method according to claim 8, characterized in that The self-evaluation of the candidate answers by using a preset external model to obtain a target answer corresponding to the reasoning task includes: Inputting the candidate answer as an input item to the external model for self-evaluation multiple times, and outputting the best answer of the candidate answer under each self-evaluation; According to the best answer of the candidate answer in each evaluation, the number of occurrences of each best answer is recorded; The optimal answer with the highest number of occurrences is used as the target answer corresponding to the reasoning task.

10. A system for enhancing combinatorial reasoning capabilities based on the use of a large language model tool, characterized in that: include: Task acquisition module, used to obtain reasoning tasks for questions to be answered; A tool calling module, used to select and plan an enhanced tool corresponding to the reasoning task in a preset tool application set according to the reasoning task and using a large language model; A candidate generation module, configured to use the large language model to call and execute the enhancement tool to generate candidate answers; A self-evaluation module, used to use a preset external model to self-evaluate the candidate answers to obtain a target answer corresponding to the reasoning task; The tool application set in the tool calling module includes at least one or more of the following enhanced tools: an image information processor, an equation extractor, a comparison sample constructor, a web page knowledge extractor, and a parameter knowledge generator.

11. The system according to claim 10, characterized in that The tool calling module is specifically used for: According to the reasoning task, obtaining a question-answering scenario corresponding to the reasoning task; Based on the question-answering scenario corresponding to the reasoning task, a large language model is utilized to select and plan enhanced tools for the reasoning task in different question-answering scenarios from a preset tool application set.

12. The system according to claim 10, characterized in that The candidate generation module is specifically used for: When the reasoning task includes image information, the image information processor is called and executed by using the large language model to obtain prompt information of the image information; When the reasoning task includes a mathematical expression, the formula extractor is called and executed using the large language model to obtain a calculation result of the mathematical expression; When the reasoning task includes question information, the comparison sample constructor is called and executed by the large language model to obtain the thinking process information of the question information; the web page knowledge extractor is called and executed by the large language model to obtain the web page knowledge information of the question information; the parameter knowledge generator is called and executed by the large language model to obtain the parameter knowledge information of the question information; Candidate answers are generated according to the prompt information of the image information, the calculation result of the mathematical expression, the thinking process information of the question information, the web page knowledge information and the parameter knowledge information.

13. An electronic device, characterized in that: include: at least one processor and memory; The memory and the processor are connected via a bus; The memory is used to store one or more programs; When the one or more programs are executed by the at least one processor, a method for enhancing combinatorial reasoning capability based on the use of a large language model tool as described in any one of claims 1 to 9 is implemented.

14. A computing device readable storage medium, characterized in that: An execution program is stored thereon, and when the execution program is executed, a method for enhancing combinatorial reasoning capability based on the use of a large language model tool as described in any one of claims 1 to 9 is implemented.

Images