LLM processing method and processing system for multi-hop problem and / or symbol reasoning problem

Abstract

The invention discloses an LLM processing method and system for a multi-hop problem and / or a symbol reasoning problem, and belongs to the technical field of LLM. The LLM processing method for the multi-hop problem and / or the symbol reasoning problem comprises the following steps that the multi-hop problem and / or the symbol reasoning problem are / is input, and n processing stages of the multi-hop problem and / or the symbol reasoning problem are formulated through a large language model LLM; obtaining a first temporary reasoning answer, a second temporary reasoning answer,..., an (n-2) th temporary reasoning answer and an nth reasoning plan; and outputting answers to the multi-hop question and / or the symbol reasoning question according to the multi-hop question and / or the symbol reasoning question, the first temporary reasoning answer, the second temporary reasoning answer,..., the (n-2) th temporary reasoning answer and the nth reasoning plan. According to the method, the accuracy of answering complex questions by the large language model is improved, the reasoning processing capability of the existing LLM is improved, the method is not limited to a certain specific language model, and the application scene of the large language model is greatly enriched.

Description

Technical Field

[0001] The present invention belongs to the field of LLM technology, and in particular relates to an LLM processing method and a processing system for multi-hop problems and / or symbolic reasoning problems. Background Art

[0002] In today's big model era, big language models have achieved remarkable results in various NLP (Natural Language Processing, NLP) tasks, making applications based on large language models LLM (Large Language Model, LLM) begin to flourish, such as assisted programming, psychological counseling, defect detection, etc.

[0003] Although large language models have demonstrated excellent capabilities in natural language processing tasks, it is difficult to fine-tune LLM to supplement its missing knowledge to adapt to downstream applications. This is because LLM has a high demand for high-quality training data. Typically, in order for these models to perform well on various tasks, researchers need to provide millions or even billions of labeled text fragments as training materials. However, obtaining high-quality datasets of such scale and diversity is a daunting task, which not only consumes a lot of resources, but also may lead to privacy and copyright issues. On the basis of obtaining high-quality data, it is still difficult to supplement the missing knowledge for LLM, because incorrect training methods often lead to catastrophic forgetting problems, causing the model to forget previously learned knowledge. In addition, since language usage in the real world is extremely complex and constantly changing, knowledge is updated quickly, which means that even after the missing knowledge is supplemented, the problem of missing knowledge may be faced again. In addition to the problem of lack of knowledge, LLM also faces the problems of insufficient reasoning ability and unclear reasoning trajectory. Moreover, LLM often produces wrong answers when answering some professional questions or questions related to the latest knowledge that are not included in its training data. Traditional methods directly use the original questions entered by the user for retrieval. Such retrieval methods cannot retrieve useful documents, especially for multi-hop problems and / or symbolic reasoning problems. However, a large proportion of real-life problems are multi-hop problems and / or symbolic reasoning problems. Therefore, the problem of insufficient reasoning ability also limits the application of LLM.

[0004] At present, there are technologies represented by CoT (Chain of Thought, CoT for short, such as Few-shot-CoT) and RAG (Retrieval Augmented Generation, RAG for short). CoT technology adopts the prompting method. CoT technology provides the model with a reasoning demonstration example, and generates reasoning steps based on a single call to the model to obtain the final answer. However, the generated reasoning trajectory does not match the generated answer, that is, the reasoning trajectory cannot deduce the answer. Although RAG technology can retrieve the question input by the user as a query and provide the obtained documents to LLM as supplementary knowledge to assist LLM in answering questions, in the face of complex multi-hop questions and / or symbolic reasoning problems, RAG cannot always retrieve useful documents, and the retrieved irrelevant documents will have an adverse effect on the answer. In summary, in view of the technical problems in the prior art that large model knowledge is difficult to update, the accuracy of large language models in answering multi-hop questions and / or symbolic reasoning problems is low, the model's reasoning ability is poor, and it is difficult or impossible to display the LLM reasoning trajectory well, it is urgent to provide an LLM processing solution for multi-hop problems and / or symbolic reasoning problems. Summary of the invention

[0005] The present invention application provides an LLM processing method and system for multi-hop problems and / or symbolic reasoning problems, aiming to partially or completely solve the technical problems of existing large language models in answering multi-hop problems and / or symbolic reasoning problems. The present invention application enables the large model to obtain the latest knowledge and industry knowledge in a timely manner, improves the accuracy of the large language model in answering multi-hop problems and / or symbolic reasoning problems, and is not limited to a specific large language model, which greatly enriches the application scenarios of the large language model. In order to achieve the above purpose, the present invention application adopts the following technical solutions:

[0006] In a first aspect, an LLM method for processing multi-hop problems and / or symbolic reasoning problems comprises the following steps:

[0007] Step S100: input a multi-hop problem and / or a symbolic reasoning problem, and use a large language model LLM to formulate n processing stages of the multi-hop problem and / or the symbolic reasoning problem, where n is a positive integer greater than or equal to 2;

[0008] Step S200: in the first processing stage, obtain the first reasoning plan and the first temporary reasoning answer TA1; in the second processing stage, obtain the second reasoning plan and the second temporary reasoning answer TA2, ..., in the n-1th processing stage, obtain the n-1th reasoning plan and the n-1th temporary reasoning answer TAn-1, and in the nth processing stage, obtain the nth reasoning plan;

[0009] Step S300: Output the answer to the multi-hop problem and / or the symbolic reasoning problem according to the multi-hop problem and / or the symbolic reasoning problem, the first temporary reasoning answer, the second temporary reasoning answer, ..., the n-2th temporary reasoning answer, and the nth reasoning plan.

[0010] Optionally, step S200 includes:

[0011] Step S201: In the first processing stage, reasoning and formulating a first reasoning plan, the first reasoning plan includes: a multi-hop question and / or a symbolic reasoning question Q, a question understanding definition QU, a first plan Plan and a first reasoning question SQ1; obtaining the first reasoning question SQ1, processing and determining a first temporary reasoning answer TA1 of the first reasoning question SQ1, thereby completing the acquisition of the first reasoning plan and the first temporary reasoning answer TA1;

[0012] Step S202: In the second processing stage, according to the first reasoning plan and the first temporary reasoning answer TA1, a second reasoning plan is formulated by reasoning, and the second reasoning plan includes: the problem understanding definition QU, the first temporary reasoning answer TA1, the second plan Plan and the second reasoning question SQ2; the second reasoning question SQ2 is obtained, and the second temporary reasoning answer TA2 of the second reasoning question is determined by processing, thereby completing the acquisition of the second reasoning plan and the second temporary reasoning answer TA2;

[0013] Step S203: In the third processing stage, according to the second reasoning plan and the second temporary reasoning answer TA2, the third plan is formulated by reasoning, and the third reasoning plan includes: the problem understanding definition QU, the second temporary reasoning answer TA2, the third plan Plan and the third reasoning question SQ3; the third reasoning question SQ3 is obtained, and the third reasoning temporary answer TA3 of the third reasoning question SQ3 is determined by processing, thereby completing the acquisition of the third reasoning plan and the third temporary reasoning answer TA3; ...,

[0014] Step S20n-1: In the n-1th processing stage, according to the n-2th temporary reasoning answer TAn-2 and the n-2th reasoning plan, the n-1th reasoning plan is formulated by reasoning, and the n-1th reasoning plan includes: the problem understanding definition QU, the n-2th temporary reasoning answer TAn-2, the n-1th plan Plan and the n-1th reasoning problem SQn-1; the n-1th reasoning problem SQn-1 is obtained, and the n-1th temporary reasoning answer TAn-1 of the n-1th reasoning problem SQn-1 is determined by processing;

[0015] Step S20n: In the nth processing stage, based on the n-1th reasoning plan and the n-1th temporary reasoning answer TAn-1, the nth reasoning plan is reasoned and formulated, and the nth reasoning plan includes: problem understanding definition QU, the n-1th temporary reasoning answer TAn-1, the nth plan Plan and the nth reasoning question, thereby completing the acquisition of the nth reasoning plan.

[0016] Optionally, in step S200, for multi-hop problems, the first reasoning problem SQ1, the second reasoning problem SQ2, ..., the nth reasoning problem SQn all adopt the SQ format, and the SQ format is: {"subject": the subject involved in that step of the processing stage to be executed, "question": the question generated based on the knowledge required in that step of the processing stage to be executed}; and / or, in step S200, for symbolic reasoning problems, the first reasoning problem SQ1, the second reasoning problem SQ2, ..., the nth reasoning problem SQn all adopt the Q format, and the Q format is: {"question": the question generated based on the knowledge required in that step of the processing stage to be executed}.

[0017] Optionally, in reasoning questions in SQ format, the subject can be used as a restriction condition. The question is vectorized and then retrieved as a query in the vector database Chroma. The retrieved documents and corresponding questions are input into the large language model, and the generated answer is the temporary reasoning answer.

[0018] Optionally, the first plan, the second plan, ..., the nth plan all adopt the step format, and the step format of the first plan includes: I need to know [knowledge of subject] by [subject], the step format of the second plan, ..., the nth plan includes: I need to know the [knowledge of the subject involved in last step's knowledge] by [subject involved in last step's knowledge], wherein [subject] represents the subject involved in the plan P n; [knowledge of subject] represents the knowledge that should be obtained for reasoning in our plan; [knowledge of the subject involved in last step's knowledge] is the temporary reasoning answer of the previous plan that the plan needs to obtain; the length of the temporary reasoning answer is not less than 20 characters and not more than 50 characters.

[0019] Optionally, in step S200, in the first processing stage, the first reasoning plan is displayed, in the second processing stage, the second reasoning plan is displayed, ..., in the n-1th processing stage, the n-1th reasoning plan is displayed, and in the nth processing stage, the nth reasoning plan is displayed; and / or, in step S200, when the reasoning formulation of the first reasoning plan, the second reasoning plan, ..., the n-1th reasoning plan, and the nth reasoning plan is formulated, a first prompt instruction is displayed, and the first prompt instruction is: Pay attention to the nouns that appear in the question as they represent the subjects of the knowledge you desire to obtain; when the reasoning formulation of the first reasoning plan is adjusted to the second reasoning plan, when the reasoning formulation of the second reasoning plan is adjusted to the third reasoning plan, ..., when the reasoning formulation of the n-1th reasoning plan is adjusted to the nth reasoning plan, a second prompt instruction is displayed during the adjustment process, and the second prompt instruction is: rendering the next step becomes the current step after performing current step

[0020] In a second aspect, an LLM processing system for multi-hop problems and / or symbolic reasoning problems is used to implement the LLM processing method for multi-hop problems and / or symbolic reasoning problems described in any one of the first aspects, including:

[0021] A formulation unit, for inputting a multi-hop problem and / or a symbolic reasoning problem, and formulating n processing stages of the multi-hop problem and / or the symbolic reasoning problem using a large language model LLM, where n is a positive integer greater than or equal to 2;

[0022] The plan processing unit obtains the first reasoning plan and the first temporary reasoning answer TA1 in the first processing stage; obtains the second reasoning plan and the second temporary reasoning answer TA2 in the second processing stage, ..., obtains the n-1th reasoning plan and the n-1th temporary reasoning answer TAn-1 in the n-1th processing stage, and obtains the nth reasoning plan in the n-th processing stage;

[0023] An answer output unit is used to output answers to multi-hop problems and / or symbolic reasoning problems based on the multi-hop problems and / or symbolic reasoning problems, the first temporary reasoning answer, the second temporary reasoning answer, ..., the n-2th temporary reasoning answer, and the nth reasoning plan.

[0024] Optionally, the LLM processing system for multi-hop problems and / or symbolic reasoning problems also includes: a display unit, which displays the first reasoning plan of the first processing stage, the second reasoning plan of the second processing stage,..., the n-1th reasoning plan of the n-1th processing stage, and the nth reasoning plan of the nth processing stage; the display unit is also used to display a first prompt instruction and a second instruction; the first prompt instruction is: Pay attention to the nouns that appear in the question as they represent the subjects of the knowledge you desire to obtain; the second prompt instruction is: rendering the next step becomes the current step after performing current step.

[0025] In a third aspect, the present invention application provides an LLM processing device for multi-hop problems and / or symbolic reasoning problems, comprising a memory and a processor that are communicatively connected, wherein the memory is used to store a computer program, and the processor is used to read the computer program and execute the LLM processing method for multi-hop problems and / or symbolic reasoning problems as described in any one of the first aspects.

[0026] In a fourth aspect, the present invention application provides a computer-readable storage medium having instructions stored thereon. When the instructions are executed on a computer, the LLM processing method for multi-hop problems and / or symbolic reasoning problems as described in any one of the first aspects is executed.

[0027] (1) In the present invention, the LLM processing method for multi-hop problems and / or symbolic reasoning problems includes n processing stages, each of which adopts a fixed format to maintain compactness, and the adjacent front and rear processing stages and the overall processing stage are logically compact. The n processing stages can intuitively and clearly describe the reasoning trajectory of the LLM, so that when the LLM answer fails, we can clearly distinguish the reason for the failure of the answer; the visualized reasoning trajectory also allows LLM to be applied in situations with higher security requirements (such as industrial application scenarios), and the n processing stages are conducive to understanding the LLM processing process and the reasons for the output results, so that it is also possible to judge whether the output results of the LLM processing are reasonable.

[0028] (2) In the present invention application, the LLM processing method for multi-hop problems and / or symbolic reasoning problems adopts a question-answering approach to extract knowledge. The questions generated in the LLM processing stage can be retrieved, and the subject involved in each reasoning question is used as a restriction condition. The generated answer is regarded as a temporary answer (also called extracted knowledge). The retrieved documents and the generated questions can also be input into the LLM, which significantly improves the hit rate of the original question retrieval. The temporary reasoning answer generated in the previous processing stage will be provided to the next processing stage, which alleviates the adverse effect of the retrieved irrelevant documents on the final answer of the multi-hop problem and / or symbolic reasoning problem, and alleviates the adverse effect of the LLM's lack of relevant knowledge on the final answer of the multi-hop problem and / or symbolic reasoning problem.

[0029] (3) In the present invention application, the LLM processing method for multi-hop problems and / or symbolic reasoning problems is applicable to problems that require reasoning. Complex multi-hop problems and / or symbolic reasoning problems are divided into simple problems to reduce the difficulty of the problems. The final answers are generated by answering simple questions one by one, thereby improving the accuracy of the large language model in answering complex questions. From the test results, multi-hop problems and / or symbolic reasoning problems and symbolic reasoning problems can produce good reasoning effects, improving the reasoning ability of the existing LLM, not limited to a specific language model, and are particularly suitable for large prediction models with a small number of parameters, which greatly enriches the application scenarios of large language models. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0031] Figure 1 A flowchart of the LLM processing method for multi-hop problems and / or symbolic reasoning problems applied by the present invention;

[0032] Figure 2 Schematic diagram of the LLM processing system for multi-hop problems and / or symbolic reasoning problems applied by the present invention Figure 1 ;

[0033] Figure 3 Schematic diagram of the LLM processing system for multi-hop problems and / or symbolic reasoning problems applied by the present invention Figure 2 ;

[0034] Figure 4 Schematic diagram of the LLM processing method for a multi-hop problem applied by the present invention Figure 1 ;

[0035] Figure 5 Schematic diagram of the LLM processing method for a multi-hop problem applied by the present invention Figure 2 ;

[0036] Figure 6 Schematic diagram of the LLM processing method for a multi-hop problem applied by the present invention Figure 3 ;

[0037] Figure 7 Schematic diagram of the LLM method for the two-symbol reasoning problem applied by the present invention Figure 1 .

[0038] Figure 8 Schematic diagram of the LLM method for the two-symbol reasoning problem applied by the present invention Figure 2 .

[0039] Fig. 9 A schematic diagram of the LLM processing method for multi-hop problems and / or symbolic reasoning problems applied by the present invention.

[0040] Fig.10 A schematic diagram showing the comparison of experimental answer results for the multi-hop problem applied for in the present invention after being processed using the CoT technology, RAG technology, and TDB technology on GLM-4, Qwen-turbo, and GPT-3.5-Turbo.

[0041] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments; based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0043] In the description of the present invention, it should be understood that the terms "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inside", "outside" and the like indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as a limitation on the present invention. In order to make the purpose, technical solution and advantages of the present invention clearer, the implementation methods of the present invention will be further described in detail below in conjunction with the drawings.

[0044] LLM approach to multi-hop problems and / or symbolic reasoning problems

[0045] like Figure 1 As shown, in a first aspect, an LLM processing method for multi-hop problems and / or symbolic reasoning problems comprises the following steps:

[0046] Step S100: input a multi-hop problem and / or a symbolic reasoning problem, and use a large language model LLM to formulate n processing stages of the multi-hop problem and / or the symbolic reasoning problem, where n is a positive integer greater than or equal to 2;

[0047] Step S200: in the first processing stage, obtain the first reasoning plan and the first temporary reasoning answer TA1; in the second processing stage, obtain the second reasoning plan and the second temporary reasoning answer TA2, ..., in the n-1th processing stage, obtain the n-1th reasoning plan and the n-1th temporary reasoning answer TAn-1, and in the nth processing stage, obtain the nth reasoning plan;

[0048] Specifically, step S200 includes:

[0049] Step S201: In the first processing stage, reasoning and formulating a first reasoning plan, the first reasoning plan includes: a multi-hop question and / or a symbolic reasoning question Q, a question understanding definition QU, a first plan Plan and a first reasoning question SQ1; obtaining the first reasoning question SQ1, processing and determining a first temporary reasoning answer TA1 of the first reasoning question SQ1, thereby completing the acquisition of the first reasoning plan and the first temporary reasoning answer TA1;

[0050] Step S202: In the second processing stage, according to the first reasoning plan and the first temporary reasoning answer TA1, a second reasoning plan is formulated by reasoning, and the second reasoning plan includes: the problem understanding definition QU, the first temporary reasoning answer TA1, the second plan Plan and the second reasoning question SQ2; the second reasoning question SQ2 is obtained, and the second temporary reasoning answer TA2 of the second reasoning question is determined by processing, thereby completing the acquisition of the second reasoning plan and the second temporary reasoning answer TA2;

[0051] Step S203: In the third processing stage, according to the second reasoning plan and the second temporary reasoning answer TA2, the third plan is formulated by reasoning, and the third reasoning plan includes: the problem understanding definition QU, the second temporary reasoning answer TA2, the third plan Plan and the third reasoning question SQ3; the third reasoning question SQ3 is obtained, and the third reasoning temporary answer TA3 of the third reasoning question SQ3 is determined by processing, thereby completing the acquisition of the third reasoning plan and the third temporary reasoning answer TA3; ...,

[0052] Step S20n-1: in the n-1th processing stage, according to the n-2th temporary reasoning answer TAn-2 and the n-2th reasoning plan, reasoning and formulating the n-1th reasoning plan, the n-1th reasoning plan includes: problem understanding definition QU, the n-2th temporary reasoning answer TAn-2, the n-1th plan Plan and the n-1th reasoning problem SQn-1; obtaining the n-1th reasoning problem SQn-1, processing and determining the n-1th temporary reasoning answer TAn-1 of the n-1th reasoning problem SQn-1;

[0053] Step S20n: In the nth processing stage, based on the n-1th reasoning plan and the n-1th temporary reasoning answer TAn-1, the nth reasoning plan is reasoned and formulated, and the nth reasoning plan includes: problem understanding definition QU, the n-1th temporary reasoning answer TAn-1, the nth plan Plan and the nth reasoning question, thereby completing the acquisition of the nth reasoning plan.

[0054] Step S300: Output the answer to the multi-hop problem and / or the symbolic reasoning problem according to the multi-hop problem and / or the symbolic reasoning problem, the first temporary reasoning answer, the second temporary reasoning answer, ..., the n-2th temporary reasoning answer, and the nth reasoning plan.

[0055] In some embodiments, the multi-hop problem may include a two-hop problem, a three-hop problem, a four-hop problem, a six-hop problem, an eight-hop problem, a ten-hop problem, etc. The present invention does not impose any particular restrictions on the specific type of the multi-hop problem, and technicians can choose to obtain the multi-hop problem and / or the symbolic reasoning problem according to actual conditions.

[0056] In some embodiments, by defining the question understanding QU (Question understanding, referred to as QU) setting, it is possible to avoid the first reasoning plan, the second reasoning plan, ..., the nth reasoning plan from misunderstanding the meaning of multi-hop questions and / or symbolic reasoning questions. This is because multi-hop questions and / or symbolic reasoning questions are relatively complex. If the LLM can correctly understand what the final output answer is related to, it can improve the accuracy of the answer based on the autoregressive language model.

[0057] In some embodiments, based on the multi-hop question and / or symbolic reasoning question, the 1st temporary reasoning answer, the 2nd temporary reasoning answer, ..., the n-2th temporary reasoning answer, and the nth reasoning plan, another large language model can be used for processing, so that the answer to the multi-hop question and / or symbolic reasoning question can be output.

[0058] In the LLM processing method for multi-hop problems and / or symbolic reasoning problems of the present invention, first, the multi-hop problems and / or symbolic reasoning problems are input, and n processing stages of the multi-hop problems and / or symbolic reasoning problems are formulated using the large language model LLM. The formulated n processing stages are closely related to each other and logically coherent; in the n-1 processing stages, each processing stage includes: a problem understanding definition QU, a plan Plan, a reasoning question SQ (Subject Question, abbreviated as SQ) and a temporary reasoning answer TA (Temporary Answer, abbreviated as TA), and each processing stage clearly indicates the plan Plan, reasoning question SQ and temporary reasoning answer TA involved; in addition, when facing a reasoning problem, a variety of processing methods (for example, retrieval methods) can be used to determine the temporary reasoning answer to the reasoning problem, and the required or generated reasoning problem is used as a query, so as to answer the retrieved documents. The generated temporary reasoning answer can also be called knowledge extraction, and then the next step of reasoning plan is formulated according to the temporary reasoning answer of the previous step, the reasoning plan of the previous step, and the problem understanding definition, that is, one processing stage includes: one plan, one reasoning question, and one The temporary answer is input until the nth processing stage, in which the original multi-hop question and / or symbolic reasoning question is input, the question understanding definition, the nth plan of the last time (i.e., the nth time), the 1st to n-1th temporary reasoning answers generated in the 1st processing stage to the nth processing stage, and then the large language model outputs the final answer (i.e., the answer processed by LLM for multi-hop question and / or symbolic reasoning question), which alleviates the adverse effects of LLM's lack of relevant knowledge and irrelevant documents on the answer, making LLM have different capabilities and avoiding LLM's direct use of the original question entered by the user for retrieval to cause wrong answers.

[0059] Optionally, in step S200, for multi-hop problems, the first reasoning problem SQ1, the second reasoning problem SQ2, ..., the nth reasoning problem SQn all adopt the SQ format, and the SQ format is: {"subject": the subject involved in that step of the processing stage to be executed, "question": the question generated based on the knowledge required in that step of the processing stage to be executed}; and / or, in step S200, for symbolic reasoning problems, the first reasoning problem SQ1, the second reasoning problem SQ2, ..., the nth reasoning problem SQn all adopt the Q format, and the Q format is: {"question": the question generated based on the knowledge required in that step of the processing stage to be executed}.

[0060] In some embodiments, for multi-hop problems, at least one of the first reasoning problem SQ1, the second reasoning problem SQ2, ..., and the nth reasoning problem SQn adopts the SQ format, and the SQ format is: {"subject": the subject involved in the step of the processing stage to be executed, "question": a question generated based on the knowledge required in the step of the processing stage to be executed}; and / or, for symbolic reasoning problems, the first reasoning problem SQ1, the second reasoning problem SQ2, ..., and the nth reasoning problem SQn all adopt the Q format, and the Q format is: {"question": a question generated based on the knowledge required in the step of the processing stage to be executed}. The reasoning problem SQi (i = 1, 2, ..., n, i is a positive integer) is set in the SQ format and / or the Q format, so that the fixed format of the reasoning problem can be clearly and uniformly set to maintain the closeness of the reasoning problem before and after, and the adjacent processing stages before and after and the overall processing stage are kept logically close, which also improves the rigor and processing efficiency of the reasoning problem, and facilitates the subsequent rapid retrieval of the reasoning problem to obtain a temporary reasoning answer.

[0061] Optionally, in reasoning questions in SQ format, the subject can be used as a restriction condition. The question is vectorized and then retrieved as a query in the vector database Chroma. The retrieved documents and corresponding questions are input into the large language model, and the generated answer is the temporary reasoning answer.

[0062] In some embodiments, vector retrieval requires a vector database, and Chroma is used as the vector database. During testing, the data source is the documents that come with these data sets. We use the embedding model to vectorize these documents and store them in the vector database. On this basis, using the subject as a restriction condition, filtering can be performed to ensure that the retrieved documents are definitely related to this subject, which can significantly improve the retrieval hit rate; after the question is vectorized, it can be used as a query to search in the vector database Chroma, and the corresponding retrieved documents and corresponding questions are input into the large language model (it should be noted that the large language model here may be different from the large language model LLM that specifies n processing stages), and the resulting answer is the temporary reasoning answer (i.e., the extracted knowledge);

[0063] In some embodiments, when performing a search, the Euclidean distance is used to calculate the relevance between the question and the document. That is, the question needs to be vectorized using the same embedding model, and then the distance is calculated with the vectorized document. Each time we search, we only take the first five documents with the smallest distance, because the number of documents related to each question in the test set will not exceed 5. In actual applications, the number of relevant documents required can be adjusted according to the number of relevant documents.

[0064] In the present invention application, when searching before executing a certain processing stage, the original question input by the user is not directly used for searching, but the question generated by the knowledge required for this processing stage is used for searching. The subjects involved in this processing stage are filtered as restriction conditions, which can greatly improve the hit rate of the search, and the question-and-answer method is used to obtain temporary reasoning answer extraction. The temporary reasoning answer generated by the previous processing stage will be provided to the next processing stage, which alleviates the adverse effects of the retrieved irrelevant documents on the final answers to multi-hop questions and / or symbolic reasoning questions.

[0065] Optionally, in step S200, the first plan Plan, the second plan Plan, ..., the nth plan Plan all adopt a step format, and the step format of the first plan Plan includes:

[0066] I need to know[knowledge of subject]by[subject],

[0067] The step formats of the second plan, ..., the nth plan include:

[0068] I need to know the[knowledge of the subject involved in last step's knowledge]by[subject involved in last step's knowledge],

[0069] Among them, [subject] represents the subject involved in the plan;

[0070] [knowledge of subject] indicates the knowledge that should be obtained for reasoning in our plan; [knowledge of the subject involved in last step's knowledge] is the temporary reasoning answer (also called extracted knowledge) of the last step's plan that the plan needs to obtain; the length of the temporary reasoning answer is not less than (can mean greater than or equal to) 20 characters and not more than (can mean less than or equal to) 50 characters.

[0071] In the present application, in step S200, the step format of the first plan includes: I need to know [knowledge of subject] by [subject], the step format of the second plan, ..., the nth plan includes: Need to know the [knowledge of the subject involved in last step's knowledge] by [subject involved in last step's knowledge], so that the format of the reasoning problem can be clearly and uniformly set to maintain the closeness between the second plan, ..., and the nth plan, so that the adjacent front and back processing stages and the overall processing stage are logically tight, and the rigor and processing efficiency of multiple plans are improved.

[0072] Optionally, in step S200, in the first processing stage, the first reasoning plan is displayed, in the second processing stage, the second reasoning plan is displayed, ..., in the n-1th processing stage, the n-1th reasoning plan is displayed, and in the nth processing stage, the nth reasoning plan is displayed.

[0073] In the present invention application, the overall n processing stages formulated by the large language model LLM for multi-hop problems and / or symbolic reasoning problems can intuitively and clearly visualize the reasoning trajectory of LLM, so that we can clearly distinguish the reasons for the failure of the answer when the LLM fails to answer; the visualized reasoning trajectory also allows LLM to be applied in situations with higher security requirements. The n processing stages are conducive to understanding the LLM processing process and the reasons for the output results, so that it is also possible to judge whether the output results of the LLM processing are reasonable.

[0074] Optionally, in step S200, when the first reasoning plan, the second reasoning plan, ..., the n-1th reasoning plan, and the nth reasoning plan are being reasoned, a first prompt instruction is displayed, and the first prompt instruction is: Pay attention to the nouns that appear in the question as they represent the subjects of the knowledge you desire to obtain; when the reasoning formulation of the first reasoning plan is adjusted to the second reasoning plan, when the reasoning formulation of the second reasoning plan is adjusted to the third reasoning plan, ..., when the reasoning formulation of the n-1th reasoning plan is adjusted to the nth reasoning plan, a second prompt instruction is displayed during the adjustment process, and the second prompt instruction is: rendering the next step becomes the current step after performing current step.

[0075] In some embodiments, when the first reasoning plan, the second reasoning plan, ..., the n-1th reasoning plan, and the nth reasoning plan are being formulated, a first prompt instruction may be set during the reasoning process. The first prompt instruction may be displayed or not. The setting of the first prompt instruction: Pay attention to the nouns that appear in the question as they represent the subjects of the knowledge you desire to obtain may facilitate the discovery of the subjects involved in the plan. In other embodiments, the first prompt instruction may also be: Pay attention to the nouns (eg, people, and places) that appear in the question as they represent the subjects of the knowledge you desire to obtain.

[0076] In the present invention application, when the first reasoning plan is adjusted to the second reasoning plan, when the second reasoning plan is adjusted to the third reasoning plan, ..., when the n-1 reasoning plan is adjusted to the nth reasoning plan, a second prompt instruction can be set during the adjustment process, and the second prompt instruction can be displayed or not displayed. The second prompt instruction is: rendering the next step becomes the current step after performing currentstep, which can facilitate the discovery of the adjustment from the first reasoning plan to the second reasoning plan, the adjustment from the second reasoning plan to the third reasoning plan, ..., the adjustment from the n-1 reasoning plan to the nth reasoning plan. On this basis, by displaying the first reasoning plan, the second reasoning plan, ..., the n-1 reasoning plan, the nth reasoning plan, the first prompt instruction, and the second prompt instruction, the situation of each processing stage of the multi-hop problem and / or the symbolic reasoning problem can be intuitively and conveniently demonstrated, and the reasoning trajectory of the LLM can be intuitively, clearly and visually described.

[0077] LLM system for multi-hop problems and / or symbolic reasoning problems

[0078] like Figure 2 and Figure 3 As shown, in a second aspect, an LLM processing system 100 for multi-hop problems and / or symbolic reasoning problems is used to implement the LLM processing method for multi-hop problems and / or symbolic reasoning problems described in any one of the first aspects, including:

[0079] It should be noted that the LLM processing system for multi-hop problems and / or symbolic reasoning problems applied for in the present invention is used to implement the LLM processing method for multi-hop problems and / or symbolic reasoning problems including any one of the first aspect, and accordingly also includes: all the recorded technical problems, technical solutions and technical effects of the LLM processing method for any one of the multi-hop problems and / or symbolic reasoning problems in the first aspect, which will not be repeated in the present invention application.

[0080] A formulation unit 101 is used for inputting a multi-hop problem and / or a symbolic reasoning problem, and formulating n processing stages of the multi-hop problem and / or the symbolic reasoning problem using a large language model LLM, where n is a positive integer greater than or equal to 2;

[0081] In some embodiments, after the formulation unit formulates n processing stages, n-1 iterations of processing are required before the multi-hop problem and / or symbolic reasoning problem is input and the answer to the multi-hop problem and / or symbolic reasoning problem is output. The number of processing stages n depends on the complexity of the problem input by the user. The more complex the input problem is, the more processing stages are required and the greater the number of iterations. By way of example, n can be greater than or equal to 3, n can be greater than or equal to 4, n can be greater than or equal to 5, and so on.

[0082] The plan processing unit 102 obtains the first reasoning plan and the first temporary reasoning answer TA1 in the first processing stage; obtains the second reasoning plan and the second temporary reasoning answer TA2 in the second processing stage, ..., obtains the n-1th reasoning plan and the n-1th temporary reasoning answer TAn-1 in the n-1th processing stage, and obtains the nth reasoning plan in the n-th processing stage;

[0083] In some embodiments, the large language model LLM of the formulation unit can also be understood as or referred to as the first large language model LLM1. Accordingly, the second largest language model LLM2 can be used to respectively infer and formulate the first reasoning plan, the second reasoning plan, ..., the n-1th reasoning plan and the nth reasoning plan; the third largest language model LLM3 can be used to respectively process and determine the first temporary reasoning answer TA1, the second temporary reasoning answer TA2, ..., the n-1th temporary reasoning answer TA1 and the nth temporary reasoning answer TA1. The second largest language model LLM2 and the third largest language model LLM3 cooperate with each other to eventually complete the acquisition of the first temporary reasoning answer, the second temporary reasoning answer, ..., the n-2th temporary reasoning answer and the nth reasoning plan.

[0084] The answer output unit 103 is used to output the answer to the multi-hop problem and / or the symbolic reasoning problem according to the multi-hop problem and / or the symbolic reasoning problem, the first temporary reasoning answer, the second temporary reasoning answer, ..., the n-2th temporary reasoning answer, and the nth reasoning plan.

[0085] In some embodiments, a fourth language model LLM4 may be used, which jointly outputs answers to multi-hop questions and / or symbolic reasoning questions based on the multi-hop questions and / or symbolic reasoning questions, the first temporary reasoning answers, the second temporary reasoning answers, ..., the n-2th temporary reasoning answers, and the nth reasoning plan.

[0086] In the LLM processing system for multi-hop problems and / or symbolic reasoning problems of the present invention, a formulation unit is used to formulate n processing stages of the multi-hop problems and / or symbolic reasoning problems using a large language model LLM according to the multi-hop problems and / or symbolic reasoning problems input by the user, and the formulated n processing stages are closely related to each other and logically coherent; a planning processing unit, in which each processing stage includes: a problem understanding definition QU, a plan Plan, a reasoning question SQ (Subject Question, abbreviated as SQ) and a temporary reasoning answer TA (Temporary Reasoning Answer TA) in n-1 processing stages Each processing stage clearly indicates the plan, reasoning question SQ and temporary reasoning answer TA involved. The temporary reasoning answer, reasoning plan and problem understanding definition of the previous step are used to formulate the reasoning plan of the next step, that is, one processing stage includes: one plan, one reasoning question, one temporary answer, until the last nth processing stage; the answer output unit inputs the original multi-hop question and / or symbolic reasoning question, the problem understanding definition, the nth plan of the last time (i.e., the nth time), the temporary reasoning answers from the 1st processing stage to the nth processing stage generated from the 1st processing stage to the nth processing stage, and then completes the large language model to output the final answer (i.e., the answer processed by LLM for multi-hop questions and / or symbolic reasoning questions), which alleviates the adverse effects of LLM's lack of relevant knowledge and irrelevant documents on the answer, so that LLM has different capabilities and avoids LLM directly using the original question input by the user for retrieval to cause incorrect answers.

[0087] Optionally, the LLM processing system for multi-hop problems and / or symbolic reasoning problems also includes: a display unit, displaying the first reasoning plan of the first processing stage, displaying the second reasoning plan of the second processing stage,..., the n-1th reasoning plan of the n-1th processing stage, and displaying the nth reasoning plan of the nth processing stage.

[0088] In the present application, the whole of the n processing stages formulated by the large language model LLM for multi-hop problems and / or symbolic reasoning problems can be visually described through the display unit to intuitively and clearly describe the reasoning trajectory of the LLM. The display unit allows us to clearly distinguish the reasons for the failure of the LLM answer when the LLM fails to answer; the visualized reasoning trajectory given by the display unit also allows LLM to be applied in situations with higher security requirements. The n processing stages are conducive to understanding the LLM processing process and the reasons for the output results, so that it is also possible to judge whether the output results of the LLM processing are reasonable.

[0089] Optionally, the display unit 104 is further used to display a first prompt instruction and a second instruction; the first prompt instruction is: Pay attention to the nouns that appear in the question as they represent the subjects of the knowledge you desire to obtain; the second prompt instruction is: rendering thenext step becomes the current step after performing current step.

[0090] In some embodiments, when the first reasoning plan, the second reasoning plan, ..., the n-1th reasoning plan, and the nth reasoning plan are formulated, a first prompt instruction may be set in the reasoning formulation process, and the first prompt instruction may be displayed or not. For example, the display unit 104 may display the first prompt instruction: Pay attention to the nouns that appear in the question as they represent the subjects of the knowledge you desire to obtain, which may facilitate the discovery of the subject involved in the plan;

[0091] In some embodiments, when the first reasoning plan is inferred to be adjusted to the second reasoning plan, when the second reasoning plan is inferred to be adjusted to the third reasoning plan,..., when the n-1th reasoning plan is inferred to be adjusted to the nth reasoning plan, a second prompt instruction can be set during the adjustment process, and the second prompt instruction can be displayed or not. Exemplarily, the display unit 104 can also provide a second prompt instruction at the same time. The second prompt instruction is: rendering the next step becomes the current step after performing current step, which can facilitate the discovery of the adjustment from the first reasoning plan to the second reasoning plan, the adjustment from the second reasoning plan to the third reasoning plan,..., the adjustment from the n-1th reasoning plan to the nth reasoning plan.

[0092] On this basis, the display unit can display the first reasoning plan, the second reasoning plan,..., the n-1th reasoning plan, the nth reasoning plan, the first prompt instruction, the second prompt instruction, and can intuitively and conveniently demonstrate the situation of each processing stage of multi-hop problems and / or symbolic reasoning problems, and intuitively and clearly visualize the reasoning trajectory of LLM and the problems in the processing process.

[0093] Examples of testing LLM approaches to multi-hop problems and / or symbolic reasoning problems

[0094] (1) Multi-hop problem

[0095] Taking the GLM-4 model as an example, a multi-hop question is: Q: What genre is the record label of the performer of So Long, See You Tomorrow associated with? The LLM processing system for multi-hop questions and / or symbolic reasoning problems is used to process the multi-hop question as follows:

[0096] like Figure 4 As shown, the first reasoning plan includes: multi-hop questions and / or symbolic reasoning questions, QU (i.e. Question understand), a 3-step first plan Plan and a first reasoning question SQ ("subject":"So long, See You Tomorrow","question":"Who is the performer of So Long, See You Tomorrow by SoLong, See You Tomorrow"), subject is used as a restriction condition, requiring that the document processed (for example, retrieved) must contain the string So Long, See You Tomorrow, the above question and document are input into the third language model LLM3 to generate an answer of no more than 50 characters.

[0097] like Figure 5 As shown, the first temporary reasoning answer generated by the third largest language model LLM3 can be used: The performer is Bombay Bicycle Club; Figure 6As shown, the second reasoning plan includes: QU (i.e., Question understand), the first temporary reasoning answer, the second plan Plan of step 2, and the second reasoning question SQ ("subject":"Bombay Bicycle Club","question":"What record The performer is label does Bob belong to?"). Obviously, the format of the first reasoning question and the second reasoning question is selected in the SQ format. The first temporary reasoning answer in the first reasoning plan and the first plan Plan of step 3 can be adjusted by the second language model LLM2 to formulate the first reasoning plan into the second reasoning plan, and so on, and the second temporary reasoning answer and the third reasoning plan can be obtained. Finally, the fourth language model LLM4 can output the final answer of the multi-hop question and / or symbolic reasoning question according to the multi-hop question and / or symbolic reasoning question, the first temporary reasoning answer, the second temporary reasoning answer, and the third reasoning plan. The final answer is: The genre is jaz.

[0098] (2) Symbolic reasoning problems

[0099] In this invention application, in addition to being applicable to real-world problems, the invention application can also be applied to symbolic reasoning problems. Whether symbolic reasoning problems can be solved well is an important factor in applying LLM to complex problems such as mathematics, because mathematical problems can often be transformed into symbolic reasoning problems. The two symbolic reasoning problems come from the Coin Flip and Last Letter Concatenation datasets, respectively. These two datasets are widely used to verify the symbolic reasoning ability of LLM.

[0100] Take the GLM-4 model as an example. Figure 7 and Figure 8 As shown in Figure 1, the first and second processing stages of the LLM approach to the symbolic reasoning problems Coin Flip and Last Letter Concatenation. Figure 7 Middle (a), Figure 8 As shown in (a), in the first and second processing stages of the LLM processing method for the symbolic reasoning problem Coin Flip, the format of the first and second reasoning problems is selected in Q format, and the third language model LLM3 can directly answer the answer to the first reasoning problem. Similarly, Figure 7 Middle (b), Figure 8As shown in (b), in the first and second processing stages of the LLM processing method for the symbolic reasoning problem Last Letter Concatenation, the formats of the first and second reasoning questions are selected in Q format, and the third language model LLM3 can also directly answer the answer to the first reasoning question.

[0101] (3) Multi-hop problem and symbolic reasoning problem

[0102] like Fig. 9 As shown, after using the LLM processing method and processing system of multi-hop problems and / or symbolic reasoning problems and a combination of symbolic reasoning problems, the final answers to the multi-hop problems and / or symbolic reasoning problems and symbolic reasoning problems can be directly obtained, and the final answers can be displayed to the user.

[0103] Therefore, in the present invention application, there is little difference between the processing processes of symbolic reasoning problems and multi-hop problems. The LLM processing method and processing system of multi-hop problems and / or symbolic reasoning problems have good universality. The LLM processing method and processing system of multi-hop problems and / or symbolic reasoning problems improve the accuracy of LLM in answering complex problems in the real world, especially multi-hop problems and / or symbolic reasoning problems, avoids the problems of catastrophic forgetting and unexplainable output caused by fine-tuning the model, and expands the application scope of LLM.

[0104] (4) Comparison of GLM-4, Qwen-turbo, and GPT-3.5-Turbo processing experiments for multi-hop problems

[0105] GLM-4 and Qwen-turbo are domestic models. They use different architectures and are trained on different data sets. GPT-3.5-turbo is the most widely used model. Taking the multi-hop problem as an example, the same test conditions were used to test GLM-4, Qwen-turbo and GPT-3.5-Turbo, respectively, using CoT technology, RAG technology, and TDB technology to output 9 test answer results.

[0106] like Fig.10As shown, from the results of the 9 test answers, we can see that all the 9 test answers are wrong and failed. The reason is that CoT technology, RAG technology, and TDB technology often encounter the problem of lack of relevant knowledge. RAG technology uses a retrieval method to provide relevant knowledge for LLM, but RAG technology only performs one retrieval before the start of reasoning, and cannot retrieve all documents that are helpful in answering questions, resulting in the failure to provide buffer information for LLM; the application of CoT technology on GPT-3.5-turbo will lead to the problem of misunderstanding the meaning of the question, resulting in the complete failure of GPT-3.5-turbo's reasoning analysis of multi-hop problems. However, the LLM processing method for multi-hop problems and / or symbolic reasoning problems applied by the present invention (named: MultistageRetrieval Method, hereinafter referred to as MRM method) produces the final answer result: eight, which is completely correct.

[0107] (5) Experiments on multi-hop dataset and symbol dataset processing

[0108] Taking multi-hop datasets: Musique, HotpotQA, 2WikiMultihopQA, and symbolic datasets: Coin Flip and Last Letter Concatenation as examples, these multi-hop datasets and symbolic datasets represent complex problems in the real world.

[0109] For multi-hop problems, the multi-hop data set can sample 2000 data respectively, and conduct experiments with GLM-4, Qwen-turbo and GPT-3.5-Turbo, using Direct prompting, TDB, Few-shot CoT, RAG and MRM methods respectively; for symbolic reasoning problems, the symbolic data set can sample 500 data respectively, using the GLM-4 model, and applying Direct prompting, TDB, Few-shot CoT, RAG without Retrieval, Zero-shot PS+, Self-consistency, Zero shot CoT and MRM methods.

[0110] Therefore, from the experimental results in Table 1 to Table 4, it can be seen that the LLM processing method for multi-hop problems and / or symbolic reasoning problems applied in the present invention significantly improves the accuracy of GLM-4, Qwen-turbo and GPT-3.5-Turbo in answering multi-hop problems, and the LLM processing method for multi-hop problems and / or symbolic reasoning problems applied in the present invention also shows the highest accuracy in symbolic reasoning problems (see Table 4), verifying that the LLM processing method for multi-hop problems and / or symbolic reasoning problems has good effectiveness and accuracy.

[0111] Table 1 Comparison of experimental results of different processing methods of Musique dataset on GLM-4, Qwen-turbo and GPT-3.5-Turbo

[0112] Treatment GPT-3.5-turbo Qwen-turbo GLM-4 Direct prompting 22.55％ 7.45％ 18.70％ TDB 22.30％ 6.55％ 18.85％ Few-shot CoT 24.90％ 9.20％ 22.35％ RAG 41.75％ 29.20％ 38.35％ MRM 50.25％ 51.60％ 58.15％

[0113] Table 2 Comparison of experimental results of different processing methods of HotpotQA dataset on GLM-4, Qwen-turbo and GPT-3.5-Turbo

[0114]

[0115]

[0116] Table 3 Comparison of experimental results of different processing methods on 2WikiMultihopQA dataset on GLM-4, Qwen-turbo and GPT-3.5-Turbo

[0117] Treatment GPT-3.5-turbo Qwen-turbo GLM-4 Direct prompting 33.00％ 12.85％ 30.55％ TDB 31.20％ 12.30％ 35.25％ Few-shot CoT 44.10％ 17.35％ 33.10％ RAG 40.95％ 25.30％ 40.45％ MRM 65.35％ 59.80％ 78.05％

[0118] Table 4 Comparison of experimental results of different processing methods on the symbolic reasoning dataset on GLM-4

[0119]

[0120] (4) Ablation test

[0121] Ablation experiments are conducted on multi-hop datasets: Musique, HotpotQA, and 2WikiMultihopQA to prove that each unit or part of the LLM processing system is indispensable for multi-hop problems and / or symbolic reasoning problems.

[0122] It can be seen from Table 5 that if the temporary reasoning answer is removed, the accuracy will be reduced. Removing the retrieval will also reduce the accuracy. The reason is that removing the temporary reasoning answer will cause the retrieved irrelevant documents to affect the output, and removing the retrieval cannot solve the problem of lack of knowledge. If the instruction (such as the first instruction and / or the second instruction) is removed, the accuracy will also be reduced. The reason is that outputting in a specific / specified format can facilitate the parsing of the intermediate output. The lack of instructions will increase the number of cases where the output cannot be parsed, which in turn leads to failure in the intermediate processing process.

[0123] Table 5 Ablation test results of Musique, HotpotQA, and 2WikiMultihopQA datasets

[0124] Dataset Remove temporary reasoning answers Remove Search Remove Instructions MRM Musique 52.00％ 28.67％ 46.67％ 54.67％ HotpotQA 66.67％ 46.67％ 76.67％ 81.33％ 2WikiMutihopQA 73.33％ 36.67％ 72.00％ 76.67％

[0125] LLM processing device for multi-hop problems and / or symbolic reasoning problems

[0126] In a third aspect, the present invention application provides an LLM processing device for multi-hop problems and / or symbolic reasoning problems, comprising a memory and a processor that are communicatively connected, wherein the memory is used to store a computer program, and the processor is used to read the computer program and execute the LLM processing method for multi-hop problems and / or symbolic reasoning problems as described in any one of the first aspects.

[0127] In some embodiments, the LLM processing device for multi-hop problems and / or symbolic reasoning problems includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in any of the LLM processing methods for multi-hop problems and / or symbolic reasoning problems in the first aspect are implemented.

[0128] Computer readable storage medium

[0129] In a fourth aspect, the present invention application provides a computer-readable storage medium having instructions stored thereon. When the instructions are executed on a computer, the LLM processing method for multi-hop problems and / or symbolic reasoning problems as described in any one of the first aspects is executed.

[0130] Those skilled in the art will understand that the schematic diagram is merely an example of a terminal device and does not constitute a limitation on the terminal device. The terminal device may include more or fewer components than shown in the diagram, or a combination of certain components, or different components. For example, the terminal device may also include input and output devices, network access devices, buses, etc.

[0131] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire terminal device.

[0132] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the terminal device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0133] Wherein, if the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0134] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.

[0135] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. An LLM processing method for multi-hop problems and / or symbolic reasoning problems, characterized in that: The following steps are involved: Step S100: input a multi-hop problem and / or a symbolic reasoning problem, and use a large language model LLM to formulate n processing stages of the multi-hop problem and / or the symbolic reasoning problem, where n is a positive integer greater than or equal to 2; Step S200: in the first processing stage, obtain the first reasoning plan and the first temporary reasoning answer TA1; in the second processing stage, obtain the second reasoning plan and the second temporary reasoning answer TA2, ..., in the n-1th processing stage, obtain the n-1th reasoning plan and the n-1th temporary reasoning answer TAn-1; in the nth processing stage, obtain the nth reasoning plan; Step S300: Output the answer to the multi-hop problem and / or the symbolic reasoning problem according to the multi-hop problem and / or the symbolic reasoning problem, the first temporary reasoning answer, the second temporary reasoning answer, ..., the n-2th temporary reasoning answer, and the nth reasoning plan.

2. The LLM processing method for multi-hop problems and / or symbolic reasoning problems according to claim 1, characterized in that: Step S200 includes: Step S201: In the first processing stage, reasoning and formulating a first reasoning plan, the first reasoning plan includes: a multi-hop question and / or a symbolic reasoning question Q, a question understanding definition QU, a first plan Plan and a first reasoning question SQ1; obtaining the first reasoning question SQ1, processing and determining a first temporary reasoning answer TA1 of the first reasoning question SQ1, thereby completing the acquisition of the first reasoning plan and the first temporary reasoning answer TA1; Step S202: In the second processing stage, according to the first reasoning plan and the first temporary reasoning answer TA1, a second reasoning plan is formulated by reasoning, and the second reasoning plan includes: the problem understanding definition QU, the first temporary reasoning answer TA1, the second plan Plan and the second reasoning question SQ2; the second reasoning question SQ2 is obtained, and the second temporary reasoning answer TA2 of the second reasoning question is determined by processing, thereby completing the acquisition of the second reasoning plan and the second temporary reasoning answer TA2; Step S203: In the third processing stage, according to the second reasoning plan and the second temporary reasoning answer TA2, the third plan is formulated by reasoning, and the third reasoning plan includes: the problem understanding definition QU, the second temporary reasoning answer TA2, the third plan Plan and the third reasoning question SQ3; the third reasoning question SQ3 is obtained, and the third reasoning temporary answer TA3 of the third reasoning question SQ3 is determined by processing, thereby completing the acquisition of the third reasoning plan and the third temporary reasoning answer TA3; ..., Step S20n-1: In the n-1th processing stage, according to the n-2th temporary reasoning answer TAn-2 and the n-2th reasoning plan, the n-1th reasoning plan is formulated by reasoning, and the n-1th reasoning plan includes: the problem understanding definition QU, the n-2th temporary reasoning answer TAn-2, the n-1th plan Plan and the n-1th reasoning problem SQn-1; the n-1th reasoning problem SQn-1 is obtained, and the n-1th temporary reasoning answer TAn-1 of the n-1th reasoning problem SQn-1 is determined by processing; Step S20n: In the nth processing stage, based on the n-1th reasoning plan and the n-1th temporary reasoning answer TAn-1, the nth reasoning plan is reasoned and formulated, and the nth reasoning plan includes: problem understanding definition QU, the n-1th temporary reasoning answer TAn-1, the nth plan Plan and the nth reasoning question, thereby completing the acquisition of the nth reasoning plan.

3. The LLM processing method for multi-hop problems and / or symbolic reasoning problems according to claim 2, characterized in that: In step S200, for multi-hop problems, the first reasoning problem SQ1, the second reasoning problem SQ2, ..., the nth reasoning problem SQn all adopt the SQ format, and the SQ format is: {"subject": the subject involved in that step of the processing stage to be executed, "question": the question generated based on the knowledge required in that step of the processing stage to be executed}; and / or, in step S200, for symbolic reasoning problems, the first reasoning problem SQ1, the second reasoning problem SQ2, ..., the nth reasoning problem SQn all adopt the Q format, and the Q format is: {"question": the question generated based on the knowledge required in that step of the processing stage to be executed}.

4. The LLM processing method for multi-hop problems and / or symbolic reasoning problems according to claim 3, characterized in that: In reasoning questions in the SQ format, the subject can be used as a restriction condition. The question is vectorized and used as a query to search in the vector database Chroma. The retrieved documents and corresponding questions are input into the large language model, and the generated answer is the temporary reasoning answer.

5. The LLM processing method for multi-hop problems and / or symbolic reasoning problems according to claim 2, characterized in that: The first plan, the second plan, ..., the nth plan all use the step format; The step format of Plan 1 includes: I need to know [knowledge of subject] by [subject]; The step format of Plan 2, ..., Plan n includes: I need to know the [knowledge of the subject involved in last step's knowledge] by [subject involved in last step's knowledge], where [subject] refers to the subject involved in the plan; [knowledge of subject] represents the knowledge that should be acquired for reasoning in our plan; [knowledge of the subject involved in last step's knowledge] is the temporary reasoning answer of the previous plan required for the plan; The length of the provisional reasoning answer should be no less than 20 characters and no more than 50 characters.

6. The LLM processing method for multi-hop problems and / or symbolic reasoning problems according to claim 5, characterized in that: In step S200, in the first processing stage, the first reasoning plan is displayed, in the second processing stage, the second reasoning plan is displayed, ..., in the n-1th processing stage, the n-1th reasoning plan is displayed, and in the nth processing stage, the nth reasoning plan is displayed; And / or, in step S200, when the first reasoning plan, the second reasoning plan, ..., the n-1th reasoning plan, and the nth reasoning plan are being reasoned, a first prompt instruction is displayed, and the first prompt instruction is: Pay attention to the nouns that appear in the question as they represent the subjects of the knowledge you desire to obtain; when the reasoning formulation of the first reasoning plan is adjusted to the second reasoning plan, when the reasoning formulation of the second reasoning plan is adjusted to the third reasoning plan, ..., when the reasoning formulation of the n-1th reasoning plan is adjusted to the nth reasoning plan, a second prompt instruction is displayed during the adjustment process, and the second prompt instruction is: rendering the next step becomes the current step after performing current step.

7. An LLM processing system for multi-hop problems and / or symbolic reasoning problems, used to implement the LLM processing method for multi-hop problems and / or symbolic reasoning problems according to any one of claims 1 to 6, characterized in that: include: A formulation unit, for inputting a multi-hop problem and / or a symbolic reasoning problem, and formulating n processing stages of the multi-hop problem and / or the symbolic reasoning problem using a large language model LLM, where n is a positive integer greater than or equal to 2; The plan processing unit obtains the first reasoning plan and the first temporary reasoning answer TA1 in the first processing stage; obtains the second reasoning plan and the second temporary reasoning answer TA2 in the second processing stage, ..., obtains the n-1th reasoning plan and the n-1th temporary reasoning answer TAn-1 in the n-1th processing stage, and obtains the nth reasoning plan in the n-th processing stage; An answer output unit is used to output answers to multi-hop problems and / or symbolic reasoning problems based on the multi-hop problems and / or symbolic reasoning problems, the first temporary reasoning answer, the second temporary reasoning answer, ..., the n-2th temporary reasoning answer, and the nth reasoning plan.

8. The LLM processing system for multi-hop problems and / or symbolic reasoning problems according to claim 7, characterized in that: The LLM processing system for multi-hop problems and / or symbolic reasoning problems further comprises: a display unit, the display unit displays a first reasoning plan for a first processing stage, a second reasoning plan for a second processing stage, ..., an n-1th reasoning plan for an n-1th processing stage, and an nth reasoning plan for an nth processing stage; The display unit is further used to display a first prompt instruction and a second instruction; the first prompt instruction is: Pay attention to the nouns that appear in the question as they represent the subjects of the knowledge you desire to obtain; the second prompt instruction is: rendering the next step becomes the current step after performing current step.

9. An LLM processing device for multi-hop problems and / or symbolic reasoning problems, characterized in that: It comprises a memory and a processor which are communicatively connected, wherein the memory is used to store a computer program, and the processor is used to read the computer program and execute the LLM processing method for the multi-hop problem and / or the symbolic reasoning problem as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the LLM processing method for multi-hop problems and / or symbolic reasoning problems described in any one of claims 1 to 6 is executed.

Images