Question and answer method, device and equipment based on large language model

Abstract

The invention provides a question answering method, device and equipment based on a large language model, and the method comprises the steps: determining a corresponding answering process based on the obtained question content, and determining a plurality of keywords and questions corresponding to a current stage; inputting the keyword into a keyword retrieval model to obtain a first retrieval result, inputting the question into a vector retrieval model to obtain a second retrieval result, and inputting the first retrieval result and the second retrieval result into a fine arrangement model to obtain a corresponding reference document; based on the question content, the answer process and the reference document, determining corresponding answers, and summarizing the answers to the generated answers; judging whether a subsequent stage in the reflection answering process needs to be adjusted or not, if so, adjusting, repeating the steps, and determining an answer corresponding to the next stage; and traversing each stage to obtain a corresponding answer, and taking the generated answer as a final answer corresponding to the question content. According to the method, the illusion problem of the LLM can be reduced, and the credibility of the final answer is improved.

Description

Technical Field

[0001] The present application relates to the field of computer data processing technology, and in particular to a question-answering technology based on a large language model. Background Art

[0002] Since the emergence of LLM (Large Language Model), the model may learn wrong information or false knowledge due to factors such as insufficient training data or low quality, or the model may have some unknown defects or biases that cause the model to produce inaccurate results during processing, or the model may learn some rules or patterns that do not match the actual situation during training, and over-generalize, which may cause LLM to have hallucination problems.

[0003] In order to improve the illusion problem of LLM, the general idea is to use RAG (Retrieval-Augmented Generation) technology, which splits the external knowledge text into different granularities to form a multi-angle summary, and stores it in the knowledge base in the form of a vector. LLM searches the knowledge base according to the questions entered by the user, obtains relevant knowledge information, and integrates it into LLM, and then generates the final answer to the corresponding question by LLM. Since RAG technology usually uses a single search before generating an answer, and there is no reflection process on the processing logic, when generating an answer, there may be insufficient retrieval content to support the answer details, or the overall logic of the processing is not strong, resulting in detailed errors in the generated answer, resulting in low credibility. Especially for some complex and large-scale questions that require model summary and understanding. Summary of the invention

[0004] The purpose of this application is to provide a question-answering method, device and equipment based on a large language model, so as to at least partially solve the technical problems in the prior art that the large language model has weak overall logic when generating answers to some questions, and the generated answers may have errors in details, resulting in low credibility.

[0005] According to an embodiment of one aspect of the present application, a question answering method based on a large language model is provided, wherein the method comprises:

[0006] Based on the obtained question content, determine an answer process corresponding to the question content, and based on the question content and the answer process, determine a number of keywords and a number of questions corresponding to the current stage of the answer process;

[0007] Input the plurality of keywords into a keyword search model to obtain a first search result, input the plurality of questions into a vector search model to obtain a second search result, and input the first search result and the second search result into a refined ranking model to obtain a reference document corresponding to the current stage;

[0008] Based on the question content, answer process and reference documents, determine the answer corresponding to the current stage and summarize it into the generated answer;

[0009] D. Based on the question content, answer process and generated answers, reflect whether the subsequent stages in the answer process need to be adjusted. If necessary, adjust them, and repeat the above steps for the next stage of the answer process to determine the answer corresponding to the next stage;

[0010] E traverses each stage of the answering process, repeats the above steps, obtains the corresponding answer for each stage, and takes the generated answer that summarizes the answers of the last stage as the final answer corresponding to the question content.

[0011] Optionally, the step of determining an answer process corresponding to the question content based on the obtained question content includes:

[0012] The obtained question content is input into the answer process prompting project in the large language model to obtain the answer process corresponding to the question content.

[0013] Optionally, the determining of a number of keywords and a number of questions corresponding to the current stage of the answering process based on the question content and the answering process includes:

[0014] The question content and the answer process are respectively input into the keyword generation prompt project and the question generation prompt project in the large language model to obtain a number of keywords and a number of questions corresponding to the current stage of the answer process.

[0015] Optionally, the step of inputting the plurality of keywords into a keyword search model to obtain a first search result comprises:

[0016] The plurality of keywords are input into a keyword search model, and a search is performed based on a preset knowledge base to obtain a first search result.

[0017] Optionally, the step of inputting the plurality of questions into a vector search model to obtain a second search result comprises:

[0018] The plurality of questions are input into a vector retrieval model, and a search is performed based on a preset knowledge base to obtain a second search result.

[0019] Optionally, the determining the answer corresponding to the current stage based on the question content, answer process and reference documents includes:

[0020] The question content, answer process and reference documents are input into the answer prompt project in the large language model to obtain the answer corresponding to the current stage.

[0021] Optionally, the reflecting on whether a subsequent stage in the answering process needs to be adjusted based on the question content, the answering process and the generated answer includes:

[0022] The question content, answer process and generated answer are input into the reflection prompt project in the large language model to reflect whether the subsequent stages in the answer process need to be adjusted.

[0023] Optionally, if a subsequent stage of the answering process needs to be adjusted, the method includes:

[0024] When the number of stages in the adjusted answer process meets the preset threshold, no more reflection is performed.

[0025] Optionally, the method comprises:

[0026] If the generated answer meets the preset conditions, no further reflection will be performed.

[0027] Optionally, the step of taking the generated answer that summarizes the answers in the last stage as the final answer corresponding to the question content includes:

[0028] A content redundancy removal operation is performed on the generated answer that summarizes the answers in the last stage, and the generated answer after redundancy removal is used as the final answer corresponding to the question content.

[0029] According to another embodiment of another aspect of the present application, a question-answering device based on a large language model is provided, the device comprising:

[0030] The first module is used to determine an answer process corresponding to the question content based on the obtained question content, and determine a number of keywords and a number of questions corresponding to the current stage of the answer process based on the question content and the answer process;

[0031] The second module is used to input the plurality of keywords into the keyword search model to obtain a first search result, and input the plurality of questions into the vector search model to obtain a second search result, and input the first search result and the second search result into the refined ranking model to obtain a reference document corresponding to the current stage;

[0032] The third module is used to determine the answer corresponding to the current stage based on the question content, answer process and reference documents, and summarize the generated answers;

[0033] The fourth module is used to reflect whether the subsequent stages of the answer process need to be adjusted based on the question content, the answer process and the generated answers. If necessary, the adjustments are made, and for the next stage of the answer process, the answers corresponding to the next stage are determined through the above-mentioned related modules;

[0034] The fifth module is used to obtain the answers corresponding to each stage, and summarize the generated answers in the last stage as the final answers corresponding to the question content.

[0035] Compared with the prior art, the present application provides a question-answering method based on a large language model, characterized in that the method includes: based on the acquired question content, determining the answer process corresponding to the question content, and based on the question content and the answer process, determining several keywords and several questions corresponding to the current stage of the answer process; inputting the several keywords into the keyword retrieval model to obtain a first retrieval result, and inputting the several questions into the vector retrieval model to obtain a second retrieval result, and inputting the first retrieval result and the second retrieval result into the refined ranking model to obtain a reference document corresponding to the current stage; based on the question content, the answer process and the reference document, determining the answer corresponding to the current stage, and summarizing it into a generated answer; based on the question content, the answer process and the generated answer, reflecting on whether the subsequent stages in the answer process need to be adjusted, and if necessary, adjusting it, and repeating the above steps for the next stage of the answer process to determine the answer corresponding to the next stage; traversing each stage of the answer process, repeating the above steps to obtain the answer corresponding to each stage, and taking the generated answer that summarizes the answers of the last stage as the final answer corresponding to the question content. Through this method, the answer process for the question is determined, and the keywords and extended questions related to the question are searched for each stage of the answer process, and the reference documents and answers for each stage are obtained, which are summarized into the final answer. The search part of LLM is used to enhance the search, and the generation part of LLM is combined with the answers generated up to the current stage to reflect on the answer process. The overall logic of the generated answer can be enhanced, and the detailed errors of the final answer can be reduced, thereby improving the credibility of the final answer. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0037] Figure 1 A schematic diagram showing a flow chart of a question-answering method based on a large language model according to one aspect of the present application;

[0038] Figure 2 A schematic diagram of a question-answering device based on a large language model according to another aspect of the present application is shown;

[0039] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0040] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0041] In a typical configuration of each embodiment of the present application, each device, each trusted party of the system and / or each module of the apparatus may include one or more processors (CPU), input / output interface, network interface and memory, as well as related software to ensure that related functions are implemented.

[0042] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0043] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0044] In the prior art, in order to improve or even solve the illusion problem of LLM, RAG technology is usually used. By splitting the external knowledge text into different granularities to form a multi-angle summary, the summary is stored in the knowledge base in the form of a vector. LLM searches the knowledge base according to the question input by the user, obtains relevant knowledge information (for example, obtains several most similar knowledge text fragments that meet the conditions), and integrates them into LLM, and then LLM generates the final answer to the corresponding question. Since RAG technology usually uses a single search before generating an answer, and there is no reflection process for processing logic, when generating an answer, the search content may not be sufficient to support the details of the answer, or the overall logic of the processing is not strong, so that the generated answer has errors in details, resulting in low credibility and may not meet the requirements. Especially for some complex and long-length questions that require model summary and understanding.

[0045] In order to solve the above technical problems, the present application provides a question-answering method, device and equipment based on a large language model. The question-answering method first determines the answer process for the question, and then searches for keywords and extended questions related to the question for each stage of the answer process, obtains reference documents for each stage, and obtains answers for each stage, which are summarized into the final answer. After obtaining the answer for the current stage, the answer process is reflected on in combination with the generated answer to determine whether the subsequent stage needs to be adjusted. In the retrieval part of the LLM, multiple rounds of retrieval are performed according to the answer process to achieve retrieval enhancement. In the generation part of the LLM, the answer process is also reflected on to enhance the overall logic of the generated answer, which can improve the credibility of the final answer.

[0046] In this application, the device can be deployed in a computer device and / or terminal device that integrates LLM and related software and hardware environments, and the computer device includes but is not limited to a personal computer, a laptop, an industrial computer, a network host, a single network server, a set of multiple network servers, etc.; the terminal device includes but is not limited to a smart phone, a tablet computer, etc. Here, the computer device and / or terminal device are only examples, and other existing or future devices and / or resources that are applicable to this application should also be included in the scope of protection of this application, and are hereby included by reference.

[0047] In order to further illustrate the technical means adopted by the present application and the effects achieved, the technical solution of the present application is clearly and completely described below in combination with the accompanying drawings and preferred embodiments.

[0048] Figure 1 A schematic flow chart of a question-answering method based on a large language model according to one aspect of the present application is shown, wherein the method of one embodiment includes:

[0049] S101 determines an answering process corresponding to the question content based on the obtained question content, and determines a number of keywords and a number of questions corresponding to the current stage of the answering process based on the question content and the answering process;

[0050] S102: inputting the plurality of keywords into a keyword search model to obtain a first search result, inputting the plurality of questions into a vector search model to obtain a second search result, and inputting the first search result and the second search result into a refined ranking model to obtain a reference document corresponding to the current stage;

[0051] S103 determines the answer corresponding to the current stage based on the question content, answer process and reference documents, and summarizes the generated answers;

[0052] S104: Based on the question content, the answer process and the generated answer, reflect whether the subsequent stage in the answer process needs to be adjusted. If necessary, adjust it, and repeat the above steps for the next stage of the answer process to determine the answer corresponding to the next stage;

[0053] S105 traverses each stage of the answering process, repeats the above steps, obtains the answer corresponding to each stage, and uses the generated answer that summarizes the answers of the last stage as the final answer corresponding to the question content.

[0054] The method embodiment may be executed by device 100, wherein device 100 is a computer device and / or terminal device integrated with LLM and related software and hardware environment.

[0055] In this method embodiment, in step S101, the LLM in the device 100 can obtain the question content based on the text input by the user, determine the answer process corresponding to the question content based on the question content, and determine several keywords and several questions corresponding to the current stage of the answer process in combination with the question content and the answer process.

[0056] Optionally, in step S101, the process of determining an answer corresponding to the question content based on the obtained question content includes:

[0057] The obtained question content is input into the answer process prompting project in the large language model to obtain the answer process corresponding to the question content.

[0058] First, the LLM needs to think about how to answer the question content. The device 100 can input the obtained question content into the answer process prompt project in the LLM, let the LLM think about the process of answering the question content, and generate an answer process corresponding to the question content. The answer process prompt project may include input prompts (i.e., input prompts) and output prompts (i.e., input prompts). An exemplary template of the input prompt and output prompt of the answer process prompt project is as follows:

[0059] Input prompt:

[0060] ```

[0061] user:

[0062] The existing problems are as follows:

[0063] {Question content}

[0064] Which steps are needed to answer the above questions? Please refer to the following format:

[0065] 1. Historical background

[0066] 2. Narrative perspective 1

[0067] 3. Narrative perspective 2

[0068] 4. Summary

[0069] ```

[0070] Output prompt:

[0071] ```

[0072] 1. Historical background

[0073] 2. Narrative perspective 1

[0074] 3. Narrative perspective 2

[0075] 4. Summary

[0076] ```

[0077] In an optional embodiment, the answer process corresponding to the question content can also be determined based on the question content and the initial reference document. Wherein, when determining the answer process, in addition to the question content, the initial reference document can also be combined, wherein the question content can be input into a vector retrieval model, and the search is performed based on a preset knowledge base, and the relevant knowledge information retrieved and filtered out from the preset knowledge base is used as the initial reference document. Wherein, massive knowledge data can be collected in advance, and each piece of knowledge data can be split according to different granularities. For the document fragments obtained after the split, they can be summarized from different directions through LLM, and multi-angle document summaries from different directions can be formed, and then the massive multi-angle document summaries can be assembled into a knowledge base. Another exemplary template for the input prompt of the answer process prompt project is as follows:

[0078] Input prompt:

[0079] ```

[0080] user:

[0081] The existing problems are as follows:

[0082] {Question content}

[0083] Please refer to the following information to answer:

[0084] {Initial Reference Document}

[0085] Which steps are needed to answer the above questions? Please refer to the following format:

[0086] 1. Historical background

[0087] 2. Narrative perspective 1

[0088] 3. Narrative perspective 2

[0089] 4. Summary

[0090] ```

[0091] Optionally, in step S101, the determining of a number of keywords and a number of questions corresponding to the current stage of the answering process based on the question content and the answering process includes:

[0092] The question content and the answer process are respectively input into the keyword generation prompt project and the question generation prompt project in the large language model to obtain a number of keywords and a number of questions corresponding to the current stage of the answer process.

[0093] Among them, after determining the answer process of the corresponding question content including several stages, in order to reduce the illusion problem of LLM, starting from the first stage of the answer process, at each stage of the answer process, the corresponding keywords and related questions are determined respectively, and corresponding enhanced retrieval is performed. Among them, at the current stage of the answer process, the question content and the answer process are respectively input into the keyword generation prompt project and the question generation prompt project in LLM to obtain several keywords and several questions corresponding to the current stage of the answer process. Among them, the keyword generation prompt project and the question generation prompt project can both include input prompts (i.e., input prompts) and output prompts (i.e., input prompts). The templates of the input prompt and output prompt of an exemplary keyword generation prompt project are as follows:

[0094] Input prompt:

[0095] ```

[0096] user:

[0097] The existing problems are as follows:

[0098] {Question content}

[0099] Please answer the following questions:

[0100] {Generated answer flow}

[0101] Now we need to answer {current stage}. To answer this question, we need to search for keywords in the knowledge base. Please give them in python.json format. The reference output is:

[0102] {

[0103] “keyword”:[“keyword 1”, “keyword 2”]

[0104] }

[0105] If the search is complete, return empty.

[0106] assistant:

[0107] ```

[0108] Output prompt:

[0109] ```

[0110] {

[0111] “keyword”:[“generated keyword 1”, “generated keyword 2”]

[0112] }

[0113] ```

[0114] Among them, the templates of the input prompt and output prompt of an exemplary question generation prompt project are as follows:

[0115] Input prompt:

[0116] ```

[0117] User:

[0118] The existing problems are as follows:

[0119] {Question content}

[0120] Please answer the following questions:

[0121] {Generated answer flow}

[0122] Now we need to answer {current stage}. To answer this question, we need to find other questions in the knowledge base. Please give them in python.json format. The reference output is:

[0123] {

[0124] "keyword":["Generated Problem 1","Generated Problem 2"]

[0125] }

[0126] If the search is complete, return empty.

[0127] assistant:

[0128] ```

[0129] Output prompt:

[0130] ```

[0131] {

[0132] "answer": ["generated question 1", "generated question 2"]

[0133] }

[0134] ```

[0135] In an optional embodiment, several keywords and several questions corresponding to the current stage of the answer process can also be determined based on the question content, reference documents, answer process and generated answers. When determining several keywords and several questions corresponding to the current stage of the answer process, in addition to the question content and the question-answer process, reference documents and generated answers corresponding to each stage before the current stage can also be combined, wherein the generated answers are a summary of the answers generated at each stage before the current stage of the answer process. Another exemplary template for the input prompt of the keyword generation prompt project is as follows:

[0136] Input prompt:

[0137] ```

[0138] user:

[0139] The existing problems are as follows:

[0140] {Question content}

[0141] Please refer to the following information to answer:

[0142] {Reference Documentation}

[0143] Please answer the following questions:

[0144] {Generated answer flow}

[0145] The generated responses are:

[0146] {Generated Answer}

[0147] Now we need to answer {current stage}. To answer this question, we need to search for keywords in the knowledge base. Please give them in python.json format. The reference output is:

[0148] {

[0149] “keyword”:[“keyword 1”, “keyword 2”]

[0150] }

[0151] If the search is complete, return empty.

[0152] assistant:

[0153] ```

[0154] Another example question generation prompt project input prompt template is as follows:

[0155] Input prompt:

[0156] ```

[0157] User:

[0158] The existing problems are as follows:

[0159] {Question content}

[0160] Please refer to the following information to answer:

[0161] {Reference Documentation}

[0162] Please answer the following questions:

[0163] {Generated answer flow}

[0164] The generated responses are:

[0165] {Generated Answer}

[0166] Now we need to answer {current stage}. To answer this question, we need to find other questions in the knowledge base. Please give them in python.json format. The reference output is:

[0167] {

[0168] "keyword":["Generated Problem 1","Generated Problem 2"]

[0169] }

[0170] If the search is complete, return empty.

[0171] assistant:

[0172] ```

[0173] After determining a number of keywords and a number of questions corresponding to the current stage of the answering process, continuing in this method embodiment, in step S102, the device 100 can input the several keywords into the keyword retrieval model to obtain a first retrieval result, input the several questions into the vector retrieval model to obtain a second retrieval result, and then input the first retrieval result and the second retrieval result into the refined ranking model to obtain a reference document corresponding to the current stage.

[0174] Optionally, in step S102, inputting the plurality of keywords into a keyword search model to obtain a first search result includes:

[0175] The plurality of keywords are input into a keyword search model, and a search is performed based on a preset knowledge base to obtain a first search result.

[0176] In step S102, the plurality of keywords may be input into a keyword search model, and a search may be performed based on a preset knowledge base, and the relevant knowledge information retrieved and screened from the preset knowledge base may be used as a first search result.

[0177] Optionally, in step S102, inputting the plurality of questions into a vector search model to obtain a second search result includes:

[0178] The plurality of questions are input into a vector retrieval model, and a search is performed based on a preset knowledge base to obtain a second search result.

[0179] The several questions may be input into a vector retrieval model, and a search may be performed based on a preset knowledge base, and the relevant knowledge information retrieved and screened from the preset knowledge base may be used as the second search result.

[0180] Continuing with this method embodiment, in step S103, the device 100 may generate an answer corresponding to the current stage of the answer process based on the question content, the answer process and the reference document corresponding to the current stage of the answer process, and summarize the answer into the generated answers.

[0181] Optionally, in step S103, determining the answer corresponding to the current stage based on the question content, answer process and reference documents includes:

[0182] The question content, answer process and reference documents are input into the answer prompt project in the large language model to obtain the answer corresponding to the current stage.

[0183] The question content, answer process and reference documents corresponding to the current stage can be input into the answer prompt project in LLM to obtain the answer corresponding to the current stage. An exemplary template of the answer prompt project is as follows:

[0184] ```

[0185] user:

[0186] The existing problems are as follows:

[0187] {Question content}

[0188] Please refer to the following information to answer:

[0189] {Reference Documentation}

[0190] Please answer the following questions:

[0191] {Generated answer flow}

[0192] You now need to answer {current stage}. Please make sure that the generated content is coherent and logical.

[0193] ```

[0194] In an optional embodiment, the answer corresponding to the current stage can also be determined based on the question content, reference documents, answer process and generated answers. When determining the answer corresponding to the current stage, in addition to the question content, answer process and reference documents corresponding to the current stage, the generated answers of each stage before the current stage can also be combined. Another exemplary template of the answer prompt project is as follows:

[0195] ```

[0196] user:

[0197] The existing problems are as follows:

[0198] {Question content}

[0199] Please refer to the following information to answer:

[0200] {Reference Documentation}

[0201] Please answer the following questions:

[0202] {Generated answer flow}

[0203] The generated answers are:

[0204] {Generated Answer}

[0205] You now need to answer {current stage}. Please make sure that the generated content is coherent and logical.

[0206] ```

[0207] Continuing in this method embodiment, in step S104, the device 100 may reflect on whether the subsequent stages after the current stage in the answer process need to be adjusted based on the question content, the answer process, and the generated answers including the answers corresponding to the current stage, and if necessary, make adjustments. Regardless of whether the subsequent stages in the answer process are adjusted, for the next stage of the answer process, the next stage of the answer process is taken as the current stage, and the above steps are repeated to determine the corresponding keywords and questions, obtain the corresponding reference documents, and then determine the answers corresponding to the next stage to obtain new generated answers.

[0208] Optionally, in step S104, reflecting on whether a subsequent stage in the answering process needs to be adjusted based on the question content, the answering process and the generated answer includes:

[0209] The question content, answer process and generated answer are input into the reflection prompt project in the large language model to reflect whether the subsequent stages in the answer process need to be adjusted.

[0210] The question content, answer process and generated answers can be input into the reflection prompting project in the LLM to reflect whether the subsequent stages of the answering process need to be adjusted. An exemplary template of the reflection prompting project is as follows:

[0211] ```

[0212] user:

[0213] The existing problems are as follows:

[0214] {Question content}

[0215] Please answer the following questions:

[0216] {Generated answer flow}

[0217] The generated answers are:

[0218] {Generated Answer}

[0219] Now we need to reflect on whether there are any problems with the existing answer process. If there are any problems, we need to modify them. Please provide the subsequent answer process to ensure that the generated content is coherent and logical.

[0220] ```

[0221] In an optional embodiment, it is also possible to reflect on whether the subsequent stages after the current stage in the answer process need to be adjusted based on the question content, reference documents, answer process and generated answers. When reflecting on whether the subsequent stages after the current stage in the answer process need to be adjusted, in addition to the question content, the current answer process and the generated answers, it is also possible to combine the reference documents corresponding to the current stage or the reference documents corresponding to each stage including the current stage. Another exemplary template of the reflection prompt project is as follows:

[0222] ```

[0223] user:

[0224] The existing problems are as follows:

[0225] {Question content}

[0226] Please refer to the following information to answer:

[0227] {Reference Documentation}

[0228] Please answer the following questions:

[0229] {Generated answer flow}

[0230] The generated answers are:

[0231] {Generated Answer}

[0232] Now we need to reflect on whether there are any problems with the existing answer process. If there are any problems, we need to modify them. Please provide the subsequent answer process to ensure that the generated content is coherent and logical.

[0233] ```

[0234] If the question is complex, LLM may need more rounds of reflection, and constantly adjust the subsequent stages of the answer process after understanding and summarizing. In order to avoid the answer process taking too long, it is necessary to balance the credibility of the answer and the efficiency of the answer process.

[0235] Optionally, in step S104, if a subsequent stage of the answering process needs to be adjusted, the method includes:

[0236] When the number of stages in the adjusted answer process meets the preset threshold, no more reflection is performed.

[0237] A threshold for the number of stages may be preset. When the number of stages in the adjusted answering process meets the preset threshold, no further reflection is performed and no further adjustment is made to the answering process.

[0238] Optionally, in step S104, if the generated answer meets the preset conditions, no further reflection is performed.

[0239] Among them, preset conditions can be set. For example, the generated answers are sufficient to show that there is no need to continue the adjustment stage (for example, the answers corresponding to multiple consecutive stages that have been generated are all negative). If the generated answers meet the preset conditions, no reflection will be conducted and the answer process will no longer be adjusted.

[0240] Continuing with this method embodiment, in step S105, the device 100 may repeat the above steps for each stage of the answering process to obtain the answer corresponding to each stage, and use the generated answer that summarizes the answers of the last stage as the final answer corresponding to the question content, so as to complete a high-credibility answer to the question content input by the user based on LLM.

[0241] Optionally, in step S105, the generated answer that summarizes the answers in the last stage as the final answer corresponding to the question content includes:

[0242] A content redundancy removal operation is performed on the generated answer that summarizes the answers in the last stage, and the generated answer after redundancy removal is used as the final answer corresponding to the question content.

[0243] In order to obtain a clearer, more concise and more reliable final answer, the generated answer that summarizes the answers in the last stage may be subjected to a content de-redundancy operation. For example, repeated content in the answers in different stages may be de-redundanted, and the generated answer after de-redundancy may be used as the final answer corresponding to the content of the question.

[0244] Through this method embodiment and / or optional embodiment, based on the preset knowledge base, multiple rounds of enhanced search are performed in the search part of LLM for the question content input by the user to enrich the search content. In the generation part of LLM, the answer process is also reflected and adjusted to enhance the overall logic and obtain the final answer, which can improve the credibility of the final answer.

[0245] Figure 2 A schematic diagram of a question-answering device based on a large language model according to another aspect of the present application is shown, wherein the device comprises:

[0246] The first module 210 is used to determine an answer process corresponding to the question content based on the obtained question content, and determine a number of keywords and a number of questions corresponding to the current stage of the answer process based on the question content and the answer process;

[0247] The second module 220 is used to input the plurality of keywords into the keyword search model to obtain a first search result, and input the plurality of questions into the vector search model to obtain a second search result, and input the first search result and the second search result into the refined ranking model to obtain a reference document corresponding to the current stage;

[0248] The third module 230 is used to determine the answer corresponding to the current stage based on the question content, the answer process and the reference document, and summarize it into the generated answer;

[0249] The fourth module 240 is used to reflect whether the subsequent stage of the answer process needs to be adjusted based on the question content, the answer process and the generated answer, and if necessary, make adjustments, and determine the answer corresponding to the next stage of the answer process through the above-mentioned related modules;

[0250] The fifth module 250 is used to obtain the answer corresponding to each stage, and summarize the generated answer in the last stage as the final answer corresponding to the question content.

[0251] In the device embodiment, the device is integrated into the apparatus 100 in the aforementioned method embodiment and / or optional embodiment.

[0252] In the embodiment of the device, through the first module 210 of the device, the question content can be obtained according to the text input by the user, and the answer process corresponding to the question content can be determined according to the question content, and in combination with the question content and the answer process, several keywords and several questions corresponding to the current stage of the answer process can be determined.

[0253] Continuing with the device embodiment, through the second module 220 of the device, the multiple keywords can be input into the keyword retrieval model to obtain a first retrieval result, the multiple questions can be input into the vector retrieval model to obtain a second retrieval result, and then the first retrieval result and the second retrieval result can be input into the refined ranking model to obtain the reference document corresponding to the current stage.

[0254] Continuing in the device embodiment, the third module 230 of the device can generate an answer corresponding to the current stage of the answer process according to the question content, the answer process and the reference document, and summarize the answer into the generated answer. The current generated answer is a summary of the answers of each stage before the current stage of the answer process.

[0255] Continuing in the device embodiment, through the fourth module 240 of the device, it is possible to reflect whether the subsequent stages after the current stage in the answer process need to be adjusted according to the question content, the answer process and the generated answers including the answers corresponding to the current stage, and if necessary, make adjustments. Regardless of whether the subsequent stages in the answer process are adjusted, for the next stage of the answer process, the next stage of the answer process is taken as the current stage, and through the above-mentioned related modules, the corresponding keywords and questions are determined, the corresponding reference documents are obtained, and then the answers corresponding to the next stage are determined to obtain new generated answers.

[0256] For each stage of the answering process, the corresponding answer for each stage can be determined through the above-mentioned related modules. Continuing in the device embodiment, through the fifth module 250 of the device, the answer corresponding to each stage can be obtained and summarized, and the generated answer summarizing the answers of the last stage is used as the final answer corresponding to the question content, so as to complete a high-credibility answer to the question content input by the user based on the LLM.

[0257] In each embodiment and / or optional embodiment of the above-mentioned device, the unmentioned parts in the method steps executed by each module of the device are the same as those in the above-mentioned related method embodiments and will not be repeated here.

[0258] According to another aspect of the present application, a computer-readable medium is provided, wherein the computer-readable medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to implement the aforementioned method embodiments.

[0259] It should be noted that each method embodiment and / or optional embodiment in the present application does not strictly limit the order in which each step is performed, as long as each method embodiment can solve the defects existing in the prior art, achieve the purpose of the invention of the present application, and obtain beneficial effects. Each method embodiment in the present application can be implemented in software and / or a combination of software and hardware. The software program involved in the present application can be executed by a processor to implement the steps or functions of the above-mentioned embodiments. Similarly, the software program (including related data structures) of the present application can be stored in a computer-readable recording medium.

[0260] In addition, part or all of the present application may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present application through the operation of the computer. The program instruction for calling the method of the present application may be stored in a fixed or removable recording medium, and / or transmitted through a data stream in a broadcast or other signal-bearing medium, and / or stored in a working memory of a computer device that runs according to the program instruction.

[0261] According to an embodiment of another aspect of the present application, a question-answering device based on a large language model is provided, the device comprising: a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to run the methods and / or technical solutions of the aforementioned embodiments.

[0262] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the device claim can also be implemented by one unit or device through software and / or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A question-answering method based on a large language model, characterized in that: The method comprises: Based on the obtained question content, determine an answer process corresponding to the question content, and based on the question content and the answer process, determine a number of keywords and a number of questions corresponding to the current stage of the answer process; Input the plurality of keywords into a keyword search model to obtain a first search result, input the plurality of questions into a vector search model to obtain a second search result, and input the first search result and the second search result into a refined ranking model to obtain a reference document corresponding to the current stage; Based on the question content, answer process and reference documents, determine the answer corresponding to the current stage and summarize it into the generated answer; Based on the question content, answer process and generated answers, reflect on whether the subsequent stages in the answer process need to be adjusted. If necessary, make adjustments and repeat the above steps for the next stage of the answer process to determine the answer corresponding to the next stage; Traverse each stage of the answer process, repeat the above steps, obtain the corresponding answer for each stage, and use the generated answer that summarizes the answers of the last stage as the final answer corresponding to the question content.

2. The method according to claim 1, characterized in that The process of determining an answer corresponding to the question content based on the obtained question content includes: The obtained question content is input into the answer process prompting project in the large language model to obtain the answer process corresponding to the question content.

3. The method according to claim 1, characterized in that: The determining of a number of keywords and a number of questions corresponding to the current stage of the answering process based on the question content and the answering process includes: The question content and the answer process are respectively input into the keyword generation prompt project and the question generation prompt project in the large language model to obtain a number of keywords and a number of questions corresponding to the current stage of the answer process.

4. The method according to claim 1, characterized in that The inputting the plurality of keywords into the keyword search model to obtain the first search result comprises: The plurality of keywords are input into a keyword search model, and a search is performed based on a preset knowledge base to obtain a first search result.

5. The method according to claim 1, characterized in that Inputting the plurality of questions into the vector search model to obtain the second search result comprises: The plurality of questions are input into a vector retrieval model, and a search is performed based on a preset knowledge base to obtain a second search result.

6. The method according to claim 1, characterized in that Determining the answer corresponding to the current stage based on the question content, the answer process and the reference document includes: The question content, answer process and reference document are input into the answer prompt project in the large language model to obtain the answer corresponding to the current stage.

7. The method according to claim 1, characterized in that Based on the question content, answer process and generated answers, reflecting on whether subsequent stages in the answer process need to be adjusted includes: The question content, answer process and generated answer are input into the reflection prompt project in the large language model to reflect whether the subsequent stages in the answer process need to be adjusted.

8. The method according to claim 1, characterized in that If the subsequent stages of the response process need to be adjusted, the methods described include: When the number of stages in the adjusted answer process meets the preset threshold, no more reflection is performed.

9. The method according to claim 1, characterized in that: If the generated answer meets the preset conditions, no further reflection will be performed.

10. The method according to claim 1, characterized in that The generated answer that summarizes the answers in the last stage as the final answer corresponding to the question content includes: A content redundancy removal operation is performed on the generated answer that summarizes the answers in the last stage, and the generated answer after redundancy removal is used as the final answer corresponding to the question content.

11. A question-answering device based on a large language model, characterized in that: The device comprises: The first module is used to determine an answer process corresponding to the question content based on the obtained question content, and determine a number of keywords and a number of questions corresponding to the current stage of the answer process based on the question content and the answer process; The second module is used to input the plurality of keywords into the keyword search model to obtain a first search result, and input the plurality of questions into the vector search model to obtain a second search result, and input the first search result and the second search result into the refined ranking model to obtain a reference document corresponding to the current stage; The third module is used to determine the answer corresponding to the current stage based on the question content, the answer process and the reference document, and summarize the generated answers; The fourth module is used to reflect whether the subsequent stages of the answer process need to be adjusted based on the question content, the answer process and the generated answers. If necessary, the adjustments are made, and for the next stage of the answer process, the answers corresponding to the next stage are determined through the above-mentioned related modules; The fifth module is used to obtain the answers corresponding to each stage, and summarize the generated answers in the last stage as the final answers corresponding to the question content.

12. A computer readable medium, characterized in that Computer readable instructions are stored thereon, and the computer readable instructions are executed by a processor to implement the method according to any one of claims 1 to 10.

13. A question-answering device based on a large language model, characterized in that: The device comprises: one or more processors; and A memory storing computer readable instructions which, when executed, cause the processor to perform the operations of the method of any one of claims 1 to 10.

Images