Question and answer method and question and answer system

Abstract

This invention provides a question-answering method and system that ensures the completeness and feasibility of problem-solving solutions. The question-answering method includes the following steps: Obtaining the question; Executing multiple LLMs to answer the question based on the mining direction setting, exploration direction weight, and verification direction weight, selectively performing solution exploration mining or solution verification mining to generate solution mining results; Determining whether the solution mining results meet the application settings based on the solution completeness calculated by the updated exploration direction weight and the solution feasibility calculated by the updated verification direction weight, to determine whether to stop iteratively mining the question; When mining the question stops, outputting the pre-mined solutions formed by all iterations of solution mining results as a problem-solving solution library.

Description

Technical Field

[0001] This invention relates to a question-answering method, and more particularly to a question-answering method and system applicable to multiple large language models. Background Technology

[0002] Users can provide the same problem to different large language models (LLMs) to brainstorm and find a complete and feasible optimal solution. To avoid knowledge illusion and instability in the LLMs, current optimization methods include, for example, providing each LLM with K prompts at once (i.e., top-K prompts), having each LLM provide multiple prompts (i.e., self-random sampling), or calculating the confidence of all prompts (i.e., Avg-Confidence). Knowledge illusion refers to the LLM proposing infeasible answers based on non-existent knowledge; the corresponding optimization method mainly involves a certain degree of verification to ensure the feasibility of the solution. Instability means that even if the LLM's answer is a feasible solution, the LLM may not always output the same answer; the corresponding optimization method mainly involves a certain degree of exploration to ensure the completeness of the solution.

[0003] However, because current optimization methods process the responses output by individual LLMs, the optimized solutions are limited by the output of a single LLM. In other words, current optimization methods can only optimize the behavior of a single LLM, and cannot optimize the set of solutions integrated by the entire LLM. Therefore, current optimization methods cannot guarantee the feasibility and completeness of the final overall solution provided to the user.

[0004] Furthermore, since each LLM possesses different capabilities and characteristics, the contribution of an individual LLM to the overall problem-solving solution varies. Therefore, rather than using multiple LLMs to discover solutions individually, selecting an LLM suitable for the overall solution optimization direction from among multiple LLMs is a more effective way to utilize resources to optimize the feasibility and completeness of the overall problem-solving solution. Summary of the Invention

[0005] This invention provides a question-answering method applicable to multiple large language models, and can ensure the completeness and feasibility of the problem-solving solution.

[0006] The question-answering method of this invention includes the following steps: Obtaining the question. Based on the mining direction setting, exploration direction weight, and verification direction weight, multiple LLMs are executed to answer the question, selectively performing solution exploration mining or solution verification mining to generate solution mining results, and updating the exploration direction weight and verification direction weight accordingly. Based on the solution completeness calculated by the updated exploration direction weight and the solution feasibility calculated by the updated verification direction weight, it is determined whether the solution mining results meet the application settings, and whether to stop iterating the question mining. When question mining stops, a pre-mined solution formed by all iterations of solution mining results is output as a solution library.

[0007] This invention also provides a question-answering system applicable to multiple large language models. The question-answering system includes a storage device and a processor. The storage device stores multiple modules. The processor is coupled to the storage device. The processor is used to execute the multiple modules and multiple LLMs to perform the aforementioned question-answering method.

[0008] Based on the above, the question-answering method and system of this invention perform solution mining (including solution exploration mining and solution verification mining) through multiple LLMs. They can evaluate the completeness of answers based on solution exploration mining and the feasibility of answers based on solution verification mining, according to application settings. Thus, the question-answering method can optimize the pre-mined solutions integrated by multiple LLMs, thereby ensuring the completeness and feasibility of the overall solution (i.e., the solution library).

[0009] To make the above features and advantages of the present invention more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the operation of a question-and-answer system according to an embodiment of the present invention; Figure 2 This is a circuit block diagram of a question-and-answer system according to an embodiment of the present invention; Figure 3 This is a flowchart illustrating a question-and-answer method according to an embodiment of the present invention; Figures 4A to 4C It is based on the present invention Figure 3 The flowchart shown in the embodiment illustrates the steps of answering questions to conduct solution mining; Figure 5 It is based on the present invention Figure 4B The flowchart shown in the embodiment illustrates the steps for verifying the repetition scheme; Figures 6A to 6B It is based on the present invention Figure 4CThe flowchart shown in the embodiment illustrates the relevant processing steps for determining the exploration weight corresponding to the selected LLM and verifying whether the weight can be 0.

[0011] Explanation of icon numbers 101~10n: Large Language Model (LLM); 100: Question and Answer System; 110: Processor; 120: Storage device; 130: Problem-solving solution library; 221: Excavation direction selection module; 222: Problem-solving strategy discovery module; 223: Termination judgment module; 230: Pre-excavation plan; 240: Valid verification scheme temporary storage area; 411: Solution Mining and Exploration Template; 412: Solution Mining and Verification Template; 421: Select LLM (the object of inquiry); 610: Explore the target value of the weight; 620: Verify the target weight value; A0: Results of solution discovery; A1: Repeated solutions; A12: Valid verification scheme; A2: New proposal; Q0: Question; S310~S360, S410~S494, S510~S570, S611~S682: Steps. Detailed Implementation

[0012] Reference will now be made in detail to exemplary embodiments of the invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same component symbols are used in the drawings and description to denote the same or similar parts.

[0013] Figure 1 This is a schematic diagram illustrating the operation of a question-and-answer system according to an embodiment of the present invention. (Reference) Figure 1The question-answering system 100 is applicable to multiple large language models (LLMs) 101~10n, where n is a positive integer. Each LLM 101~10n can act as an independent agent and has its own optimization method 1~optimization method n for its own answers. The question-answering system 100 can be applied to situations where the same question Q0 is the problem to be solved, and can use multiple LLMs 101~10n to brainstorm to find all feasible answers (i.e., problem-solving solutions, hereinafter referred to as solutions), and build a solution library 130 based on it. In this way, the question-answering system 100 can push a set of optimal solutions to the user based on the solution library 130.

[0014] Specifically, the question-answering system 100 filters solutions based on the overall required content and the individual capabilities of each LLM 101~10n. This allows for the selection of a single LLM (e.g., LLM 101) to find a solution to question Q0 in different iterations of the mining process. The system then individually evaluates the completeness and feasibility of each solution. Furthermore, the system 100 aggregates and reflects on the solutions generated across all iterations based on multiple weights associated with the completeness and feasibility of the solutions found by each LLM 101~10n. This process is then used to recursively optimize the filtering process to achieve the overall evaluation of solution completeness and feasibility.

[0015] Please refer to the above. Figure 2 , Figure 2 This is a circuit block diagram of a question-and-answer system according to an embodiment of the present invention. The question-and-answer system 100 includes a processor 110 and a storage device 120. The storage device 120 stores a plurality of LLMs 101-10n. The storage device 220 stores a pre-mining scheme 230 generated by the plurality of LLMs 101-10n according to the question Q0, and also stores a solution library 130 constructed based on the pre-mining scheme 230.

[0016] The storage device 120 also stores multiple modules 221-223. These modules 221-223 include a mining direction selection module 221, a problem-solving scheme mining module 222, and a termination judgment module 223. These modules 221-223 can be implemented, for example, in programming languages ​​such as Python, Java, JSON (JavaScript Object Notation), Extensible Markup Language (XML), or YAML to implement the functions of filtering, updating, and various calculations of this invention, but the invention is not limited thereto. The storage device 120 can be, for example, Dynamic Random Access Memory (DRAM), Flash memory, Non-Volatile Random Access Memory (NVRAM), or a combination of these.

[0017] Processor 110 is coupled to storage device 120. Processor 110 executes multiple modules 221-223 to perform solution mining and various calculations based on the same problem Q0, and aggregates and reflects on all solutions of multiple LLMs 101-10n, thereby constructing a complete and feasible solution library 130. Processor 120 may be, for example, a signal converter, a field programmable gate array (FPGA), a central processing unit (CPU), or other programmable general-purpose or special-purpose microprocessor, digital signal processor (DSP), programmable controller, application-specific integrated circuit (ASIC), programmable logic device (PLD), or other similar device or combination of these devices, which can load and execute computer program related firmware or software to perform functions such as extraction, filtering, updating, and various calculations.

[0018] Figure 3 This is a flowchart illustrating a question-and-answer method according to an embodiment of the present invention. (Reference) Figure 2 as well as Figure 3The question-answering system 100 executes steps S310~S360 through processor 110 to execute multiple LLMs 101~10n, and finds feasible pre-mining solutions 230 based on the same question Q0 to build a solution library 130.

[0019] In step S310, processor 110 obtains problem Q0.

[0020] In step S320, the processor 110 executes the digging direction selection module 221 to determine whether the current digging round is the first digging. If the determination result of step S320 is yes, the processor 110 executes step S341 to explore the digging according to a preset plan. If the determination result of step S320 is no, the processor 110 executes steps S330-S340 to select the digging direction.

[0021] In step S330, the processor 110 executes the mining direction selection module 221, so that the mining direction selection module 221 selects a mining direction based on the mining direction setting, the exploration direction weight, and the verification direction weight. The mining direction indicates whether the scheme mining performed by the processor 110 is scheme exploration mining or scheme verification mining.

[0022] Regarding solution exploration, since an open-ended problem Q0 may have k feasible solutions, a problem-solving expert (e.g., LLM 101) will iteratively query problem Q0 multiple times to find m potential answers to cover the range of k solutions as much as possible. k is an unknown positive integer dependent on problem Q0, and m is a positive integer. LLM 101 may, for example, obtain 10 answers (i.e., 10 solutions) for problem Q0, of which 8 answers are repeated in previous iterations. The coverage rate of these answers, i.e., 80%, can be used as the overall solution exploration status (i.e., exploration rate) for individual evaluation of problem Q0 by LLM 101, which is the exploration completion rate of the overall solutions output by all LLM 101~10n. The average of the multiple exploration completion rates individually evaluated by multiple LLM 101~10n methods can be used as the evaluation of the completeness of the overall solutions for problem Q0.

[0023] Regarding solution validation mining, for an open-ended question Q0, since the m potential answers obtained by the problem-solving expert (e.g., LLM 101) may be infeasible solutions due to insufficient consideration or knowledge illusion, LLM 101 needs to validate these m potential answers one or more times. For example, LLM 101 might obtain 10 answers (i.e., 10 solutions) for question Q0, of which 8 answers are repeatedly obtained in historical iterations. Further, if the number of times 2 of the 8 answers has been repeatedly obtained (i.e., validated) reaches a validation threshold, then LLM 101 considers the coverage rate between the 2 answers and the 8 answers, i.e., 25%, as the overall solution validation status (i.e., validation rate) for question Q0 in an individual evaluation of LLM 101, which is the validation completion rate of the overall solutions output by all LLM 101~10n. The average of the multiple validation completion rates individually evaluated by multiple LLM 101~10n can be used to evaluate the feasibility of the overall solution for question Q0.

[0024] In this embodiment, the mining direction setting can be, for example, a user-defined parameter, and includes an exploration resource ratio and a verification resource ratio. The exploration resource ratio indicates the resource allocation preference for scheme exploration mining. The verification resource ratio indicates the resource allocation preference for scheme verification mining. The exploration direction weight indicates the overall scheme's exploration incompleteness (i.e., exploration direction weight = 100% - exploration completion), and is the weight by which the processor 110 selects to execute scheme exploration mining for multiple LLMs 101~10n. The exploration direction weight has a preset initial weight (e.g., 0.5). The verification direction weight indicates the overall scheme's verification incompleteness (i.e., verification direction weight = 100% - verification completion), and is the weight by which the processor 110 selects to execute scheme verification mining for multiple LLMs 101~10n. The verification direction weight has a preset initial weight (e.g., 0.5).

[0025] The excavation direction selection module 221 calculates the product of the exploration direction weight and the exploration resource ratio (hereinafter referred to as the first product). For example, the excavation direction selection module 221 multiplies the exploration direction weight (e.g., a preset 0.5) by a set exploration resource ratio (e.g., 1). , so as to obtain the first product.

[0026] The mining direction selection module 221 also calculates the product of the verification direction weight and the verification resource ratio (hereinafter referred to as the second product). For example, the mining direction selection module 221 multiplies the verification direction weight (e.g., a preset 0.5) by a set verification resource ratio (e.g., 4), for example... , to obtain the second product.

[0027] Next, based on formula (1), the mining direction selection module 221 determines the mining direction according to the random number "r", the first product and the second product, so as to selectively perform scheme exploration mining or scheme verification mining. The random number "r" is a real number between 0 and 1.

[0028] …Formula (1) When the random number "r" satisfies formula (1), the mining direction selection module 221 selects the mining direction as scheme exploration mining; otherwise, it selects the mining direction as scheme verification mining. Assume the first product between the exploration direction weight and the exploration resource ratio is 0.5, and the second product between the verification direction weight and the verification resource ratio is 1. If the random number "r" generated by the mining direction selection module 221 is 0.3, then based on the above formula (2), 0.3 < 0.5 / (0.5+1). The mining direction selection module 221 selects the mining direction as scheme exploration mining.

[0029] In step S340, processor 110 executes solution mining module 222, causing solution mining module 222 to execute multiple LLMs 101~10n to answer question Q0 based on the mining direction determined in step S330, selectively performing solution exploration mining or solution verification mining. In step S341, solution mining module 222 executes multiple LLMs 101~10n to answer question Q0 to perform solution exploration mining, generating solution mining results and updating the exploration direction weights accordingly. In step S342, solution mining module 222 executes multiple LLMs 101~10n to answer question Q0 to perform solution verification mining, generating solution mining results and updating the verification direction weights accordingly.

[0030] Since the exploration direction weights can be used as the basis for calculating the overall solution's exploration completion rate (i.e., exploration rate), they can be updated, for example, based on the mining results (i.e., solution mining results) during multiple iterations of the mining process. Similarly, the validation direction weights can be used as the basis for calculating the overall solution's validation completion rate (i.e., validation rate). Validation direction weights can be updated, for example, based on the solution mining results. For operational details regarding updating exploration and validation direction weights, please refer to [link to relevant documentation]. Figures 4A to 4C The embodiment describes steps S480-S483 and S490-S494.

[0031] In step S350, processor 110 executes termination judgment module 223, so that termination judgment module 223 determines whether the solution mining result meets the application settings based on the solution completeness calculated by the updated exploration direction weight and the solution feasibility calculated by the updated verification direction weight, and determines whether to stop the iteration of mining problem Q0 by multiple LLMs 101~10n.

[0032] In this embodiment, the application settings can be, for example, user-defined parameters, and include a solution exploration threshold and a solution verification threshold. The solution exploration threshold and the solution verification threshold respectively indicate the degree of coverage of all current solutions (i.e., solution mining results) with the solution exploration completion degree (i.e., the calculation result based on the updated exploration direction weights) and the solution verification completion degree (i.e., the calculation result based on the updated verification direction weights).

[0033] If the judgment result of step S350 is negative, it means that at least one of the solution exploration completion rate and solution verification completion rate of the current mining round has not met the application settings, that is, the completeness and feasibility of the solution mining results have not yet reached the target. At this time, the processor 110 re-executes step S330 to continue the next round of solution mining.

[0034] When the judgment result of step S350 is yes, it means that the completion rate of scheme exploration and scheme verification in the current mining round has met the application settings, that is, the completeness and feasibility of the scheme mining results have reached the target. The processor 110 continues to step S360.

[0035] In step S360, when mining of problem Q0 stops, processor 110 executes termination judgment module 223, so that termination judgment module 223 outputs the pre-mined solution 230 formed by the solution mining results of all iterations as the solution library 130. That is, processor 110 stores all feasible solutions after aggregation and reflection in solution library 130.

[0036] It is worth mentioning that by performing solution exploration mining on the same question Q0 using multiple LLMs 101~10n, the question-answering system 100 can evaluate whether the completeness of the answers from all solutions (i.e., the solution mining results) is sufficient. By performing solution validation mining on the same question Q0 using multiple LLMs 101~10n, the question-answering system 100 can also evaluate whether the feasibility of the answers from the solution mining results is sufficient. In this way, the question-answering system 100 can aggregate and reflect on the pre-mined solutions 230 integrated by multiple LLMs 101~10n in all mining rounds, thereby optimizing the completeness and feasibility of the overall solutions (i.e., the solution library 130).

[0037] Figures 4A to 4C It is based on the present invention Figure 3 The embodiment illustrates a flowchart of the steps involved in answering a question to discover solutions. (See also...) Figure 2 as well as Figures 4A to 4C The question-answering system 100 executes the solution mining module 222 through the processor 110 to perform steps S340~S342 and S410~S494, and provides examples illustrating the implementation details of step S341 regarding solution exploration mining and step S342 regarding solution verification mining.

[0038] In step S410, when the solution mining module 222 selects to perform solution exploration mining in step S341, the solution mining module 222 randomly selects the selected LLM 421 from the multiple LLMs 101~10n as the question object based on the multiple exploration weights corresponding to the multiple LLMs 101~10n respectively. Each LLM 101~10n has its own exploration weight to indicate the incomplete rate of solution exploration mined for answering question Q0.

[0039] In detail, based on formulas (2-1) and (2-2), the solution mining module 222 randomly selects one of the multiple LLMs 101~10n (e.g., the i-th LLM) as the question object according to the random number "r" and multiple exploration weights of multiple LLMs 101~10n, that is, selects LLM 421. The random number "r" is a real number between 0 and 1.

[0040] …Formula (2-1) …Formula (2-2) For example, suppose there are 5 LLM 101~10n instances, and the exploration weights for these 5 LLM 101~105 are 0.7, 0.2, 0.3, 0.8, and 0.5 respectively. If the random number "r" generated by the solution mining module 222 is 0.5, then based on the above formulas (2-1) and (2-2), we know that (0.7+0.2+0.3) / 2.5 0.5 (0.7+0.2+0.3+0.8) / 2.5. Thus, the solution discovery module 222 selects the fourth LLM 104 as the query target (i.e., selects LLM 421). Note that, without loss of generality, the “≤” in formula (2-1) can also be changed to “<”. Correspondingly, the “<” in formula (2-2) can be changed to “≤”.

[0041] Furthermore, the solution mining module 222 also accesses the solution mining exploration template 411 and continues steps S430-S440. The solution mining exploration template 411 indicates how to design questions based on the elements of the input information and the problem-solving preferences of the application domain.

[0042] Similarly, in step S420, when the solution mining module 222 selects to perform solution verification mining in step S342, the solution mining module 222 randomly selects the selected LLM 421 from the multiple LLMs 101~10n as the question object based on the multiple verification weights corresponding to the multiple LLMs 101~10n respectively. Each LLM 101~10n has its own verification weight to indicate the incomplete rate of solution verification mined for answering question Q0.

[0043] In detail, the solution mining module 222 can replace the "exploration weight" in formulas (2-1) and (2-2) with "verification weight". Based on the modified formulas (2-1) and (2-2), the solution mining module 222 randomly selects one of the multiple LLMs 101 to 10n (e.g., the i-th LLM) as the question object according to the random number "r" and multiple verification weights of multiple LLMs 101 to 10n, that is, selects LLM 421. In addition, the solution mining module 222 also accesses the solution mining verification template 412 and continues with steps S430 to S440. The solution mining verification template 412 indicates how to design questions based on the elements of the input information and the elements of the known solutions to be verified, in order to meet the problem-solving preferences of the application domain.

[0044] In step S430, the solution mining module 222 selects LLM 421 (i.e., the question object) based on the solution mining preference information in the application settings to construct question Q0 according to the solution mining preference information. The solution mining preference information indicates user-defined preference elements and may include, for example, at least one of the problem boundaries, solution objectives, available resources, and solution constraints in the application domain.

[0045] Problem boundaries indicate the scope of the problem to be constructed and the scope of the tasks. If problem Q0 is [a representative work scenario type of corporate management consultant], the scope of the field may be, for example, [based on the role of corporate management consultant, excluding the work of technical consultant and legal consultant], and the scope of the tasks may be, for example, [including practical project execution and high-level strategic planning, etc., in the form of project scenarios, rather than the scope of a single tool or technology].

[0046] The problem-solving objective indicates the core value, coverage, and usability of the problem to be constructed. If problem Q0 is [a representative work scenario type for corporate management consultants], the core value could be, for example, [establishing representative scenarios and using them to classify the scope of corporate management consultant work], the coverage could be, for example, [scenarios that can cover most of the typical tasks of consultant work], and the usability could be, for example, [a framework for describing scenario types and serving as input for "subsequent classification frameworks"].

[0047] Available resources indicate the tool and computational limitations of the problem to be built. The tool limitations may be, for example, based on user preferences or needs, restricting the selected LLM 421 to using only partial knowledge sources such as internal knowledge (e.g., pre-trained knowledge of the LLM), external case libraries (e.g., RAG link retrieval), and / or the Internet (e.g., internet search tools). The computational limitations may be, for example, limiting the length of the output of the selected LLM 421 to control execution time or required resources.

[0048] The scheme's constraints specify the validation method, structuring, and general applicability of the problem to be constructed. If problem Q0 is exemplified by [a representative work scenario type for enterprise management consultants], the validation method could be, for example, that [each scenario must be able to be mapped to one or two brief descriptions of real-world knowledge or cases, and that the scenario can be understood and used by consulting practitioners]. The structuring could, for example, require selecting the LLM 421 output [presented in a structured framework (including major category structures and subcategory structures) and in JSON format]. The general applicability could, for example, require selecting the LLM 421 output [a framework structure of a generalizable type, applicable to different scenarios in different industries].

[0049] Thus, in the selected LLM 421 solution exploration mining round, the LLM 421 solution mining exploration template 411 is selected to construct problem Q0 based on solution mining preference information. Solution mining exploration template 411 is, for example: [Based on the definition of "problem boundary", based on the resources of "available resources", and under the constraints of "solution constraints", find feasible solutions that meet the "problem-solving objectives"].

[0050] In the selected mining round of LLM 421 to perform solution verification mining, LLM 421 is selected to construct problem Q0 based on solution mining verification template 412 according to solution mining preference information. Solution mining verification template 412 is, for example: [Check whether the following "known solutions" meet the "problem-solving objectives" and satisfy the constraints of "solution constraints", wherein, if necessary, other feasible solutions can be proposed based on the resources of "available resources" and according to the definition of "problem boundaries"]. In some other embodiments, solution mining verification template 412 may adopt solution mining exploration template 411.

[0051] In step S440, during the mining round where LLM 421 is selected for solution exploration mining, the solution mining module 222 executes the selected LLM 421 to answer question Q0 to perform solution exploration mining, and generates a solution mining result A0 accordingly. The solution mining module 222 also updates the exploration weight corresponding to the selected LLM 421 based on the generated solution mining result A0, and also updates the overall solution's exploration direction weight (i.e., the basis for calculating exploration completion). During the mining round where LLM 421 is selected for solution verification mining, the solution mining module 222 executes the selected LLM 421 to answer question Q0 to perform solution verification mining, and generates a solution mining result A0 accordingly. The solution mining module 222 also updates the verification weight corresponding to the selected LLM 421 based on the generated solution mining result A0, and also updates the overall solution's verification direction weight (i.e., the basis for calculating verification completion).

[0052] Specifically, in step S450, the solution mining module 222 determines whether the solution mining result A0 is an empty set. If the determination result of step S450 is yes, it means that the selected LLM 421 did not output any answer (i.e., solution) for question Q0. The solution mining module 222 continues to step S490. Otherwise, the solution mining module 222 continues to step S470.

[0053] In step S490, when the solution mining result A0 is an empty set, the solution mining module 222 further determines whether the mining direction determined in step S330 (i.e., the solution mining direction executed by the selected LLM 421) is solution exploration mining or solution verification mining. When the mining direction is solution exploration mining, the solution mining module 222 executes step S491 to set the exploration weight corresponding to the selected LLM 421 to 0, and then continues with step S493 to update the exploration direction weight of the overall solution (i.e., the basis for calculating the exploration completion rate). When the mining direction is solution verification mining, the solution mining module 222 executes step S492 to set the verification weight corresponding to the selected LLM 421 to 0, and then continues with step S494 to update the verification direction weight of the overall solution (i.e., the basis for calculating the verification completion rate).

[0054] It is worth noting that when the solution mining direction is solution exploration mining, the verification weight corresponding to the selected LLM 421 will not be updated, nor will the overall solution verification direction weight be updated. Similarly, when the solution mining direction is solution verification mining, the exploration weight corresponding to the selected LLM 421 will not be updated, nor will the overall solution exploration direction weight be updated. Steps S480~S483 follow the same processing logic: when the solution mining direction is solution exploration mining, only the exploration weight corresponding to the selected LLM 421 is updated; and when the solution mining direction is solution verification mining, only the verification weight corresponding to the selected LLM 421 is updated.

[0055] In step S470, when the solution mining result A0 is a non-empty set, the solution mining module 222 classifies the solution mining result A0 into duplicate solutions A1 and new solutions A2. Duplicate solutions A1 are solutions that already exist in all existing solutions (i.e., pre-mined solutions 230). New solutions A2 are solutions that do not yet exist in all existing solutions (i.e., pre-mined solutions 230).

[0056] Next, when the solution mining result A0 is a non-empty set, the solution mining module 222 continues with steps S472~S473 and S480~S483. The solution mining module 222 verifies the duplicate solution A1 based on the classified duplicate solution A1 and the new solution A2, and adds the new solution A2 to the pre-mined solution 230, thereby updating the pre-mined solution 230. The solution mining module 222 also updates the exploration weight and verification weight corresponding to the selected LLM 421 based on the mining direction determined in step S330 (i.e., the solution mining direction executed by the selected LLM 421).

[0057] In detail, for the new solution A2, in step S472, the solution mining module 222 adds the new solution A2 to the pre-mined solution 230. Thus, the solution mining module 222 obtains all updated solutions (i.e., the pre-mined solution 230). Then, the solution mining module 222 executes step S480 to determine whether the mining direction determined in step S330 (i.e., the solution mining direction executed by the selected LLM 421) is solution exploration mining or solution verification mining. When the mining direction is solution exploration mining, the solution mining module 222 executes step S481 to update the exploration weight corresponding to the selected LLM 421 according to the new solution A2, and then continues with step S493 to update the exploration direction weight of the overall solution and update the overall solution exploration completion rate accordingly.

[0058] Regarding updating the exploration weight corresponding to the selected LLM 421, in this embodiment, the exploration weight corresponding to the selected LLM 421 is updated based on the number of new solutions A2 in solution mining result A0 compared to the total number of solutions (i.e., the number of solutions in solution mining result A0). In some cases, when the number of solutions in solution mining result A0 is 0, the exploration weight corresponding to the selected LLM 421 is updated to 0 (e.g., Figure 4C (See step S491). Regarding the exploration direction weights for updating the overall plan (i.e., the basis for calculating exploration completion), in this embodiment, as follows... Figure 4C As shown in step S493, the exploration direction weight of the overall scheme (i.e., the basis for calculating the exploration completion rate) is updated based on the average value of the multiple exploration weights corresponding to multiple LLMs 101~10n.

[0059] For example, suppose there are 3 LLMs 101~10n, and the current exploration weights of LLMs 101~103 are 0.8, 0.7, and 0.9 respectively. In the i-th round of solution exploration, suppose LLM 101 is randomly selected to select LLM 421 for solution exploration of question Q0, and the solution exploration result A0 generated by selecting LLM 421 includes 2 duplicate solutions A1 and 2 new solutions A2, where i is a positive integer. Taking question Q0 as an example of [representative work scenario type for enterprise management consultants], the 2 duplicate solutions A1 could be answers to "proposal briefing and senior management communication" and "strategic planning and transformation design", and the 2 new solutions A2 could be answers to "enterprise diagnosis and problem analysis" and "operational process improvement and efficiency enhancement".

[0060] Thus, the solution mining module 222 calculates the ratio between the number of new solutions A2 (i.e., 2) and the number of solution mining results A0 (i.e., 4), which is 2 / 4 = 0.5, to update the ratio to the exploration weight corresponding to LLM 101. The solution mining module 222 also calculates the average of the current exploration weight of LLM 101 (i.e., 0.5), the current exploration weight of LLM 102 (i.e., 0.7), and the current exploration weight of LLM 103 (i.e., 0.9), which is (0.5 + 0.7 + 0.9) / 3 = 0.7, to update the average to the overall solution exploration direction weight (i.e., 0.7, which is the expected proportion of obtaining new solutions A2 when the question-answering system 100 selects to execute the solution exploration mining direction), and updates the overall solution exploration completion rate to 0.3 (i.e., exploration completion rate = 100% - exploration direction weight).

[0061] Continuing the example above, in the (i+1)th round of solution exploration and mining, assume that LLM 103 is randomly selected as the selected LLM 421, and the solution mining result A0 generated by selecting LLM 421 is an empty set, or the solution mining result A0 only includes 0 new solutions A2. In this case, the solution mining module 222 correspondingly executes step S491 or step S481 to update the exploration weight corresponding to LLM 103 to 0. The solution mining module 222 further continues with step S493 to calculate the average value between the current exploration weight of LLM 101 (i.e., 0.5), the current exploration weight of LLM 102 (i.e., 0.7), and the current exploration weight of LLM 103 (i.e., 0), which is (0.5+0.7+0) / 3=0.4, to update the average value as the overall solution exploration direction weight (i.e., 0.4), and update the overall solution exploration completion degree to 0.6.

[0062] For the duplicate solution A1, in step S473, the solution mining module 222 verifies the duplicate solution A1. Details regarding the verification of the duplicate solution A1 can be found in [reference needed]. Figure 5 Description of the embodiment. Then, the solution mining module 222 executes step S480 to determine whether the mining direction determined in step S330 (i.e., the solution mining direction executed by the selected LLM 421) is solution exploration mining or solution verification mining. When the mining direction is solution verification mining, the solution mining module 222 further executes step S482 to determine whether the repeating solution A1 is an empty set. When the repeating solution A1 is an empty set, the solution mining module 222 continues with step S492 to set the verification weight corresponding to the selected LLM 421 to 0, and then continues with step S494. On the other hand, when the repeating solution A1 is a non-empty set, the solution mining module 222 executes step S483 to update the verification weight corresponding to the selected LLM 421 according to the repeating solution A1, and then continues with step S494 to update the verification direction weight of the overall solution and update the verification completion degree of the overall solution accordingly.

[0063] Regarding updating the validation weight corresponding to the selected LLM 421, in this embodiment, the validation weight corresponding to the selected LLM 421 is updated based on the number of valid validations in the scheme mining result A0 relative to the number of duplicate schemes A1. In some cases, when the number of scheme mining results A0 is 0, or when the number of duplicate schemes A1 is 0, the validation weight corresponding to the selected LLM 421 is updated to 0 (e.g., Figure 4C (See step S492). Regarding the verification direction weights for updating the overall scheme (i.e., the basis for calculating verification completion), in this embodiment, as follows... Figure 4CAs shown in step S494, the verification direction weight of the overall scheme (i.e., the basis for calculating the verification completion rate) is updated based on the average value of the multiple verification weights corresponding to the multiple LLMs 101~10n.

[0064] For example, suppose there are 3 LLMs 101~10n, and the current validation weights of LLMs 101~103 are 0.8, 0.9, and 0.7 respectively. In the i-th round of scheme validation mining, suppose LLM 101 is randomly selected as the selected LLM 421, and the scheme mining result A0 generated by selecting LLM 421 includes 2 repeated schemes A1 and 2 new schemes A2, where i is a positive integer. Repeated schemes A1 include 2 repeated schemes "X" and "Y". Suppose the number of samplings of repeated scheme "X" (e.g., 6 times), that is, the number of times that selected LLM 421 answers "X" to question Q0 based on scheme mining preference information, has met the valid validation threshold indicated by the application settings. Thus, repeated scheme "X" is an invalid validation. If the sampling number for the repeated option "Y" is, for example, 3 times, which is the number of times the LLM 421 selects the option mining preference information to answer "Y" to question Q0, then the valid validation threshold indicated by the application settings has not yet been met. Thus, "Y" is considered valid validation.

[0065] Thus, the solution mining module 222 calculates the ratio between the number of valid verifications (i.e., 1) in the solution mining result A0 and the number of duplicate solutions A1 (i.e., 2), which is 1 / 2 = 0.5, to update the ratio to the verification weight corresponding to LLM 101. The solution mining module 222 also calculates the average of the current verification weights of LLM 101 (i.e., 0.5), LLM 102 (i.e., 0.9), and LLM 103 (i.e., 0.7), which is (0.5 + 0.9 + 0.7) / 3 = 0.7, to update the average to the overall solution verification direction weight (i.e., 0.7, which is the expected proportion of valid verifications obtained when the question-answering system 100 selects the direction to perform solution verification mining), and updates the overall solution verification completion rate to 0.3 (i.e., verification completion rate = 100% - verification direction weight).

[0066] Continuing the example above, in the (i+1)th round of solution verification mining, assume that LLM 102 is randomly selected to select LLM 421, and the solution mining result A0 generated by selecting LLM 421 is an empty set, or the solution mining result A0 includes 0 duplicate solutions (i.e., the number of duplicate solutions A1 is 0). Thus, the solution mining module 222 executes step S492 to update the verification weight corresponding to LLM 102 to 0. The solution mining module 222 also calculates the average value between the current verification weight of LLM 101 (i.e., 0.5), the current verification weight of LLM 102 (i.e., 0), and the current verification weight of LLM 103 (i.e., 0.7), which is (0.5+0+0.7) / 3=0.4, to update the average value as the overall solution verification direction weight (i.e., 0.4), and updates the overall solution verification completion degree to 0.6.

[0067] Continuing the example above, in the (i+2)th round of solution verification mining, assume that LLM 103 is randomly selected as the selected LLM 421, and the duplicate solutions A1 included in the solution mining result A0 generated by selecting LLM 421 are not an empty set, but the included duplicate solutions A1 are invalid verifications with 0 valid verification times. Thus, the solution mining module 222 executes step S483 to update the verification weight corresponding to LLM 103 to 0. The solution mining module 222 also calculates the average value between the current verification weight of LLM 101 (i.e., 0.5), the current verification weight of LLM 102 (i.e., 0), and the current verification weight of LLM 103 (i.e., 0), which is (0.5+0+0) / 3=0.17, to update the average value as the overall solution verification direction weight (i.e., 0.17), and updates the overall solution verification completion degree to 0.83.

[0068] After the solution mining module 222 executes step S493 or S494 to update the exploration direction weight and exploration completion degree of the overall solution or the verification direction weight and verification completion degree of the overall solution, the solution mining module 222 continues with step S460. In step S460, the solution mining module 222 determines whether the exploration weight and verification weight corresponding to the selected LLM 421 are allowed to be 0, so as to set the exploration weight and verification weight corresponding to the selected LLM 421. When the determination result of step S460 is yes, the solution mining module 222 does not change the exploration weight and verification weight corresponding to the selected LLM 421 (i.e., it is allowed to be 0). When the determination result of step S460 is no, in step S461, the solution mining module 222 sets the exploration weight and verification weight corresponding to the selected LLM 421. The operation details of step S461 can be found in [reference needed]. Figures 6A to 6BDescription of the embodiments.

[0069] Figure 5 It is based on the present invention Figure 4B The flowchart illustrating the steps for verifying the reproducibility scheme in the embodiment is shown in the example. (See also...) Figure 2 as well as Figure 5 The question-answering system 100 executes the solution mining module 222 via the processor 110 to perform steps S510 to S570, and provides an example of step S473 regarding the details of verifying the repeating solution A1.

[0070] In step S510, the solution mining module 222 obtains the solution to be verified based on the repeated solution A1. For example, the solution mining module 222 may extract each solution one by one from the repeated solution A1 as the solution to be verified.

[0071] In step S520, the solution mining module 222 determines whether the solution to be verified is an empty set. When the solution to be verified is a non-empty set, the solution mining module 222 obtains the target solution based on the solution to be verified and all existing solutions (i.e., pre-mined solutions 230). The solution mining module 222 also determines whether to accept the solution to be verified as a valid verification solution based on the number of votes from multiple LLMs 101~10n for the target solution and the voting verification threshold, so as to store the solution to be verified in the valid verification solution temporary storage area 240.

[0072] Specifically, when the solution to be verified is a non-empty set, in step S531, the solution mining module 222 constructs a query description of the solution to be verified. In step S532, the solution mining module 222 extracts a matching solution from all currently mined solutions (i.e., pre-mined solutions 230) based on the constructed query description as the target solution. In step S533, the solution mining module 222 selects the LLM (i.e., ...) as the target solution. Figure 4A And as shown in 4B, the number of responses to the target solution (LLM 421) increases the number of votes for the target solution accordingly. "Vote count +1" indicates that the target solution appears once. The solution mining module 222 also updates all existing solutions (i.e., pre-mined solutions 230) based on the responses to the target solution.

[0073] In step S534, the solution mining module 222 determines whether the number of votes in step S533 is less than or equal to the voting verification threshold. The voting verification threshold may be included in the application settings, indicating the minimum number of votes required to satisfy valid verification, and may be set as the number of verification thresholds. In this embodiment, the voting verification threshold may be determined, for example, based on the number of multiple LLM101~10n. The voting verification threshold may be set to the stated number (i.e., the total number n), or it may be set to different proportions such as 1 / 2 or 2 / 3 of the total number n.

[0074] If the judgment result of step S534 is negative, it means that the number of times the selected LLM's response to the target solution has repeatedly occurred (i.e., verified) has not yet reached the verification threshold. In other words, the solution to be verified obtained by the repeated knowledge A1 obtained by the selected LLM in this solution mining provides further effective help in verifying the target solution. Therefore, the solution to be verified belongs to the valid verification solution A12. Solution mining module 222 continues with step S535.

[0075] In step S535, the solution mining module 222 identifies the target solution whose voting count is closer to the voting verification threshold as the valid verification solution A12. The solution mining module 222 also stores the valid verification solution A12 in the valid verification solution temporary storage area 240, and then re-executes step S510 to continue verifying other solutions.

[0076] On the other hand, if the judgment result of step S534 is yes, it means that the number of times the LLM answers the target solution repeatedly (i.e., verification) has reached the verification threshold. In other words, the solution to be verified selected from the repeated knowledge A1 obtained by the LLM in this solution mining can no longer provide further effective help in verifying the target solution. Therefore, the solution to be verified is not valid verification knowledge. The solution mining module 222 re-executes step S510 to continue verifying other solutions to be verified.

[0077] In step S560, when the solution to be verified is an empty set, the solution mining module 222 takes the solution to be verified stored in the valid verification solution temporary storage area 240 as the valid verification solution A12, and outputs the valid verification solution A12 accordingly.

[0078] In step S570, the solution mining module 222 calculates the ratio between the number of valid verification solutions A12 and the number of duplicate solutions A1. The solution mining module 222 updates the verification weight corresponding to the selected LLM 421 based on this ratio, as explained in step S473 regarding updating the verification weight corresponding to the selected LLM 421.

[0079] Figures 6A to 6B It is based on the present invention Figure 4C The flowchart shown in this embodiment illustrates the processing steps related to determining the exploration weight corresponding to the selected LLM and verifying whether the weight can be 0. (See reference) Figure 2 as well as Figure 5 The question-answering system 100 executes the problem-solving solution mining module 222 through the processor 110 to perform steps S611 to S682, and provides an example illustrating the details of step S461 regarding setting the exploration weight and the verification weight in the scenario where the exploration weight and verification weight corresponding to the selected LLM 421 are not allowed to be 0.

[0080] When the exploration weight corresponding to the selected LLM 421 is not allowed to be 0, the solution mining module 222 executes steps S611, S630, S641, and S651~S652 to calculate the exploration weight target value 610. The solution mining module 222 calculates the exploration weight target value 610 based on the number of multiple LLMs 101~10n and the multiple exploration weights corresponding to each LLM 101~10n. The solution mining module 222 modifies the exploration weight corresponding to the selected LLM 421 to the exploration weight target value 610 to avoid the situation where the exploration weight is 0.

[0081] Specifically, in step S611, the solution mining module 222 selects the smallest (i.e., the minimum exploration weight) from the multiple exploration weights corresponding to the multiple LLMs 101~10n. The minimum exploration weight is a decimal between 0 and 1, or 1. In step S630, the solution mining module 222 calculates the number "n" of the multiple LLMs 101~10n. The number "n" is a positive integer.

[0082] In step S641, the solution mining module 222 determines whether the minimum exploration weight is greater than the reciprocal of the number "n" of the multiple LLMs 101~10n (i.e., 1 / n), and uses the reciprocal "1 / n" or the minimum exploration weight as the exploration weight target value 610. Specifically, when the determination result of step S641 is yes, in step S651, the solution mining module 222 sets the exploration weight target value 610 to the reciprocal of the number "n" of the multiple LLMs 101~10n (i.e., 1 / n). Conversely, in step S652, the solution mining module 222 sets the exploration weight target value 610 to the minimum exploration weight.

[0083] When the verification weight corresponding to the selected LLM 421 is not allowed to be 0, the solution mining module 222 executes steps S612, S630, S642, and S653~S654 to calculate the verification weight target value 620. The solution mining module 222 calculates the verification weight target value 620 based on the number of multiple LLMs 101~10n and the multiple verification weights corresponding to each LLM 101~10n. The solution mining module 222 modifies the verification weight corresponding to the selected LLM 421 to the verification weight target value 620 to avoid the situation where the verification weight is 0.

[0084] Specifically, in step S612, the solution mining module 222 selects the smallest (i.e., the minimum verification weight) from the multiple verification weights corresponding to the multiple LLMs 101~10n. The minimum verification weight is a decimal between 0 and 1, or 1. In step S630, the solution mining module 222 calculates the number "n" of the multiple LLMs 101~10n.

[0085] In step S642, the solution mining module 222 determines whether the minimum verification weight is greater than the reciprocal of the number "n" of the multiple LLMs 101~10n (i.e., 1 / n), and uses the reciprocal "1 / n" or the minimum verification weight as the verification weight target value 620. Specifically, when the determination result of step S642 is yes, in step S653, the solution mining module 222 sets the verification weight target value 620 to the reciprocal of the number "n" of the multiple LLMs 101~10n (i.e., 1 / n). Conversely, in step S654, the solution mining module 222 sets the verification weight target value 620 to the minimum verification weight.

[0086] In step S661, the solution mining module 222 retrieves the exploration weight corresponding to the selected LLM 421. In step S671, the solution mining module 222 determines whether the exploration weight corresponding to the selected LLM 421 is 0. If the determination in step S671 is yes, the solution mining module 222 stores the calculated exploration weight target value 610, which is the minimum exploration weight or the reciprocal of the number of knowledge nodes "n" (i.e., 1 / n). In step S681, the solution mining module 222 sets the exploration weight corresponding to the selected LLM 421 to the exploration weight target value 610.

[0087] In step S662, the solution mining module 222 retrieves the verification weight corresponding to the selected LLM 421. In step S672, the solution mining module 222 determines whether the verification weight corresponding to the selected LLM 421 is 0. If the determination in step S672 is yes, the solution mining module 222 stores the calculated verification weight target value 620, which is the minimum verification weight or the reciprocal of the number of knowledge nodes "n" (i.e., 1 / n). In step S682, the solution mining module 222 sets the verification weight corresponding to the selected LLM 421 to the verification weight target value 620.

[0088] In summary, the question-answering method and system of this invention utilize multiple LLMs to perform solution exploration mining on the same question, enabling the system to evaluate the completeness of the overall solution's answer based on user settings. By performing solution verification mining on the same question using multiple LLMs, the system can evaluate the feasibility of the overall solution's answer based on user settings. Furthermore, the system can use the completeness and feasibility of the overall solution's answer as a calculation basis to randomly select either solution exploration mining or solution verification mining as the optimization direction. Based on the optimization direction, the system iteratively mines solutions by randomly selecting one of the multiple LLMs as the question object (i.e., selecting the LLM) according to the multiple exploration weights and multiple verification weights of the multiple LLMs. The system can construct questions based on application settings, mine solutions for the question object based on the questions to optimize the content of the overall solution, and estimate the solution exploration rate and solution verification rate of the question object based on the mining results, thereby updating the exploration weight and verification weight of the question object and further updating the completeness and feasibility of the overall solution's answer. In this way, the question-answering system can optimize the completeness and feasibility of the overall solution (i.e., the solution library).

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A question-and-answer method, characterized in that, include: Get the question; Based on the mining direction setting, exploration direction weight, and verification direction weight, multiple large language models are executed to answer the question, selectively performing solution exploration mining or solution verification mining to generate solution mining results, and updating the exploration direction weight and the verification direction weight. Based on the completeness of the scheme calculated by the updated exploration direction weights and the feasibility of the scheme calculated by the updated verification direction weights, it is determined whether the scheme mining results meet the application settings, so as to determine whether to stop the iteration of mining the problem. as well as When mining the problem stops, output the pre-mined solutions formed by the mining results of all iterations of the solution as a solution library.

2. The question-and-answer method according to claim 1, characterized in that, When the multiple large language models selectively perform scheme exploration mining, the question answering method further includes: Based on the multiple exploration weights corresponding to the multiple large language models, a selected large language model is randomly chosen as the question object; and The selected large language model is executed to answer the question in order to perform solution exploration mining, thereby generating the solution mining results, and the exploration weight and exploration direction weight corresponding to the selected large language model are updated.

3. The question-and-answer method according to claim 2, characterized in that, The exploration weight corresponding to the selected large language model is updated based on the number of new schemes in the scheme mining results compared to the number of scheme mining results. When the number of scheme mining results is 0, the exploration weight corresponding to the selected large language model is updated to 0.

4. The question-and-answer method according to claim 2, characterized in that, The exploration direction weights are updated based on the average of the multiple exploration weights corresponding to the multiple large language models.

5. The question-and-answer method according to claim 1, characterized in that, When the multiple large language models selectively perform scheme verification mining, the question answering method further includes: Based on the multiple validation weights corresponding to the multiple large language models, a selected large language model is randomly chosen as the question object; and The selected large language model is executed to answer the question in order to perform scheme verification mining, thereby generating the scheme mining results, and the verification weight and the verification direction weight corresponding to the selected large language model are updated.

6. The question-and-answer method according to claim 5, characterized in that, The validation weight corresponding to the selected large language model is updated based on the number of valid validations in the scheme mining results relative to the number of duplicate schemes. When the number of scheme mining results is 0, or when the number of duplicate schemes is 0, the validation weight corresponding to the selected large language model is updated to 0.

7. The question-and-answer method according to claim 5, characterized in that, The verification direction weights are updated based on the average of the multiple verification weights corresponding to the multiple large language models.

8. The question-and-answer method according to claim 1, characterized in that, The selected large language model from the multiple large language models is chosen as the question object and executed to answer the question to generate the solution mining results. When the mining result of the scheme is an empty set, the question-answering method further includes: When the multiple large language models selectively perform scheme exploration mining, the exploration weight corresponding to the selected large language model is set to 0, and the exploration direction weight is updated; When the multiple large language models selectively perform scheme verification mining, the verification weight corresponding to the selected large language model is set to 0, and the verification direction weight is updated.

9. The question-and-answer method according to claim 8, characterized in that, When the exploration weight corresponding to the selected large language model is not allowed to be 0, the question answering method further includes: Based on the number of the multiple large language models and the multiple exploration weights corresponding to the multiple large language models, calculate the target value of the exploration weight; Modify the exploration weights corresponding to the selected large language model to the target value of the exploration weights.

10. The question-and-answer method according to claim 9, characterized in that, The target value of the exploration weight is the reciprocal of the number of the plurality of large language models, or the minimum exploration weight among the plurality of exploration weights corresponding to the plurality of large language models.

11. The question-and-answer method according to claim 8, characterized in that, When the validation weight corresponding to the selected large language model is not allowed to be 0, the question answering method further includes: Calculate the target value of the verification weight based on the number of the multiple large language models and the multiple verification weights corresponding to the multiple large language models; Modify the validation weights corresponding to the selected large language model to the target validation weight value.

12. The question-and-answer method according to claim 11, characterized in that, The target value of the verification weight is the reciprocal of the number of the plurality of large language models, or the minimum verification weight among the plurality of verification weights corresponding to the plurality of large language models.

13. The question-and-answer method according to claim 8, characterized in that, When the mining result of the scheme is a non-empty set, the question-answering method further includes: The results of the scheme mining are classified into duplicate schemes and new schemes; Based on the repeating scheme and the new scheme, verify the repeating scheme and add the new scheme to the pre-mining scheme to update the pre-mining scheme; and The exploration weight and the verification weight corresponding to the selected large language model are updated according to the scheme mining direction of the selected large language model.

14. The question-and-answer method according to claim 13, characterized in that, Also includes: Determine whether the exploration weight and the verification weight corresponding to the selected large language model are allowed to be 0, so as to set the exploration weight and the verification weight corresponding to the selected large language model.

15. The question-and-answer method according to claim 13, characterized in that, In the operation of verifying the repeatability scheme, the question-and-answer method further includes: Based on the aforementioned repetition scheme, obtain the scheme to be verified; When the scheme to be verified is a non-empty set, a target scheme is obtained based on the scheme to be verified and the pre-mined scheme; and, based on the number of votes and the voting verification threshold of the responses of the multiple large language models to the target scheme, it is determined whether to store the scheme to be verified as a valid verification scheme in the valid verification scheme temporary storage area. When the set of schemes to be verified is empty, the schemes to be verified stored in the temporary storage area of ​​the valid verification schemes are taken as the valid verification schemes; and Calculate the ratio between the number of valid validation schemes and the number of duplicate schemes to update the validation weights corresponding to the selected large language model.

16. The question-and-answer method according to claim 15, characterized in that, The voting verification threshold is determined based on the number of the multiple large language models.

17. The question-and-answer method according to claim 1, characterized in that, Also includes: Based on the scheme mining preference information in the application settings, the selected large language model from the multiple large language models is executed to construct the question according to the scheme mining preference information. The scheme mining preference information mentioned above includes at least one of the following: problem boundary, problem-solving objective, available resources, and scheme constraints.

18. The question-and-answer method according to claim 1, characterized in that, The mining direction setting includes the proportion of resources to be explored and the proportion of resources to be verified. The execution of multiple large language models to answer the question is determined based on a random number, the product of the exploration direction weight and the exploration resource ratio, and the product of the verification direction weight and the verification resource ratio, to selectively perform solution exploration mining or solution verification mining.

19. A question-answering system applicable to multiple large language models, characterized in that, include: Storage device, storing multiple modules; as well as A processor, coupled to the storage device, is configured to execute the plurality of modules and the plurality of large language models to perform the question-answering method as described in claim 1.

Images