Question and answer method and device based on large language model, medium and program product

Abstract

The invention provides a question answering method and device based on a large language model, a medium and a program product. The method can comprise the following steps: acquiring question information aiming at a large language model, and respectively inputting the question information into at least two large language model instances; and in the process of receiving the alternative answers returned by the large language model instances, if a part of the alternative answers are received and the part of the alternative answers meet a preset accuracy rate condition, determining reply information for the question information based on the part of the alternative answers.

Description

Technical Field

[0001] One or more embodiments of this specification relate to the field of artificial intelligence technology, and in particular, to a question-answering method, device, medium, and program product based on a large language model. Background Art

[0002] In the field of natural language processing technology, large language models (LLMs) play an important role. They can understand and generate human language, thereby performing a variety of tasks such as text generation, question answering, and translation. However, despite the remarkable achievements of large language models in many aspects, hallucinations may still occur in the process of generating results, that is, large language models may generate content that is inconsistent with user instructions or logically unreasonable, resulting in incorrect output results. In addition, the time consumption of large language models in generating responses is unstable. Even for the same large language model for the same task request, the time required to generate results each time fluctuates greatly. This uncertainty in response time and result accuracy seriously affects the user experience of large language models. Summary of the invention

[0003] In view of this, one or more embodiments of this specification provide the following technical solutions:

[0004] According to a first aspect of one or more embodiments of this specification, a question answering method based on a large language model is proposed, comprising:

[0005] Get question information for large language models;

[0006] Inputting the question information into at least two large language model instances respectively;

[0007] In the process of receiving the alternative answers returned by each large language model instance, if some alternative answers have been received and the partial alternative answers have met the preset accuracy conditions, the reply information for the question information is determined based on the partial alternative answers.

[0008] According to a second aspect of one or more embodiments of this specification, an electronic device is provided, including:

[0009] A processor; a memory for storing processor-executable instructions; wherein the processor implements the steps of the method as described in the first aspect by running the executable instructions.

[0010] According to a third aspect of one or more embodiments of the present specification, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0011] According to a fourth aspect of one or more embodiments of the present specification, a computer program product is provided, comprising a computer program / instruction, which implements the steps of the method described in the first aspect when executed by a processor.

[0012] It can be seen from the above embodiments that this specification inputs the question information into at least two large language model instances, and combines the preset accuracy conditions. Even if some of the large language model instances have output errors, it can ensure that the correct reply information is determined based on the output of other large language model instances, which helps to reduce the errors caused by the unreliability of a single alternative answer, thereby significantly improving the accuracy and robustness of the reply information. At the same time, based on the above preset accuracy conditions, when some of the received alternative answers have met the preset accuracy conditions, the reply information can be determined as soon as possible without waiting for all large language model instances to return alternative answers. Therefore, it is possible to reduce the fluctuation of the response time of the large language model on the basis of ensuring the accuracy and robustness of the reply information. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic diagram of response time provided by an exemplary embodiment.

[0014] Figure 2 It is a schematic diagram of the architecture of a question-answering system based on a large language model provided by an exemplary embodiment.

[0015] Figure 3 It is a flowchart of a question-answering method based on a large language model provided by an exemplary embodiment.

[0016] Figure 4 It is a schematic diagram of multiple large language model instances provided by an exemplary embodiment.

[0017] Figure 5 It is another schematic diagram of response time provided by an exemplary embodiment.

[0018] Figure 6 It is a schematic diagram of the operation process of a question-answering system provided by an exemplary embodiment.

[0019] Figure 7 The present invention is a flowchart of a multi-way voting unit determining reply information provided by an exemplary embodiment.

[0020] Figure 8 It is a structural schematic diagram of a device provided by an exemplary embodiment.

[0021] Fig. 9 It is a block diagram of a question-answering device based on a large language model provided by an exemplary embodiment. DETAILED DESCRIPTION

[0022] Large language models are widely used in intelligent question-answering systems due to their excellent language understanding and generation capabilities. In related technologies, in the process of using large language models to complete tasks, large language models may generate content that is inconsistent with user instructions or logically unreasonable, resulting in the output of erroneous results. For example, the user instruction is "Please check out late." In the process of executing the user's instruction, the large language model may only capture the check-out information, resulting in the user's intention being understood as checking out, and the erroneous result is output to the downstream system for execution. Obviously, this erroneous result is contrary to the user's original intention, and executing this erroneous result will cause an unpleasant experience for the user. In addition, the time consumption of the large language model in generating responses is unstable. Even if the same large language model is used for the same task request, the time required to generate the result each time fluctuates greatly. Figure 1 is a schematic diagram of response time provided by an exemplary embodiment. Figure 1 , the horizontal axis represents the number of times the large language model is called, and the vertical axis represents the response time in milliseconds. Figure 1 It can be seen that when the same question is input into the same large language model multiple times at different times, the response time obtained will fluctuate greatly, which greatly affects the user experience.

[0023] In view of this, this specification proposes a question-answering method based on a large language model, which can significantly improve the accuracy and robustness of reply information while reducing the fluctuation of response time.

[0024] Figure 2 FIG. 1 is a schematic diagram of the architecture of a question-answering system based on a large language model provided by an exemplary embodiment. Figure 2 As shown, the system architecture may include a server 21, a network 22, and several electronic devices, such as a PC (Personal Computer) 23, a mobile phone 24, and the like.

[0025] The server 21 may be a physical server including an independent host, or the server 21 may be a virtual server carried by a host cluster. During operation, the server 21 may run a server-side program of the question-answering system to realize the server end of the question-answering system.

[0026] PC 23 and mobile phone 24 are only some types of electronic devices that users can use. In fact, users can obviously also use electronic devices such as the following types: tablet devices, laptops, PDAs (Personal Digital Assistants), wearable devices (such as smart glasses, smart watches, etc.), etc., and one or more embodiments of this specification do not limit this. During operation, the electronic device can run the client-side program of the question-and-answer system to implement the client of the question-and-answer system. Among them, the client application of the above-mentioned question-and-answer system can be started and run on the electronic device. The client-side program can be a native application installed on the electronic device, or the client-side program can be a small program, a quick application or other similar forms. Of course, when using web page technologies such as HTML5 or similar, the relevant functions can be implemented through the page displayed by the browser. The browser here can be an independent browser application or a browser module embedded in certain applications.

[0027] As for the network 22 for interaction between electronic devices such as PC 23 and mobile phone 24 and server 21, it is possible to select a wired or wireless network to achieve communication based on the communication mode supported by the corresponding electronic device, and this specification does not limit this. For example, PC 23 can support both wired and wireless communication, so it can use a wired or wireless network to achieve communication as needed, while mobile phone 24 usually only supports wireless communication, so it can use a wireless network to achieve communication.

[0028] based on Figure 2 In the question-answering system shown, a user can initiate a question information / task request to an electronic device such as PC 23 or mobile phone 24, so that the electronic device such as PC 23 or mobile phone 24 submits the received question information / task request to server 21. Server 21 can run the question-answering method of this specification for the received question information / task request, obtain reply information for the question information or task execution result for the task request, and return the reply information or task execution result to the electronic device that submitted the question information.

[0029] In another embodiment, the question-answering system based on the large language model may only include electronic devices such as PC 23 and mobile phone 24. In this case, the large language model may be deployed locally on electronic devices such as PC 23 and mobile phone 24. Taking PC 23 as an example, PC 23 runs the question-answering method of this specification for the received question information and obtains reply information for the question information.

[0030] The following will be combined Figure 3 , the technical solution of this specification is described in detail.

[0031] Figure 3FIG. 1 is a flowchart of a question-answering method based on a large language model provided by an exemplary embodiment. Figure 3 As shown, the method may include the following steps:

[0032] Step 302: Obtain question information for the large language model.

[0033] A large language model usually refers to a pre-trained language model with a very large number of parameters. This type of model can capture the complex structure and patterns of the language by training on large-scale text data, thereby demonstrating excellent performance in a variety of natural language processing tasks. Question information refers to information input into a large language model instance, which can be information input by a user in an electronic device that supports intelligent question-and-answer functions, including but not limited to: questions such as "What is the answer to this multiple-choice question?", "What is the weather in City A today?", or task requests such as "check out" and "book a high-speed rail ticket to City A". Among them, the user's input method can be text input, voice input, and other methods, which are not limited in this manual.

[0034] Alternatively, the question information may be information obtained by the question-answering system after processing the user input. For example, if the user input is "What is the weather in City A today?", the question-answering system may process the user input and convert the user input into more standardized and formatted information such as "Query the weather in City A", and use the converted information "Query the weather in City A" as the question information for the large language model. By converting the user input into more standardized and formatted question information, the speed and accuracy of the large language model instance's understanding of the question information can be improved, thereby improving the accuracy and response speed of the final output alternative answers.

[0035] Step 304: input the question information into at least two large language model instances respectively.

[0036] The solution in this specification requires calling a large language model to understand the question information and generate answers to the question information. A large language model instance refers to a specific instance created based on a large language model, which is a specific application of a large language model in a specific scenario. Each large language model instance can run independently on different devices. The question information is input into at least two large language model instances respectively, so that the question information is processed in parallel by at least two large language model instances, and alternative answers are generated for the question information respectively.

[0037] In one embodiment, at least two large language model instances may correspond to the same large language model, in which case at least two large language model instances are built based on the infrastructure of the same large language model. Although the infrastructure of at least two large language model instances is the same, due to the random initialization during the training process and slight differences in the training data, there will be some slight differences between different large language model instances. Therefore, at least two large language model instances will also have different understandings of the same question information. This approach only requires the maintenance of a large language model, which helps reduce the overhead of deploying and maintaining large language models.

[0038] In addition, at least two large language model instances may also correspond to different large language models respectively. Due to the different infrastructures of different large language models, such as BERT (Bidirectional Encoder Representations from Transformers) uses a pure encoder architecture design, and the GPT (Generative Pre-trained Transformer) series uses a pure decoder architecture design. In addition, the training data and training methods of different large language models are different. Therefore, in this case, the alternative answers generated by at least two large language model instances for the same question information may be significantly different. This method uses multiple large language model instances corresponding to different large language models to answer the same question information, which can obtain more diverse alternative answers and cover a wider range of alternative answers, thereby helping to determine more accurate response information from the diverse alternative answers.

[0039] In one embodiment, different question information may belong to different categories, for example, "book a hotel room" belongs to the hotel service category, and "what is the answer to this college entrance examination multiple-choice question" belongs to the education category. Different categories of question information have different levels of complexity, resulting in different numbers of large language model instances that need to be called. In order to avoid the two situations of too many large language model instances being called, resulting in a waste of resources, or too few large language model instances affecting the correctness of the reply information, the staff of the intelligent question-answering system can pre-define the required number of large language model instances corresponding to each alternative category, and then after obtaining the question information, determine the predefined required number of large language model instances corresponding to the question information according to the target category to which the question information belongs.

[0040] In addition, the required number of large language model instances corresponding to each alternative category can also be determined based on the answer accuracy of each large language model instance for each alternative category. Among them, the answer accuracy of each large language model instance for each alternative category can be determined based on the historical usage experience of each large language model or the data of the test phase. For example, for any alternative category, multiple test tasks under any alternative category can be input into an initial number of large language model instances to obtain the predicted answers output by each large language model instance for each test task. The answer accuracy corresponding to any alternative category is calculated based on the similarity between the predicted answer and the standard answer corresponding to the corresponding test task, and then the aforementioned initial number is adjusted based on the answer accuracy, until the answer accuracy corresponding to any alternative category reaches a preset threshold under the joint action of the adjusted number of large language model instances, at which time the adjusted number can be determined as the required number of large language model instances corresponding to any alternative category.

[0041] Figure 4 FIG. 1 is a schematic diagram of multiple large language model instances provided by an exemplary embodiment. Figure 4 As shown, the question-answering system includes an instruction distribution module 402 and an instruction voting module 404. The instruction distribution module 402 is responsible for receiving user instructions and processing the user instructions to obtain question information for the large language model. Then, the target category to which the question information belongs is determined, and the number of large language model instances to be called and the accuracy condition corresponding to the target category are determined based on the target category. Assuming that three large language model instances need to be called, the instruction distribution module 402 inputs the question information into the three large language model instances respectively. The large language model instances 1-3 will respectively generate alternative answers for the question information and return the alternative answers to the instruction voting module 404. In the process of receiving the alternative answers returned by the three large language model instances respectively, the instruction voting module 404 can determine the reply information for the question information based on the received partial alternative answers. In addition, the parameters in the corresponding large language model instance can be automatically adjusted according to the alternative answers received in the instruction voting module 404, so that the corresponding large language model instance outputs an alternative answer with a higher accuracy rate based on the adjusted parameters.

[0042] Step 306, in the process of receiving the alternative answers returned by each large language model instance, if some alternative answers have been received and the partial alternative answers have met the preset accuracy conditions, then the reply information for the question information is determined based on the partial alternative answers.

[0043] Since each large language model instance takes different time to respond to the same question information, the large language model instances usually return alternative answers one after another. Of course, if there are multiple large language model instances that take exactly the same time to respond to the same question information, then the multiple large language model instances will return alternative answers at the same time. In the process of receiving each alternative answer, the accuracy of some of the received alternative answers can be evaluated. If the evaluation result shows that the part of the alternative answers has met the preset accuracy condition, the reply information for the question information can be determined based on the part of the alternative answers.

[0044] The above embodiment inputs the question information into at least two large language model instances and combines the preset accuracy conditions. Even if some of the large language model instances have output errors, it can ensure that the correct answer information is determined based on the output of other large language model instances, which helps to reduce errors caused by unreliable single alternative answers, thereby significantly improving the accuracy and robustness of the answer information. At the same time, based on the above preset accuracy conditions, when some of the received alternative answers have met the preset accuracy conditions, the answer information can be determined as soon as possible without waiting for all large language model instances to return alternative answers. Therefore, it is possible to reduce the fluctuation of the response time of the large language model on the basis of ensuring the accuracy and robustness of the answer information.

[0045] Combine the following Figure 5 Further explanation of the stability of the response time of this scheme: Assume Figure 1 The same problem in is problem P, using a large language model instance A. Figure 1 On the basis of the embodiment shown, the question-answering scheme of this specification is used to answer question P, for example, question P is input into large language model instance A and large language model instance B respectively, and then the answer information for question P is determined from the alternative answers returned by the two large language model instances. Figure 5 , Figure 5 The figure shows the fluctuation of response time in the above reply process. Figure 5In , the horizontal axis represents the number of calls of each large language model instance, and the vertical axis represents the response time in milliseconds. Among them, the long and short lines represent the fluctuation of the response time of the large language model instance A during the 100 replies, the dotted line represents the fluctuation of the response time of the large language model instance B during the 100 replies, and the solid line represents the fluctuation of the response time of determining the answer information for question P from the alternative answers. Obviously, the curve represented by the solid line is smoother than the other two curves, and the fluctuation amplitude of its response time is smaller, which indicates that the response time fluctuation of the method of determining the answer information from the alternative answers is smaller and the overall value of the response time is smaller, that is, the response speed is faster. This is because the large language model instance A and the large language model instance B are used to answer question P at the same time. If the alternative answer returned by a single large language model instance has met the preset accuracy condition, the reply information can be determined from the alternative answer as soon as possible without waiting for both large language model instances to return the alternative answers, thereby greatly improving the response speed and reducing the response time.

[0046] In one embodiment, an accuracy threshold may be defined in the preset accuracy condition. By predefining the accuracy threshold as a quantitative standard, it is helpful to improve the accuracy of the final determined answer information according to an objective specific value. When evaluating the accuracy of some of the received alternative answers, the answer accuracy corresponding to each of the received alternative answers is first determined. The answer accuracy corresponding to any alternative answer is used to indicate the probability that the answer content of any alternative answer is the correct answer to the question information. If the first alternative answer with a corresponding answer accuracy greater than or equal to the accuracy threshold appears, it can be determined that the accuracy condition has been met, and the answer content of the alternative answer is determined as the answer information. For example, assuming that the accuracy threshold is 90%, the question information is input into three large language model instances respectively. Large language model instance 1 first returns alternative answer A, and its corresponding answer accuracy is 88% (less than the accuracy threshold). Immediately afterwards, large language model instance 3 returns alternative answer B, and its corresponding answer accuracy is 90.7% (greater than the accuracy threshold). It is determined that the accuracy condition has been met, and the answer content "B" of alternative answer B can be determined as the answer information.

[0047] In addition, although the alternative answers are not exactly the same, the answer contents of some alternative answers may be very similar or even the same. For example, for the alternative answers "book tomorrow's air ticket" and the alternative answer "book a flight to City A", although the two alternative answers are different, their answer contents are both "booking air tickets", rather than other contents such as booking high-speed rail tickets. Therefore, the reply information can be determined according to the comprehensive accuracy corresponding to the content of each answer received. Exemplarily, the comprehensive accuracy corresponding to each answer content can be counted based on the reply accuracy corresponding to each alternative answer. The comprehensive accuracy corresponding to any answer content is used to indicate the probability that any answer content is the correct answer to the question information. For example, assuming that the question information is a calculation multiple-choice question, the calculation multiple-choice question is input into N large language model instances (N≥2) respectively. Among the received alternative answers, there are M alternative answers B (M≤N) and L alternative answers A (M+L≤N). And there are K large language model instances that do not return alternative answers (K=NML). Among them, the answer content of alternative answer B is "B", and the answer content of alternative answer A is "A". Assuming that the accuracy rate of the response corresponding to each alternative answer is 80%, the comprehensive accuracy rate corresponding to each answer content can be calculated separately according to the Bayesian probability. The statistical principle can refer to the following formulas (1) to (2):

[0048]

[0049] Among them, P("B") represents the comprehensive accuracy corresponding to the answer content "B", and P("A") represents the comprehensive accuracy corresponding to the answer content A.

[0050] When the first corresponding answer content with a comprehensive accuracy greater than or equal to the accuracy threshold appears, it is determined that the accuracy condition has been met, and the answer content is determined as the reply information. If the statistical results show that the comprehensive accuracy corresponding to all answer contents does not meet the accuracy threshold, it is possible to wait for other large language model instances that have not returned alternative answers to return alternative answers, and re-count the comprehensive accuracy corresponding to each answer content based on the newly received alternative answers, until the first corresponding answer content with a comprehensive accuracy greater than or equal to the accuracy threshold appears, and the answer content is determined as the reply information.

[0051] In this embodiment, when the first alternative answer and / or answer content that meets the accuracy condition appears, the corresponding answer content is directly determined as the reply information, which significantly improves the response speed and reduces the user's waiting time while ensuring the accuracy of the reply information, thereby helping to improve the user experience.

[0052] In one embodiment, after the reply information is determined when the preset accuracy condition is met, the large language model instance that has not returned the alternative answer can be notified to terminate the processing of the question information. Because the determined reply information has met the accuracy condition, the large language model that has not returned the alternative answer no longer needs to return the alternative answer. This method helps to save system resources and reduce system overhead. And notifying other large language model instances that have not returned alternative answers to terminate processing means that there is no need to wait for these large language models to finish running, and the aforementioned determined reply information can be directly returned to the user or the downstream system can be called based on the reply information, which significantly improves the overall response speed.

[0053] In one embodiment, the accuracy of the reply corresponding to each alternative answer received can be determined by the category to which the question information belongs. Exemplarily, the target category to which the question information belongs in the preset alternative category is first determined. For example, the keywords / words contained in the question information can be extracted to identify the target category to which the question information belongs based on the keywords / words. The alternative categories may include: hotel service category, travel service category, common sense category, education category, etc. Usually, the reply accuracy of different large language model instances for the same alternative category is not exactly the same, and the reply accuracy of the same large language model instance for different alternative categories is not exactly the same. The reply accuracy of any large language model instance for any alternative category can be understood as: the probability that the answer output by the large language model instance for the question information under any alternative category is the correct answer. For example, the reply accuracy of a large language model instance for the education category is 85%. This means that an education-related multiple-choice question (such as a multiple-choice question for the middle school entrance examination) is input into the large language model instance, and the answer output by the large language model instance is B, and the probability that the answer B is the correct answer to the aforementioned multiple-choice question is 85%.

[0054] The alternative categories can be pre-set by those skilled in the art according to the actual application scenarios of the question-answering system, and this specification does not limit this. After the pre-selected categories are set, the answer accuracy of each large language model instance for each alternative category can be pre-determined. In addition, in order to improve the accuracy of the reply information under each alternative category, the accuracy threshold can be pre-defined for each alternative category. In the process of executing the scheme of this specification, the accuracy threshold predefined for the target category is determined according to the target category to which the question information belongs. At the same time, the reply accuracy of the corresponding large language model instance for the target category is determined as: the reply accuracy corresponding to the alternative answer returned by the large language model instance. For example, it is pre-determined that the reply accuracy of large language model instance 1 for the target category is 90%, the reply accuracy of large language model instance 2 for the target category is 85%, and the reply accuracy of large language model instance 3 for the target category is 96%. Then, if the alternative answer to the question information returned by large language model instance 2 is received, it can be determined that the reply accuracy corresponding to the alternative answer is 85%.

[0055] In this embodiment, accuracy thresholds are defined in advance for each alternative category, and the answer accuracy of each major language model instance for each alternative category is determined in advance. Then, the corresponding accuracy thresholds and the answer accuracy corresponding to the alternative answers returned by the major language model instances for the question information are determined according to the target category to which the question information belongs. This helps to simplify the process of determining the answer accuracy corresponding to each alternative answer and improve the reliability of the determined answer accuracy.

[0056] In one embodiment, the answer accuracy of any large language model instance for any candidate category can be determined as follows: obtain at least two test tasks under any candidate category and the standard answers corresponding to each test task. Then input the at least two test tasks into any large language model instance respectively, and obtain the predicted answers output by the large language model instance for each test task respectively. Taking the education category as any candidate category as an example, the college entrance examination questions of the past ten years can be obtained as test tasks, and the standard answers corresponding to each college entrance examination question need to be obtained. Then the college entrance examination questions of the past ten years are respectively input into the same large language model instance, and the predicted answers output by the large language model instance for each college entrance examination question are obtained. The answer accuracy is calculated according to the similarity between each predicted answer and the corresponding standard answer, and the calculation result is used as the answer accuracy of any large language model instance for any candidate category. Among them, there are many similarity calculation methods, such as cosine similarity calculation, Euclidean distance calculation, etc. The method of calculating the answer accuracy based on multiple similarities may include: taking any one of the standard deviation, weighted average, and median of multiple similarities as the answer accuracy. This specification does not limit the specific similarity calculation method and answer accuracy calculation method.

[0057] Alternatively, at least two test tasks may be input into another large language model instance that belongs to the same large language model as any of the aforementioned large language model instances, and the predicted answers output by the other large language model instance for each test task are obtained, and then the answer accuracy is calculated based on the similarity between the predicted answers and the corresponding standard answers. Since any large language model instance and another large language model instance are constructed based on the same large language model, the difference between the two large language model instances is small. By introducing another large language model, the amount of test data can be effectively increased, and the reliability of the calculated answer accuracy can be improved.

[0058] The above method tests the large language model instance through the test task, which helps to more objectively and accurately determine the response accuracy of each large language model instance for each candidate category.

[0059] In one embodiment, a quantity threshold is also defined in the preset accuracy condition. Before the first corresponding answer content whose comprehensive accuracy is greater than or equal to the accuracy threshold appears, if the number of received alternative answers corresponding to any answer content increases to greater than or equal to the quantity threshold, it is determined that the accuracy condition has been met, and any answer content is determined as the reply information. For example, assuming that the quantity threshold is 3, the question information "arrange for late check-out tomorrow morning" is input into 6 large language model instances respectively. Table 1 shows the order returned by the 4 large language model instances and the alternative answers:

[0060] Return order Large language model instance Alternative answers Answer content of the alternative answers 1 Large language model example 2 Late check-out Late check-out 2 Large language model example 4 check out check out 3 Large language model example 3 Late check-out Late check-out 4 Large language model example 1 Late check-out Late check-out

[0061] Table 1

[0062] Combined with Table 1, it is assumed that after receiving the alternative answers returned by the large language model instance 1, it is statistically found that the first corresponding answer content with a comprehensive accuracy greater than or equal to the accuracy threshold has not yet appeared, and the number of alternative answers corresponding to the answer content "late check-out" has been received is 3 (equal to the number threshold), therefore, "late check-out" can be determined as the reply information. In the above process, when the number of alternative answers corresponding to any answer content has increased to greater than or equal to the number threshold, it means that the possibility of any answer content being the correct answer is the highest. Therefore, the above method can not only make decisions quickly and improve response speed, but also select the answer content with the highest possibility in multi-way decision-making.

[0063] Since the comprehensive accuracy of the aforementioned answer content "late check-out" is less than the accuracy threshold, we can continue to wait for other large language model instances that have not returned alternative answers to return alternative answers, and re-count the comprehensive accuracy of each answer content based on the newly received alternative answers. If the first corresponding answer content with a comprehensive accuracy greater than or equal to the accuracy threshold appears, and the answer content is different from the answer content determined based on the quantity threshold, then the reply information needs to be updated based on the first corresponding answer content with a comprehensive accuracy greater than or equal to the accuracy threshold. In this way, the accuracy of the final reply information can be further improved.

[0064] In one embodiment, similar to the accuracy threshold, the quantity threshold can also be determined by the target category to which the question information belongs. Exemplarily, the quantity threshold for each candidate category can be predefined, for example, the quantity threshold for each candidate category can be predefined based on the historical experience of each candidate category in the question-answering process. Then, based on the target category to which the question information belongs, the quantity threshold corresponding to the target category is determined as the quantity threshold in the accuracy condition.

[0065] In addition, the accuracy of the responses of each candidate category in each language model instance can be combined to determine the quantity threshold of each candidate category, so as to determine the answer content with higher accuracy in each candidate category based on the quantity threshold. By predefining the quantity threshold of each candidate category in a targeted manner, it is helpful to determine a more reliable answer content based on a reliable quantity threshold when the accuracy threshold is not met.

[0066] In one embodiment, after the reply information is determined, the reply information can be returned to the questioning user corresponding to the question information. For example, in the case where the question information is a question such as "What is the answer to this multiple-choice question?" or "What is the weather like in City A today?", the reply information is used to indicate the answer corresponding to the question, so the reply information can be directly returned to the questioning user. In addition, the question information may come from task requests such as "check out" or "book a high-speed rail ticket to City A". In this case, the reply information is used to indicate the task intent contained in the task request. Then, after the reply information is determined, it is necessary to execute the target task corresponding to the task intent, such as calling the corresponding application to execute the target task, and return the task execution result to the user who submitted the task request.

[0067] In one embodiment, accuracy evaluation information for the reply information may be obtained. After the reply information or the task execution result is returned to the user, the user's evaluation information for the received feedback is obtained, and the accuracy evaluation information for the reply information is determined based on the evaluation information. Then, the accuracy threshold and / or quantity threshold in the accuracy condition is adjusted based on the accuracy evaluation information.

[0068] For example, if the accuracy evaluation information for the reply information indicates that the reply information is not accurate enough, then the quantity threshold and / or the number of large language model instances required to be called for the question information can be appropriately increased, so that the question-answering scheme of this specification is executed based on more large language model instances, which helps to determine more accurate reply information. Alternatively, the reply accuracy of each large language model instance for the target category to which the question information belongs can be appropriately adjusted. For example, the reply accuracy of each large language model instance for the target category to which the question information belongs can be appropriately reduced, so as to call more large language model instances to execute the question-answering scheme of this specification.

[0069] If the accuracy evaluation information of the reply information shows that the reply information is relatively accurate and the accuracy of the reply information is much higher than the preset expectation, then the accuracy threshold can be appropriately lowered to improve the response speed and reduce the response time.

[0070] In this embodiment, dynamically adjusting the accuracy threshold and / or quantity threshold in the accuracy condition according to the accuracy evaluation information helps to determine more accurate answer information based on the adjusted accuracy condition, while further improving the response speed of the question and answer system.

[0071] Combination Figure 4 In the illustrated embodiment, the question-answering system includes an instruction distribution module 402 and an instruction voting module 404. Based on the functions of the two modules, the instruction distribution module 402 can be further subdivided into a user instruction dispatching center and an instruction parsing center. The user instruction dispatching center is used to receive user instructions and dispatch the user instructions to the instruction parsing center according to the intention contained in the user instructions, so that the instruction parsing center can parse the user instructions and generate question information for the large language model. Figure 6 , Figure 6 FIG. 1 is a schematic diagram of the operation process of a question-answering system provided by an exemplary embodiment. Figure 6 As shown, the question-answering system includes a user instruction dispatching center 602, an instruction parsing center 604 and an analysis center 606. The analysis center 606 is equivalent to the instruction voting module 404, which is used to determine the answer information for the question information from the received alternative answers. The operation process of the question-answering system can be divided into the instruction intention recognition stage, the instruction analysis stage and the instruction execution stage.

[0072] For example, after receiving the user instruction, the user instruction dispatch center 602 may perform preliminary intention recognition on the received user instruction. There are three execution routes to choose from according to different recognition results:

[0073] Route ①: For simple user instructions such as "What is the answer to the multiple-choice question", the user instruction dispatch center 602 can identify the simple intention contained in such simple user instructions. Therefore, when the recognition result shows that the user instruction contains a simple intention, the user instruction dispatch center 602 will send the user instruction to the instruction parsing center 604 to execute the question-answering scheme of this specification on the user instruction and obtain the reply information for the user instruction. Specifically, the instruction parsing center 604 parses the received user instruction, extracts important / key information, and obtains the question information for the large language model. Then, the instruction parsing center 604 sends the question information to the data compression / decompression unit 6061 in the analysis center 606. The data compression / decompression unit 6061 can construct corresponding prompt words based on the question information and compress the question information at the same time. The purpose of compressing the question information here is to reduce the amount of data and improve the response speed of the large language model instance. The prompt word is used to instruct the large language model instance to generate an alternative answer corresponding to the question information. The data compression / decompression unit 6061 will associate the prompt word with the compressed question information and send it to the request scheduling unit 6062. The request scheduling unit 6062 is used to request the call of at least two large language model instances, and input the information set consisting of the compressed question information and prompt words into each large language model instance called. Then, the multi-way voting unit 6063 receives the alternative answers returned by the major language model instances. In the process of receiving the alternative answers, if the multi-way voting unit 6063 has received some alternative answers and the part of the alternative answers has met the preset accuracy conditions, it can determine the reply information for the question information based on the part of the alternative answers, and return the reply information to the data compression / decompression unit 6061. Among them, the specific process of the multi-way voting unit 6063 determining the reply information can be found in Figure 7 The illustrated embodiment is not described in detail here. The data compression / decompression unit 6061 decompresses the reply information and returns the decompressed reply information to the instruction parsing center 604. If the decompressed reply information indicates that there is no need to call other application systems, the instruction parsing center 604 can directly return the decompressed reply information to the questioning user corresponding to the question information. If the question information originates from a task request submitted by the user, and the decompressed reply information is used to indicate the task intent contained in the task request, the instruction parsing center 604 can call the corresponding API to control the application system to execute the target task corresponding to the task intent, and return the task execution result to the corresponding user. ( Figure 6 The route for returning the decompressed reply information / task execution results to the user is not shown)

[0074] Route ②: The user instruction dispatch center 602 cannot identify the intention contained in the complex user instruction. For example, for "the price of apples", the user instruction dispatch center 602 cannot identify whether "apple" refers to fruit or a brand of an electronic product. Therefore, when the recognition result shows that the user instruction contains a complex intention, it is necessary to first execute the question-answering scheme of this specification for the user instruction to identify the instruction intention contained in the user instruction. Specifically, the user instruction dispatch center 602 will send the user instruction to the data compression / decompression unit 6061 in the analysis center 606. The data compression / decompression unit 6061 can construct a corresponding prompt word based on the user instruction, and at the same time convert the user instruction into question information and compressed question information for the large language model. The data compression / decompression unit 6061 will associate the prompt word with the compressed question information and send it to the request dispatch unit 6062. The request dispatch unit 6062 will call at least two large language model instances, and input the information set consisting of the compressed question information and the prompt word into each large language model instance called. Then, the multi-way voting unit 6063 receives the alternative answers returned by each large language model instance. In the process of receiving the alternative answers, if the multi-way voting unit 6063 has received some alternative answers and the partial alternative answers have met the preset accuracy condition, it can determine the reply information for the question information based on the partial alternative answers, and return the reply information to the data compression / decompression unit 6061. The data compression / decompression unit 6061 decompresses the reply information and returns the decompressed reply information to the user instruction dispatch center 602. The decompressed reply information is used to indicate the instruction intention contained in the user instruction.

[0075] Then, the user instruction dispatch center 602 will send the instruction intent obtained in the above process to the instruction analysis center 604 to execute the question-answering solution of this specification on the instruction intent and obtain the reply information for the instruction intent. The process can refer to the corresponding content of route 1 and will not be repeated here.

[0076] Route ③: A question database can be deployed in the question-and-answer system, which is used to store questions / task requests submitted by users in the past, as well as the user intentions corresponding to each question / task request. After receiving the user instruction, the user instruction dispatch center 602 can search the question database to see whether there is a question / task request that matches the user instruction. If so, the user intention corresponding to the retrieved question / task request can be sent to the instruction parsing center 604 to execute the question-and-answer solution of this specification for the user intention and obtain reply information for the user intention. This process can refer to the corresponding content of Route 1 and will not be repeated here. By retrieving user intentions in the question database, the efficiency of identifying the intentions contained in user instructions can be significantly improved.

[0077] The retrieval process may include: the user instruction dispatch center 602 sends the user instruction to the data cleaning unit 608. The data cleaning unit 608 cleans the user instruction, filters out irrelevant information such as stop words, and sends the cleaned instruction to the data evaluation and filtering unit 610. The data evaluation and filtering unit 610 retrieves the question / task request that matches the cleaned instruction in the question database, and when a matching question / task request is retrieved, the user intention corresponding to the retrieved question / task request is evaluated. Here, it is mainly to evaluate whether the obtained user intention matches the user instruction. If the two match, the obtained user intention is returned to the user instruction dispatch center 602, so that the user instruction dispatch center 602 sends the received user intention to the instruction parsing center 604. If the two do not match, it is necessary to execute route ① or route ②. The evaluation operation can ensure that the retrieved user intention is relatively accurate, and then the question and answer solution of this specification is executed based on the accurate user intention.

[0078] Figure 7 This is a schematic diagram of a process for a multi-way voting unit to determine a reply message provided by an exemplary embodiment. Figure 7 , the process is executed by a multi-way voting unit and may include the following steps:

[0079] S701: receiving candidate answers returned by a large language model instance.

[0080] S702: Determine the response accuracy rate corresponding to each received candidate answer.

[0081] As described above, the answer accuracy of the corresponding large language model instance for the target category to which the question information belongs can be determined as: the answer accuracy of the candidate answers returned by the large language model instance.

[0082] In this embodiment, when a certain large language model instance in S701 is the first large language model instance to return an alternative answer, the answer accuracy rate corresponding to the alternative answer returned by the certain large language model instance is determined.

[0083] When a certain large language model instance in S701 is not the first large language model instance that returns an alternative answer, the comprehensive accuracy rate corresponding to each answer content is counted respectively according to the reply accuracy rate corresponding to each alternative answer.

[0084] S703: Determine whether there is a first corresponding alternative answer whose response accuracy is greater than or equal to the accuracy threshold, or a first corresponding answer content whose comprehensive accuracy is greater than or equal to the accuracy threshold.

[0085] If it is determined that it does not exist, jump to S704; if it is determined that it exists, jump to S705.

[0086] In this embodiment, when a certain large language model instance in S701 is the first large language model instance to return an alternative answer, if the answer accuracy corresponding to the alternative answer returned by the large language model instance is greater than or equal to the accuracy threshold, jump to S705. If the answer accuracy corresponding to the alternative answer returned by the large language model instance is less than the accuracy threshold, jump to S704.

[0087] In the case where a certain large language model instance in S701 is not the first large language model instance that returns an alternative answer, if the statistical result shows that the comprehensive accuracy rate corresponding to a certain answer content is greater than or equal to the accuracy rate threshold, jump to S705. If the statistical result shows that the comprehensive accuracy rates corresponding to all answer contents are less than the accuracy rate threshold, jump to S704.

[0088] S704: Count the number of received alternative answers corresponding to each answer content.

[0089] S705: Notify the large language model instance that has not returned an alternative answer to terminate the processing of the question information.

[0090] S706: Determine response information for the question information from some candidate answers that meet the accuracy threshold, and return the response information to the corresponding application system.

[0091] S707: Determine whether there is an answer content for which the number of received corresponding candidate answers is greater than or equal to the number threshold.

[0092] If it exists, jump to S708; if it does not exist, jump to S711.

[0093] S708: Determine the answer content whose received number of corresponding candidate answers is greater than or equal to the number threshold as reply information, and return the reply information to the corresponding application system.

[0094] S709: Determine whether there is a large language model instance that does not return an alternative answer.

[0095] If yes, then jump to S710 and wait for the large language model instance that has not returned the candidate answer to return the candidate answer. If no, then the answer to the question information is successfully completed.

[0096] S710: Wait for the large language model instance that has not returned an alternative answer to return an alternative answer.

[0097] S711: Determine whether there is a large language model instance that does not return an alternative answer.

[0098] If yes, then jump to S710. If no, then jump to S712.

[0099] S712: Notify the corresponding application system of the result of the execution failure.

[0100] Figure 7 The illustrated embodiment calls at least two large language model instances to generate alternative answers in parallel. Even if some of the large language model instances have output errors, it can ensure that the correct reply information is determined based on the output of other large language model instances, which helps to reduce errors caused by unreliable single alternative answers, thereby significantly improving the accuracy and robustness of the reply information. At the same time, by determining the reply information from some of the received alternative answers, the reply information can be determined as early as possible without waiting for all large language model instances to return alternative answers. Therefore, it is possible to reduce the fluctuation of the response time of the large language model while ensuring the accuracy and robustness of the reply information.

[0101] Figure 8 is a schematic structural diagram of a device provided by an exemplary embodiment. Figure 8 At the hardware level, the device includes a processor 802, an internal bus 804, a network interface 806, a memory 808, and a non-volatile memory 810, and may also include hardware required for other functions. One or more embodiments of this specification may be implemented based on software, such as the processor 802 reading the corresponding computer program from the non-volatile memory 810 into the memory 808 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0102] Please refer to Fig. 9 , the question-answering device based on the large language model can be applied to Figure 8 The device shown in the figure can realize the technical solution of this specification. The question-answering device based on the large language model can include: an acquisition unit 901, an input unit 902 and a determination unit 903.

[0103] The acquisition unit 901 is configured to acquire question information for the large language model.

[0104] The input unit 902 is configured to input the question information into at least two large language model instances respectively.

[0105] The determination unit 903 is configured to determine, during the process of receiving the alternative answers returned by each large language model instance, if some alternative answers have been received and the some alternative answers have met the preset accuracy conditions, then the reply information for the question information is determined based on the some alternative answers.

[0106] Optionally, an accuracy threshold is defined in the accuracy condition; the determination unit 903 is specifically used to: determine the reply accuracy corresponding to each received alternative answer; when the first corresponding alternative answer whose reply accuracy is greater than or equal to the accuracy threshold appears, determine that the accuracy condition has been met, and determine the answer content of the alternative answer as the reply information; and / or determine the reply accuracy corresponding to each received alternative answer, and count the comprehensive accuracy corresponding to each answer content according to the determined reply accuracy; when the first corresponding answer content whose comprehensive accuracy is greater than or equal to the accuracy threshold appears, determine that the accuracy condition has been met, and determine the answer content as the reply information.

[0107] Optionally, the device further comprises:

[0108] The notification unit 904 is configured to notify the large language model instance that has not returned an alternative answer to terminate the processing of the question information when the reply information is determined.

[0109] Optionally, determining the response accuracy corresponding to each received alternative answer includes: based on the target category to which the question information belongs in the preset alternative categories, and the predetermined response accuracy of each large language model instance for each alternative category, determining the response accuracy of the corresponding large language model instance for the target category as: the response accuracy of the alternative answer returned by the large language model instance for the question information; wherein the accuracy threshold is an accuracy threshold pre-defined for the target category.

[0110] Optionally, the answer accuracy of any large language model instance for any alternative category is determined in the following manner: obtaining at least two test tasks under any alternative category and the standard answers corresponding to each test task; inputting the at least two test tasks into any large language model instance or another large language model instance belonging to the same large language model, and obtaining the predicted answers output by the corresponding large language model instance for each test task; calculating the answer accuracy based on the similarity between the predicted answers and the corresponding standard answers, as the answer accuracy of any large language model instance for any alternative category.

[0111] Optionally, a quantity threshold is also defined in the accuracy condition; the determination unit 903 is also used for: before the first corresponding answer content with a comprehensive accuracy greater than or equal to the accuracy threshold appears, if the number of received alternative answers corresponding to any answer content increases to greater than or equal to the quantity threshold, it is determined that the accuracy condition has been met, and any answer content is determined as the reply information; in the case that the first corresponding answer content with a comprehensive accuracy greater than or equal to the accuracy threshold is not any of the answer contents, the reply information is updated based on the first corresponding answer content with a comprehensive accuracy greater than or equal to the accuracy threshold.

[0112] Optionally, the quantity threshold is determined in the following manner: according to the target category to which the question information belongs in the preset candidate categories and the predefined quantity thresholds for each candidate category, the quantity threshold corresponding to the target category is determined as the quantity threshold in the accuracy condition.

[0113] Optionally, the device further comprises:

[0114] The evaluation unit 905 is configured to obtain accuracy evaluation information for the reply information, and adjust the accuracy threshold and / or the quantity threshold based on the accuracy evaluation information.

[0115] Optionally, the device further comprises:

[0116] The return unit 906 is configured to return the reply information to the questioning user corresponding to the question information; or, when the question information originates from a task request submitted by the user and the reply information is used to indicate the task intent contained in the task request, execute the target task corresponding to the task intent and return the task execution result to the corresponding user.

[0117] Optionally, the at least two large language model instances correspond to the same large language model; or, the at least two large language model instances correspond to different large language models respectively.

[0118] Based on the same concept as the above method, this specification also provides an electronic device, including: a processor; a memory for storing processor executable instructions; wherein the processor implements the steps of the method described in any of the above embodiments by running the executable instructions.

[0119] Based on the same concept as the above method, this specification also provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0120] Based on the same concept as the above method, this specification also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any of the above embodiments when executed by a processor.

Claims

1. A question-answering method based on a large language model, characterized in that: include: Get question information for large language models; Inputting the question information into at least two large language model instances respectively; In the process of receiving the alternative answers returned by each large language model instance, if some alternative answers have been received and the partial alternative answers have met the preset accuracy conditions, the reply information for the question information is determined based on the partial alternative answers.

2. The method according to claim 1, characterized in that The accuracy condition defines an accuracy threshold; if some alternative answers have been received and the some alternative answers have met the preset accuracy condition, then based on the some alternative answers, answer information for the question information is determined, including: Determine the response accuracy corresponding to each received alternative answer; when the first alternative answer whose corresponding response accuracy is greater than or equal to the accuracy threshold appears, determine that the accuracy condition has been met, and determine the answer content of the alternative answer as the response information; and / or, Determine the reply accuracy corresponding to each received alternative answer, and count the comprehensive accuracy corresponding to each answer content according to the determined reply accuracy; when the first corresponding answer content with a comprehensive accuracy greater than or equal to the accuracy threshold appears, determine that the accuracy condition has been met, and determine the answer content as the reply information.

3. The method according to claim 2, characterized in that Also includes: When the answer information is determined, the large language model instance that has not returned an alternative answer is notified to terminate the processing of the question information.

4. The method according to claim 2, characterized in that: The step of determining the response accuracy rate corresponding to each received candidate answer includes: According to the target category to which the question information belongs in the preset candidate categories, and the predetermined answer accuracy rates of the major language model instances for each candidate category, the answer accuracy rate of the corresponding large language model instance for the target category is determined as: the answer accuracy rate of the candidate answer returned by the large language model instance for the question information; The accuracy threshold is a pre-defined accuracy threshold for the target category.

5. The method according to claim 4, characterized in that The accuracy of the response of any large language model instance for any candidate category is determined as follows: Obtain at least two test tasks under any of the candidate categories and standard answers corresponding to each test task; Inputting the at least two test tasks into any one of the large language model instances or another large language model instance belonging to the same large language model, respectively, and obtaining a prediction answer output by the corresponding large language model instance for each test task; The answer accuracy is calculated according to the similarity between the predicted answer and the corresponding standard answer to serve as the answer accuracy of any large language model instance for any candidate category.

6. The method according to claim 2, characterized in that The accuracy condition also defines a quantity threshold; if some alternative answers have been received and the some alternative answers have met the preset accuracy condition, then based on the some alternative answers, answer information for the question information is determined, further comprising: Before the first corresponding answer content with a comprehensive accuracy greater than or equal to the accuracy threshold appears, if the number of received alternative answers corresponding to any answer content increases to be greater than or equal to the number threshold, it is determined that the accuracy condition has been met, and any answer content is determined as the reply information; When the first corresponding answer content whose comprehensive accuracy is greater than or equal to the accuracy threshold is not any of the answer contents, the reply information is updated based on the first corresponding answer content whose comprehensive accuracy is greater than or equal to the accuracy threshold.

7. The method according to claim 6, characterized in that The quantity threshold is determined in the following manner: According to the target category to which the question information belongs in the preset candidate categories and the predefined quantity thresholds for each candidate category, the quantity threshold corresponding to the target category is determined as the quantity threshold in the accuracy condition.

8. The method according to claim 6, characterized in that Also includes: Acquire accuracy evaluation information for the reply information, and adjust the accuracy threshold and / or the quantity threshold based on the accuracy evaluation information.

9. The method according to claim 1, characterized in that: Also includes: Returning the answer information to the questioning user corresponding to the question information; or, In the case where the question information originates from a task request submitted by a user and the reply information is used to indicate the task intent contained in the task request, a target task corresponding to the task intent is executed and the task execution result is returned to the corresponding user.

10. The method according to claim 1, characterized in that The at least two large language model instances correspond to the same large language model; Alternatively, the at least two large language model instances correspond to different large language models respectively.

11. An electronic device, comprising: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method according to any one of claims 1 to 10 by executing the executable instructions.

12. A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 10.

13. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

Images