Large model deployment method and device
Through the strategy of filtering and sorting based on the estimated accuracy and resource consumption of large models, the problem of insufficient deployment strategies of large models in Q&A scenarios is solved, and the effects of fast feedback and low resource consumption are achieved.
Patent Information
- Application Number
- CN202411074975.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-08-07
AI Technical Summary
There is a general large-model deployment strategy in the Q&A scenario, which can not only quickly feedback answers that meet user needs, but also consume less resources during the Q&A interaction process.
By receiving user input questions, filter and sort based on the estimated accuracy and resource consumption of the big model, select the appropriate big model to answer until the user needs are met or the preset conditions are met.
It realizes quick feedback of answers that meet user needs in Q&A interaction, while reducing resource consumption and improving user experience.
Smart Images

Figure CN118607643B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to machine learning, online learning and large language models, and more particularly to a large model deployment method and device thereof. Background Art
[0002] Large language models (also known as big models) have made a lot of progress in recent years, and many different types of big models have performed well. However, this progress also comes at the cost of increased resource consumption and increased latency during training and reasoning, which poses challenges to the actual deployment of large language models. For different user questions, how to choose a suitable big model to answer is an issue that needs to be considered.
[0003] In the process of realizing the concept of the present invention, the inventors found that there is currently a lack of a relatively universal large-model deployment strategy in the question-and-answer scenario, which can not only quickly feedback answers that meet user needs based on the questions input by the user, but also consume fewer resources during the question-and-answer interaction process. Summary of the invention
[0004] In view of this, the present invention provides a large model deployment method and device thereof.
[0005] One aspect of the present invention provides a large model deployment method, including: receiving a question to be answered input by a user during an Nth question-answering interaction; based on the estimated accuracy of each large model in the first large model set in answering the question, screening K large models from the first large model set to obtain a second large model set, wherein the first large model set is determined in the N-1th question-answering interaction, and the estimated accuracy is determined based on the number of times each large model in the first large model set is called in the first N-1 question-answering interactions and the answer to the question after being called, K is a positive integer, and K is less than the number of large models in the first large model set; when it is determined that there is no question to be answered in the cache, or when it is determined that there is a question to be answered in the cache and the answer corresponding to the question to be answered does not meet the user's needs, based on calling each large model in the second large model set According to the estimated resources required to answer the questions, each large model in the second largest model set is sorted to obtain the third largest model set, wherein M answered questions and answered answers corresponding to each of the M answered questions are stored in the cache, N and M are positive integers greater than or equal to 1, and the estimated resources are obtained based on the estimated consumption when each large model in the second largest model set is called to answer questions in the first N-1 question-and-answer interactions; and the target large models are determined in sequence from the third largest model set according to the sorting, so that the target answer is output after the question to be answered is input into the target large model, until the target answer meets the user's needs or the target answers output by the K large models in the third largest model set do not meet the user's needs, then stop determining the target large model from the third largest model set and complete the Nth question-and-answer interaction.
[0006] According to an embodiment of the present invention, sorting includes ascending sorting, K≥2; determining a target large model from the third largest model set according to the sorting, so that after the question to be answered is input into the target large model, a target answer is output, until the target answer meets the user's needs or the target answers output by the K large models in the third largest model set do not meet the user's needs, stopping determining the target large model from the third largest model set, including: determining the first large model in the third largest model set after ascending sorting as the first target large model, so that after the question to be answered is input into the first target large model, a first target answer is output; sending the first target answer to the user, so that the user returns a first feedback result based on the first target answer; and when it is determined that the first feedback result indicates that the first target answer does not meet the user's needs In this case, the second largest model in the third largest model set after ascending sorting is determined as the second target large model, so that after the question to be answered is input into the second target large model, the second target answer is output, wherein the first target large model and the second target large model are different; the second target answer is sent to the user, so that the user returns a second feedback result based on the second target answer; when it is determined that the second feedback result represents that the second target answer meets the user's needs, stop determining the target large model from the third largest model set and complete the Nth question-answer interaction; and when it is determined that all K large models in the third largest model set are called and the target answers output by the K large models do not meet the user's needs, stop determining the target large model from the third largest model set and complete the Nth question-answer interaction.
[0007] According to an embodiment of the present invention, the large model deployment method also includes: for each large model in the second large model set: respectively counting the consumption when calling the large model to answer questions and the number of times the large model is called in the first N-1 question-and-answer interactions; and based on the consumption when calling the large model to answer questions and the number of times the large model is called in the first N-1 question-and-answer interactions, determining the estimated resources required to call the large model to answer questions.
[0008] According to an embodiment of the present invention, based on the consumption when calling the big model to answer questions in the first N-1 question-answer interactions and the number of times the big model is called, the estimated resources required to be consumed when calling the big model to answer questions are determined, including: determining the average estimated resources according to the ratio of the consumption when calling the big model to answer questions in the first N-1 question-answer interactions and the number of times the big model is called; determining the correction resources used to correct the average estimated resources based on the preset number of answer questions, the number of times the big model is called and preset parameters; and determining the estimated resources required to be consumed when calling the big model to answer questions according to the difference between the average estimated resources and the correction resources.
[0009] According to an embodiment of the present invention, the big model deployment method also includes: when determining that the Nth question-and-answer interaction is completed, based on the consumption of calling the target big model to answer unanswered questions, updating the consumption of calling the target big model to answer questions in the previous N-1 question-and-answer interactions; updating the number of times the target big model was called in the previous N-1 question-and-answer interactions; and updating the estimated resources required to consume when calling the target big model to answer questions based on the updated consumption of calling the target big model to answer questions in the previous N-1 question-and-answer interactions and the updated number of times the target big model was called in the previous N-1 question-and-answer interactions.
[0010] According to an embodiment of the present invention, the large model deployment method also includes: for each large model in the first large model set: counting the number of times the large model is called in the first N-1 question-and-answer interactions and the number of times the answers to the questions answered after the large model is called meet the user's needs; and based on the number of times the answers to the questions answered after the large model is called meet the user's needs and the number of times the large model is called, determining the estimated accuracy of the large model's answer to the question.
[0011] According to an embodiment of the present invention, the big model deployment method also includes: when it is determined that the Nth question-and-answer interaction is completed, based on the target answer, updating the number of times the answers to questions answered after the target big model is called in the previous N-1 question-and-answer interactions meet user needs; updating the number of times the target big model is called in the previous N-1 question-and-answer interactions; and updating the estimated accuracy of the target big model's answers to questions based on the number of times the answers to questions answered after the target big model is called in the previous N-1 question-and-answer interactions after the update meet user needs and the number of times the target big model is called in the previous N-1 question-and-answer interactions after the update.
[0012] According to an embodiment of the present invention, based on the estimated accuracy of each large model in the first large model set in answering questions, K large models are screened from the first large model set to obtain a second large model set, including: according to the estimated accuracy of each large model in the first large model set in answering questions, each large model in the first large model set is sorted in descending order to obtain a sorting result; and according to the sorting result, the first K large models are screened from the first large model set to obtain a second large model set.
[0013] According to an embodiment of the present invention, the large model deployment method also includes: collecting P answered questions generated within a historical time period and the answered answers corresponding to each of the P answered questions; determining the probability of each of the P answered questions being generated within the historical time period; screening M answered questions whose probabilities meet preset conditions from the P answered questions, and storing the M answered questions and the answered answers corresponding to each of the M answered questions in a cache, where P≥M and P is a positive integer; when it is determined that the Nth question-answering interaction is completed, updating the cache based on the target answer and the question to be answered.
[0014] Another aspect of the present invention provides a large model deployment device, including: a receiving module, used to receive the unanswered questions input by the user during the Nth question-answer interaction; a screening module, used to screen K large models from the first large model set based on the estimated accuracy of each large model in the first large model set in answering the questions, to obtain a second large model set, wherein the first large model set is determined in the N-1th question-answer interaction, and the estimated accuracy is determined based on the number of times each large model in the first large model set is called in the first N-1 question-answer interactions and the answers to the questions answered after being called, K is a positive integer, and K is less than the number of large models in the first large model set; a sorting module, used to sort the large models based on the call when it is determined that there is no unanswered question in the cache, or when it is determined that there is a unanswered question in the cache and the answer corresponding to the unanswered question does not meet the user's needs. The estimated resources required for each large model in the second largest model set to answer questions are sorted to obtain the third largest model set, wherein M answered questions and answered answers corresponding to each of the M answered questions are stored in the cache, N and M are both positive integers, and the estimated resources are obtained based on the estimated consumption when each large model in the second largest model set is called to answer questions in the first N-1 question-and-answer interactions; and a determination module is used to determine the target large model from the third largest model set in sequence according to the sorting, so as to input the question to be answered into the target large model and then output the target answer, until the target answer meets the user's needs or the target answers output by the K large models in the third largest model set do not meet the user's needs, then stop determining the target large model from the third largest model set and complete the Nth question-and-answer interaction.
[0015] According to an embodiment of the present invention, when it is determined that there is no unanswered question input by the user in the cache, or when it is determined that there is an unanswered question in the cache and the answer corresponding to the unanswered question does not meet the user's needs, based on the accuracy of the large model answering the question, the large model set that meets the conditions is screened, and then based on the resources consumed when calling the large model to answer the question, the large models in the large model set are sorted, and the target large model is determined in turn from the sorted large model set, and the target large model is used to answer the unanswered question input by the user. Through the cache mechanism, it is possible to achieve that only when there is no unanswered question input by the user in the cache, or although there is an unanswered question input by the user in the cache, but the answer corresponding to the unanswered question does not meet the user's needs, the target large model needs to be determined, and the resources consumed when calling the large model to answer the question can be reduced with the help of the cache, and the answer that meets the user's needs can be quickly fed back. In addition, through the large model strategy selection mechanism, both the accuracy of the large model answering the question and the resources consumed when answering the question are considered, so that the target large model determined can accurately answer the question and consume less resources when answering the question. The large model deployment strategy that combines the caching mechanism with the large model strategy selection mechanism can improve the user experience and solve the problem of quickly feeding back answers that meet user needs based on the questions entered by the user, while also consuming fewer resources during the question-answering interaction process. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:
[0017] Figure 1 A flowchart of a large model deployment method according to an embodiment of the present invention is shown;
[0018] Figure 2 A schematic diagram of question-answer interaction according to an embodiment of the present invention is shown;
[0019] Figure 3 A schematic diagram of question-answer interaction according to another embodiment of the present invention is shown;
[0020] Figure 4 A block diagram of a large model deployment device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0021] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.
[0022] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.
[0023] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.
[0024] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).
[0025] In the process of realizing the concept of the present invention, the inventors found that there is currently a lack of a relatively universal large-model deployment strategy in the question-and-answer scenario, which can not only quickly feedback answers that meet user needs based on the questions input by the user, but also consume fewer resources during the question-and-answer interaction process.
[0026] Based on this, an embodiment of the present invention provides a large model deployment method and device. The method includes: receiving a question to be answered input by a user during the Nth question-answering interaction; based on the estimated accuracy of each large model in the first large model set in answering the question, screening K large models from the first large model set to obtain a second large model set, wherein the first large model set is determined in the N-1th question-answering interaction, and the estimated accuracy is determined based on the number of times each large model in the first large model set is called in the first N-1 question-answering interactions and the answer to the question after being called, K is a positive integer, and K is less than the number of large models in the first large model set; when it is determined that there is no question to be answered in the cache, or when it is determined that there is a question to be answered in the cache and the answer corresponding to the question to be answered does not meet the user's needs, based on the consumption required to call each large model in the second large model set to answer the question The estimated resources consumed are sorted for each large model in the second largest model set to obtain the third largest model set, wherein the cache stores M answered questions and answered answers corresponding to each of the M answered questions, N and M are positive integers greater than or equal to 1, and the estimated resources are obtained based on the estimated consumption when each large model in the second largest model set is called to answer questions in the first N-1 question-and-answer interactions; and the target large models are determined in sequence from the third largest model set according to the sorting, so that the target answer is output after the question to be answered is input into the target large model, until the target answer meets the user's needs or the target answers output by the K large models in the third largest model set do not meet the user's needs, then stop determining the target large model from the third largest model set and complete the Nth question-and-answer interaction.
[0027] The following will be passed Figure 1~Figure 3 The large model deployment method of the embodiment of the present invention is described in detail.
[0028] Figure 1 A flow chart of a large model deployment method according to an embodiment of the present invention is shown.
[0029] like Figure 1 As shown, the large model deployment method 100 includes operations S110 to S140.
[0030] In operation S110, a question to be answered input by a user during the Nth question-answer interaction is received.
[0031] In operation S120, based on the estimated accuracy of each large model in the first large model set in answering the question, K large models are screened from the first large model set to obtain a second large model set.
[0032] In operation S130, when it is determined that there are no unanswered questions in the cache, or when it is determined that there are unanswered questions in the cache and the answers to the unanswered questions do not meet the user's needs, each large model in the second largest model set is sorted based on the estimated resources required to call each large model in the second largest model set to answer the question, to obtain the third largest model set.
[0033] In operation S140, the target large model is determined in sequence from the third largest model set according to the sorting, so that the target answer is output after the question to be answered is input into the target large model. When the target answer meets the user's needs or the target answers output by the K large models in the third largest model set do not meet the user's needs, the target large model is determined from the third largest model set to complete the Nth question-answering interaction.
[0034] According to an embodiment of the present invention, the first large model set is determined in the N-1th question-answer interaction. For example, the large models participating in the question-answer interaction in the N-1th question-answer interaction can be determined as the first large model set. If N is 1, it can be the first question-answer interaction, and the N-1th question-answer interaction can be understood as the initialization state. The first large model set can be determined by the initialization state. For example, a plurality of randomly determined large models can be determined as the first large model set. , where l L represents the Lth largest model l in the first largest model set. The probability of each question being entered by the user , the accuracy of each large model in answering the questions to be answered And the consumption required for each large model to answer the questions to be answered are all unknown, among which, is a sub-Gaussian random noise with expectation 0 and .
[0035] According to an embodiment of the present invention, the estimated accuracy is determined based on the number of times each large model in the first large model set is called in the first N-1 question-answer interactions and the answers to the questions answered after being called. For example, the estimated accuracy can be determined based on the proportion of answers that meet the needs of users after each large model in the first large model set is called in the first N-1 question-answer interactions. If N is 1, the estimated accuracy can be randomly determined by the random number of times each large model in the first large model set is called in the initialization state and the answers to the questions answered after being called.
[0036] According to an embodiment of the present invention, K is a positive integer, and K is less than the number of large models in the first large model set. K can be configured according to actual question-answering interaction requirements and is not limited here.
[0037] According to an embodiment of the present invention, M answered questions and answered answers corresponding to each of the M answered questions are stored in the cache, and both N and M are positive integers greater than or equal to 1.
[0038] According to an embodiment of the present invention, during the Nth question-answer interaction, when receiving the unanswered question input by the user, it can be determined whether there is the unanswered question in the cache. If it is determined that there is the unanswered question in the cache, the answered answer corresponding to the unanswered question is sent to the user. If it is determined that there is no unanswered question in the cache, or if it is determined that there is a unanswered question in the cache and the answer corresponding to the unanswered question does not meet the user's needs, the target large model is determined.
[0039] According to an embodiment of the present invention, the estimated resources may be obtained based on the estimated consumption when each large model in the second large model set is called to answer questions in the first N-1 question-answering interactions. For example, the consumption when each large model answers questions may include but is not limited to at least one of the following: electricity fees, network fees, fees required for use rights, etc.
[0040] According to an embodiment of the present invention, sorting may include ascending sorting.
[0041] According to an embodiment of the present invention, the target large model may be determined in sequence from the third largest model set in ascending order, starting from the first one in the order and in a direction from small to large.
[0042] According to an embodiment of the present invention, when it is determined that there is no unanswered question input by the user in the cache, or when it is determined that there is an unanswered question in the cache and the answer corresponding to the unanswered question does not meet the user's needs, based on the accuracy of the large model answering the question, the large model set that meets the conditions is screened, and then based on the resources consumed when calling the large model to answer the question, the large models in the large model set are sorted, and the target large model is determined in turn from the sorted large model set, and the target large model is used to answer the unanswered question input by the user. Through the cache mechanism, it is possible to achieve that only when there is no unanswered question input by the user in the cache, or although there is an unanswered question input by the user in the cache, but the answer corresponding to the unanswered question does not meet the user's needs, the target large model needs to be determined, and the resources consumed when calling the large model to answer the question can be reduced with the help of the cache, and the answer that meets the user's needs can be quickly fed back. In addition, through the large model strategy selection mechanism, both the accuracy of the large model answering the question and the resources consumed when answering the question are considered, so that the target large model determined can accurately answer the question and consume less resources when answering the question. The large model deployment strategy that combines the caching mechanism with the large model strategy selection mechanism can improve the user experience and solve the problem of quickly feeding back answers that meet user needs based on the questions entered by the user, while also consuming fewer resources during the question-answering interaction process.
[0043] According to an embodiment of the present invention, the large model deployment method may include the following: Figure 1 In addition to operations S110 to S140 shown, the following operations may also be included: for each large model in the second large model set: separately counting the consumption when calling the large model to answer questions and the number of times the large model is called in the first N-1 question-and-answer interactions; and based on the consumption when calling the large model to answer questions and the number of times the large model is called in the first N-1 question-and-answer interactions, determining the estimated resources required to call the large model to answer questions.
[0044] According to an embodiment of the present invention, the average consumption of calling the large model to answer questions can be determined based on the ratio of the consumption of calling the large model to answer questions in the first N-1 question-answering interactions to the number of times the large model is called. The average consumption is determined as the estimated resources required to be consumed when calling the large model to answer questions.
[0045] According to an embodiment of the present invention, by adopting a statistical method, the consumption of calling a large model to answer questions can be estimated, which is conducive to continuously optimizing the deployment of the large model during the question-answering interaction.
[0046] According to an embodiment of the present invention, the inventors found that although using a statistical method to determine the average consumption as the estimated resources required to consume when calling a large model to answer questions can simply and quickly estimate the consumption, the estimated resources determined in this way have a large error, which is not conducive to the deployment of large models.
[0047] In addition, the inventors also found that the multi-armed bandit problem is a classic equilibrium exploration-defense dilemma problem in online learning, which is mainly used to provide a better selection strategy to improve efficiency. In the Bandit model, there are usually multiple arms (or entries), each with a different average return. The entire model needs to make multiple rounds of decisions for optimization. In each round, the model needs to select the arm to be pulled in this round based on the historical information of the previous rounds. After that, the model will receive feedback information about the arm it selected to update the model, find the arm with the largest average return among all arms, and maximize the cumulative return of all rounds. However, in the question-and-answer interaction process of the present invention, the goal of deploying a large model is to have low consumption and high accuracy when calling the deployed large model to answer questions.
[0048] Based on this, another embodiment of the present invention determines the estimated resources required to call the large model to answer questions based on the consumption when calling the large model to answer questions in the first N-1 question-and-answer interactions and the number of times the large model is called, and may include the following operations: determining the average estimated resources based on the ratio of the consumption when calling the large model to answer questions in the first N-1 question-and-answer interactions and the number of times the large model is called; determining the corrected resources used to correct the average estimated resources based on the preset number of answer questions, the number of times the large model is called and preset parameters; and determining the estimated resources required to call the large model to answer questions based on the difference between the average estimated resources and the corrected resources.
[0049] According to an embodiment of the present invention, the preset number of times to answer questions may be determined according to the number of questions to be answered that the user is allowed to input in the question-answering system within a preset optimization period. The preset optimization period may be, for example, one day. The preset parameter may be a parameter that is adjustable during the question-answering interaction.
[0050] According to an embodiment of the present invention, based on the number of preset answer questions, the number of times the large model is called, and the preset parameters, determining the correction resource for correcting the average estimated resource may include the following operations: determining the logarithm of the number of preset answer questions. Determining the correction resource based on the ratio of the logarithm of the number of preset answer questions to the number of times the large model is called and the preset parameters. For example, the ratio of the logarithm of the number of preset answer questions to the number of times the large model is called may be squared and multiplied by the preset parameters to obtain the correction resource.
[0051] For example, the estimated resources can be expressed as shown in the following formula (1):
[0052] (1)
[0053] in, Indicates that the large model l at the tth question-answer interaction j estimated resources; Indicates that the large model l is called in the first t-1 question-answer interactions j consumption when answering questions; Indicates that the large model l in the first t-1 question-answer interactions j The number of times it was called; represents the preset parameter, where x has no actual meaning and is a symbol used to distinguish it from the preset accuracy parameter in the following formula (2); T represents the number of preset answer questions.
[0054] According to an embodiment of the present invention, by using corrected resources to correct the average estimated resources, the estimated resources determined in this way can be close to more real resources, reduce the error of the estimated resources, facilitate the deployment of large models, and solve problems that the multi-armed tiger (Bandit) model cannot solve.
[0055] According to an embodiment of the present invention, the large model deployment method may include the following: Figure 1 In addition to operations S110 to S140 shown, the following operations may also be included: for each large model in the first large model set: counting the number of times the large model is called in the first N-1 question-answer interactions and the number of times the answers to the questions answered by the large model after being called meet the user's needs; and determining the estimated accuracy of the large model's answer to the question based on the number of times the answers to the questions answered by the large model after being called meet the user's needs and the number of times the large model is called.
[0056] According to an embodiment of the present invention, the estimated accuracy of the large model answering questions can be determined based on the ratio of the number of times the answers to questions that meet user needs after the large model is called in the first N-1 question-answer interactions to the number of times the large model is called.
[0057] According to an embodiment of the present invention, by adopting a statistical method, the accuracy of the large model in answering questions can be estimated, which is conducive to continuously optimizing the deployment of the large model during the question-answering interaction process.
[0058] However, the inventors also found that although the accuracy can be simply and quickly estimated by using a statistical method to determine the probability that the answer to the question after the large model is called meets the user's needs, the estimated accuracy determined in this way has a large error, which is not conducive to the deployment of the large model. According to an embodiment of the present invention, in the question-and-answer interaction process of the present invention, the goal of deploying a large model is to have low consumption and high accuracy when calling the deployed large model to answer questions. Based on this, the average estimated accuracy can also be determined based on the ratio of the number of times the answer to the question after the large model is called in the first N-1 question-and-answer interactions to the number of times the large model is called; based on the preset number of times the question is answered, the number of times the large model is called, and the preset accuracy parameters, the corrected accuracy used to correct the average estimated accuracy is determined; and the estimated accuracy of the large model answering the question is determined based on the sum of the average estimated accuracy and the corrected accuracy.
[0059] According to an embodiment of the present invention, the preset number of times of answering questions may be determined according to the number of questions to be answered that the user is allowed to input in the question-answering system within a preset optimization period. The preset optimization period may be, for example, one day. The preset accuracy parameter may be a parameter that is adjustable during the question-answering interaction.
[0060] According to an embodiment of the present invention, a method for determining the corrected accuracy may include an operation: the logarithm of the number of preset answer questions may be determined. The corrected accuracy may be determined based on the ratio of the logarithm of the number of preset answer questions to the number of times the large model is called and a preset accuracy parameter. For example, the square root of the ratio of the logarithm of the number of preset answer questions to the number of times the large model is called may be multiplied by the preset accuracy parameter to obtain the corrected accuracy.
[0061] For example, the estimated accuracy can be expressed as shown in the following formula (2):
[0062] (2)
[0063] in, Indicates that the large model l at the tth question-answer interaction j The accuracy of the estimate; Indicates that the large model l in the first t-1 question-answer interactions j The number of times the answer to the question after being called meets the user's needs; Indicates that the large model l in the first t-1 question-answer interactions j The number of times it was called; represents the preset accuracy parameter, where y has no actual meaning and is a symbol used to distinguish it from the preset parameter in the above formula (1); T represents the number of preset answer questions.
[0064] According to an embodiment of the present invention, by using the corrected accuracy to correct the average estimated accuracy, the estimated accuracy determined thereby can be close to a more realistic accuracy, thereby reducing the error of the estimated accuracy and facilitating the deployment of large models.
[0065] According to an embodiment of the present invention, for the above Figure 1 Operation S120, based on the estimated accuracy of each large model in the first large model set in answering the question, screens K large models from the first large model set to obtain a second large model set, may include the operations of: sorting each large model in the first large model set in descending order according to the estimated accuracy of each large model in the first large model set in answering the question to obtain a sorting result; and screening the first K large models from the first large model set according to the sorting result to obtain a second large model set.
[0066] According to an embodiment of the present invention, based on the estimated accuracy determined during the first N-1 question-answer interactions, accuracy screening is performed during the Nth question-answer interaction, which can improve the accuracy of answering questions by the large model.
[0067] According to an embodiment of the present invention, the large model deployment method may include the following: Figure 1 In addition to operations S110 to S140 shown, the following operations may also be included: when it is determined that the Nth question-and-answer interaction is completed, based on the consumption of calling the target large model to answer unanswered questions, updating the consumption of calling the target large model to answer questions in the previous N-1 question-and-answer interactions; updating the number of times the target large model was called in the previous N-1 question-and-answer interactions; and updating the estimated resources required to consume when calling the target large model to answer questions based on the updated consumption of calling the target large model to answer questions in the previous N-1 question-and-answer interactions and the updated number of times the target large model was called in the previous N-1 question-and-answer interactions.
[0068] According to an embodiment of the present invention, the consumption when calling the target big model to answer unanswered questions can be determined based on the electricity fee, network fee, fee required for usage rights, etc. when the target big model answers unanswered questions during the actual interaction process.
[0069] According to an embodiment of the present invention, when it is determined that the Nth question-answering interaction is completed, the consumption of calling the target large model to answer unanswered questions is determined. The consumption is cumulatively summed with the consumption of calling the target large model to answer questions in the previous N-1 question-answering interactions to obtain the consumption of calling the target large model to answer questions in the previous N question-answering interactions, that is, the consumption of calling the target large model to answer questions in the previous N-1 question-answering interactions after update.
[0070] According to an embodiment of the present invention, by counting the number of times the target big model is called during the Nth question-and-answer interaction, and then cumulatively summing the number of times the target big model is called this time and the number of times the target big model is called in the previous N-1 question-and-answer interactions, the number of times the target big model is called in the previous N question-and-answer interactions is obtained, that is, the number of times the target big model is called in the previous N-1 question-and-answer interactions after update.
[0071] According to an embodiment of the present invention, based on the consumption when calling the target large model to answer questions in the updated first N-1 question-and-answer interactions and the number of times the target large model is called in the updated first N-1 question-and-answer interactions, a method for updating the estimated resources required to be consumed when calling the target large model to answer questions may be similar to the above-mentioned method of determining the estimated resources required to be consumed when calling the large model to answer questions for each large model in the second large model set in the present invention, and will not be repeated here.
[0072] According to an embodiment of the present invention, by updating in real time the estimated resources required to call the target big model to answer questions in the previous question-and-answer interaction when each question-and-answer interaction is completed, it is beneficial to continuously optimize the selection of the big model from the direction of consumption when calling the big model to answer questions.
[0073] According to an embodiment of the present invention, the large model deployment method may include the following steps: Figure 1 In addition to operations S110 to S140 shown, the following operations may also be included: when it is determined that the Nth question-and-answer interaction is completed, based on the target answer, updating the number of times the target big model is called to answer questions in the previous N-1 question-and-answer interactions to meet user needs; updating the number of times the target big model is called in the previous N-1 question-and-answer interactions; and updating the estimated accuracy of the target big model in answering questions based on the updated number of times the target big model is called to answer questions in the previous N-1 question-and-answer interactions to meet user needs and the updated number of times the target big model is called in the previous N-1 question-and-answer interactions.
[0074] According to an embodiment of the present invention, when it is determined that the Nth question-and-answer interaction is completed, it can be determined whether the target answer meets the user's needs. In the case of meeting the user's needs, the number of times the answer to the question after the target large model is called in the first N-1 question-and-answer interactions meets the user's needs and the number of times the user's needs are met in the Nth question-and-answer interaction can be cumulatively summed to obtain the number of times the answer to the question after the target large model is called in the first N question-and-answer interactions meets the user's needs, that is, the updated number of times the answer to the question after the target large model is called in the first N-1 question-and-answer interactions meets the user's needs. In the case of not meeting the user's needs, the number of times the answer to the question after the target large model is called in the first N-1 question-and-answer interactions meets the user's needs can be used as the number of times the answer to the question after the target large model is called in the first N question-and-answer interactions meets the user's needs, that is, the number of times the answer to the question after the target large model is called in the first N-1 question-and-answer interactions meets the user's needs is not updated.
[0075] According to an embodiment of the present invention, the method for determining the estimated accuracy of answering questions for the updated target large model may be similar to the method for determining the estimated accuracy of answering questions for each large model in the first large model set, and will not be repeated here.
[0076] According to an embodiment of the present invention, by updating in real time the accuracy of answering questions after the target big model is called in the previous question-and-answer interaction when each question-and-answer interaction is completed, it is beneficial to continuously optimize the selection of the big model from the direction of the accuracy of the big model in answering questions.
[0077] Figure 2 A schematic diagram of question-answer interaction according to an embodiment of the present invention is shown.
[0078] According to an embodiment of the present invention, the sorting may include ascending sorting. The large model deployment method of the embodiment of the present invention may be executed by a question-answering interactive system.
[0079] Regarding the above Figure 1 In operation S140, a target large model is determined from the third largest model set according to the sorting, so that the target large model outputs a target answer after the question to be answered is input into the target large model until the target answer meets the user's needs or the target answers output by the K large models in the third largest model set do not meet the user's needs, and then the target large model is stopped from being determined from the third largest model set, which may include the following steps: Figure 2 Operation S210 to operation S280 are shown.
[0080] In operation S210, the first largest model in the third largest model set after ascending sorting is determined as the first target large model, so that the first target answer is output after the question to be answered is input into the first target large model.
[0081] In operation S220, the first target answer is sent to the user.
[0082] In operation S230, the user returns a first feedback result based on the first target answer.
[0083] In operation S240, when it is determined that the first feedback result indicates that the first target answer does not meet the needs of the user, the second largest model in the third largest model set after ascending sorting is determined as the second target large model, so that the second target answer can be output after the question to be answered is input into the second target large model.
[0084] In operation S250, the second target answer is sent to the user.
[0085] In operation S260, the user returns a second feedback result based on the second target answer.
[0086] In operation S270, when it is determined that the second feedback result represents that the second target answer meets the needs of the user, the determination of the target large model from the third large model set is stopped, and the Nth question-answer interaction is completed.
[0087] In operation S280, when it is determined that all K large models in the third large model set are called and the target answers output by the K large models do not meet the user's needs, stop determining the target large model from the third large model set and complete the Nth question-answer interaction.
[0088] According to an embodiment of the present invention, the first target macromodel is different from the second target macromodel. The first feedback result is used to characterize whether the first target answer meets the needs of the user.
[0089] According to an embodiment of the present invention, the question-answering interactive system can determine a first target macromodel, and the first target macromodel answers the question to be answered to obtain a first target answer. When the user receives the first target answer, he can determine whether the first target answer is relevant to the question to be answered. If so, it can be determined that the user's needs are met; if not, it can be determined that the user's needs cannot be met.
[0090] According to an embodiment of the present invention, when it is determined that the second feedback result indicates that the second target answer does not meet the needs of the user, the target large models can be determined in sequence starting from the third largest model in the third largest model set until the target answer meets the user needs or the target answers output by the K large models in the third largest model set do not meet the user needs.
[0091] According to an embodiment of the present invention, by combining the ranking of large models in the third large model set with user feedback results, it is possible to reduce the time and consumption of users in obtaining answers that meet their needs in the current question-and-answer interaction, thereby improving the user experience.
[0092] According to an embodiment of the present invention, the large model deployment method may include the following steps: Figure 1 In addition to operations S110 to S140 shown, the following operations may also be included: collecting P answered questions generated within a historical time period and the answered answers corresponding to each of the P answered questions; determining the probability of each of the P answered questions being generated within the historical time period; selecting M answered questions whose probabilities meet preset conditions from the P answered questions, and storing the M answered questions and the answered answers corresponding to each of the M answered questions in a cache, wherein P≥M, and P is a positive integer.
[0093] According to an embodiment of the present invention, when it is determined that the Nth question-answer interaction is completed, the cache is updated based on the target answer and the question to be answered.
[0094] According to an embodiment of the present invention, the historical time period may be a period of time in the past. Answered questions may be questions that have been answered by the question-answering system in the past period of time. The preset condition may refer to a probability condition that the answered questions in the cache need to satisfy. For example, the preset condition may be a preset probability that the answered questions need to satisfy. The preset probability may be configured according to actual question-answering interaction requirements and is not limited here.
[0095] According to an embodiment of the present invention, the probability of each of the P answered questions being generated within the historical time period can be determined based on the ratio of the number of times each of the P answered questions is asked by users within the historical time period to the total number of times the P answered questions are asked by users within the historical time period.
[0096] According to an embodiment of the present invention, when it is determined that the Nth question-and-answer interaction is completed, the probability of each answered question in the previous N question-and-answer interactions can be re-determined based on the target answer and the question to be answered, and then the answered questions that meet the preset conditions can be re-screened, and the answered questions and the corresponding answered answers can be stored in the cache.
[0097] According to an embodiment of the present invention, based on the probability of answered questions generated within a historical time period, answered questions that meet preset conditions, such as questions with higher probabilities and answers, can be pre-stored in a cache. When a user inputs a question, the answer can be determined in the cache in advance, which can save the consumption of a large model in answering questions and can also speed up the response to user questions.
[0098] According to the embodiment of the present invention, the inventors have also found that although the ratio of the number of times each of the P answered questions has been asked by the user in the historical time period to the total number of times the P answered questions have been asked by the user in the historical time period can be directly determined as the probability of each of the P answered questions being generated in the historical time period, due to randomness, the probability error determined is large, which will affect the probability of the question to be answered input by the user appearing in the cache.
[0099] In addition, considering that in the question-answer interaction process of the present invention, cache storage resources are limited, it is necessary to store as many answered questions with a higher probability as possible and answered answers corresponding to the answered questions.
[0100] Based on this, the average estimated probability can be determined based on the ratio of the number of times each of the P answered questions is asked by the user in the historical time period to the total number of times the P answered questions are asked by the user in the historical time period. A revised probability for revising the average estimated probability is determined based on the number of preset answer questions, the number of times each of the P answered questions is asked by the user in the historical time period, and a preset probability parameter; and the probability of each of the P answered questions being generated in the historical time period is determined based on the sum of the average estimated probability and the revised probability.
[0101] According to an embodiment of the present invention, the preset probability parameter may be a parameter that can be adjusted during the question-answer interaction process.
[0102] According to an embodiment of the present invention, the revised probability may be determined based on the ratio of the logarithm of the number of times the preset answer question is answered to the number of times each of the P answered questions is asked by the user in the historical time period and a preset probability parameter. For example, the square root of the ratio of the logarithm of the number of times the preset answer question is answered to the number of times each of the P answered questions is asked by the user in the historical time period may be multiplied by the preset probability parameter to obtain the revised probability.
[0103] For example, the total number of times that P answered questions q are asked by users in the historical time period may be t, that is, t question-answering interactions are performed in the historical time period.
[0104] The probability that each answered question q among P answered questions will be generated in the historical time period It can be expressed as shown in the following formula (3):
[0105] (3)
[0106] in, represents the number of times each answered question q among P answered questions has been asked by users in the historical time period; Indicates preset parameters.
[0107] Figure 3 A schematic diagram of question-answer interaction according to another embodiment of the present invention is shown.
[0108] Large model deployment methods can include Figure 1 In addition to the operations S110 to S140 shown in FIG. Figure 3 The question-answer interaction process of operation S310 to operation S380 is shown.
[0109] In operation S310, a user inputs a question to be answered.
[0110] In operation S320, when it is determined that a question to be answered input by the user is received, it is determined whether there is a question to be answered in the cache.
[0111] In operation S330, when it is determined that there is a question to be answered in the cache, a solved answer corresponding to the question to be answered is sent to the user.
[0112] In operation S340, the user returns a target feedback result based on the solved answer.
[0113] In operation S350, when it is determined that the target feedback result represents that the solved answer does not meet the user's needs, the target large model is determined in sequence from the third large model set according to the sorting, so as to output the target answer after the question to be answered is input into the target large model.
[0114] In operation S360, the target answer is sent to the user.
[0115] In operation S370, the user returns a target feedback result based on the target answer.
[0116] In operation S380, when it is determined that the target feedback result indicates that the target answer meets the needs of the user, the answered answer corresponding to the unanswered question in the cache is updated to the target answer.
[0117] According to an embodiment of the present invention, when it is determined that the target feedback result represents that the solved answer meets the needs of the user, there is no need to update the cache.
[0118] According to an embodiment of the present invention, when it is determined that the target answers answered by each large model in the third large model set do not meet the needs of the user, there is no need to update the cache.
[0119] According to an embodiment of the present invention, the answers corresponding to the questions stored in the cache are updated in real time through the interaction process, which is beneficial to quickly respond to the questions input by the user during the question-and-answer interaction, while providing the user with answers that meet the user's needs, thereby improving the user experience.
[0120] Figure 4 A block diagram of a large model deployment device according to an embodiment of the present invention is shown.
[0121] like Figure 4 As shown, the large model deployment device 400 includes a receiving module 410 , a screening module 420 , a sorting module 430 and a determining module 440 .
[0122] The receiving module 410 is used to receive the question to be answered input by the user during the Nth question-answering interaction.
[0123] The screening module 420 is used to screen K large models from the first large model set based on the estimated accuracy of each large model in the first large model set in answering questions, so as to obtain the second large model set, wherein the first large model set is determined in the N-1th question-and-answer interaction, and the estimated accuracy is determined based on the number of times each large model in the first large model set is called in the first N-1 question-and-answer interactions and the answers to the questions answered after being called, and K is a positive integer, and K is less than the number of large models in the first large model set.
[0124] The sorting module 430 is used to sort each large model in the second largest model set based on the estimated resources required to consume when calling each large model in the second largest model set to answer the question when it is determined that there are no unanswered questions in the cache, or when it is determined that there are unanswered questions in the cache and the answers corresponding to the unanswered questions do not meet the user's needs, so as to obtain the third largest model set, wherein the cache stores M answered questions and answered answers corresponding to each of the M answered questions, N and M are both positive integers greater than or equal to 1, and the estimated resources are obtained based on the estimated consumption when calling each large model in the second largest model set to answer questions in the first N-1 question-and-answer interactions.
[0125] The determination module 440 is used to determine the target large model from the third largest model set in sequence according to the sorting, so as to output the target answer after the question to be answered is input into the target large model, until the target answer meets the user's needs or the target answers output by the K large models in the third largest model set do not meet the user's needs, then stop determining the target large model from the third largest model set and complete the Nth question-answer interaction.
[0126] According to an embodiment of the present invention, any multiple modules of the receiving module 410, the screening module 420, the sorting module 430 and the determination module 440 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the receiving module 410, the screening module 420, the sorting module 430 and the determination module 440 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the receiving module 410, the screening module 420, the sorting module 430 and the determination module 440 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.
[0127] It should be noted that the large model deployment device part in the embodiment of the present invention corresponds to the large model deployment method part in the embodiment of the present invention. The description of the large model deployment device part specifically refers to the large model deployment method part, which will not be repeated here.
[0128] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.
[0129] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination. The scope of the present invention is defined by the attached claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.
Claims
1. A large model deployment method, characterized in that: The method comprises: Receive the unanswered question input by the user during the Nth question-answer interaction; Based on the estimated accuracy of each large model in the first large model set in answering questions, K large models are selected from the first large model set to obtain a second large model set, wherein the first large model set is determined in the N-1th question-answering interaction, and the estimated accuracy is determined based on the number of times each large model in the first large model set is called in the first N-1th question-answering interaction and the answers to the questions answered after being called, and K is a positive integer, and K is less than the number of large models in the first large model set; When it is determined that the unanswered question does not exist in the cache, or when it is determined that the unanswered question exists in the cache and the answer corresponding to the unanswered question does not meet the user's needs, based on the estimated resources required to be consumed when calling each of the large models in the second large model set to answer the question, each large model in the second large model set is sorted to obtain a third large model set, wherein the cache stores M answered questions and answered answers corresponding to each of the M answered questions, N and M are both positive integers, and the estimated resources are obtained based on the estimated consumption when calling each large model in the second large model set to answer the question in the first N-1 question-and-answer interactions; and The target large model is determined in sequence from the third largest model set according to the sorting, so that the target answer is output after the question to be answered is input into the target large model, until the target answer meets the needs of the user or the target answers output by the K large models in the third largest model set do not meet the needs of the user, then the target large model is stopped from being determined from the third largest model set, and the Nth question-answering interaction is completed.
2. The method according to claim 1, characterized in that The sorting includes ascending sorting, K≥2; The step of determining the target large model from the third large model set in accordance with the sorting so as to output a target answer after inputting the question to be answered into the target large model until the target answer meets the needs of the user or the target answers output by the K large models in the third large model set do not meet the needs of the user, and then stopping determining the target large model from the third large model set includes: The first largest model in the third largest model set after the ascending sorting is determined as the first target large model, so that the first target answer is output after the question to be answered is input into the first target large model; Sending the first target answer to the user, so that the user returns a first feedback result based on the first target answer; In the case where it is determined that the first feedback result indicates that the first target answer does not meet the needs of the user, the second largest model in the third largest model set after the ascending sorting is determined as the second target large model, so that the second target answer is output after the question to be answered is input into the second target large model, wherein the first target large model and the second target large model are different; sending the second target answer to the user, so that the user returns a second feedback result based on the second target answer; If it is determined that the second feedback result indicates that the second target answer satisfies the needs of the user, stop determining the target large model from the third large model set, and complete the Nth question-answer interaction; and When it is determined that all K large models in the third large model set are called and the target answers output by the K large models do not meet the needs of the user, stop determining the target large model from the third large model set and complete the Nth question-answer interaction.
3. The method according to claim 1, characterized in that The method further comprises: For each large model in the second largest model set: The consumption of calling the large model to answer questions and the number of times the large model is called in the first N-1 question-answering interactions are counted respectively; and Based on the consumption of calling the large model to answer questions in the first N-1 question-and-answer interactions and the number of times the large model is called, the estimated resources required to be consumed when calling the large model to answer questions are determined.
4. The method according to claim 3, characterized in that The consumption of calling the large model to answer questions and the number of times the large model is called in the first N-1 question-answering interactions to determine the estimated resources required to be consumed when calling the large model to answer questions include: Determine the average estimated resources according to the ratio of the consumption of calling the large model to answer questions in the first N-1 question-answering interactions and the number of times the large model is called; Determining a correction resource for correcting the average estimated resource based on the number of preset answer questions, the number of times the large model is called, and preset parameters; and The estimated resources required to be consumed when calling the large model to answer the question are determined based on the difference between the average estimated resources and the corrected resources.
5. The method according to claim 3, characterized in that: The method further comprises: When it is determined that the Nth question-answer interaction is completed, based on the consumption of calling the target large model to answer unanswered questions, the consumption of calling the target large model to answer questions in the first N-1 question-answer interactions is updated; Updating the number of times the target large model is called in the first N-1 question-answer interactions; and Based on the consumption when calling the target big model to answer questions in the updated first N-1 question-and-answer interactions and the number of times the target big model is called in the updated first N-1 question-and-answer interactions, the estimated resources required to call the target big model to answer questions are updated.
6. The method according to claim 1, characterized in that The method further comprises: For each large model in the first large model set: Counting the number of times the large model is called in the first N-1 question-answer interactions and the number of times the answers to the questions answered by the large model after being called meet the user's needs; and The estimated accuracy of the big model in answering the question is determined based on the number of times the answer to the question after the big model is called meets the user's needs and the number of times the big model is called.
7. The method according to claim 6, characterized in that The method further comprises: When it is determined that the Nth question-and-answer interaction is completed, based on the target answer, the number of times the answer to the question answered by the target large model after being called in the first N-1 question-and-answer interactions satisfies the user's needs is updated; Updating the number of times the target large model is called in the first N-1 question-answer interactions; and Based on the number of times the answers to questions answered by the target big model after being called in the updated first N-1 question-and-answer interactions meet the user's needs and the number of times the target big model is called in the updated first N-1 question-and-answer interactions, the estimated accuracy of the target big model in answering questions is updated.
8. The method according to claim 1, characterized in that The method selects K large models from the first large model set based on the estimated accuracy of answering questions by each large model in the first large model set to obtain the second large model set, including: According to the estimated accuracy of each large model in answering the question in the first large model set, sorting each large model in the first large model set in descending order to obtain a sorting result; and According to the ranking result, the first K large models are screened from the first large model set to obtain the second large model set.
9. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: Collecting P answered questions generated in a historical time period and answered answers corresponding to each answered question in the P answered questions; Determine a probability of each of the P answered questions being generated within the historical time period; Selecting M answered questions whose probabilities meet a preset condition from the P answered questions, and storing the M answered questions and the answered answers corresponding to each answered question in the M answered questions in the cache, wherein P≥M, and P is a positive integer; When it is determined that the Nth question-answer interaction is completed, the cache is updated based on the target answer and the question to be answered.
10. A large model deployment device, characterized in that: The device comprises: A receiving module, used to receive the question to be answered input by the user during the Nth question-answering interaction; A screening module, configured to screen K large models from the first large model set based on the estimated accuracy of each large model in answering questions to obtain a second large model set, wherein the first large model set is determined in the N-1th question-answering interaction, the estimated accuracy is determined based on the number of times each large model in the first large model set is called in the first N-1th question-answering interaction and the answers to the questions answered after being called, and K is a positive integer, and K is less than the number of large models in the first large model set; a sorting module, for sorting each large model in the second large model set based on the estimated resources consumed when each large model in the second large model set is called to answer the question, to obtain a third large model set, when it is determined that the unanswered question does not exist in the cache, or when it is determined that the unanswered question exists in the cache and the answer corresponding to the unanswered question does not meet the user's needs, wherein the cache stores M answered questions and answered answers corresponding to each of the M answered questions, N and M are both positive integers, and the estimated resources are obtained based on the estimated consumption when each large model in the second large model set is called to answer the question in the first N-1 question-answer interactions; and A determination module is used to determine the target large model from the third largest model set in sequence according to the sorting, so as to output the target answer after the question to be answered is input into the target large model, until the target answer meets the needs of the user or the target answers output by the K large models in the third largest model set do not meet the needs of the user, then stop determining the target large model from the third largest model set and complete the Nth question-answering interaction.
