Intelligent question answering method and system based on multi-model collaboration

Abstract

The invention discloses an intelligent question answering method and system based on multi-model collaboration. The method comprises the following steps: S1, preprocessing a question text input by a user; s2, performing collaborative reasoning on the preprocessed text through large-scale language models, and generating candidate answers by utilizing complementarity among the large-scale language models; s3, a question type is judged based on a semantic classification and rule feature combined judgment method; s4, when the question type is closed, counting the number of votes supported by the large language model of the candidate answers through a confidence-based dynamic voting method, and taking the candidate answer with the highest number of votes as a final answer; when the question type is open, screening the candidate answers through a consistency quantitative evaluation method and outputting a final answer; and S5, dynamically adjusting the weight of the large language model based on the correctness data of the output final answer, and optimizing the collaborative reasoning ability of the large language model, thereby effectively improving the efficiency and precision of outputting question and answer results by multiple models.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automation conversion, in particular to an intelligent question and answer method and system based on multi-model cooperation. BACKGROUND

[0002] In the field of natural language to machine command automation conversion, the existing technology mainly relies on a single large language model to complete the question and answer generation task. Although such models perform excellently in general scenarios, their limitations are gradually exposed in actual complex applications, especially when dealing with professional knowledge-intensive or multi-step logical reasoning tasks, the system performance decreases significantly.

[0003] Single model has low accuracy on complex problems: existing systems usually generate answers directly by a single large language model, but when dealing with complex problems involving professional knowledge or requiring multi-step logical reasoning, the model performance decreases significantly. The root cause lies in the difficulty of training data to cover vertical fields (such as medicine, law, engineering, etc.), and the lack of support for deep logical connections in the model architecture. For example, in the medical diagnosis scenario, the model may generate incorrect recommendations due to the lack of rare case data; in the engineering parameter analysis, invalid instructions are output due to the inability to associate multi-variable logic. This error rate problem seriously restricts the application value of the system in high reliability scenarios.

[0004] Multi-model answer aggregation lacks reliability evaluation: existing technologies generate answers in parallel through multiple models to make up for the shortcomings of a single model, but lack systematic coordination and unified evaluation methods, resulting in differences in semantics or logic between answers. Existing methods rely more on majority voting or manual intervention, which cannot quantify answer consistency and is difficult to achieve dynamic optimization of model weights. For example, a superficially reasonable but actually incorrect answer may be incorrectly selected, affecting system reliability. Due to the lack of effective semantic analysis or knowledge verification methods, the efficiency and accuracy of multi-model cooperation do not meet expectations, creating a technical bottleneck for stable improvement of answer quality.

[0005] In summary, the existing technology cannot balance model capability and answer reliability, and cannot meet the needs of high-precision question and answer scenarios. SUMMARY

[0006] The purpose of the present application is to solve the technical problem of improving answer reliability in complex question and answer scenarios, and to propose an intelligent question and answer method and system based on multi-model cooperation.

[0007] The technical problem of the present application is solved by the following technical solution:

[0008] An intelligent question and answer method based on multi-model cooperation, comprising the following steps:

[0009] S1, pre-processing the problem text input by the user;

[0010] S2. Perform collaborative reasoning on the preprocessed text using at least two large language models, and generate candidate answers by leveraging the complementarity between the large language models.

[0011] S3. Determine the problem type based on a joint judgment method of semantic classification and rule features;

[0012] S4. When the question type is closed, the number of votes supported by the large language model for candidate answers is counted using a dynamic voting method based on confidence, and the candidate answer with the highest number of votes is taken as the final answer; when the question type is open, the candidate answers are filtered and the final answer is output using a consistency quantification evaluation method.

[0013] S5. Based on the correctness data of the final output answer, dynamically adjust the weights of the large language model to optimize the collaborative reasoning ability of the large language model.

[0014] In some embodiments, the following technical features are also included:

[0015] In step S1, the preprocessing includes the following steps:

[0016] The problem text is cleaned, and keywords and entities are extracted using a named entity recognition model based on a pre-trained language model.

[0017] In some embodiments, the following technical features are also included:

[0018] Step S2 specifically includes the following steps: Distributing the text to the large language model based on the features of the preprocessed text using a dynamic scheduling algorithm, wherein the features include text length, extracted entity types, and estimated logical complexity; the large language model includes a general model trained on a general corpus, a domain model fine-tuned with domain data, and a reasoning model optimized with logical rules.

[0019] In some embodiments, the following technical features are also included:

[0020] In step S3, the joint determination method of semantic classification and rule features includes an intent classification method and a rule matching method. The intent classification method specifically includes the following steps:

[0021] Based on a pre-trained language model, the question is encoded into a semantic embedding vector, and the probability distribution of closed or open questions is output.

[0022] The rule matching method specifically includes the following steps:

[0023] The system uses regular expressions to detect and match numerical values ​​with standardized entities, and identifies and extracts logical keywords based on a keyword list.

[0024] In some embodiments, the following technical features are also included:

[0025] In step S4, the confidence-based dynamic voting method includes the following steps:

[0026] S41. Iterate through each candidate answer and count the number of votes for each candidate answer and the corresponding large language model;

[0027] S42. If there is only one candidate answer with the highest number of votes, output that answer and update the historical correct count of all large language models; if there are at least two candidate answers with the highest number of votes, retrieve the historical correct count of the large language model corresponding to the candidate answer with the highest number of votes, select the candidate answer corresponding to the one with the highest historical correct count in the large language model as the final answer, and update the historical correct count of the large language model corresponding to the final answer.

[0028] In some embodiments, the following technical features are also included:

[0029] In step S4, the consistency quantification evaluation method includes the following steps:

[0030] S43. Construct a cosine similarity matrix and calculate the consistency score of candidate answers using the cosine similarity matrix;

[0031] S44. Set a dynamic threshold based on the distribution characteristics of the semantic similarity matrix. When the consistency score is less than the dynamic threshold or there are at least two candidate answers with the highest consistency scores, filter by level and output the final answer.

[0032] In some embodiments, the following technical features are also included:

[0033] In step S44, the hierarchical filtering specifically includes the following steps:

[0034] Candidate answers are sorted in descending order based on consistency score. If there are at least two candidate answers with the highest consistency score, a second descending order is performed based on the logical structure parsing index, using dictionary matching counts. If the logical density is still the same, a third descending order is performed based on text length, prioritizing longer answers. A confidence-weighted median selection algorithm is used to select the answer at the median position from the sorted list. If N is even, the N / 2th candidate answer is taken as the final answer (index position starts from 0, and index N / 2-1 is taken when N is even). If it is still impossible to distinguish, an answer is randomly selected.

[0035] In some embodiments, the following technical features are also included:

[0036] In step S43, the formula for the cosine similarity matrix is:

[0037]

[0038] Where v i and v j Let S be the answer vector and matrix S. ij S represents the semantic similarity between answers i and j. ij The value range of v is [0,1]. ik Indicates v i The component in the k-th dimension, v jk Let vj be the component in the k-th dimension.

[0039] In some embodiments, the following technical features are also included:

[0040] In step S43, the formula for the consistency score is: C i =(Σ j ≠i S ij ) / (N-1), where N is the number of answers; in step S44, the dynamic threshold τ=max(τ_min,τ_{base}-k·σ) (where τ_min is the preset lower limit: such as 0.6 for closed-ended questions and 0.4 for open-ended questions; τ_{base} is pre-configured according to question type: 0.9 for closed-ended questions and 0.7 for open-ended questions; σ is the standard deviation of similarity among all answers; k∈[0,1] is the adjustment coefficient.

[0041] The technical problem of the present invention is also solved by the following technical solution:

[0042] A multi-model collaborative intelligent question answering system that implements the above method includes a preprocessing module, a model invocation module, an intent classification module, a rule matching module, an answer output module, and a feedback loop optimization module.

[0043] The preprocessing module is used to preprocess the question text input by the user;

[0044] The model invocation module is used to invoke a large language model to perform collaborative reasoning on the preprocessed text and generate candidate answers.

[0045] The intent classification module and rule matching module are used to determine the question type;

[0046] The answer output module is used to filter candidate answers according to the question type and output the final answer;

[0047] The feedback loop optimization module is used to dynamically adjust the weights of the large language model based on the correctness data of the final answer.

[0048] The present invention has the following beneficial effects:

[0049] This invention proposes an intelligent question-answering method and system based on multi-model collaboration. Through technical features such as question text preprocessing, collaborative reasoning of the preprocessed text using large language models, generation of candidate answers using the complementarity of large language models, determination of question type based on intent classification and rule matching methods, counting of large language model support votes for candidate answers using a confidence-based dynamic voting method when the question type is closed, and screening of candidate answers using a consistency quantification evaluation method when the question type is open, a scalable model collaboration framework is established. This framework enables adaptive and high-precision answers to complex questions in vertical domains. Based on large language model complementarity → dynamic hierarchical decision-making → semantic consistency quantification → feedback optimization, the synergistic effect of complementary reasoning and dynamic consistency evaluation of large language models is achieved, thereby effectively improving the efficiency and accuracy of multi-model output question-answering results.

[0050] In addition, some embodiments also have the following beneficial effects:

[0051] This invention employs a confidence-weighted median selection algorithm to select the answer at the median position from a sorted list. If N is even, the N / 2th candidate answer is taken as the final answer. The index position starts from 0, and when N is even, the index is N / 2-1. This approach, combined with logical density and text length sorting, ensures that answer selection balances logical rigor and text integrity for open-ended questions.

[0052] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0053] Figure 1 This is a flowchart of the intelligent question-answering method based on multi-model collaboration in an embodiment of the present invention;

[0054] Figure 2 This is a flowchart of the intelligent question-answering system based on multi-model collaboration in an embodiment of the present invention;

[0055] Figure 3 This is a flowchart of the multi-model collaborative reasoning and problem type determination method in an embodiment of the present invention;

[0056] Figure 4 This is a flowchart of the dynamic voting method in an embodiment of the present invention;

[0057] Figure 5 This is a flowchart of the consistency quantification evaluation method in the embodiments of the invention. Detailed Implementation

[0058] The present invention will be further described below with reference to the accompanying drawings and preferred embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0059] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0060] This invention proposes an intelligent question-answering method based on multi-model collaboration. This method systematically coordinates the outputs of multiple models and quantifies their consistency. Through multi-model collaboration and dynamic consistency evaluation, it addresses the core problems of high error rates in single-model natural language to machine command conversion and the lack of reliability assessment for multi-model parallelism. To address the shortcomings of existing technologies that rely on a single large language model, such as insufficient domain adaptability and limited logical reasoning capabilities, this invention proposes integrating heterogeneous models and leveraging their complementarity to generate diverse candidate answers. Simultaneously, a semantic consistency evaluation module is designed to quantify answer reliability through vector space mapping.

[0061] This invention proposes a method for enhancing question-answering accuracy based on multi-model collaborative consistency using artificial intelligence. Through a hierarchical approach involving multi-model parallel reasoning, consistency quantification evaluation, and majority voting decision-making, it systematically improves the accuracy of answers to complex questions. The core content of this invention is as follows:

[0062] This invention addresses the high error rate of single models on complex problems by providing an intelligent question-answering system based on multi-model collaboration. The system includes a preprocessing module, a model invocation module, an intent classification module, a rule matching module, an answer output module, and a feedback loop optimization module. The preprocessing module preprocesses the user-input question text. The model invocation module invokes a large language model to perform collaborative reasoning on the preprocessed text, generating candidate answers. The intent classification and rule matching modules determine the question type. The answer output module filters candidate answers based on the question type and outputs the final answer. The feedback loop optimization module dynamically adjusts the weights of the large language model based on the correctness data of the final answer. The workflow diagram of the intelligent question-answering system is shown below. Figure 2As shown, the intelligent question-answering system utilizes a dynamic collaborative framework to concurrently invoke large-scale language models based on different training objectives. These models include a general model trained on a general corpus, a domain model fine-tuned with domain data, and a reasoning model optimized with logical rules. The system leverages the complementarity between these models to generate diverse candidate answers. This dynamic collaborative framework is an intelligent decision-making framework capable of coordinating parallel reasoning from multiple heterogeneous language models in real time, dynamically evaluating output consistency, and optimizing weights based on feedback. Through hierarchical integration of multi-model outputs, semantic verification, and weight adaptation mechanisms, this framework ensures the system generates highly reliable answers to complex questions.

[0063] The user inputs a question, and each model independently processes the question through a unified input preprocessing module (multi-model parallel reasoning, e.g., model A outputs answer A, model B outputs answer B, and model C outputs answer C). Preliminary results are generated based on semantic embedding parsing, and the reasoning model verifies the logical consistency of the answers using a knowledge graph (e.g., using the UMLS medical knowledge base). Answer integration employs a weighted fusion algorithm based on semantic similarity, dynamically adjusting model weights to ensure semantic consistency and logical rationality of the output answers. In medical question-answering scenarios, the general model generates preliminary diagnostic suggestions, the domain model supplements specialized knowledge through fine-tuning using medical literature, and the reasoning model verifies the rationality of the diagnostic path using a medical knowledge graph, significantly reducing the error rate caused by insufficient data coverage or weak logical capabilities. Compared to the limitations of single-model or simple voting methods in existing technologies, this invention provides high-quality answers for subsequent decision-making through dynamic collaboration and semantic verification methods.

[0064] For closed-ended questions (such as fact queries and numerical calculations), the intelligent question-answering system employs a dynamic voting method, statistically analyzing the number of votes received by multiple models to output the answer with the highest number of votes. For example, in the "drug dosage calculation" scenario, the intelligent question-answering system integrates the results of multiple models and outputs the dosage recommendation with the highest consistency. For open-ended questions (such as logical reasoning and semantically ambiguous questions), the intelligent question-answering system quantifies the reliability of answers using a consistency quantification evaluation method: it uses Sentence-BERT (SBERT, a sentence embedding representation model) to generate semantic embedding vectors for the answers, calculates the cosine similarity matrix, and judges consistency based on a weighted average score. If the score exceeds a dynamic threshold, a consensus answer is output; if it is in a divergence mode, the intelligent question-answering system uses a median selection algorithm based on confidence weighting to select the answer with the highest median score and generates prompts through a divergence quantification module, explaining multiple potential reasonable explanations. For example, in "disease inference with ambiguous symptoms," the intelligent question-answering system filters reasonable diagnoses and reminds doctors to pay attention to multiple possible causes. Compared to the limitations of static voting or single semantic evaluation in existing technologies, this scheme significantly improves the reliability of answers and the transparency of decision-making by combining semantic similarity quantification with a median strategy.

[0065] This invention provides an intelligent question-answering method based on multi-model collaboration, such as... Figure 1 As shown, it includes the following steps:

[0066] S1. Preprocess the user-input question text; the intelligent question answering system cleans the input question text through the preprocessing module (using jieba word segmentation and a unified format based on the domain terminology library) and extracts keywords and entities based on the BERT (Bidirectional Encoder Representations from Transformers, a pre-trained language model) NER (Named Entity Recognition) model.

[0067] S2. Collaborative reasoning is performed on the preprocessed text using at least two large language models with different training objectives, leveraging the complementarity between the large language models to generate diverse candidate answers. The core of this approach is to intelligently select the most suitable subset of models for processing based on the immediate characteristics of the question, achieving load balancing and specialized processing. Specifically, this includes the following sub-steps:

[0068] S21. Feature Extraction and Analysis: The algorithm receives the preprocessing results from S1 (including cleaned text, extracted keywords, and entity lists). It then calculates the text length (number of characters) of the problem, analyzes the entity type distribution (such as the number and type of entities like "disease," "drug," and "symptom" in medical problems), and the density of logical keywords (such as the frequency of occurrence of words like "why," "how," "prove," and "analyze"), thereby comprehensively estimating the logical complexity of the problem.

[0069] S22. Scheduling Decision and Routing: Based on the above characteristics, the routing problem is solved according to the preset scheduling strategy.

[0070] Strategy 1 (Domain Entity Based): For questions that extract a large number of domain-specific entities (such as medical, legal, and engineering terms), priority is given to routing to the corresponding domain model to obtain accurate professional knowledge. For example, if the question contains entities such as "myocardial infarction" and "dosage", it will be routed to the medical domain model.

[0071] Strategy 2 (based on logical requirements): For questions with high density of logical keywords, long text, and high complexity (such as those containing phrases like "explain why", "please deduce", "how to analyze"), priority is given to scheduling to the inference model to leverage its advantages in logical reasoning and causal analysis.

[0072] Strategy 3 (Default and Load Balancing): For general problems that cannot be clearly categorized using the strategies above, or when the request queue of the target domain model / inference model is too long or the load is too high, requests are distributed to the general model. The algorithm monitors the request load of each model in real time (such as the number of tasks being processed and the average response time). If a model is overloaded, new requests are distributed to similar models or general models with lower loads, thus achieving load balancing of the system.

[0073] S23. Parallel Inference and Answer Collection: Based on scheduling decisions, the question text is distributed to one or more selected models. Each model independently generates an answer and outputs it in a standardized format (such as JSON) for answer integration in subsequent steps.

[0074] like Figure 3 As shown, the preprocessed text is distributed to different models (e.g., model A, model B, model C) through a dynamic scheduling algorithm (based on question type) for multi-model collaborative reasoning. Each model independently generates an answer and standardizes it into JSON (JavaScript Object Notation, a lightweight data exchange format), and outputs the standardized answer for subsequent calculations.

[0075] S3. A joint determination method based on semantic classification and rule features is used to determine the problem type. Specifically, such as... Figure 3 As shown, the intelligent question-answering system determines the question type through an intent classification module and a rule matching module. The joint determination method of semantic classification and rule features includes an intent classification method and a rule matching method. The intent classification module uses the intent classification method, specifically including the following steps: encoding the question into a semantic embedding vector based on BERT (BERT-base-Chinese), and outputting the probability distribution of closed-ended questions (such as numerical queries and fact verification) or open-ended questions (such as cause analysis). The rule matching module uses the rule matching method, specifically including the following steps: detecting and matching numerical values ​​with standardized entities through regular expressions, and identifying and extracting logical keywords based on a keyword list, further enhancing the robustness of the judgment.

[0076] S4. When the question type is closed-ended, a dynamic voting method based on confidence is used to count the number of votes supported by the large language model for candidate answers, and the candidate answer with the highest number of votes is taken as the final answer. When the question type is open-ended, a consistency quantification evaluation method is used to filter candidate answers and output the final answer. Specifically, after the question type is determined, when the question type is closed-ended, the dynamic voting method based on confidence is triggered to count the number of votes supported by the large language model for candidate answers, and the candidate answer with the highest number of votes is taken as the final answer, outputting the answer with the highest votes. For example, in the medical Q&A question 'drug dosage query', the result with the highest consistency is output. When the question type is open-ended, the open-ended question enters the consistency quantification evaluation module. The consistency quantification evaluation module uses the consistency quantification evaluation method to calculate the cosine similarity score using Sentence-BERT to select the optimal solution. For example, in the legal consultation question 'contract clause interpretation', the answer with semantic consistency is selected.

[0077] In some embodiments, a dynamic voting method is used to select the final answer for closed-ended questions. The dynamic voting method includes the following steps:

[0078] S41. Iterate through each candidate answer, count the number of votes for each candidate answer, and determine the corresponding large-scale language model. For example... Figure 4 As shown, during the initialization of the intelligent question answering system (large language model correct count table, counter), the historical correct counts of each large language model need to be recorded (read from the database, using a uniform assignment strategy during cold start, initial value = 1, to avoid division by zero errors; a memory dictionary correct_counts is created). In addition, the system has a periodic decay mechanism (e.g., executed weekly), applying a decay factor to the historical correct counts correct_count of all models: correct_count = correct_count * (1 - decay_rate), where the decay rate decay_rate is preset to 0.1 to prevent overfitting of historical data and maintain the timeliness of model weight evaluation. Before each vote, each candidate answer in the input large language model answer list (JSON format) is traversed, and the number of votes (answer_votes) for each candidate answer and its corresponding generating model (models_answers) (large language model) are counted. For example, if both large language models A and B generate the answer "Paris", then "Paris" has 2 votes, and the corresponding answer list is [A, B].

[0079] S42. If there is only one candidate answer with the highest number of votes, output that answer and update the historical correct counts of all large language models. If there are at least two candidate answers with the highest number of votes, retrieve the historical correct counts of the large language models corresponding to the candidate answers with the highest number of votes, select the candidate answer corresponding to the one with the highest historical correct count among the large language models, and update the historical correct count of the large language model corresponding to the final answer. The maximum number of votes is determined by `max_votes`, a list of candidate answers is filtered out, and it is determined whether the candidate answer is unique. If the candidate answer with the highest number of votes is unique, output that answer directly, and iterate through all its corresponding generating models, incrementing their historical correct counts by 1 (reflecting collaborative contribution) (i.e., updating the historical correct counts of all large language models for that answer). For example, if the answer "Paris" is generated by models A and B, both are rewarded.

[0080] If there are at least two candidate answers with the highest number of votes (i.e., not unique, such as "Paris" and "London" both receiving 2 votes), then for each candidate answer with the highest number of votes, retrieve the score (max_score) of the largest language model with the highest historical correctness count. Finally, select the candidate answer corresponding to the largest language model with the highest historical score (if multiple candidate answers still exist, only the first retrieved candidate answer corresponding to the largest language model with the highest historical correctness count is selected) as the final answer, and only the optimal model contributing to this answer (which updates the historical correctness count of the largest language model with the highest historical correctness count) receives a +1 reward. For example, if model A has a higher historical correctness count than model B, then answer A is given priority.

[0081] In some embodiments, a consistency quantification evaluation method is used to screen the final answers to open-ended questions. The consistency quantification evaluation method includes the following steps:

[0082] S43. Construct a cosine similarity matrix and calculate the consistency score of candidate answers using the cosine similarity matrix. For example... Figure 5 As shown, the intelligent question answering system uses the Sentence-BERT model to map candidate answers to semantic embedding codes v. i The dimension is d (determined by the Sentence-BERT model configuration). Each answer is preprocessed (punctuation removed, terminology standardized) and then input into the model to generate a vector v. i (The semantic embedding vector of the i-th answer), construct the semantic embedding matrix V∈R N×dWhere R is the set of real numbers, a standard notation in matrix mathematics, used in this embodiment to declare the numerical type of the embedded vector as real; N is the number of candidate answers, the number of rows in the matrix, with each answer corresponding to one row; d is the embedding dimension, the number of columns in the matrix, defined by the Sentence-BERT model. For example, in the "symptom cause analysis" scenario, the answer "viral infection" is mapped to vector v1.

[0083] The cosine similarity matrix S between the answers is calculated using the following formula:

[0084]

[0085] Where v i and v j The answer vector. The semantic embedding matrix V∈R N×d , is a symmetric matrix (S) ij =S ji ), matrix S ij S represents the semantic similarity between answers i and j. ij The value range is [-1, 1], v ik Indicates v i The component in the k-th dimension, v jk Let vj be the component in the k-th dimension.

[0086] S44. Based on the distribution characteristics of the semantic similarity matrix, a dynamic threshold is set. When the consistency score is less than the dynamic threshold or there are at least two candidate answers with the highest consistency scores, the final answer is output through hierarchical filtering. For each answer i, the consistency score C is calculated. i , defined as the average similarity between answer i and other answers: C i =(Σ j ≠i S ij The intelligent question-answering system, based on the distribution characteristics of the semantic similarity matrix, sets a dynamic threshold τ = max(τ_min, τ_{base} - k·σ), where τ_min is a preset lower limit: such as 0.6 for closed-ended questions and 0.4 for open-ended questions; τ_{base} is pre-configured according to question type: 0.9 for closed-ended questions and 0.7 for open-ended questions; σ is the standard deviation of the similarity among all answers; k∈[0,1] is an adjustment coefficient that balances the inclusiveness and strictness of answers, solving the problem of reasonable discrepancies being missed or incorrect answers being accepted due to traditional fixed thresholds. When answers are highly consistent (σ < 0.1), k = 0 is used to maintain strictness (such as directly using τ = 0.9 in the drug dosage calculation scenario); when answers are significantly divergent (σ > 0.3), k = 1.5 is used to reduce the threshold (such as reducing τ from 0.7 to 0.65 in the symptom analysis scenario) to avoid over-filtering reasonable differences.

[0087] If C exists iIf the score is ≥τ, output the answer with the highest score; otherwise, use the median strategy. For example, if C is the answer to "virus infection", then... i If 0.85 ≥ τ, then it is output.

[0088] When the consistency score is C i When one or more answers have the same score, the intelligent question-answering system filters answers according to the following hierarchical filtering process:

[0089] Press C i Sort the candidate answers in descending order;

[0090] If the scores are the same, a second descending sort is performed based on the indicators of logical structure parsing. Logical density is defined as the number of causal conjunctions (such as "therefore" and "because") in the answer, which is counted by dictionary matching.

[0091] If the logical density is still the same, sort the text in descending order a third time, prioritizing longer answers;

[0092] A confidence-weighted median selection algorithm is used to select the answer at the median position from the sorted list. If N is even, the N / 2th candidate answer is selected as the final answer, with the index starting from 0. When N is even, the index N / 2-1 is used; for example, if N=4, the second answer is selected. If a distinction still cannot be made, an answer is randomly selected. For example, in "Symptom Cause Analysis," the intelligent question-answering system selects the median answer "Virus Infection" from the sorted answer list as the output, along with a divergence hint.

[0093] S5. Based on the correctness data of the final output answer, dynamically adjust the weights of the large language model to optimize its collaborative reasoning ability. Specifically, this involves integrating the authenticity of user feedback (e.g., setting "helpful / not helpful" buttons; to filter noise, the system executes cleaning rules: ignoring repeated feedback from a single user in the short term; for user feedback, only accepting a signal when negative (or positive) feedback for a particular answer is confirmed by more than 3 independent users; and filtering out automatic verification results corresponding to answers with a model confidence level below a preset threshold (e.g., 0.6), the objectivity of automatic verification (knowledge base / search engine results or authoritative website information), and the real-time nature of the model's own confidence level to perform a multi-dimensional correctness evaluation of the final answer, thus obtaining the correctness data for the final answer.

[0094] Experimental Example 1

[0095] This experimental example sets up a comparative experiment on the method of dynamically adjusting the weights of a large language model in the feedback loop optimization module, using the exponential decay method and the sliding window method for comparison and verification. This experimental example is based on three months of historical interaction data (approximately 30,000 question-and-answer records) from the medical vertical field, with the accuracy rate of answers to closed-ended questions (such as drug dosage calculation) as the core indicator. The formula used for the exponential decay method is:

[0096] w t =w t-1 ·e -λ·error

[0097] Where λ = 0.9, w is the decay rate, t is time, and error is the recent error rate. The exponential decay method causes the historical correctness of a large language model to decrease exponentially over time, enabling rapid response to new error patterns; the sliding window method uses the accuracy rate of the most recent 500 answers as a fixed statistical measure, updating the weights with the mean within the window. The results show:

[0098] In terms of response speed: the exponential decay method has a significant advantage in scenarios with sudden changes in data distribution (such as the addition of rare disease Q&A), with the error rate decreasing by 19.2% within 1 week, while the sliding window method takes 2 weeks to achieve a similar effect (17.8%).

[0099] In terms of stability: the sliding window method exhibits less volatility in long-term operation, with a quarterly average error rate standard deviation of 0.7%, which is better than the exponential decay method's 1.5%.

[0100] Overall performance evaluation: The exponential decay method showed faster initial improvement (F1 value +8.3% in the first month), but the sliding window method performed more stably in the later period (F1 value reached 92.1% in the third month, higher than the exponential decay method's 90.7%).

[0101] The conclusions show that the exponential decay method is suitable for scenarios requiring rapid adaptation to new knowledge distributions (such as the diagnosis and treatment of sudden illnesses), while the sliding window method is more suitable for the continuous optimization of steady-state knowledge bases (such as routine drug queries). The system in this embodiment ultimately adopts a hybrid strategy—a default sliding window ensures stability, and when the error rate is detected to rise sharply by 15%, it automatically switches to the exponential decay mode to achieve a balance between efficiency and robustness.

[0102] Experiment Example 2

[0103] In medical diagnostic scenarios, the system of this invention employs a strict threshold strategy for high reliability requirements (such as drug dosage calculation): the base value τ in the dynamic threshold formula is set as the threshold value. baseThe threshold is set to 0.95 (higher than the default 0.9), and the adjustment coefficient k is reduced to 0.5 to ensure that only highly consistent answers are output (e.g., rejecting divergent answers directly when σ < 0.05). In legal consultation scenarios (such as contract interpretation), to accommodate reasonable disagreements, the basic threshold for open-ended questions is reduced to 0.65, and k is increased to 2.0 (e.g., the threshold is reduced to 0.55 when σ > 0.4). Answers containing conditional words such as "if" and "then" are prioritized based on logical density to ensure the rigor of legal logic. Tests show that the error rate is reduced by 23% in medical scenarios and the rate of missing reasonable disagreements is reduced by 18% in legal scenarios.

[0104] Experimental Example 3

[0105] When new engineering domain models are needed (fine-tuned based on IEEE literature), the system of this embodiment seamlessly integrates through a standardized JSON interface. During the cold start phase, uniform weight allocation (initial value = 1) is used. In the "multivariate mechanical fault diagnosis" task, the initial accuracy of the new model is only 68%. After two weeks of feedback optimization (1200 rounds of dynamic voting), its weight is increased to 0.42. Combined with the general model (weight 0.33) and the inference model (weight 0.25), the accuracy for complex problems increases from 71% to 89%. In a typical case, the new model supplements knowledge of bearing thermal deformation parameters and corrects the general model's erroneous inference regarding the "vibration frequency-temperature correlation." The correct diagnosis is output after consistency evaluation (Ci = 0.82 ≥ τ), verifying the effectiveness of modular expansion.

[0106] Compared with the prior art, the embodiments of the present invention have the following effects:

[0107] Significantly improves the accuracy of answers to complex questions: By using multiple different models, the complementary coverage of factual knowledge and causal reasoning ability is utilized to reduce errors caused by insufficient data or capabilities of a single model, thereby improving the accuracy of answers to complex questions.

[0108] The intelligent question-answering system optimizes its accuracy through a self-enhancing feedback loop: It records the historical correct answers of each model using a dynamic voting method, prioritizing answers from highly reliable models. For open-ended questions, a median strategy, combined with logical density and text length sorting, ensures that answer selection balances logical rigor and textual completeness. As the accuracy of the intelligent question-answering system improves, its high-quality answer data serves as feedback signals, continuously feeding back into the model training and weight optimization modules. Over the long term, the model dynamically adjusts its collaboration strategy based on feedback data, gradually optimizing inference efficiency and knowledge coverage, forming a positive loop of "accuracy improvement → feedback optimization → enhanced collaboration efficiency." This ultimately results in an exponential increase in answer accuracy over time (tests show an improvement of over 6.7% in closed-ended questions, 16.9% in open-ended questions, and 17.2% in vertical domain scenarios; the intelligent question-answering system achieves self-enhancing through the feedback loop, with an additional 3.6% improvement in open-ended question accuracy after 8 weeks, demonstrating the effectiveness of the "accuracy → optimized collaboration strategy → higher accuracy" closed loop).

[0109] Compared with the prior art, the embodiments of the present invention have the following advantages:

[0110] Heterogeneous Model Collaborative Reasoning Framework: "Large-scale language models" are a powerful foundational model type in the current field of artificial intelligence and are one of the foundational models used in this invention embodiment. "Heterogeneous model" emphasizes the diversity and differences of models, referring to a collection of models of different types, architectures, training objectives, or trained on different data. "Integrated heterogeneous model" refers to a system or method that combines multiple heterogeneous models according to specific strategies and architectures to work collaboratively; the aim is to leverage the complementary advantages of each model, maximizing strengths and minimizing weaknesses, to obtain more accurate results than a single model. This invention embodiment designs a systematic collaborative framework that integrates heterogeneous models (general models based on general corpora, domain models based on vertical domain data, and reasoning models based on knowledge graphs). Through the division of training objectives, it generates diverse candidate answers and utilizes the complementarity of knowledge coverage and reasoning capabilities to solve the high error rate problem of single models in vertical domains and complex logic.

[0111] Dynamic hierarchical decision-making technology: This technology dynamically distinguishes between closed-ended problems (such as numerical calculations) and open-ended problems (such as multivariate analysis) through an intent classification model (based on BERT encoding) and rule matching (regular expression detection of numerical and logical keywords), triggering majority voting or consensus assessment modules accordingly. Existing technologies lack dynamic hierarchical processing; this invention improves efficiency and accuracy through differentiated decision-making and ensures logical consistency through semantic verification.

[0112] A semantic consistency quantification evaluation method is proposed: Sentence-BERT is used to map answers to semantic embedding vectors, a similarity matrix is ​​constructed, a consistency score (average similarity between answers) is calculated, and a threshold is used to select the optimal solution. Existing technologies rely on simple voting, which cannot quantify consistency. The method in this invention addresses disagreements in open-ended questions through semantic quantification, thereby improving the reliability of answers.

[0113] Structured answer filtering and modular expansion methods: When there is insufficient consistency or a tie, answers are processed through structured filtering rules.

[0114] The text is sorted by consistency score, number of causal conjunctions, and text length. The median selection method (taking the answer at the middle position of the sorted list) is used to determine the output to balance the impact of extreme values.

[0115] The tie-breaker mechanism is based on the first occurrence position index, which distinguishes it from manual intervention and ensures objectivity.

[0116] It supports seamless integration of new domain models (via standardized JSON input) without weight adjustment. Existing technologies lack structured filtering and objective adjudication; the method in this invention fills and expands upon this gap.

[0117] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0118] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0119] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0120] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0121] The above description provides a further detailed explanation of the present invention in conjunction with specific / preferred embodiments, and it should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various substitutions or modifications can be made to these described embodiments without departing from the concept of the present invention, and all such substitutions or modifications should be considered within the scope of protection of the present invention. In the description of this specification, the reference to terms such as "an embodiment," "some embodiments," "preferred embodiment," "example," "specific example," or "some examples," etc., indicates that the specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples. Although the embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions, and modifications can be made herein without departing from the scope of protection of the patent application.

Claims

1. An intelligent question and answer method based on multi-model cooperation, characterized in that, Comprise the following steps: S1, pre-processing the question text input by the user; S2, cooperative reasoning of the pre-processed text by at least two large language models, and generating candidate answers by the complementarity between large language models; S3, judging the question type based on the joint determination method of semantic classification and rule characteristics; S4, when the question type is closed, counting the number of votes of large language models supporting the candidate answers by the dynamic voting method based on confidence, and taking the candidate answer with the highest number of votes as the final answer; When the question type is open, the candidate answers are screened and the final answer is output by the consistency quantitative evaluation method; S5, based on the correctness data of the output final answer, dynamically adjusting the weight of the large language model, and optimizing the cooperative reasoning ability of the large language model.

2. The method of claim 1, wherein, In step S1, the pre-processing comprises the following steps: The question text is cleaned, and the key words and entities are extracted by the named entity recognition model based on the pre-trained language model.

3. The method of claim 1, wherein, Step S2 specifically comprises the following steps: based on the features of the pre-processed text, the text is distributed to the large language model by a dynamic scheduling algorithm, wherein the features include text length, extracted entity type, and estimated logical complexity; the large language model includes a general model trained by general corpus, a domain model fine-tuned by domain data, and an inference model optimized by logical rules.

4. The method of claim 1, wherein, In step S3, the joint determination method of semantic classification and rule characteristics comprises an intent classification method and a rule matching method, and the intent classification method specifically comprises the following steps: The question is encoded into a semantic embedding vector based on a pre-trained language model, and the probability distribution of closed or open questions is output; The rule matching method specifically comprises the following steps: Detect and match numerical values and standardized entities by regular expressions, and identify and extract logical keywords based on a keyword list.

5. The method of claim 1, wherein, In step S4, the dynamic voting method based on confidence comprises the following steps: S41, traverse each candidate answer, count the number of votes of each candidate answer and the corresponding large language model; S42, if the candidate answer with the highest number of votes is only one, output the answer and update the historical correct number of all large language models; if the candidate answer with the highest number of votes is at least two, retrieve the historical correct number of the large language model corresponding to the candidate answer with the highest number of votes, select the candidate answer corresponding to the large language model with the highest historical correct number as the final answer, and update the historical correct number of the large language model corresponding to the final answer.

6. The method of claim 5, wherein, In step S4, the consistency quantitative evaluation method comprises the following steps: S43, construct a cosine similarity matrix, and calculate the consistency score of the candidate answer by the cosine similarity matrix; S44, according to the distribution characteristics of the semantic similarity matrix, set a dynamic threshold, when the consistency score is less than the dynamic threshold or the candidate answer with the highest consistency score is at least two, output the final answer by hierarchical screening.

7. The method of claim 6, wherein, In step S44, the hierarchical screening specifically comprises the following steps: The candidate answers are ranked in descending order according to the consistency score, if the candidate answer with the highest consistency score has at least two, the second descending order is sorted based on the index of logical structure analysis, and the count of dictionary matching is performed; if the logical density is still the same, the third descending order is sorted according to the text length, and the longer answer is preferred; the answer at the median position is selected from the sorted list by using a median selection algorithm based on confidence weighting, if N is even, the N / 2th candidate answer is taken as the final answer, wherein the index position starts from 0, and when N is even, the index N / 2-1 is taken; if it is still indistinguishable, an answer is randomly selected.

8. The method of claim 6, wherein, In step S43, the formula of the cosine similarity matrix is: where v i and v j are the answer vectors, matrix S ij represents the semantic similarity of answers i and j, S ij has a value range of [0, 1], v ik represents the component of v i in the kth dimension, and v jk represents the component of vj in the kth dimension.

9. The method of claim 6, wherein, In step S43, the formula of the consistency score is: C i = (Σ j ≠i S ij ) / (N-1), N is the number of answers; in step S44, the dynamic threshold τ = max(τ_min, τ_{base}-k·σ), where τ_min is a preset lower limit: 0.6 for closed questions and 0.4 for open questions; τ_{base} is preconfigured according to the question type: 0.9 for closed questions and 0.7 for open questions; σ is the standard deviation of the similarity between all answers; and k ∈ [0, 1] is an adjustment coefficient.

10. A multi-model collaboration based intelligent question answering system implementing the method of any one of claims 1-9. The pre-processing module, the model calling module, the intention classification module, the rule matching module, the answer output module and the feedback loop optimization module are included. The pre-processing module is used for pre-processing the question text input by the user; The model calling module is used for calling a large language model to perform collaborative reasoning on the pre-processed text to generate candidate answers; The intention classification module and the rule matching module are used for judging the question type; The answer output module is used for screening the candidate answers according to the question type and outputting a final answer; The feedback loop optimization module is used for dynamically adjusting the weight of the large language model according to the correctness data of the final answer.

Images