Audit compliance ai knowledge base multi-engine fusion retrieval and question answering method and system

Abstract

The application discloses an audit compliance AI knowledge base multi-engine fusion retrieval and question and answer method and system, belongs to the technical field of artificial intelligence and knowledge management, and analyzes a user question through a double-engine intention recognition mechanism and generates a structured retrieval plan; based on the plan, recall operations of four channels of semantic retrieval, keyword matching, structured query and FAQ matching are executed in parallel; five-stage intelligent fusion sorting is executed on the multi-channel recall results, including dynamic weight distribution, adaptive score standardization, differentiated truncation, multi-source consensus bonus and coarse-grained filtering; multi-layer quality bottom processing is executed on the results after fusion sorting, including diversity optimization, multi-level confidence response, full-link observability and feedback closed loop, to generate a final answer and output; the scheme can dynamically adjust the retrieval strategy according to the problem type, systematically fuse multi-source retrieval results, and effectively improve the accuracy and reliability of knowledge retrieval and question and answer in the field of audit compliance.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and knowledge management technology, specifically involving a multi-engine fusion retrieval and question-answering method and system for audit compliance AI knowledge base. Background Technology

[0002] In the field of audit compliance, companies and accounting firms frequently need to consult a large number of professional documents, including regulations, standards, audit manuals, case libraries, and internal operating guidelines. With the advancement of digital transformation, more and more organizations are establishing enterprise-level AI knowledge base systems, hoping to improve the work efficiency of auditors through intelligent retrieval and question-and-answer mechanisms.

[0003] However, existing enterprise-level knowledge base systems generally adopt a "single-path" approach in their retrieval strategies, meaning they use the same retrieval and ranking mechanism indiscriminately for all user questions. A typical existing technical solution includes the following process: First, after the user inputs a question, the system sends it to both a BERT-based semantic retrieval module and an Elasticsearch (distributed search and analytics engine) keyword matching module for retrieval; then, the two retrieval results are merged and scored and ranked using a unified inverse fractional fusion (RRF) formula; finally, the ranked results are fed into a large language model to generate the answer. This approach lacks an intent recognition stage during the response process, making it unable to distinguish between different question types such as exact queries, semantic understanding, or common question-and-answer questions. It also lacks systematic control over answer quality, easily leading to content illusions or simple rejection when relevant information is not available in the knowledge base. Furthermore, it lacks a closed-loop mechanism for user feedback to optimize system parameters.

[0004] The aforementioned existing technical solutions have the following main drawbacks: First, the retrieval strategy lacks specificity. Different types of questions have significantly different dependencies on retrieval channels. For example, exact query questions heavily rely on structured query and keyword matching channels, but have a lower dependence on semantic retrieval channels; semantic understanding questions, on the other hand, heavily rely on semantic retrieval channels. Existing solutions indiscriminately send all types of questions to the same combination of retrieval channels, failing to select the optimal channel strategy based on the question type, thus limiting the relevance and accuracy of the retrieval results.

[0005] Second, the multi-channel fusion ranking uses a uniform weight. Different retrieval channels differ fundamentally in their scoring mechanisms, score distributions, and reliability. Existing solutions employ a uniform RRF fusion strategy, ignoring the differences in scoring scales between channels and the varying reliability requirements of different question types. This results in the fusion ranking results failing to accurately reflect the retrieval quality of each channel. For example, for exact query questions, the hit results from the structured query channel should receive a higher fusion weight, but the uniform RRF strategy cannot reflect this difference.

[0006] Third, there is a lack of a systematic quality assurance mechanism. Existing solutions lack the ability to optimize for diverse results, easily leading to highly repetitive results at the top of the rankings; they lack the ability to provide tiered responses based on confidence levels, failing to provide differentiated prompts to users when search results are of insufficient quality; they lack end-to-end observability, making it difficult to pinpoint the problem when search results are poor; and they lack a user feedback-driven self-optimization loop, meaning that even if user feedback is collected, it cannot automatically analyze the root causes of problems and adjust system parameters, preventing the system's search performance from continuously improving during use.

[0007] Therefore, there is an urgent need for an intelligent retrieval and question-answering solution that can dynamically adjust retrieval strategies based on question type, differentiate and rank multi-path recall results, and systematically ensure answer quality. Summary of the Invention

[0008] To address the shortcomings of the existing technologies mentioned above, this application provides a multi-engine fusion retrieval and question-answering method and system for an audit compliance AI knowledge base.

[0009] Firstly, this application proposes a multi-engine fusion retrieval and question-answering method for an audit compliance AI knowledge base, including the following steps: Get the question text entered by the user; The intent recognition results are obtained by performing intent recognition on the question text. A structured retrieval plan is generated based on the intent recognition results. The structured retrieval plan includes channel selection information and retrieval content information. The channel selection information is used to indicate which retrieval channel needs to be activated from multiple retrieval channels. The retrieval content information includes the core retrieval text after referential resolution and entity standardization, domain filter tags for limiting the retrieval scope, and exact matching objects for exact matching. Based on the structured retrieval plan, recall operations for multiple retrieval channels are executed in parallel to obtain the recall result set corresponding to each retrieval channel; The recall result set from multiple retrieval channels is subjected to multi-stage fusion and sorting processing to obtain a fusion and sorted result sequence; Perform multi-level quality fallback processing on the merged and sorted result sequence to generate the final answer and output it.

[0010] Secondly, this application proposes a multi-engine fusion retrieval and question-answering system for an audit compliance AI knowledge base, including: The intent recognition module is used to receive the question text input by the user, perform intent recognition on the question text, and generate a structured search plan. The structured search plan includes channel selection information and search content information. The multi-channel recall module is used to execute recall operations of multiple search channels in parallel based on a structured search plan, and obtain the recall result set corresponding to each search channel. The fusion sorting module is used to perform multi-stage fusion sorting processing on the recall result sets of multiple retrieval channels to obtain the fusion sorted result sequence; The quality fallback module is used to perform multi-layer quality fallback processing on the merged and sorted result sequence, generate the final answer and output it. The feedback loop module is used to collect user feedback on the final answer, perform root cause analysis based on the feedback information, and automatically adjust system parameters according to the root cause analysis results.

[0011] Thirdly, this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described above.

[0012] The beneficial effects of this invention are: The system automatically identifies question types and generates structured retrieval plans that include channel selection and tag filtering through a dual-engine intent recognition mechanism, enabling differentiated retrieval and significantly improving the relevance and targeting of results. A five-stage intelligent fusion ranking mechanism is employed, utilizing dynamic weight allocation, adaptive Sigmoid score standardization, differentiated truncation, multi-source consensus scoring based on the channel independence matrix, and coarse-grained noise reduction to systematically eliminate differences in scoring scales and credibility across different channels. Furthermore, a four-layer quality assurance mechanism is constructed, encompassing MMR (maximum marginal relevance) diversity optimization, multi-level confidence tiered response, and a fully observable root cause feedback loop, ensuring the system possesses highly reliable, interpretable, and continuously self-optimizing answer quality assurance capabilities. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the overall process of the present invention.

[0014] Figure 2 This is a schematic diagram of the dual-engine intent recognition processing flow in an embodiment of the present invention.

[0015] Figure 3 This is a schematic diagram of the five-stage fusion sorting process in an embodiment of the present invention.

[0016] Figure 4 This is a schematic diagram of the processing flow of the four-layer quality fallback mechanism in an embodiment of the present invention.

[0017] Figure 5 This is a system diagram of the present invention. Detailed Implementation

[0018] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein; rather, these embodiments are provided so that a more thorough understanding of the invention can be achieved and that the full scope of the invention can be conveyed to those skilled in the art.

[0019] Firstly, this application proposes a multi-engine fusion retrieval and question-answering method for an audit compliance AI knowledge base, such as... Figure 1 As shown, it includes the following steps: Get the question text entered by the user; The intent recognition results are obtained by performing intent recognition on the question text. A structured retrieval plan is generated based on the intent recognition results. The structured retrieval plan includes channel selection information and retrieval content information. The channel selection information is used to indicate which retrieval channel needs to be activated from multiple retrieval channels. The retrieval content information includes the core retrieval text after referential resolution and entity standardization, domain filter tags for limiting the retrieval scope, and exact matching objects for exact matching. Furthermore, the step of performing intent recognition on the question text to obtain intent recognition results, and generating a structured retrieval plan based on the intent recognition results, includes: The intent recognition of the question text adopts a dual-engine mechanism, which includes an AI parsing engine and a rule engine; The AI ​​parsing engine uses a large language model to perform semantic analysis on the question text and outputs intent type, entity information and knowledge domain labels. The rule engine performs pattern matching on the question text based on a predefined set of rules and outputs the matching results. Consistency judgment is made based on the output results of the AI ​​parsing engine and the rule engine. When the judgment results are consistent, a structured retrieval plan is generated using the consistent results. When the judgment results are inconsistent, two sets of retrieval plans are generated in parallel and intent confidence weights are introduced in the subsequent fusion stage.

[0020] The AI ​​parsing engine sends user questions to a large language model, which performs semantic analysis to identify the following information: the core intent of the question (precise query / semantic understanding / common question and answer / mixed type); key entities in the question (such as regulatory numbers, clause numbers, audit standard names, etc.); pronouns and contextual information in the question; and the knowledge domain tags to which the question applies (such as laws and regulations, audit cases, and compliance manuals, etc.). Rule Engine: Based on predefined rule templates, it performs pattern matching on questions to help identify structured query intents containing specific numbering formats; FAQ matching intents containing common question templates; and domain classifications containing professional terminology keywords. The analysis results of the two engines are mutually verified and backed up: when the intent judgments of the AI ​​parsing engine and the rule engine are consistent, the result of the AI ​​parsing engine shall prevail; when the two are inconsistent, retrieval plans are generated for the two intents respectively, and then the best one is selected by fusion and ranking.

[0021] Technical implementation details of the dual-engine collaboration mechanism: The specific algorithm for consistency judgment is as follows: Intent categories are vectorized, defining a four-dimensional one-hot intent vector: Exact Query = [1,0,0,0], Semantic Understanding = [0,1,0,0], Common Question Answering = [0,0,1,0], and mixed types are represented as probability distribution vectors of each intent category (e.g., when exact query and semantic understanding are mixed, it can be represented as [0.45,0.55,0,0], meaning exact query accounts for 0.45% and semantic understanding accounts for 0.55%), rather than a single one-hot vector. The AI ​​parsing engine outputs the intent vector. The rule engine outputs an intent vector. The degree of consistency between the two is calculated using cosine similarity:

[0022] Consistency determination rule: When When the intent categories output by the two engines are exactly the same, it is judged as "consistent," and the result of the AI ​​parsing engine is used to generate a single search plan; when When the intent categories output by the two engines differ, the result is considered "inconsistent," triggering a traffic splitting mechanism. Additionally, an auxiliary judgment dimension is introduced: key entity matching degree. This compares the Jaccard similarity of the entity sets extracted by the two engines. When the entity Jaccard similarity is ≥0.8, even if the intent categories differ, the issue is categorized as "partially consistent." In this case, the AI ​​parsing engine becomes the main engine, and the rule engine becomes the secondary engine. The intent category judgment result of the main engine is used as the intent type of the retrieval plan, while simultaneously integrating the entity information extracted by the secondary engine. The master-slave relationship between the main and secondary engines is a pre-configured fixed relationship; that is, the AI ​​parsing engine is always the main engine, and the rule engine is always the secondary engine. This fixed configuration is based on the fact that the AI ​​parsing engine, based on a large language model, can output more granular intent types, entity information, and knowledge domain labels. Its semantic understanding capability is superior to that of the rule engine based on rule matching; therefore, the intent judgment of the AI ​​parsing engine is used as the main intent graph. The specific fusion algorithm for the entity information of the fusion sub-engine is as follows: the entity set extracted by the rule engine and the entity set extracted by the AI ​​parsing engine are combined, and entities in the rule engine with a matching confidence score higher than a preset entity confidence score threshold are added to the entity information extracted by the AI ​​parsing engine. The merged entity information is used as the entity constraint condition for the retrieval content information in the structured retrieval plan to guide the accurate matching and tag filtering of subsequent retrieval channels.

[0023] Inconsistent execution mechanism: When the intents of the two engines are inconsistent, a parallel execution strategy is adopted, and the specific process is as follows: Step 1: Based on the intent judgments of the AI ​​parsing engine and the rule engine, generate an intent retrieval plan based on the AI ​​engine and an intent retrieval plan based on the rule engine, respectively.

[0024] Step 2: Two search plans are submitted in parallel to four recall channels. Each search plan independently performs channel selection, tag filtering, and exact matching to generate its own candidate result set.

[0025] Step 3: Merge the two sets of candidate results into a unified candidate pool, with each result carrying a source tag. intent The "ai" or "rule" is used to distinguish the source in the subsequent fusion and sorting stage.

[0026] Step 4: In the five-stage fusion and ranking stage, differentiate the results from different intent sources: introduce an intent confidence weight W. intent Among them, the AI ​​engine's intent confidence W ai The entropy value is calculated based on the output probability distribution of the large language model (the lower the entropy, the higher the confidence). The intent confidence W of the rule engine is then calculated. ruleBased on the rule-based matching confidence field. During fusion and sorting, the final score for each result = base fusion score × (1 + W) intent ×α intent ), where α intent The source adjustment factor (default 0.15) ensures that results from high-confidence intended sources receive a moderate advantage in ranking.

[0027] Step 5: If the two retrieval plans have overlapping document fragments in the Top-N results (i.e. the same fragment is recalled by both plans at the same time), then apply an additional dual-intent consensus score to the fragment. The score is equal to the base score × β (β is 1.5 by default, which is higher than the β=1.0 of the ordinary multi-source consensus score), which reflects the higher reliability of "two different intent judgment paths hitting at the same time".

[0028] After intent recognition is completed, a structured retrieval plan is output, containing fields as shown in Table 1: Table 1: Field Example Table

[0029] For example, when a user asks "the provisions of Auditing Standard No. 1131 regarding audit sampling", the intent recognition module automatically identifies: precise matching according to the Auditing Standard No. 1131 number; marking the property rights type as "laws and regulations"; and prioritizing precise matching through the structured query channel in subsequent steps.

[0030] Based on the structured retrieval plan, recall operations for multiple retrieval channels are executed in parallel to obtain the recall result set corresponding to each retrieval channel; Furthermore, the multiple retrieval channels include a semantic retrieval channel, a keyword matching channel, a structured query channel, and a FAQ matching channel; The semantic retrieval channel generates text vectors based on a pre-trained embedding model and performs similarity retrieval in a vector database; The keyword matching channel performs full-text keyword matching based on the BM25 algorithm; The structured query channel supports both exact and fuzzy matching of document numbers and clause levels; The FAQ matching channel performs fast matching based on an independent FAQ vector library.

[0031] Among them, multiple retrieval channels include, first channel: semantic retrieval channel; The user question and all document fragments in the knowledge base are converted into high-dimensional vectors (e.g., 1024-dimensional vectors) using a pre-trained embedding model; the cosine similarity (or inner product similarity) between the question vector and the document fragment vector is calculated in the vector database; the top-K document fragments with the highest similarity are returned; the vector dimension is 1024, and the fragmented document paragraphs are used as the retrieval unit.

[0032] Second channel: Keyword matching channel; The classic BM25 algorithm is used to perform full-text keyword matching on the user's question; the document fragment with the highest overlap with the question keywords is retrieved from the inverted index; the BM25 algorithm scores the keyword matching results by considering term frequency, inverse document frequency and document length normalization; and the top-K document fragments with the highest BM25 scores are returned.

[0033] Third channel: Structured query channel; For queries containing structured information such as clause number, regulation number, and document number, the query conditions are extracted from the exact match items output by the intent recognition module; the structured index is used to perform an exact search according to fields such as clause number (e.g., "Article 23") and regulation number; and the exact matching document fragments are returned. This channel has the highest accuracy but the lowest recall rate. Fourth channel: FAQ matching channel; Maintain a Frequently Asked Questions (FAQ) database to store frequently asked user questions and their standard answers; perform semantic similarity matching between user questions and questions in the FAQ database; when the matching degree exceeds a threshold, directly return the corresponding standard answer or related document fragment; this channel is suitable for handling common user process-related and operational questions, with the fastest response speed.

[0034] The FAQ matching channel reuses the Embedding Model (the same 1024-dimensional pre-trained model) of the semantic retrieval channel, but maintains an independent FAQ vector library. The specific technical solution is as follows: Independent Vector Library Design: The FAQ vector library and the knowledge base vector library are physically isolated and stored in different collections within the vector database. Each record in the FAQ Collection includes: familiar_id (FAQ number), question_text (original question text), question_vector (question vector), answer_text (standard answer), related_doc_ids (list of related document IDs), hit_count (historical hit count), and last_updated (last update time).

[0035] The technical basis for reusing the Embedding Model is that FAQ questions and knowledge base document fragments use the same embedding model to ensure semantic space consistency and avoid the waste of computational resources caused by maintaining two sets of models. The FAQ vector library and the knowledge base vector library are distinguished by Collection isolation and namespace prefixes (faq_ vs kb_). During retrieval, only the FAQCollection is queried, so as not to be confused with the results of the knowledge base.

[0036] Matching process: After the user's question is vectorized by the Embedding Model, the cosine similarity with all FAQ question vectors is calculated in the FAQ Collection, and the top-1 result with the highest similarity is taken as the best matching candidate.

[0037] Threshold setting and dynamic adjustment: The threshold is determined based on the FAQ matching threshold θ. faq The initial value is determined based on the F1-score optimization of the historical validation set. The specific process is as follows: collect 500 labeled validation samples (question, FAQ question), and then... faq A grid search is performed within the value range [0.5, 0.9] (step size 0.05) to calculate the F1-score for each θ value. The θ corresponding to the maximum F1-score is taken as the initial threshold. Experiments have verified that the initial threshold θ... faq The optimal F1-score is achieved when the F1 value is 0.72 (F1=0.87).

[0038] Dynamic adjustment mechanism of threshold: threshold θ faq Related to the feedback closed-loop mechanism of this invention, online calibration based on user feedback is supported. The specific calibration logic is as follows: Statistically analyze user feedback data after the most recent N=200 (this value was determined through ablation experiments within the range N∈{100,150,200,250,300} with the goal of predicting stability based on positive feedback rate) FAQ channel hits, and calculate the positive feedback rate (positive_rate = number of likes / (number of likes + number of dislikes)). When positive_rate > 0.85, it indicates that the current threshold is too conservative (potentially missing valid FAQs), and θ is adjusted accordingly. faq Lower it by 0.02 (lower limit 0.60); when positive_rate < 0.60, it indicates that the current threshold is too lenient (more false positives), so adjust θ. faq Increase by 0.02 (maximum 0.85). Calibration cycle is once a week.

[0039] Handling below the threshold: When the similarity of the best match of the FAQ is lower than the threshold θ faq When a match is found to be unavailable, the result is not returned directly but is downgraded: the FAQ match is added to the candidate pool with a low weight (weight coefficient 0.3) for subsequent fusion and ranking. If the FAQ result enters the Top-N after fusion and ranking, it is displayed as a reference in the answers; otherwise, it is discarded. This design ensures that the FAQ channel does not completely waste recall results when the matching is uncertain, while also preventing low-quality FAQ answers from being directly presented to users.

[0040] Cold start and update strategy for the FAQ repository: Cold start strategy: The initial FAQ library is built from two sources: First, a collection of frequently asked questions in the audit and compliance field (approximately 200 questions) compiled manually, covering common scenarios such as audit process operation, report filling guidelines, and applicable regulations; Second, user questions from the past 6 months are automatically extracted from historical question and answer logs and semantically clustered using a clustering algorithm (DBSCAN algorithm, eps=0.15). Cluster centers containing ≥10 similar questions are extracted as FAQ candidates, which are then manually reviewed and added to the library.

[0041] Runtime automatic update strategy: Continuously monitor user question logs. If a new question is detected to have been asked more than 5 times by different users in the last 30 days, and there are no questions with a similarity ≥ θ in the current FAQ library, the update strategy will be implemented. faq When a match is found, the question is marked as a "FAQ candidate". FAQ candidates enter a manual review queue. Once approved, domain experts write standard answers and add them to the database, while simultaneously vectorizing them and storing them in the FAQ Collection. This mechanism allows the FAQ database to continuously expand as it is used, covering emerging and frequently asked questions.

[0042] Each channel has an independent timeout limit. If a channel does not return a result within the timeout period, the recall result for that channel is considered empty, which does not affect the normal operation of other channels and ensures that the overall response time is controllable.

[0043] The recall result set from multiple retrieval channels is subjected to multi-stage fusion and sorting processing to obtain a fusion and sorted result sequence; Furthermore, the multi-stage fusion sorting process includes the following five stages: In the first phase, dynamic weights are assigned to the recall results of each retrieval channel based on the intent type in the structured retrieval plan. In the second stage, the recall results of each retrieval channel are standardized to eliminate the differences in scoring scales between different retrieval channels. Furthermore, in the second stage, the score standardization process adopts a non-linear mapping method based on the Sigmoid function. The temperature parameter of the Sigmoid function is dynamically calculated and determined according to the standard deviation of the score of the most recent preset number of retrieval results for each retrieval channel.

[0044] In the third stage, the number of differential retentions is determined based on the characteristics of each retrieval channel, and the standardized results of each channel are truncated. In the fourth stage, multi-source consensus scoring is applied to results from different retrieval channels, and consensus scores are increased for results that are hit by multiple retrieval channels simultaneously. Furthermore, in the fourth stage, the multi-source consensus scoring process introduces a channel independence matrix. The channel independence matrix records the degree of independence between each retrieval channel. For results that are hit by multiple retrieval channels at the same time, the consensus score is the base score multiplied by the sum of the independence coefficients between the corresponding channels.

[0045] In the fifth stage, coarse-grained filtering and noise reduction are performed on the fused results to remove low-quality results.

[0046] When four channels are recalled in parallel, a five-stage intelligent fusion ranking algorithm is used to comprehensively rank the results, instead of using the traditional unified RRF formula: Phase 1: Dynamic weight allocation; Based on the question type output by the intent recognition module, the fusion weights of each retrieval channel are dynamically allocated. Different question types correspond to different channel weight configurations, as shown in Table 2: Table 2: Channel Weight Configuration Table

[0047] For example, for exact query questions, the structured query channel is assigned the highest weight of 0.45; for FAQ questions, the FAQ matching channel is assigned the highest weight of 0.65. This weighting ensures that different types of questions can obtain optimal results from the most appropriate channel.

[0048] Phase Two: Score Standardization; Because the four channels use different scoring algorithms (cosine similarity for semantic retrieval, BM25 score for keyword matching, exact match confidence for structured queries, and semantic match for FAQ matching), the score scales and distributions of each channel differ significantly. The following steps were used to standardize the scores: Min-Max normalization: Min-Max normalization is performed on the recall score in each channel, mapping the score of each channel to the [0,1] interval; Non-linear compression: The normalized scores are non-linearly compressed using the Sigmoid function or a similar compression function to eliminate the skewed distribution of scores in each channel (e.g., the semantic retrieval channel may return a large number of high-scoring results, while the score distribution of the keyword matching channel is more dispersed). Standardization and unified scale: After standardization, the scores of each channel can be compared on the same scale, providing a basis for subsequent weighted fusion.

[0049] Specifically, the adaptive parameters for score standardization and anomaly handling are as follows: Adaptive setting of Sigmoid temperature parameters: Instead of using fixed Sigmoid parameters, the temperature parameters are adaptively determined based on the historical score distribution of each channel. The specific technical solution is as follows: Distribution statistics: Maintain the normalized score distribution statistics for the most recent N=1000 searches for each channel, including the mean and standard deviation.

[0050] Temperature parameter calculation: Temperature parameters Automatically calculated based on the channel's fraction distribution characteristics:

[0051] in, This represents the temperature parameter of the Sigmoid mapping function. This represents the standard deviation of the scores of the most recent 1,000 search results for this search channel; Design basis: When At this point, the slope of the inflection point of the Sigmoid function exactly matches the standard deviation of the score distribution in that channel, making the sensitivity of the compression curve to score differences consistent with the actual dispersion of scores within the channel.

[0052] Channel score distribution is concentrated: Small → Small → Steep compression curve → Amplify minute differences; Channel score distribution is dispersed: Big → Large compression curve → flat compression extreme value.

[0053] Special handling of outliers: Channels may return extreme outlier scores (e.g., cosine similarity > 1.0 or negative BM25 score) due to timeouts, network anomalies, or indexing errors. Outlier detection and correction are performed before standardization. Outlier detection: For the recall result score set of each channel, calculate the interquartile range IQR = Q3 - Q1, and mark the scores that exceed the range [Q1 - 3 × IQR, Q3 + 3 × IQR] as outliers.

[0054] Outlier correction: Outliers are not discarded directly, but truncated to the boundary value. For example, if the normalized score of a semantic search result is 1.5 (exceeding the upper bound Q3 + 3 × IQR = 1.2), it is corrected to 1.2 before sigmoid compression.

[0055] Channel-level anomaly handling: When more than 50% of the results of a channel are marked as anomalies, the channel is determined to be in an abnormal state. The fusion weight of all results of the channel is temporarily multiplied by a degradation factor of 0.5. At the same time, a channel anomaly alarm is recorded in the full-link log and the operation and maintenance personnel are notified to investigate.

[0056] Phase 3: Channel Differentiation K-Parameters; The automated method for determining the K value: The K value is not set manually, but is automatically determined based on the Precision@K curve of each channel on the validation set. The specific technical solution is as follows: Validation set construction: Construct a validation set containing 2000 labeled entries, covering four intent types.

[0057] Precision@K curve plotting: For each channel, calculate the Precision@K value (i.e., the proportion of the first K results containing the correct answer) on the validation set for different K values ​​(K=1,2,3,...,100).

[0058] Inflection point detection algorithm: The optimal K value is determined using an inflection point detection method based on the second derivative. The specific calculation process is as follows: Smooth the Precision@K curve (sliding window mean, window size = 5); calculate the first difference of the smoothed curve. ; Calculate the second-order difference ,in, =Precision@K represents the proportion of the first K results retrieved after recalling a certain channel that contain the correct answer. =Precision@(K+1), representing the same proportion when taking the first K+1 items; optimal K value = argmax K |Δ 2 P(K)| represents the K value with the largest absolute value of the second-order difference; this point corresponds to the inflection point where Precision@K changes from "rapid growth" to "gradual flattening".

[0059] Experimental results: Semantic retrieval channel: inflection point K=62, rounded down to K=60; Keyword matching channel: inflection point K=38, rounded down to K=40; Structured query channel: inflection point K=9, rounded down to K=10; FAQ matching channel: inflection point K=4, rounded down to K=5.

[0060] Phase Four: Multi-source consensus scoring; If the same document fragment is retrieved by multiple retrieval channels simultaneously, a consensus score is applied to that fragment. The consensus score is calculated as follows: For a document fragment that is recalled simultaneously by N channels, its consensus score = base score × (N-1), where the base score is a configurable parameter; The paragraphs that are simultaneously recalled by three channels (such as being hit by semantic retrieval, keyword matching, and structured query) receive the highest consensus score. Phase 5: Coarse-grained filtration and noise reduction; After the fusion and sorting are completed, coarse-grained filtering is performed to eliminate low-quality and mismatched results: Low-score elimination: Discard results whose fusion scores are significantly lower than the threshold to avoid low-relevance results interfering with subsequent processing; Non-matching blacklist: Identifies and blacklists results that do not match the semantics of the question at all; Top-Z retention: The top Z results with the highest fusion scores (e.g., the top 20) are retained and sent to the subsequent quality fallback stage for fine sorting and screening.

[0061] Perform multi-level quality fallback processing on the merged and sorted result sequence to generate the final answer and output it.

[0062] Furthermore, the multi-layer quality fallback processing includes: the first layer of result diversity optimization processing, which uses the maximum marginal relevance algorithm to perform redundancy removal and reordering on the fused and sorted result sequence, and iteratively selects results from the result sequence that are highly relevant to the problem and have low similarity to the already selected results.

[0063] Results diversity optimization: After intelligent sorting, it's easy for the top results to all come from different fragments of the same document, leading to a single perspective in the answers. Therefore, the Maximum Marginal Relevance (MMR) algorithm is used for diversity optimization. Step 1: For multiple fragments from the same document, retain only the fragment with the highest fusion score in the document and remove redundant fragments from the same document; Step 2: In the deduplicated candidate set, the MMR algorithm is used to achieve a balance between relevance and diversity. The formula for calculating MMR is:

[0064] in ∈ Selected set, Candidate segments With the question similarity, for Maximum similarity with the selected fragment set, For balance parameters ; Step 3: Iteratively select evidence using the MMR algorithm to ensure that the top 5 pieces of evidence recommended to the user come from different documents and cover different perspectives, thus avoiding a single perspective in the answer.

[0065] Furthermore, the multi-layered quality fallback processing also includes: a second-layer multi-level confidence response processing, which compares the final confidence score of the first result in the merged and sorted result sequence with preset high confidence thresholds and low confidence thresholds: when the final confidence score is higher than the high confidence threshold, a complete answer is directly generated based on the search results; when the final confidence score is lower than the low confidence threshold, the user is prompted that no relevant information supporting the answer was found in the knowledge base and is advised to adjust the question wording; when the final confidence score is between the low confidence threshold and the high confidence threshold, a confidence prompt is attached to the generated answer, informing the user that the answer is uncertain; the multi-layered quality fallback processing also further includes collecting user feedback on the final answer, performing root cause analysis based on the feedback information, and automatically adjusting the parameters in the method according to the root cause analysis results, wherein the root cause analysis is implemented based on a classification model, the classification model takes a feature vector of preset dimensions as input, the feature vector includes question type, the score of the first result in the ranking, the result diversity ratio, and the number of retrieval channels, and outputs the corresponding root cause category.

[0066] Multi-level confidence response processing: The highest score in the multi-level confidence response mechanism is clearly defined as the final weighted score after the five stages are merged and ranked. The value range of this score is [0,1], where 1.0 represents the highest confidence level, as shown in Table 3 below: Table 3: Confidence Level Corresponding Strategy Table

[0067] The thresholds of 0.3 and 0.6 are not arbitrarily set, but are automatically determined based on the optimal classification performance of the validation set. The specific technical solution is as follows: Validation set construction: Collect 1000 triplet validation data (question, search result, user satisfaction rating), with user satisfaction ratings ranging from 1 to 5 (1-2 points = dissatisfied, 3 points = neutral, 4-5 points = satisfied).

[0068] ROC curve analysis: Using "user satisfaction (4-5 points)" as the positive class, plot the ROC curve (response operating characteristic) of the fusion score, and calculate the true positive rate (TPR) and false positive rate (FPR) at different thresholds.

[0069] The Youden index determines the high confidence lower limit: The Youden index J = TPR - FPR, and the score corresponding to the maximum value of J is taken as the high confidence lower limit. Verification showed that the Youden index reaches its maximum value J = 0.71 when the score = 0.62, which is rounded down to 0.6 as the high confidence lower limit. At this threshold, TPR = 0.89 and FPR = 0.18, meaning that 89% of satisfactory answers are correctly classified as high confidence, and only 18% of unsatisfactory answers are incorrectly classified as high confidence.

[0070] Determining the lower confidence level: Using "User Dissatisfaction (1-2 points)" as the positive class, an ROC curve was plotted and the Youden index was calculated. Verification showed that the Youden index reaches its maximum value at a score of 0.28, which was rounded down to 0.3 as the lower confidence level upper limit. At this threshold, 88% of unsatisfactory answers were correctly classified as low confidence.

[0071] F1-score validation: Under the final threshold configuration (<0.3 low, 0.3-0.6 medium, >0.6 high), the macro average F1-score for the three categories is 0.83, validating the effectiveness of the threshold configuration.

[0072] Root cause analysis: Automatically analyze the root causes of poor answer quality (such as improper segmentation strategy, unreasonable search channel weights, inaccurate tag filtering, etc.). The specific algorithm and parameter mapping for root cause analysis: A root cause classification model based on feature vectors is disclosed, which automatically infers the most likely root cause category from user feedback using machine learning algorithms. The specific technical solution is as follows; Input feature vector: Extract the following 12-dimensional feature vector from the retrieval log of each negative feedback (downvote / error): Table 4: 12-Dimensional Feature Vector Table

[0073] Root cause category definition: Define 6 types of root causes: Table 5: Six Types of Root Causes

[0074] Classification Model: The LightGBM (Lightweight Gradient Boosting Machine) model was used as the root cause classifier and trained on a pre-labeled training set of 3000 (feature vectors, root cause labels). The model output is the probability distribution of each root cause category, and the root cause with the highest probability is taken as the classification result. The model achieved a classification accuracy of 82.6% and a Top-2 hit rate of 94.3% on the test set.

[0075] Establish a mapping relationship between root cause categories and optimizable parameters to automate the adjustment from root cause to specific parameters: Table 6: Root Cause-Parameter Mapping Table

[0076] Parameter optimization execution: After the root cause analysis is completed, the relevant parameters of the knowledge retrieval engine are automatically adjusted (such as adjusting channel weights, modifying fragment lengths, and optimizing tag filtering rules) to improve the accuracy of answers to similar questions in the future.

[0077] This embodiment uses the scenario of an auditor from an accounting firm querying auditing standards as an example to illustrate the complete implementation process of the present invention. When conducting audit work, the auditor needs to input a question into the knowledge base system, such as: "According to the provisions of 'Chinese Certified Public Accountant Auditing Standard No. 1323—Related Parties,' what material misstatement risks should a certified public accountant pay attention to when performing audit procedures on related party transactions?" This question involves both precise standard number citation and open-ended semantic understanding requirements, belonging to a typical mixed-type audit compliance query scenario, which fully demonstrates the advantages of the multi-engine fusion retrieval method of the present invention.

[0078] The following section provides a detailed explanation of this technical solution using examples from specific application scenarios.

[0079] Step S1: Obtain the question text input by the user; The system receives natural language question text input by auditors through a user interface. In this embodiment, the question text is: "According to the provisions of 'Chinese Certified Public Accountant Auditing Standard No. 1323—Related Parties,' what material misstatement risks should a certified public accountant pay attention to when performing audit procedures on related party transactions?" The system preprocesses the question text, including text cleaning, length truncation, and session context association. After preprocessing, the system generates a unique query identifier for this query, used for end-to-end tracing.

[0080] It should be understood that the specific implementation of session context association in this embodiment is as follows: The system maintains a dialogue history buffer with a sliding window size of 5 rounds. When the user inputs a new question, the system appends the historical dialogue information in the buffer to the current question text in the form of structured fields, so that the intent recognition and retrieval stages can obtain sufficient contextual information for reference resolution and semantic completion. Each record in the dialogue history buffer contains a role identifier, message text, and timestamp. When the buffer overflows, old records are removed according to the principle of earliest timestamp priority. The 5-round window size was determined through ablation experiments on an actual audit dialogue dataset. Experimental results show that when the window is too small (less than 3 rounds), the accuracy of reference resolution decreases significantly, and when the window is too large (more than 8 rounds), the introduced noise information begins to interfere with the accuracy of intent recognition.

[0081] Step S2: Dual-engine intent recognition and structured retrieval plan generation; like Figure 2 As shown, the system simultaneously sends the preprocessed question text into the AI ​​parsing engine and the rule engine for parallel processing.

[0082] The AI ​​parsing engine performs semantic analysis based on a large language model. Specifically, the system constructs a prompt template that includes role definitions, output format instructions, and multiple sets of examples. The role definition section explicitly requires the large language model to act as an expert in analyzing retrieval intent in the audit and compliance field. The output format instructions require the return of structured analysis results in JSON format. The examples section provides typical analysis examples for precise query types, semantic understanding types, and common question-and-answer types. The prompt template also embeds a dictionary containing approximately two hundred professional terms in the audit and compliance field, covering abbreviations of standard names, abbreviations of audit procedures, and aliases for compliance inspection items, to assist the large language model in entity recognition and domain disambiguation.

[0083] In this embodiment, after analyzing the input question, the AI ​​parsing engine outputs the following analysis results: the intent type is "mixed type," encompassing both a precise reference to Standard No. 1323 and an open-ended semantic understanding of the risk of material misstatement in related-party transactions; the identified entities include "Chinese Certified Public Accountant Auditing Standard No. 1323—Related Parties" (document reference entity) and "Risk of Material Misstatement" (concept entity); the sub-intent probability distribution is 0.45% for precise queries and 0.55% for semantic understanding; the knowledge domain labels are "Auditing Standards" and "Related-Party Transactions." The confidence score of the above analysis results is 0.85.

[0084] The rule engine performs pattern matching on the question text based on a predefined rule set. Each rule in the rule set is stored in JSON format and includes fields such as rule identifier, rule name, rule type, regular expression pattern, confidence level, and entity extraction template. The initial rule base contains approximately fifty rules, covering common guideline number query patterns, audit procedure query patterns, and high-frequency FAQ matching patterns. In this embodiment, the rule engine identifies the pattern containing guideline number "1323" in the question through regular expression matching, sets the confidence level of the precise query sub-intent to 0.90, and the confidence level of the semantic understanding sub-intent to 0.75.

[0085] Specifically, in this embodiment, the rule engine employs a composite strategy of "priority hierarchy + confidence weighting" for conflict resolution. First, all matched rules are grouped according to their preset priority hierarchy, which is divided into three levels from high to low. Within the same priority hierarchy, if multiple rules match, the weighted average of the confidence scores of each rule is calculated as the overall confidence score for that hierarchy. If the score difference between different intent categories is less than a preset similarity threshold (0.1 in this embodiment), the intent type is determined to be "mixed type." This conflict resolution strategy can provide reasonable judgments in scenarios with ambiguous rule boundaries, avoiding misjudgments of intent caused by matching a single rule.

[0086] After the AI ​​parsing engine and the rule engine output their analysis results, the system performs a consistency check on the results of the two engines. The consistency check is achieved as follows: First, the intent category information output by the two engines is converted into intent category vectors (using one-hot encoding), and then the cosine similarity of the two vectors is calculated. Simultaneously, the Jaccard similarity of the entity sets identified by the two engines is calculated as an auxiliary judgment indicator. When the cosine similarity exceeds a preset consistency threshold (0.8 in this embodiment) and the Jaccard similarity exceeds 0.6, the results of the two engines are determined to be consistent. In this embodiment, both engines identify that the question contains both precise query and semantic understanding intents, and the entity sets have a high degree of overlap, leading to consistent judgment results. Therefore, the system takes the comprehensive result output by the two engines (primarily based on the result of the AI ​​parsing engine, combined with the high-confidence precise query judgment of the rule engine) to generate the final structured retrieval plan.

[0087] In this embodiment, the structured search plan is organized in JSON format and includes the following key fields: Query Identifier field, the value of which is a unique query_id generated by the system; Search Content field, the value of which is the core search text after referencing and entity standardization; Channel Activation List field, indicating the combination of search channels to be activated, in this embodiment all four channels are activated (semantic search, keyword matching, structured query, and FAQ matching); Domain Filter Tag field, the value of which is "Auditing Standards" and "Related Party Transactions", used to limit the knowledge domain scope of the search; Exact Match Object field, the value of which is "Chinese Certified Public Accountant Auditing Standard No. 1323", used to guide the structured query channel to perform exact search; Channel Priority Ranking field, arranged in the order of structured query first, semantic search second, keyword matching third, and FAQ matching last.

[0088] As a supplementary implementation, when the output format of the AI ​​parsing engine does not conform to the expected JSON Schema, the system performs a three-step fault tolerance process: First, it uses regular expressions to extract JSON fragments from the original output text of the AI ​​engine; second, it verifies the extracted JSON fragments against the preset JSON Schema, and if the verification fails, it proceeds to the third step; third, it discards the output result of the AI ​​engine, completely rolls back to the result generated by the rule engine to generate the retrieval plan, and marks this query as an "AI engine rollback event" and records it in the observation log.

[0089] Step S3: Parallel recall via four channels; Based on the channel activation list in the structured retrieval plan, the system initiates recall operations for four retrieval channels in parallel. The recall operations for the four channels are executed asynchronously and in parallel: the system publishes the retrieval plan to the message queue, the recall service of each channel independently consumes messages and performs the retrieval, and after the retrieval is completed, the results are written to the cache aggregator. The fusion sorting service listens for the completion events of each channel through a publish / subscribe pattern.

[0090] The recall process for each channel is as follows: First, the semantic retrieval channel. This channel uses a pre-trained embedding model to convert the retrieved text into a fixed-dimensional dense vector representation. Then, using this vector as the query vector in the vector database, a cosine similarity retrieval is performed, returning the top K text fragments with the highest similarity and their similarity scores. In this embodiment, K is set to 60. The vector database adopts a segmented indexing strategy, with the index stored in segments according to knowledge domain labels. During retrieval, similarity calculation is performed only on the segments that match the domain filter labels, thereby controlling computational overhead while ensuring recall coverage.

[0091] Second, the keyword matching channel. This channel performs word segmentation on the retrieved text using the BM25 algorithm, and then performs full-text keyword matching on the inverted index. The BM25 algorithm comprehensively considers three factors: term frequency, inverse document frequency, and document length normalization, outputting a matching score for each document fragment with the retrieved content. In this embodiment, the K value for the keyword matching channel is set to forty. It should be noted that the word segmentation process in this embodiment has been dictionary-enhanced for professional terms in the field of audit compliance, treating compound terms such as "related party transactions," "material misstatement risk," and "Audit Standard No. 1323" as whole terms for indexing and matching, avoiding a decrease in search accuracy due to improper word segmentation.

[0092] Third, the structured query channel. This channel receives the exact match object ("Chinese Certified Public Accountant Auditing Standard No. 1323") from the retrieval plan and performs a structured retrieval of the document number. In specific implementation, the system adopts a three-level fault-tolerant matching strategy: the first level performs exact matching, querying document records in the relational database according to the standardized number; if the exact matching has no results, it enters the second level to perform standardized matching, converting full-width numbers, Chinese numbers, and various separators in the search number into a standard format before querying; if the standardized matching still has no results, it enters the third level to perform edit distance fuzzy matching, allowing fuzzy matching with an edit distance not exceeding 2, and the confidence of the fuzzy matching result is attenuated according to the method of "1.0 minus the edit distance multiplied by 0.15". When none of the three levels of matching can obtain results, the structured query channel automatically degrades to the semantic retrieval channel, and a degraded mark and a weight attenuation coefficient (0.8) are attached to the degraded retrieval result. In this embodiment, due to the standardized format of the standard number "1323", the first-level exact matching successfully locates the target document.

[0093] Fourth, the FAQ matching channel. This channel reuses the embedding model of the semantic retrieval channel, but performs the retrieval in an independent FAQ vector library. The FAQ vector library is physically isolated from the vector library of the general knowledge base, and FAQ entries are manually reviewed and updated regularly. In this embodiment, the initial similarity threshold for FAQ matching is set to 0.72, which is the optimal result determined through F1-score grid search. The system supports dynamic adjustment of the threshold: after collecting 200 user feedbacks, the positive feedback rate of the FAQ matching results in the most recent 200 feedbacks is calculated. When the positive feedback rate is higher than 0.85 for two consecutive evaluation periods, the threshold is automatically increased by 0.02; when the positive feedback rate is lower than 0.60 for two consecutive evaluation periods, the threshold is automatically decreased by 0.02. The dynamic adjustment range of the threshold is limited to between 0.60 and 0.85 to avoid excessive adjustment leading to fluctuations in matching quality.

[0094] Each channel has an independent timeout control mechanism: 1500 milliseconds for semantic retrieval, 800 milliseconds for keyword matching, 500 milliseconds for structured query, and 300 milliseconds for FAQ matching. The system's global timeout is 3000 milliseconds. Timeout control is implemented through the cache's TTL (Time To Live) mechanism: each channel sets a TTL consistent with its channel timeout when writing recall results to the cache, and the fusion sorting service only reads results that have not expired when triggered. Timeout channels are treated as returning an empty set and do not affect the normal participation of other channels' results in subsequent fusion.

[0095] Step S4: Five-stage intelligent fusion sorting; like Figure 3 As shown, after the recall results from the four retrieval channels are aggregated into the fusion ranking service, the system sequentially performs the following five stages of fusion ranking processing. The processing procedure for each stage is described in detail below with reference to specific data from this embodiment.

[0096] Phase 1: Dynamic Weight Allocation. The system retrieves the corresponding channel weight configuration from a pre-defined weight table based on the intent type (mixed type in this example) and sub-intent probability distribution (0.45 for exact query and 0.55 for semantic understanding) determined in the structured retrieval plan. The weight table maintains independent channel weight matrices for different intent types. Typical configurations for exact query are: structured query channel weight 0.45, keyword matching channel weight 0.25, semantic retrieval channel weight 0.20, and FAQ matching channel weight 0.10. Typical configurations for semantic understanding are: semantic retrieval channel weight 0.40, keyword matching channel weight 0.25, structured query channel weight 0.20, and FAQ matching channel weight 0.15. For mixed types, the system weights the two types of weights according to the sub-intent probability distribution. That is, the final weight of each channel equals its weight in the exact query category multiplied by the exact query sub-intent probability, plus its weight in the semantic understanding category multiplied by the semantic understanding sub-intent probability. After synthesis, the channel weights are normalized to ensure the sum of the weights is one.

[0097] In this embodiment, dynamic weight allocation also incorporates an online learning mechanism for adaptive adjustment. Specifically, the system models the channel weight adjustment problem as a contextual multi-armed slot machine problem: each retrieval channel is considered a selectable "arm," and the channel weight corresponds to the probability of selecting that arm. Contextual features include dimensions such as question type, question length, domain label, and query time. The system utilizes the LinUCB (Linear Upper Confidence Bound) algorithm for online learning in each retrieval: for each channel, an expected return estimate is calculated based on its contextual feature vector and historical feedback data, plus an exploration reward term. The product of the exploration reward term and the exploration coefficient determines the degree of balance between "utilizing" the current optimal weight and "exploring" potentially better weights. The exploration coefficient is optimized to 2.0 through A / B testing, achieving a reasonable balance between utilization and exploration. When the cumulative number of retrievals for a certain type of question reaches one hundred, the system triggers a weight parameter update, with each update step limited to 0.02.

[0098] Phase Two: Score Standardization. Due to the fundamental differences in the scoring mechanisms of different retrieval channels—the semantic retrieval channel outputs a cosine similarity score (typically ranging from 0.5 to 1.0), the BM25 channel outputs a term matching score (its value is significantly affected by factors such as document length), and the structured query channel outputs a confidence score—directly weighting and summing the raw scores from each channel would result in the channel with the larger score scale dominating the final ranking. Therefore, this phase standardizes the scores for each channel separately.

[0099] In this embodiment, the standardization process employs a nonlinear mapping method based on the Sigmoid function, mapping the original scores of each channel to a uniform scale range of 0-1. The key to Sigmoid mapping lies in the setting of the temperature parameter, which determines the steepness of the mapping function in the central region of the score distribution. In this embodiment, the temperature parameter is not a fixed value, but is dynamically calculated and determined based on the standard deviation of the scores from the most recent thousand retrievals for each channel. The logic of this adaptive mechanism is as follows: when the score dispersion of a certain channel is large, a larger temperature parameter is used to make the mapping function smoother, preserving the distinguishability between scores; when the score dispersion is small, a smaller temperature parameter is used to make the mapping function steeper, increasing the difference between scores.

[0100] in, This represents the temperature parameter of the Sigmoid mapping function. This represents the standard deviation of the scores of the most recent thousand search results for this search channel.

[0101] In addition, the standardization phase includes an outlier handling sub-step: First, the interquartile range of the recall results scores for each channel is calculated. Scores exceeding the range of the lower quartile minus three times the interquartile range or the upper quartile plus three times the interquartile range are identified as outliers, and these outliers are truncated and corrected. If more than 50% of the recall results for a certain channel are identified as outliers, an additional weighting factor of 0.5 is applied to the overall weight of that channel, and an alarm is triggered to indicate the possible channel anomaly.

[0102] Phase 3: Truncation of Differentiated Retention Quantities for Each Channel. Different retrieval channels have different recall precision characteristics: semantic retrieval channels have a higher probability of introducing noise into results ranked lower, therefore their retention quantity should be moderate; structured query channels typically have fewer results but extremely high precision, therefore all results should be retained as much as possible. In this embodiment, the differentiated retention quantity for each channel is determined based on the second derivative inflection point detection method of the Precision@K curve on the offline validation set. Specifically, the curves of Precision@K versus K for each channel are plotted on the validation set, the second derivative of the curve is calculated at each point, and the K value corresponding to the inflection point where the second derivative changes from negative to positive (i.e., the starting point of accelerated Precision decline) is taken as the retention quantity for that channel. The K values ​​determined by this method are: semantic retrieval channel 60, keyword matching channel 40, structured query channel 10, and FAQ matching channel 15.

[0103] During actual retrieval execution, the system also supports fine-tuning the K-value based on domain tags and question length. For example, when the domain tag is "audit case," the K-value for the semantic retrieval channel is increased by 10 from the base value because case documents are typically long and have many paragraphs; when the question text length exceeds 100 characters, the K-values ​​for both the semantic retrieval and keyword matching channels are increased by 5 each due to the large amount of information in the question. This fine-tuning strategy ensures sufficient recall coverage under different retrieval scenarios.

[0104] Phase Four: Multi-Source Consensus Bonus. This phase awards additional consensus bonuses to document fragments simultaneously hit by multiple retrieval channels. Unlike existing technologies that only linearly award bonuses based on the number of hit channels, this embodiment introduces a channel independence matrix mechanism. The channel independence matrix is ​​a 4x4 symmetric matrix that records the degree of independence between each pair of retrieval channels. The degree of independence is determined based on a combination of two dimensions: the first dimension is the algorithmic difference between the channel retrieval mechanisms (e.g., semantic retrieval is based on vector similarity while BM25 is based on word frequency statistics, resulting in a higher degree of difference); the second dimension is the overlap rate of channel results in historical retrievals (the lower the overlap rate, the higher the independence). Each element in the matrix ranges from 0 to 1, with larger values ​​indicating greater independence between the two channels.

[0105] The consensus score for a given document fragment is calculated as follows: First, all retrieval channels that hit the document fragment are identified. Then, the pairwise independence coefficients between these channels are read from the channel independence matrix and summed. Finally, the basic consensus score is multiplied by this sum. In this embodiment, the basic consensus score is 0.08, which is determined by parameter search on the validation set using the F1-scores of the consensus hit results' Precision and Recall as targets. For example, if a document fragment is hit by both the semantic retrieval channel and the structured query channel, and the independence coefficients of these two channels are 0.85, then the consensus score for this fragment is 0.08 * 0.85 = 0.068. This differentiated scoring mechanism based on channel independence ensures that joint hits from independent channels with significantly different retrieval mechanisms receive more meaningful scores, while joint hits from redundant channels with similar mechanisms do not receive excessively high scores.

[0106]

[0107] in, This indicates the bonus points for consensus. This represents the base score. Channels can be retrieved from the channel independence matrix. and The independence coefficient between them; Phase 5: Coarse-grained Filtering and Noise Reduction. This phase performs final filtering on the fused results, eliminating low-quality results. This embodiment employs a dual filtering strategy: first, filtering based on a relative threshold eliminates all results with a fusion score lower than 0.15 times the score of the top-ranked result; then, statistical distribution filtering is used to eliminate results with a fusion score lower than the mean of all results minus twice the standard deviation. The union of the two filtering strategies is taken. Furthermore, this phase includes a semantic-based blacklist identification sub-step: calculating the cosine similarity between each result and the original question text; if the similarity is lower than a hard threshold of 0.10, the result is considered semantically completely mismatched with the question and is filtered out; simultaneously, a domain semantic conflict rule table is used to filter results containing content that clearly contradicts the question's domain labels. After this fifth phase of filtering, this embodiment ultimately retains fifteen high-quality document fragments for the subsequent answer generation stage.

[0108] Step S5: Four-layer quality fallback processing and answer generation; like Figure 4 As shown, the system performs four layers of quality fallback processing on the result sequence after the five-stage fusion and sorting, and finally generates the answer and outputs it to the user.

[0109] First layer: Result diversity optimization. Before feeding the result sequence into the large language model to generate the answer, the system first performs redundancy removal and reordering on the result sequence to ensure that the context information finally provided to the large language model is both highly relevant to the question and diverse in content, avoiding the answer biased towards a single source due to highly repetitive context information.

[0110] This embodiment employs the Maximum Marginal Relevance (MMR) algorithm for diversity optimization. The core idea of ​​the MMR algorithm is to iteratively select results from the candidate result set, simultaneously considering the relevance of the candidate result to the original problem and the redundancy of the candidate result within the already selected result set. A tradeoff parameter λ controls the balance between relevance and diversity. In practice, the system uses a step-by-step strategy to improve computational efficiency: first, document-level deduplication is performed, retaining only the fragment with the highest fusion score from multiple fragments from the same document to avoid multiple fragments from the same document occupying too much space in the context; then, MMR iterative selection is performed on the deduplicated set, calculating the MMR score of each candidate result in each iteration, and adding the candidate result with the highest score to the selected set, until the selection quantity reaches a preset upper limit or the candidate set is empty.

[0111] In this embodiment, the trade-off parameter λ is dynamically set according to the question type. Specifically, for exact query questions, λ is set to 0.85, prioritizing relevance; for semantic understanding questions, λ is set to 0.70, allowing more room for diversity; for FAQ questions, λ is set to 0.60, further increasing diversity; and for mixed-type questions, λ is set to 0.65. These λ values ​​were determined through multiple rounds of A / B testing, with user satisfaction as the objective function. This embodiment ultimately selected eight document fragments with high relevance and good diversity as the input context for the large language model.

[0112] The second layer: Multi-level confidence response. Before calling the large language model to generate the answer, the system first evaluates the final confidence score of the first-ranked result. The confidence score is derived from the final score of the first-ranked result after the five-stage fusion ranking, and this score comprehensively reflects the multi-channel hit rate, consensus bonus, and the quality level after filtering. The system compares this score with preset high-confidence thresholds and low-confidence thresholds, and selects the corresponding response strategy.

[0113] In this embodiment, the high confidence threshold is set to 0.60, and the low confidence threshold is set to 0.30. These two thresholds are the optimal dividing points determined by the validation set ROC curve and the Youden index. Furthermore, the thresholds are not fixed. After collecting 500 user feedback responses, the system calculates the positive feedback rate of the most recent 500 responses. When the positive feedback rate exceeds 0.90, the high confidence threshold is lowered by 0.05; when the positive feedback rate is below 0.50, the high confidence threshold is raised by 0.05. The threshold adjustment has a monthly calibration cycle and a cumulative adjustment limit, and an automatic rollback mechanism is configured: if the user complaint rate increases by more than 50% within one week after the adjustment, it will automatically roll back to the threshold before the adjustment.

[0114] In this embodiment, the final confidence score of the top-ranked result is 0.78, which is higher than the high confidence threshold of 0.60. Therefore, the system adopts a strategy of directly generating a complete answer. The system uses eight document fragments optimized for diversity as contextual information, assembles them together with the original question text into a prompt template, and sends it to the large language model to generate the answer. The prompt template requires the large language model to annotate the information source when generating the answer, and to actively declare uncertainty when the evidence is insufficient to support a definitive conclusion. After generating the answer, the system performs format standardization and sensitive information filtering on the answer text.

[0115] It should be understood that if the final confidence score of the first ranked result is between 0.30 and 0.60 (inclusive), the system will add a confidence level hint at the end of the answer while generating the answer, such as "Hint: The following answer is generated based on some matching content in the knowledge base, which may be uncertain. It is recommended to use it in conjunction with professional judgment." If the score is below 0.30, the system will not call the large language model to generate the answer, but will directly return a hint to the user, such as "Sorry, no relevant information that highly matches your question has been found in the current knowledge base. It is recommended that you adjust your question or contact the knowledge base administrator to supplement the relevant content."

[0116] The third layer: End-to-end observability. Throughout the entire retrieval and question-answering process, the system records the execution status of each stage, including: the AI ​​engine timeout for dual-engine intent recognition and the number of rules hit by the rule engine; the recall count and execution time (including timeout flags) for each retrieval channel; the number of intermediate results and score distribution statistics for each stage of the five-stage fusion ranking; changes in result similarity before and after MMR diversity optimization; and the final confidence score and response strategy selection. This observation data is written to the observability system in the form of structured logs. Operations personnel can retrieve complete execution chain information by querying identifiers, enabling them to quickly pinpoint the specific stage where the problem occurs when retrieval results are abnormal.

[0117] The fourth layer: a closed loop of self-assessment and user feedback from the large language model. While generating answers, the large language model self-assesses the accuracy and completeness of those answers, outputting a self-assessed confidence score between 0 and 1. The system compares this self-assessed confidence score with the confidence score at the system's retrieval level: if the deviation exceeds 0.3, the query is marked as an "abnormal confidence" event and included in the manual review queue.

[0118] After receiving an answer, users can submit feedback via the "like," "dislike," or "report error" buttons. Feedback enters a three-tiered status flow mechanism: "Pending Processing - Pending Review - Reviewed." Once operations personnel review and confirm the feedback, the root cause analysis process is triggered. Root cause analysis is based on a classification model, which takes a twelve-dimensional feature vector as input. This feature vector includes: question type encoding, the score of the first ranked result, the average score of the top five ranked results, the diversity ratio of results after MMR optimization (number of deduplicated document sources divided by the number of selected results), the recall count of each of the four retrieval channels, the user feedback type (like / dislike / report error), and the query period. The model outputs one of six root cause categories: RC1 Intent Recognition Bias, RC2 Unreasonable Channel Weights, RC3 Ranking Bias, RC4 Insufficient Retrieval Recall, RC5 Poor Answer Generation Quality, and RC6 Missing Knowledge Base Content.

[0119] There is a pre-defined mapping relationship between root causes and parameters. For example, when the root cause analysis determines it to be RC2 (unreasonable channel weights), the system automatically adjusts the corresponding channel weight parameters for the question type with an adjustment step size of 0.02; when it is determined to be RC5 (poor answer generation quality), the system automatically adjusts the prompt template configuration of the large language model. All automatic parameter adjustments are subject to safety boundary constraints: the step size for a single adjustment does not exceed 0.05 (this step size is determined through stability testing to avoid excessive single adjustment leading to drastic fluctuations in system performance); if performance rollback is triggered after adjustment, it will be automatically rolled back; the cumulative adjustment over thirty days does not exceed 20% (this upper limit is set based on online system stability experience data to ensure that parameter drift remains within a controllable range over the long term); changes with a cumulative adjustment exceeding 15% must be manually reviewed and confirmed before taking effect.

[0120] Secondly, this application proposes a multi-engine fusion retrieval and question-answering system for an audit compliance AI knowledge base, such as... Figure 5 As shown, it includes: The intent recognition module is used to receive the question text input by the user, perform intent recognition on the question text, and generate a structured search plan. The structured search plan includes channel selection information and search content information. The multi-channel recall module is used to execute recall operations of multiple search channels in parallel based on a structured search plan, and obtain the recall result set corresponding to each search channel. The fusion sorting module is used to perform multi-stage fusion sorting processing on the recall result sets of multiple retrieval channels to obtain the fusion sorted result sequence; The quality fallback module is used to perform multi-layer quality fallback processing on the merged and sorted result sequence, generate the final answer and output it. The feedback loop module is used to collect user feedback on the final answer, perform root cause analysis based on the feedback information, and automatically adjust system parameters according to the root cause analysis results.

[0121] Thirdly, this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described above.

[0122] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0123] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0124] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0125] In the embodiments provided in this disclosure, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. Multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0126] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0127] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0128] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in a computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0129] The above are merely preferred embodiments of the present invention. It should be noted that any modifications and improvements made by those skilled in the art without departing from the present technical solution should also be considered to fall within the scope of protection claimed by the present solution.

Claims

1. A multi-engine fusion retrieval and question-answering method for an audit compliance AI knowledge base, characterized in that: Includes the following steps: Get the question text entered by the user; The intent recognition results are obtained by performing intent recognition on the question text. A structured retrieval plan is generated based on the intent recognition results. The structured retrieval plan includes channel selection information and retrieval content information. The channel selection information is used to indicate which retrieval channel needs to be activated from multiple retrieval channels. The retrieval content information includes the core retrieval text after referential resolution and entity standardization, domain filter tags for limiting the retrieval scope, and exact matching objects for exact matching. Based on the structured retrieval plan, recall operations for multiple retrieval channels are executed in parallel to obtain the recall result set corresponding to each retrieval channel; The recall result set from multiple retrieval channels is subjected to multi-stage fusion and sorting processing to obtain a fusion and sorted result sequence; Perform multi-level quality fallback processing on the merged and sorted result sequence to generate the final answer and output it.

2. The method according to claim 1, characterized in that, The process of performing intent recognition on the question text to obtain intent recognition results, and generating a structured search plan based on the intent recognition results, includes: The intent recognition of the question text adopts a dual-engine mechanism, which includes an AI parsing engine and a rule engine; The AI ​​parsing engine uses a large language model to perform semantic analysis on the question text and outputs intent type, entity information and knowledge domain labels. The rule engine performs pattern matching on the question text based on a predefined set of rules and outputs the matching results. Consistency judgment is made based on the output results of the AI ​​parsing engine and the rule engine. When the judgment results are consistent, a structured retrieval plan is generated using the consistent results. When the judgment results are inconsistent, two sets of retrieval plans are generated in parallel and intent confidence weights are introduced in the subsequent fusion stage.

3. The method according to claim 2, characterized in that, The multiple retrieval channels include a semantic retrieval channel, a keyword matching channel, a structured query channel, and a FAQ matching channel; The semantic retrieval channel generates text vectors based on a pre-trained embedding model and performs similarity retrieval in a vector database; The keyword matching channel performs full-text keyword matching based on the BM25 algorithm; The structured query channel supports both exact and fuzzy matching of document numbers and clause levels; The FAQ matching channel performs fast matching based on an independent FAQ vector library.

4. The method according to claim 3, characterized in that, The multi-stage fusion sorting process includes the following five stages: In the first phase, dynamic weights are assigned to the recall results of each retrieval channel based on the intent type in the structured retrieval plan. In the second stage, the recall results of each retrieval channel are standardized to eliminate the differences in scoring scales between different retrieval channels. In the third stage, the number of differential retentions is determined based on the characteristics of each retrieval channel, and the standardized results of each channel are truncated. In the fourth stage, multi-source consensus scoring is applied to results from different retrieval channels, and consensus scores are increased for results that are hit by multiple retrieval channels simultaneously. In the fifth stage, coarse-grained filtering and noise reduction are performed on the fused results to remove low-quality results.

5. The method according to claim 4, characterized in that, In the second stage, the score standardization process adopts a non-linear mapping method based on the Sigmoid function. The temperature parameter of the Sigmoid function is dynamically calculated and determined according to the standard deviation of the score of the most recent preset number of retrieval results for each retrieval channel.

6. The method according to claim 5, characterized in that, In the fourth stage, the multi-source consensus scoring process introduces a channel independence matrix. The channel independence matrix records the degree of independence between each retrieval channel. For results that are hit by multiple retrieval channels at the same time, the consensus score is the base score multiplied by the sum of the independence coefficients between the corresponding channels.

7. The method according to claim 6, characterized in that, The multi-layered quality fallback processing includes: the first layer of result diversity optimization processing, which uses the maximum marginal relevance algorithm to perform redundancy removal and reordering on the fused and sorted result sequence, and iteratively selects results from the result sequence that are highly relevant to the problem and have low similarity to the already selected results.

8. The method according to claim 7, characterized in that, The multi-layered quality fallback processing also includes: a second-layer multi-level confidence response processing, which compares the final confidence score of the first result in the merged and sorted result sequence with preset high confidence thresholds and low confidence thresholds: when the final confidence score is higher than the high confidence threshold, a complete answer is directly generated based on the search results; when the final confidence score is lower than the low confidence threshold, the user is prompted that no relevant information supporting the answer was found in the knowledge base and is advised to adjust the question wording; when the final confidence score is between the low confidence threshold and the high confidence threshold, a confidence prompt is attached to the generated answer, informing the user that the answer is uncertain; the multi-layered quality fallback processing further includes collecting user feedback on the final answer, performing root cause analysis based on the feedback, and automatically adjusting the parameters in the method according to the root cause analysis results, wherein the root cause analysis is implemented based on a classification model, the classification model takes a feature vector of preset dimensions as input, the feature vector includes question type, the score of the first result in the ranking, the result diversity ratio, and the number of retrieval channels recalled, and outputs the corresponding root cause category.

9. An audit compliance AI knowledge base multi-engine fusion retrieval and question-answering system, characterized in that, include: The intent recognition module is used to receive the question text input by the user, perform intent recognition on the question text, and generate a structured search plan. The structured search plan includes channel selection information and search content information. The multi-channel recall module is used to execute recall operations of multiple search channels in parallel based on a structured search plan, and obtain the recall result set corresponding to each search channel. The fusion sorting module is used to perform multi-stage fusion sorting processing on the recall result sets of multiple retrieval channels to obtain the fusion sorted result sequence; The quality fallback module is used to perform multi-layer quality fallback processing on the merged and sorted result sequence, generate the final answer and output it. The feedback loop module is used to collect user feedback on the final answer, perform root cause analysis based on the feedback information, and automatically adjust system parameters according to the root cause analysis results.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-8.

Images