Knowledge base question-answering system optimization method and device based on hybrid fine tuning and multi-dimensional evaluation and readable storage medium thereof

Abstract

The invention provides a knowledge base question-answering system optimization method and device based on hybrid fine tuning and multi-dimensional evaluation and a readable storage medium thereof, and provides the following scheme: constructing a hybrid progressive fine tuning framework, fusing low rank adaptation (LoRA) and direct preference optimization (DPO), and realizing domain knowledge migration through hierarchical dynamic parameter configuration; establishing a logic-semantic-knowledge three-dimensional quantitative evaluation system, and forming closed-loop optimization by using dynamic weight fusion and a visual decision system; and designing a multi-domain prompt template library with layered parameter freezing, sparse constraint and attention driving, and realizing model lightweight and cross-domain logic constraint. According to the method, the adaptive bottleneck of a general model and domain characteristics is broken through, the small sample training efficiency and the generated content compliance are improved, the computing resource consumption is reduced, an efficient and reliable knowledge service base is provided for professional scenes such as laws, medical treatment and finance, and the technical advantages of specialization, light weight and interpretability are achieved.

Description

Technical Field

[0001] The present invention relates to the cross - field of artificial intelligence and natural language processing, in particular to an optimization method, device and readable storage medium for a knowledge - base question - answering system based on hybrid fine - tuning and multi - dimensional evaluation. Background Art

[0002] In recent years, knowledge - base question - answering systems driven by large language models have faced significant challenges in applications in vertical fields such as law, medicine, and finance: (1) Insufficient efficiency of domain knowledge transfer Although traditional full - parameter fine - tuning strategies perform stably in the open domain, it is difficult to balance general language capabilities and professional domain characteristics. For example, in legal scenarios, it is necessary to accurately identify the mutual exclusion relationship of articles; in medical scenarios, it is necessary to follow a strict logical chain of "symptom - examination - diagnosis". However, existing models adopt a global parameter update mechanism, lacking hierarchical dynamic adaptation between the middle and top layers of the Transformer (such as high - rank parameters in the middle layer to capture fine - grained semantics and low - rank parameters in the top layer to enhance logical stability), resulting in low efficiency of injecting domain semantic constraints and inference rules, often causing conflicts in clause citation or medical common - sense errors. The static rank allocation strategy further restricts the generalization ability in small - sample scenarios.

[0003] (2) Simplification of the evaluation system dimension Existing evaluations overly rely on surface indicators such as accuracy and F1 - score, ignoring core requirements such as logical self - consistency and knowledge coverage. Taking financial risk control as an example, the system needs to balance semantic relevance, regulatory compliance, and risk - warning completeness. However, the traditional BM25 algorithm only achieves literal text matching and cannot detect rule conflicts (such as contradictory transaction verification logic) or quantify entity coverage. More critically, multi - dimensional indicators lack a dynamic fusion mechanism - in legal scenarios, the weight of logical consistency needs to be strengthened, and in medical scenarios, knowledge coverage should be prioritized. The static weight strategy leads to the optimization direction deviating from the real business goals and reduces the system reliability.

[0004] (3) Imbalance between training resources and deployment requirements Full - parameter fine - tuning requires updating billions of parameters, resulting in a sharp increase in video - memory occupancy and climbing computational costs, restricting deployment on edge devices. Although low - rank adaptation (LoRA) reduces the number of parameters through matrix decomposition, existing solutions do not design dynamic configurations according to the functional differences at the model level (such as high - rank parsing of semantics in the middle layer and low - rank suppression of overfitting in the top layer). The unified rank strategy is prone to long - range reasoning semantic drift. In addition, compliance control relies on traditional reinforcement learning, suffering from problems such as sparse rewards and training fluctuations, lacking a dual - track optimization goal based on contrastive learning (such as distinguishing compliant / non - compliant answers), resulting in generated content being difficult to align with the latest industry standards.

[0005] In summary, the core contradictions of the vertical domain Q&A system are concentrated in: the adaptation conflict between the general model architecture and domain semantic rules, the matching gap between single-dimensional evaluation metrics and complex business requirements, and the balance dilemma between algorithm performance improvement and computing resource constraints. Summary of the Invention

[0006] The embodiments of the present invention provide an optimization method, device and readable storage medium for a knowledge base Q&A system based on hybrid fine-tuning and multi-dimensional evaluation. Aiming at the defects of low efficiency of domain knowledge transfer, single-dimensional evaluation system, high consumption of training and deployment resources and insufficient adaptation ability in the current technology, it is difficult to meet the professional, compliant and lightweight requirements of the vertical domain for the Q&A system and other problems.

[0007] The core technology of the present invention mainly realizes efficient transfer of domain knowledge, compliance and controllability of generated content, and optimization of resource consumption through a hybrid progressive framework that combines hierarchical dynamic LoRA fine-tuning and DPO preference optimization, combined with a three-dimensional evaluation system of "logic-semantics-knowledge" and a lightweight deployment mechanism.

[0008] In the first aspect, the present invention provides an optimization method for a knowledge base Q&A system based on hybrid fine-tuning and multi-dimensional evaluation, and the method includes the following steps: Hybrid progressive fine-tuning framework: A two-stage fine-tuning mechanism that combines low-rank adaptation and direct preference optimization, realizes domain knowledge transfer through hierarchical dynamic parameter configuration, and completes strategy alignment based on human preference data; Multi-dimensional quantitative evaluation system: Construct a three-dimensional evaluation framework including logical consistency, semantic relevance, and knowledge coverage, and generate a comprehensive evaluation result through dynamic weight fusion; Dynamic adaptation and lightweight mechanism: Adopt hierarchical parameter freezing, sparse constraints, and an attention-driven multi-domain prompt template library to achieve model lightweight and cross-domain logical constraints.

[0009] Further, the hybrid progressive fine-tuning framework includes: Perform low-rank decomposition on the pre-trained model parameter matrix through the adaptation matrix and to achieve parameter update: where , d and k are rank values; is the domain semantic encoder, which is a low-rank matrix with a dimension of d × r; is the logical reasoning decoder with a dimension of r × k, and together with forms a low-rank decomposition pair; Implement hierarchical rank configuration in the Transformer architecture: Use a higher rank value in the middle layer to capture domain semantic features, and a lower rank value in the top layer to strengthen logical reasoning ability; Introducing a contrastive learning mechanism, distinguishing high-quality answers from illegal answers based on human preference data, and constructing an optimization target L DPO Drive models aligned to industry norms.

[0010] Furthermore, the hierarchical rank configuration is specifically as follows: The middle layer uses a rank value of r=8, the top layer uses a rank value of r=4, and the total parameter update is the sum of the adaptation matrices of each layer.

[0011] Furthermore, the multi-dimensional quantitative evaluation system includes: Logical consistency assessment: Verify the compliance of answers through first-order logic expressions and production rules, and use the Z3 solver and Drools engine to perform automatic verification; Semantic relevance evaluation: Combining the BM25 algorithm with the Sentence-BERT model, the keyword coverage and vector similarity are integrated through the dynamic weight λ; Knowledge coverage evaluation: Use the Bi-LSTM-CRF model to identify domain entities and calculate coverage indicators through the intersection of entity sets:

[0012] in, The collection of entities generated for the model, is the set of annotated real entities, molecules Indicates the number of domain entities that are correctly identified.

[0013] Furthermore, logical consistency assessment also includes adversarial perturbation detection: The DeBERTa-v3 model is used to perturb the generated answers, including entity replacement and logic reversal, and the conflict rate formula is used to To verify the logical consistency; Among them, ConsistScore is the logical consistency score, and its value range is [0,1]. The higher the score, the stronger the logical self-consistency; To judge the Samples Whether it is consistent with the reference standard conflict; (⋅) is the indicator function, which is 1 when there is a conflict and 0 otherwise; n is the total number of perturbation samples generated; s i is the i-th disturbance sample; R is the original answer.

[0014] Furthermore, the dynamic adaptation and lightweight mechanism includes: Freeze the basic parameters of the pre-trained model , only update the adaptation matrix parameters, combined with sparse constraints Suppress parameter redundancy; Build a multi-domain prompt template library, dynamically match templates through the attention mechanism, and generate answers that conform to professional logic. The templates include legal clause templates, medical diagnosis path templates, and financial risk control rule templates.

[0015] Furthermore, the attention-driven template generation formula is:

[0016] where is the hidden state at the current time step; are the query generation function, key generation function, and value generation function respectively; is the dimension; P is the prompt template library.

[0017] Furthermore, the comprehensive score formula for the three-dimensional evaluation framework is:

[0018] where, is the logical consistency evaluation function; is the semantic relevance evaluation function; is the knowledge coverage evaluation function; the dynamic weights satisfy ; Q is the input question, R is the system answer, K is the knowledge base entity set, and it is automatically adjusted according to the domain characteristics.

[0019] In the second aspect, the present invention provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the above-mentioned knowledge base question and answer system optimization method based on hybrid fine-tuning and multi-dimensional evaluation.

[0020] In the third aspect, the present invention provides a readable storage medium. A computer program is stored in the readable storage medium, and the computer program includes program codes for controlling a process to execute the process, and the process includes the above-mentioned knowledge base question and answer system optimization method based on hybrid fine-tuning and multi-dimensional evaluation.

[0021] The main contributions and innovations of the present invention are as follows: 1. The efficiency of domain knowledge transfer is significantly improved The hybrid fine-tuning framework realizes the differential adaptation of fine-grained domain semantics and logical reasoning while retaining general capabilities through LoRA hierarchical dynamic rank configuration (r = 8 in the middle layer, r = 4 in the top layer). Compared with full-parameter fine-tuning, the number of trainable parameters is greatly reduced, the video memory occupancy is reduced, and the training efficiency is significantly improved in the small sample scenario. The DPO stage introduces human preference data and contrast learning to drive the model to strictly align with industry specifications (such as legal compliance and medical diagnosis logic), and the compliance and security of the generated content are significantly enhanced.

[0022] 2. The evaluation system better meets professional needs The three-dimensional quantitative evaluation framework integrates a logic rule engine (Z3 / Drools), semantic vector encoding (Sentence-BERT), and knowledge graph alignment (Bi-LSTM-CRF) to achieve hard constraint verification, deep semantic matching, and entity coverage quantification, addressing the limitations of traditional single-dimensional metrics. The dynamic weight mechanism ( ), together with the three-dimensional radar chart visualization system, supports adaptive adjustment of the evaluation focus according to domain characteristics, quickly locates logical vulnerabilities, semantic deviations, or knowledge gaps, forms a "evaluation-intervention-iteration" closed loop, and shortens the system optimization cycle.

[0023] 3. Breakthrough in lightweight and cross-domain adaptation capabilities The hierarchical parameter freezing and sparse constraint strategy ( =0.01) locks the parameters of the base model, injects domain knowledge only through a lightweight adaptation matrix, compresses the model size, and supports deployment on edge devices. The multi-domain prompt template library and attention-driven mechanism (such as legal clause citation templates, medical diagnosis path templates) significantly enhance the cross-domain adaptation ability, ensure that the generated content in scenarios such as finance and healthcare strictly follows industry norms, and improve the generalization performance.

[0024] 4. Systematically solve the core contradictions in vertical domains The three major innovation points cooperate organically, breaking through the bottlenecks of "difficulty in adapting the general model architecture to domain characteristics, mismatch between single-dimensional evaluation and complex requirements, and imbalance between resource constraints and performance improvement", providing an efficient, reliable, and interpretable knowledge service foundation for professional scenarios.

[0025] Details of one or more embodiments of the present invention are set forth in the following drawings and description to make other features, objects, and advantages of the present invention more concise and understandable. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are used to provide a further understanding of the present invention, form a part of the present invention, and the schematic embodiments and descriptions thereof are used to explain the present invention without unduly limiting the present invention. In the drawings: Figure 1 is a flowchart of an optimization method for a knowledge base question-answering system based on hybrid fine-tuning and multi-dimensional evaluation according to an embodiment of the present invention; Figure 2 is a schematic hardware structure diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0027] Here, exemplary embodiments will be described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. On the contrary, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0028] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may also be combined into a single step for description in other embodiments.

[0029] Existing vertical domain question-answering systems have low efficiency in full-parameter fine-tuning, a single evaluation metric, and high resource consumption, resulting in insufficient domain knowledge transfer, poor compliance of generated content, and difficulty in lightweight deployment.

[0030] Based on this, the present invention is based on a hybrid progressive framework that combines hierarchical dynamic LoRA fine-tuning and DPO preference optimization, and combines a "logic-semantics-knowledge" three-dimensional evaluation system and a lightweight deployment mechanism to solve the problems existing in the prior art.

[0031] Embodiment 1 The present invention aims to propose an optimization method for a knowledge base question-answering system based on hybrid fine-tuning and multi-dimensional evaluation. Specifically, referring to Figure 1 , the method includes the following steps: Step 1: In response to the problem of vertical domain knowledge transfer, this step innovatively combines low-rank adaptation (LoRA) and preference optimization (DPO) technologies to construct a two-stage progressive fine-tuning framework. The system first achieves efficient parameter update through low-rank decomposition, and then introduces human preference data to complete strategy alignment, and finally forms a domain knowledge enhanced question-answering model. The specific process is as follows: Step 1.1 Perform basic semantic adaptation through the LoRA fine-tuning mechanism. For the parameter matrix of the pre-trained language model, design a low-rank decomposition structure: introduce adaptation matrices and (where ), form a parameter update expression, and its formula is as follows:

[0032] where , d and k are rank values; is a domain semantic encoder, which is a low-rank matrix with dimensions d × r; is a logical reasoning decoder with dimensions r × k, and together with forms a low-rank decomposition pair.

[0033] In this way, the low-rank decomposition structure systematically solves the problem of balancing the efficiency and generality of vertical domain knowledge transfer through mathematical rank constraints, functional hierarchical decoupling, and dynamic adaptation in training, and has significant technological progressiveness and patent protection value.

[0034] Step 1.2 Implementation of the hierarchical adaptation strategy. Implement differential rank configuration in the Transformer architecture (further breaking through the static rank allocation defect of existing LoRA and proposing hierarchical dynamic low-rank configuration): For the middle layers (layers 6 - 9), a high rank r = 8 is adopted: The middle layers are responsible for semantic feature extraction, and the high-rank configuration allows the model to capture richer domain semantic details (such as differences in modal words like "shall / shall not" in legal provisions, and context ambiguity resolution of medical terms); For the top layers (layers 10 - 12), a low rank r = 4 is adopted: The top layers dominate logical reasoning, and the low-rank configuration suppresses overfitting through parameter regularization to ensure the logical stability of the generated answers (such as the sequential constraint of "examination first, diagnosis later" in medical diagnosis, and the rule chain of "transaction amount exceeding limit requires verification" in financial risk control).

[0035] The formula for the total parameter update amount is as follows:

[0036] where represents the total parameter update amount, that is, the sum of the update amounts of all adaptation matrices from layer 6 to layer 12, reflecting the overall parameter adjustment amplitude of the model under the hierarchical adaptation strategy; the subscript of the summation symbol Σ is the summation index, with a value range from 6 to 12, indicating the layer-by-layer accumulation of the parameter update amounts from layer 6 to layer 12; is the adaptation matrix of the th layer, with dimensions (r is 8 for the middle layers 6 - 9 and 4 for the top layers 10 - 12), responsible for mapping the input features to a low-dimensional space and capturing domain semantic features; Another adaptation matrix of the th layer, with dimensions , after multiplying with , maps the low-dimensional features back to the original parameter space to achieve the injection of domain logic rules.

[0037] The total parameter update amount is the layer-by-layer accumulation of the adaptation matrices of the middle layers (6-9 layers) and the top layers (10-12 layers). Only some layers of the Transformer are differentially adapted, which further reduces the computational cost while avoiding the destruction of general capabilities caused by global updates.

[0038] Step 1.3 Dynamic parameter update. The formula for dynamic parameter update is as follows:

[0039] in The model parameters updated by the low-rank adaptation (LoRA) method are the final parameters that combine basic general capabilities and domain adaptation adjustments, and are used for reasoning or further training of the model on specific tasks; The parameters of the frozen pre-trained model (to ensure that general capabilities are not lost) carry the general language capabilities (such as grammatical understanding, common sense reasoning, etc.) learned by the model during the pre-training phase. They remain unchanged during the current training process to ensure that general capabilities are not lost. The dynamic learning rate is used to control the amplitude of parameter updates. It can be adjusted dynamically according to the actual situation during training to balance the speed and stability of training; is the cross entropy loss function about The gradient of the loss function reflects the sensitivity to changes in Update in the direction of reducing losses; is the cross entropy loss function, which is used to measure the model under input x and true label y, and the parameters are The smaller the difference, the more accurate the model prediction.

[0040] Implementing basic parameters through gradient decomposition This mechanism increases the training convergence speed, reduces the memory usage, and achieves stable basic parameters and efficient training of adaptive parameters.

[0041] Step 1.4 Align industry standards through the DPO fine-tuning mechanism. After completing the basic semantic adaptation, this solution introduces the deep policy optimization (DPO) mechanism to strengthen the compliance of generated content through human preference data. Based on the contrastive learning theory, the optimization goal is constructed to drive the model to distinguish high-quality answers. Answer with violation , the formula is as follows:

[0042] in Represents the direct preference optimization loss function, which measures the current model and the reference model in terms of the difference in the preference for generating answers. is the model to be optimized currently, with parameters θ, representing the probability that the model generates answer y when the input is x. is the reference model with frozen parameters, serving as a comparison benchmark to prevent catastrophic forgetting caused by over-optimization, and is the probability that it generates answer y when the input is x; is the expectation taken over samples from the data distribution D. Among them, is the input (such as the question text), is the high-quality answer ("winner"), is the low-quality answer ("loser"). is the temperature coefficient (hyperparameter), which controls the exploration intensity of the strategy (the larger the value, the more sensitive to the preference difference), is the Sigmoid function, which maps the probability difference to the interval [0,1].

[0043] Step 1.5 Achieve precise preference alignment. Decompose the policy gradient into two components: positive enhancement and negative suppression, and the formula is as follows:

[0044] Among them, represents the gradient of the direct preference optimization (DPO) loss function with respect to the model parameters θ, which is used to guide the update direction of the parameters θ to optimize the model's generation behavior. is the positive gradient (enhancing compliant answers), is the negative gradient (suppressing non-compliant answers), which has characteristics such as clear direction and training stability. is a hyperparameter used to control the scaling intensity of the gradient and adjust the adjustment strength of the model for preference alignment.

[0045] This formula decomposes the policy gradient into two components with clear directions: positive enhancement (encouraging compliant answers) and negative suppression (striking non-compliant answers), enabling the model to more stably align with human preferences during training: strengthening the generation behavior that meets expectations while suppressing bad outputs, thereby improving the quality and compliance of the model's outputs.

[0046] Step 1.6 Establish prompt engineering. Based on the above-mentioned mixed fine-tuning, establish a multi-domain prompt template library, including: legal clause templates (such as "According to Article X of the XX Law..."), medical diagnosis templates (such as "For [X symptom], it is recommended to check [Y item]"), financial risk control templates (such as "When the transaction amount exceeds [X], [Y] needs to be verified"), etc.

[0047] Step 1.7 Calculate the attention weights. Select the attention-driven template and dynamically generate an adapted prompt based on the current hidden state. The formula is as follows:

[0048] where is the hidden state at the current time step; are the query generation function, key generation function, and value generation function respectively; is the dimension; P is the prompt template library.

[0049] Step 1.8 Integrate the semantic adaptation and policy alignment objectives. Add coefficient constraints. The formula is as follows:

[0050] where is the total loss function; is the weight coefficient, which adjusts the contribution of the cross-entropy loss to the total loss and controls the attention of the model to the accuracy of the basic task. is the cross-entropy loss, which measures the difference between the model's prediction result and the true label and drives the model to learn the basic input-output mapping relationship (such as text classification, question-answer matching). is the weight coefficient, which determines the importance of preference alignment in training and balances the "task accuracy" and "human preference compliance" of the model. is the direct preference optimization loss, which prompts the model to generate outputs that are more in line with human preferences (such as distinguishing between high-quality answers and low-quality answers) and aligns human values and logical rules. is the regularization coefficient, which restricts the magnitude of model parameter updates by penalizing too large , prevents overfitting, improves generalization ability, and ensures the stability of the model on new data. is the square of the Frobenius norm of the matrix , which represents the magnitude of the parameter update amount.

[0051] Its dynamic weight policy formula is as follows:

[0052] where (1) represents the first 5 rounds of training, and (2) represents after 5 rounds; Constraint condition: (Suppress parameter redundancy).

[0053] Step 2: On the basis of the hybrid fine-tuning framework, construct a multi-modal evaluation framework composed of logical consistency, semantic relevance, and knowledge coverage to form an end-to-end evaluation pipeline. The specific process is as follows: Step 2.1 Design the hierarchical structure. Input processing layer: perform dependency syntactic analysis and semantic role labeling on the question-answer pair (Q, R); Feature extraction layer: adopt a multi-channel feature parallel extraction mechanism, including logical rule matching, semantic vector encoding, and knowledge entity linking; Comprehensive calculation layer: generate the final evaluation score through dynamic weighted fusion, and its formula is as follows:

[0054] where, is the logical consistency evaluation function; is the semantic relevance evaluation function; is the knowledge coverage evaluation function; The dynamic weight satisfies ; Q is the input question, R is the system answer, K is the knowledge base entity set, and it is automatically adjusted according to the domain characteristics.

[0055] Step 2.2 The logical consistency evaluation module uses a dual-track constraint engine and an adversarial detection mechanism to systematically verify whether the generated content conforms to professional domain rules such as legal clause exclusivity and medical diagnosis paths, ensuring the compliance and logical self-consistency of the answer. The specific process is as follows: Step 2.2.1 Define two types of constraint rules. Hard constraints (legal clause exclusivity): Use first-order logic expressions to describe clause exclusivity and achieve automatic verification through the Z3 solver. Soft constraints (medical diagnosis probability chain): Use production rules to describe the diagnosis probability chain and execute it through the Drools rule engine.

[0056] where, represents universal quantification over all inputs x (such as the request content in legal Q&A). is the legal compliance determination function, which returns true if x conforms to a certain legal clause (such as "The transaction amount ≤ 5000 yuan conforms to the Measures for Payment and Settlement"); is the logical implication symbol, that is, "if... then..."; is the risk assessment function, while is the negation of the risk assessment function, which returns true if x has no risk (such as "This behavior does not trigger financial risks"); is the clinical manifestation of symptom A; is the clinical diagnosis conclusion B (such as "Diagnosed with angina"); is the production rule with probability, indicating that "when symptom A appears, there is an 80% probability of diagnosing disease B"; For example, when the input content x conforms to a certain legal clause ( is true), the corresponding behavior or conclusion must have no risk ( is false), reflecting the exclusivity of legal clauses (such as the same behavior cannot be legal and illegal at the same time).

[0057] The hard constraints use the Microsoft Z3 automated theorem prover to transform first-order logic expressions into satisfiability problems (SMT), and automatically verify whether there are logical conflicts in the generated answers. For example: Input: "According to Article 143 of the Civil Code, this contract is valid" ( is true); Verification: If the answer simultaneously contains "There is a risk of culpa in contrahendo in this contract" ( is true), then the Z3 solver determines a conflict and triggers a logical error prompt.

[0058] The soft constraints establish a probabilistic dependence relationship between symptoms and diagnoses, allowing for uncertainty in medical reasoning (e.g., the same symptom may correspond to multiple diseases, but with different probabilities), which is in line with the actual scenario of medical diagnosis.

[0059] The soft constraints use Drools to execute production rules, supporting dynamic matching of the probability chain between symptoms and diagnoses. For example: Input: "The patient complains of chest pain and palpitations" (trigger ); Rule matching: According to the probability chain of "chest pain → angina pectoris (80%), chest pain → myocardial infarction (15%)" in the knowledge base, generate a list of candidate diagnoses and probability values; Output: "It is recommended to give priority to ruling out angina pectoris (probability 80%), and at the same time, myocardial infarction needs to be excluded (probability 15%)."

[0060] Through the precise definition and technical implementation of these two types of constraint rules, this step constructs the underlying support for logical consistency evaluation, which is a core component of the "logic-semantics-knowledge" three-dimensional evaluation system, reflecting the innovation and practicality of the present invention in rule modeling in the professional field.

[0061] Step 2.2.2 improves its accuracy. Based on the answers generated by the hybrid fine-tuning model, contradiction detection verifies logical self-consistency through adversarial perturbation detection. The architecture selection uses DeBERTa-v3 (an advanced pre-trained language model, improved based on the bidirectional encoding representation (BERT) architecture), and performs directional perturbations in ways such as entity replacement, logical inversion, and context shift. The formula is as follows:

[0062] Among them, ConsistScore is the logical consistency score, with a value range of [0,1]. The higher the score, the stronger the logical self-consistency; To determine whether the th sample conflicts with the reference standard ; (⋅) is the Indicator Function, which is 1 in case of conflict and 0 otherwise; n is the total number of generated perturbed samples (e.g., 100 perturbed samples are generated through DeBERTa-v3); s i is the i-th perturbed sample (e.g., the question or answer after entity replacement); R is the original answer (i.e., the model output without perturbation).

[0063] Through the directional perturbation driven by the pre-trained model and the quantitative evaluation, this formula fills the gaps in the traditional logical verification in terms of deep semantics and automation. It is the key technical support for the "logic-semantics-knowledge" three-dimensional evaluation system, significantly improving the reliability and interpretability of the question-answering system in the vertical domain.

[0064] Step 2.3 Semantic Relevance Evaluation Module. As the core part of the evaluation system, this module adopts a hierarchical progressive design strategy, organically combining traditional retrieval algorithms with deep learning models. The overall operation process of the module can be divided into three levels: the basic retrieval layer (BM25 algorithm), the semantic parsing layer (Sentence-BERT encoding), and the decision fusion layer (dynamic weight adjustment). The three form a closed-loop feedback in the data processing flow. The specific process of the evaluation module is as follows: Step 2.3.1 Surface Semantic Matching. Use the enhanced BM25 algorithm to calculate the keyword coverage. Based on the traditional formula, a domain adaptation mechanism is introduced. The formula is as follows:

[0065] where w is the keyword in the query statement Q or an independent word in the query; is the inverse document frequency, measuring the importance of keyword w; is the term frequency of keyword w in document R; is the term frequency saturation adjustment factor (usually taking 1.2 - 2.0) to prevent the over-high weight of high-frequency words; is the document length normalization factor (usually taking 0.75 - 1.0) to suppress the natural advantage of long documents over term frequency; is the document length; is the average document length.

[0066] The domain-adaptive BM25 algorithm solves the problems of "keyword mis-matching, long document bias, and insufficient term weight" of traditional retrieval algorithms in the vertical domain through the improvements of parameter dynamicization, weight domainization, and corpus specialization. It is one of the core technologies of the "Semantic Relevance Evaluation Module" of the present invention, significantly improving the retrieval accuracy and answer accuracy of the knowledge base question-answering system in the professional field. This mechanism can be written into the patent claims as an independent technical point to protect the "adaptive optimization of retrieval algorithms based on domain characteristics" solution.

[0067] Step 2.3.2 Deep semantic matching. A domain-specific pre-trained model is used to improve the representation accuracy, and the vector similarity is generated by Sentence-BERT. The formula is as follows:

[0068] where is the vectorized representation of the query statement Q (such as the semantic vector of "how to apply for medical compensation"); is the vectorized representation of the retrieved document D (such as the semantic vector of "medical damage compensation process" in the knowledge base); is the vector dot product, which measures and the degree of projection overlap in the semantic space. The larger the value, the more similar the semantics; is its vector norm; is the cosine similarity.

[0069] This step constructs a technical link for deep semantic matching through the combination of a domain-specific pre-trained model + multi-modal vector representation + cosine similarity measurement, solves the core defect of traditional retrieval algorithms of "emphasizing word frequency and neglecting semantics", and is the key innovation point of the "semantic relevance evaluation" module of the present invention.

[0070] Step 2.3.3 On this basis, an intelligent mapping between the weight coefficient and the business scenario is established to adaptively adjust the strategy. The λ value is dynamically corrected according to the real-time accuracy feedback, and manual review is triggered when the deviation between the semantic score and the logical score > 15%. The formula is as follows:

[0071] where SemScore is the comprehensive score of semantic relevance, and its value range is [0,1]. The higher the score, the stronger the semantic relevance between Q and R. As one of the core indicators of the three-dimensional evaluation system (logic-semantics-knowledge), it provides a quantitative basis for the semantic dimension of the answer quality of the question-answering system. is the dynamic weight coefficient, which is used to adjust and the contribution ratio to meet the different requirements of different fields for "keyword matching" and "semantic understanding". is the traditional retrieval score based on keyword matching, which measures the literal matching degree between the query Q and the answer R (such as keyword coverage and word frequency weight). is the cosine similarity based on Sentence-BERT, which measures the deep semantic association between Q and R (such as the direction consistency of sentence-level semantic vectors).

[0072] This formula resolves the core contradiction of traditional semantic evaluation, which is "either overfitting to keywords or deviating from the text basis", by dynamically weighting and fusing surface and deep semantic indicators, and is one of the key innovations of the "Three-Dimensional Quantitative Evaluation System" of the present invention.

[0073] Step 2.4. Knowledge coverage evaluation module. Its main function is to ensure the knowledge integrity of the generated content through knowledge graph alignment technology. The specific process includes two key stages: entity linking and coverage calculation, forming a three-layer technology linkage with upstream and downstream modules. The specific steps are as follows: Step 2.4.1 Perform entity linking. Use the Bi-LSTM-CRF model for domain entity recognition. The formula is as follows:

[0074] where x is the input text sequence, and x i is the i-th token (such as the word "chest pain"); y is the predicted label sequence, and y i is the entity label of the i-th token (such as "B-Disease" indicating the start of a disease entity); is the bidirectional hidden state of the i-th token generated by Bi-LSTM, capturing the context semantics (such as the clinical significance of "chest pain" in a medical record); W yi is the weight vector corresponding to the label mapping the hidden state to the label space; is the score transferred from label to reflecting the dependency relationship between labels (such as only "I-Person" or the end label can follow "B-Person"); is all possible label sequences, and the denominator ensures that the sum of probabilities is 1 through normalization. This formula completely reflects the core logic of the Bi-LSTM-CRF model for entity recognition through bidirectional semantics and label transfer constraints.

[0075] This formula is the core of the probability calculation of the Bi-LSTM-CRF model, reflecting the three-layer processing logic of the entity linking task: Bidirectional semantic modeling: The hidden state is generated by Bi-LSTM, capturing the context information of words (such as the specific meaning of "myocardial infarction" in a medical record). Label transfer constraint: The parameter comes from the CRF layer, reflecting domain knowledge rules (such as a person's name entity must follow "defendant" in a legal text). Normalization processing: The denominator term normalizes the probabilities of all possible label sequences to ensure output stability.

[0076] In this embodiment, the entity linking task is to given the input text x = {x1, x2,..., x n} (such as legal provisions or medical records), and predict its corresponding entity tag sequence y = {y1, y2,..., y n} (such as "defendant", "myocardial infarction", etc.).

[0077] Step 2.4.2 Calculate the coverage. Define the coverage metric , which measures the degree of agreement between the system output and the standard knowledge base. The formula is as follows:

[0078] where is the set of entities generated by the model, is the set of true entities marked, and the numerator represents the number of domain entities correctly identified.

[0079] Step 2.5 Visualize the evaluation results: As the comprehensive presentation terminal of the evaluation system, this module forms a closed-loop feedback system with the three major evaluation modules in Steps 2.1 to 2.4. Through the three-dimensional radar chart dynamic mapping technology, the abstract evaluation indicators are transformed into a visual decision support interface. The formula is as follows:

[0080] This formula defines the three coordinate axes of the three-dimensional radar chart, corresponding to the three core dimensions of the evaluation system respectively: 1) H (LogicConsist, logical consistency) Meaning: The logical consistency score, whose value range is usually [0, 1]. The logical self-consistency of the generated content is verified through hard constraints (Z3 solver) and soft constraints (Drools engine) (such as the mutual exclusivity of legal provisions and the correctness of medical diagnosis paths).

[0081] Value conversion: After multiplying by 120, the value range is extended to [0, 120], corresponding to the scale range of the "logic axis" in the radar chart (such as the radius of the radar chart is 120 units), making the logical indicator more prominent in visualization (possibly because logical errors have a more serious impact on the system).

[0082] 2) S (SemanticRel, semantic relevance) Meaning: The semantic relevance score, which is calculated through the dynamic fusion of BM25 and Sentence-BERT ( ), and measures the semantic matching degree between the query and the answer (such as keyword coverage and deep semantic association).

[0083] Value range: directly mapped to [0,1] or normalized to the radar chart scale (such as 0-100), no need to enlarge, keep the original ratio.

[0084] 3) V (Knowledge Cover) Meaning: Knowledge coverage score, calculated by entity linking (Bi-LSTM-CRF) and knowledge graph alignment ( ), quantifying the degree of consistency between the generated content and the knowledge base entities (such as whether the medical diagnosis misses key examination items).

[0085] Value range: directly mapped to [0,1] or normalized to the radar chart scale to reflect knowledge completeness.

[0086] In this embodiment, the technology of the visual decision support interface is implemented as follows: 1. Dynamic mapping of 3D radar chart Coordinate system construction: With the center of the radar chart as the origin, the H axis (logic), S axis (semantics), and V axis (knowledge) are distributed at an angle of 120° to form an equilateral triangle coordinate system. The value of each dimension corresponds to a point on the radar chart, and connecting the three points forms a polygon. The larger the area, the better the overall performance.

[0087] Dynamic update mechanism: Receive scores of the three major evaluation modules in real time (e.g. refresh every 10 seconds) and automatically re-render radar charts; support historical data comparison (e.g. score fluctuation curves over the past 1 hour and 24 hours) to facilitate tracking of system performance changes.

[0088] 2. Linkage logic of closed-loop feedback system Problem location: If the area of ​​a certain dimension of the radar chart is significantly smaller than that of other dimensions (such as a concave V-axis), it indicates that the corresponding module has defects (such as insufficient knowledge coverage), triggering automatic diagnosis: Low logical consistency: Jump to the rule engine log, check the Z3 / Drools verification results, and locate the conflicting rules; Low semantic relevance: Analyze the similarity between BM25 keyword matching and Sentence-BERT vector, adjust the dynamic weight λ or trigger domain model fine-tuning; Low knowledge coverage: Scan entity linking results, identify uncovered knowledge base entities, automatically supplement training data, or optimize the Bi-LSTM-CRF model.

[0089] Intervention-iteration link: visual warning → defect location → parameter adjustment / data enhancement → re-evaluation → system iteration.

[0090] For example, when the knowledge coverage in a medical scenario is low, the system automatically extracts missing entities (such as "coronary angiography") from the electronic medical record library, adds them to the training set, and triggers incremental model training.

[0091] Through the 3D radar chart visualization technology, the complex multi-dimensional evaluation results are transformed into an intuitive graphical expression, which not only improves the operation and maintenance efficiency, but also drives the continuous optimization of the system through a closed-loop feedback mechanism. This module forms a technical closed-loop with the hybrid fine-tuning framework and multi-dimensional evaluation algorithm, and jointly constructs a professional and interpretable knowledge base question-answering system, providing a complete technical solution for intelligent interaction in vertical fields.

[0092] Step 3.1 System initialization and domain adaptation, pre-trained model loading: Deploy a basic language model (such as GLM4-9B), load the hybrid fine-tuning parameter matrix and initialize the domain knowledge graph. At the same time, according to the automatically constructed prompt template library P in the target domain, complete the domain feature injection through the attention-driven mechanism.

[0093] In this embodiment, the specific steps are as follows: Step 3.1.1 Pre-trained model loading Basic model deployment: Select an open-source large language model (such as GLM4-9B) as the base to provide general language understanding and generation capabilities.

[0094] Hybrid fine-tuning parameter loading: Load the total parameter update matrix obtained through LoRA+DPO hybrid fine-tuning to achieve the integration of the basic model and domain knowledge.

[0095] Domain knowledge graph initialization: Construct a knowledge graph K to store domain entities and relationships (such as the mutual exclusion relationship between legal provisions, the mapping of medical symptoms-diseases), which is used for knowledge coverage evaluation (Step 2.4) and logical rule verification (Step 2.2).

[0096] Step 3.1.2 Automatic construction of the prompt template library According to the target domain (such as law, medicine), automatically generate a structured prompt template library P. For example: Legal template: "According to Article {clause number} of the '{law name}', the legal consequence of {scene description} is {answer}"; Medical template: "When the symptoms of [{symptom list}] appear, it is recommended to give priority to the [{examination item}] examination, and the possible diagnoses include [{disease list}]". Attention-driven injection: Dynamically select the template through the attention mechanism (formula: ) to fuse the template features with the current input hidden state h t to guide the model to generate answers that conform to the domain logic.

[0097] Step 3.2 Real-time question-answering processing: For the user's question Q, after the large model answers, obtain the answer A and perform multi-dimensional evaluation. Re-answer the questions with lower scores.

[0098] In this embodiment, the user inputs a question Q. After text tokenization and vectorization, an initial answer A is generated by a large model loaded with domain parameters. Technical dependency: Model parameters after hybrid fine-tuning (see step 1.3) and the prompt template P jointly constrain the generation logic. For example, legal questions force the citation of the clause structure in the template.

[0099] Step 3.3 finally obtains the correct answer. Output a reliable answer verified through multiple rounds of evaluation to form a question-answer closed loop.

[0100] Through the closed-loop design of "initialization - question answering - evaluation - optimization", this process systematically solves the problems of efficiency, accuracy, and scalability of vertical domain question answering, and is the core execution link for the implementation of the technical solution of the present invention.

[0101] Embodiment 2 This embodiment also provides an electronic device. Refer to Figure 2 , including a memory 404 and a processor 402. A computer program is stored in the memory 404, and the processor 402 is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0102] Specifically, the above processor 402 may include a central processing unit (CPU), or a specific integrated circuit (Application Specific Integrated Circuit, abbreviated as ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0103] Among them, the memory 404 may include a mass memory 404 for data or instructions. By way of example and not limitation, the memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. In appropriate cases, the memory 404 may include removable or non-removable (or fixed) media. In appropriate cases, the memory 404 may be internal or external to the data processing device. In a particular embodiment, the memory 404 is non-volatile memory. In a particular embodiment, the memory 404 includes a read-only memory (ROM) and a random access memory (RAM). In appropriate cases, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM), or a flash memory, or a combination of two or more of these. In appropriate cases, the RAM may be a static random access memory (SRAM) or a dynamic random access memory (DRAM), where the DRAM may be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0104] The memory 404 can be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 402.

[0105] The processor 402 reads and executes the computer program instructions stored in the memory 404 to implement any of the above-described knowledge base question-answering system optimization methods based on hybrid fine-tuning and multi-dimensional evaluation.

[0106] Optionally, the above electronic device may further include a transmission device 406 and an input / output device 408. Among them, the transmission device 406 is connected to the above processor 402, and the input / output device 408 is connected to the above processor 402.

[0107] The transmission device 406 can be used to receive or send data via a network. Specific examples of the above network may include wired or wireless networks provided by the communication provider of the electronic device. In one example, the transmission device includes a network adapter (abbreviated as NIC), which can be connected to other network devices through a base station and thus communicate with the Internet. In one example, the transmission device 406 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0108] The input / output device 408 is used to input or output information.

[0109] Embodiment III This embodiment also provides a readable storage medium, in which a computer program is stored. The computer program includes program code for controlling a process to execute the process, and the process includes the knowledge base question-answering system optimization method based on hybrid fine-tuning and multi-dimensional evaluation according to Embodiment I.

[0110] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation manners, and will not be elaborated here.

[0111] Generally, various embodiments can be implemented in hardware or special circuits, software, logic, or any combination thereof. Some aspects of the present invention can be implemented in hardware, while other aspects can be implemented by firmware or software executed by a controller, a microprocessor, or other computing devices, but the present invention is not limited thereto. Although various aspects of the present invention can be shown and described as block diagrams, flowcharts, or using some other graphical representations, it should be understood that, as a non-limiting example, the blocks, devices, systems, technologies, or methods described herein can be implemented in hardware, software, firmware, special circuits or logic, general hardware or controllers, or other computing devices, or some combination thereof.

[0112] Embodiments of the present invention can be implemented by computer software, which can be executed by a data processor of a mobile device, such as in a processor entity, or by hardware, or by a combination of software and hardware. A computer software or program (also referred to as a program product), including software routines, applets, and / or macros, can be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. The computer program product can include one or more computer-executable components configured to execute the embodiments when the program runs. One or more computer-executable components can be at least one software code or a part thereof. Additionally, in this regard, it should be noted that any box in the logical flow, as Figure 1 described in [reference], can represent a program step, or interconnected logical circuits, boxes, and functions, or a combination of program steps and logical circuits, boxes, and functions. The software can be stored on physical media such as memory chips or storage blocks implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and their data variants, CDs. The physical media are non-transitory media.

[0113] Those skilled in the art should understand that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

[0114] The above embodiments only represent several implementation manners of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the appended claims.

Claims

1. A knowledge base question answering system optimization method based on hybrid fine-tuning and multi-dimensional evaluation, characterized in that: The following steps are involved: Hybrid progressive fine-tuning framework: A two-stage fine-tuning mechanism that combines low-rank adaptation and direct preference optimization, achieves domain knowledge transfer through hierarchical dynamic parameter configuration, and completes strategy alignment based on human preference data; Multi-dimensional quantitative evaluation system: build a three-dimensional evaluation framework including logical consistency, semantic relevance, and knowledge coverage, and generate comprehensive evaluation results through dynamic weight fusion; Dynamic adaptation and lightweight mechanism: Adopt hierarchical parameter freezing, sparse constraints and attention-driven multi-domain prompt template library to achieve model lightweight and cross-domain logical constraints.

2. The knowledge base question answering system optimization method based on hybrid fine-tuning and multi-dimensional evaluation as claimed in claim 1, characterized in that: The hybrid progressive fine-tuning framework includes: Pre-trained model parameter matrix Perform low-rank decomposition by adapting the matrix and Implement parameter update: ,in , d and k are rank values; is the domain semantic encoder, which is a low-rank matrix with a dimension of d × r; is a logical reasoning decoder with dimension r × k, and Constitute a low-rank decomposition pair; Implement hierarchical rank configuration in the Transformer architecture: the middle layer uses higher rank values ​​to capture domain semantic features, and the top layer uses lower rank values ​​to enhance logical reasoning capabilities; Introducing a contrastive learning mechanism, distinguishing high-quality answers from illegal answers based on human preference data, and constructing an optimization target L DPO Drive models aligned to industry norms.

3. The knowledge base question answering system optimization method based on hybrid fine-tuning and multi-dimensional evaluation as claimed in claim 2, characterized in that: The hierarchical rank configuration is specifically: The middle layer uses a rank value of r=8, the top layer uses a rank value of r=4, and the total parameter update is the sum of the adaptation matrices of each layer.

4. The knowledge base question answering system optimization method based on hybrid fine-tuning and multi-dimensional evaluation as claimed in claim 1, characterized in that: The multi-dimensional quantitative evaluation system includes: Logical consistency assessment: Verify the compliance of answers through first-order logic expressions and production rules, and use the Z3 solver and Drools engine to perform automatic verification; Semantic relevance evaluation: Combining the BM25 algorithm with the Sentence-BERT model, the keyword coverage and vector similarity are integrated through the dynamic weight λ; Knowledge coverage evaluation: Use the Bi-LSTM-CRF model to identify domain entities and calculate coverage indicators through the intersection of entity sets: ； in, The collection of entities generated for the model, is the set of annotated real entities, molecules Indicates the number of domain entities that are correctly identified.

5. The knowledge base question answering system optimization method based on hybrid fine-tuning and multi-dimensional evaluation as claimed in claim 4, characterized in that: The logic consistency assessment also includes adversarial perturbation detection: The DeBERTa-v3 model is used to perturb the generated answers, including entity replacement and logic reversal, and the conflict rate formula is used to To verify the logical consistency; Among them, ConsistScore is the logical consistency score, and its value range is [0,1]. The higher the score, the stronger the logical self-consistency; To judge the Samples Whether it is consistent with the reference standard conflict; (⋅) is the indicator function, which is 1 when there is a conflict and 0 otherwise; n is the total number of perturbation samples generated; s i is the i-th disturbance sample; R is the original answer.

6. The method for optimizing a knowledge base question answering system based on hybrid fine-tuning and multi-dimensional evaluation according to claim 1, characterized in that: The dynamic adaptation and lightweight mechanism includes: Freeze the basic parameters of the pre-trained model , only update the adaptation matrix parameters, combined with sparse constraints Suppress parameter redundancy; Build a multi-domain prompt template library, dynamically match templates through the attention mechanism, and generate answers that conform to professional logic. The templates include legal clause templates, medical diagnosis path templates, and financial risk control rule templates.

7. The method for optimizing a knowledge base question answering system based on hybrid fine-tuning and multi-dimensional evaluation as claimed in claim 6, characterized in that: The attention-driven template generation formula is: ； in is the hidden state of the current time step; They are query generation function, key generation function and value generation function respectively; is the dimension; P is the prompt template library.

8. The knowledge base question answering system optimization method based on hybrid fine-tuning and multi-dimensional evaluation according to any one of claims 1 to 7, characterized in that: The comprehensive scoring formula of the three-dimensional evaluation framework is: ； in, Evaluate functions for logical consistency; Evaluate functions for semantic relevance; is the knowledge coverage evaluation function; the dynamic weight satisfies ; Q is the input question, R is the system answer, K is the knowledge base entity set, and it is automatically adjusted according to the characteristics of the domain.

9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the knowledge base question answering system optimization method based on hybrid fine-tuning and multi-dimensional evaluation as described in any one of claims 1 to 8.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which includes a program code for controlling a process to execute a process, and the process includes a knowledge base question answering system optimization method based on hybrid fine-tuning and multi-dimensional evaluation according to any one of claims 1 to 8.

Images