Vertical large model fine tuning method based on proprietary business database

Abstract

The invention discloses a vertical large model fine tuning method based on a proprietary business database, belongs to the technical field of artificial intelligence and databases, and particularly relates to a vertical domain model training and application method for performing large model fine tuning based on self-developed database knowledge. A multi-dimensional evaluation system is established by constructing a proprietary corpus and adopting a parameter efficient fine tuning technology LoRA, and effective conversion from a general model to a proprietary database vertical domain model is realized; according to the method, only 0.5% of parameters are trained through the LoRA technology, so that the training cost and hardware requirements are remarkably reduced; the proprietary corpus is completely constructed for a specific database system, and it is ensured that the model masters proprietary grammar and operation and maintenance knowledge; the multi-dimensional evaluation method comprehensively measures the capabilities of the model in grammar, execution, semantics and the like; the fine-tuned model can be directly deployed in a production environment to meet actual business requirements.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and database technology, and in particular relates to a method for training and applying vertical domain models based on self-developed database knowledge for fine-tuning large models. Background Technology

[0002] 1. Main contents of existing technology: With the rapid development of artificial intelligence technology, large language models (LLMs) pre-trained on large-scale corpora have demonstrated outstanding performance in tasks such as natural language understanding, text generation, and code generation. In recent years, general-purpose large models such as GPT, LLaMA, and Qwen have been widely used in application scenarios such as general question answering, text summarization, and intelligent assistants.

[0003] Existing large-scale models are primarily trained on general corpora, with their knowledge sources mainly consisting of publicly available internet text, programming examples, or encyclopedic information. In terms of applications, these models support basic natural language interaction, general domain-specific question answering, and standard SQL statement generation, among other functions.

[0004] 2. Problems and shortcomings of existing technologies: (1) Lack of in-depth understanding of specialized domain knowledge; Existing general-purpose models lack a deep understanding of the internal syntax, execution mechanisms, and operational specifications of database systems. This is because model training is primarily based on publicly available general corpora, while the proprietary knowledge of specialized databases (such as self-developed databases or industry-customized databases)—including their syntax definitions, execution plans, storage engines, and logging mechanisms—is not included in the training data. Consequently, the models cannot grasp the core knowledge of these specialized domains.

[0005] (2) Insufficient ability to generate proprietary syntax; When faced with database-specific syntax tasks (such as text-to-SQL conversion, SQL performance optimization, and slow query analysis), general models often generate statements that do not conform to the proprietary database syntax specifications, or fail to recognize specific table structures, stored procedures, triggers, and other proprietary syntax elements, resulting in the generated SQL being unexecutable or producing incorrect execution results.

[0006] (3) Low accuracy rate of operation and maintenance knowledge Q&A; In specialized tasks such as database cluster operation and maintenance problem diagnosis, parameter optimization suggestions, and fault analysis, the analytical conclusions given by general models are inaccurate and the knowledge references are inconsistent, failing to meet the stringent requirements for accuracy and consistency in production environments.

[0007] (4) The contradiction between the closed nature and the generalization of knowledge; The database operations and maintenance field is characterized by significant knowledge proprietary nature. Different database systems vary considerably in their implementation details. Existing fine-tuning solutions based on publicly available SQL corpora or English question-and-answer data cannot be effectively trained for the customized syntax and operational logic of proprietary databases, making it difficult to deploy and apply models in specific systems. Summary of the Invention

[0008] The technical problem to be solved by this invention is to provide a method for fine-tuning a vertical large model based on a proprietary business database, which addresses the shortcomings of the prior art. By constructing a proprietary corpus, adopting the LoRA parameter fine-tuning technology, and establishing a multi-dimensional evaluation system, the invention achieves an effective transformation from a general model to a proprietary database vertical domain model.

[0009] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A method for fine-tuning a vertical large-scale model based on a proprietary business database is proposed. This method achieves effective transformation from a general model to a vertical domain model based on a proprietary database by constructing a proprietary corpus, employing the LoRA parameter-efficient fine-tuning technology, and establishing a multi-dimensional evaluation system. The specific steps include the following: Step 1: Construct a proprietary database corpus, which includes: Text-to-SQL Corpus: Contains structured mapping samples of natural language descriptions and proprietary database SQL statements, covering table creation statements, query statements, update statements, stored procedures, and trigger syntax; Operations and maintenance knowledge Q&A corpus: covering professional Q&A pairs for operations and maintenance scenarios such as fault diagnosis, performance optimization, configuration management, and log analysis; General knowledge Q&A corpus: includes general content such as product introductions, function descriptions, and user guides; Step 2, efficient parameter fine-tuning based on LoRA technology, includes: The Low-Rank Adaptive Restriction (LoRA) technique is used for efficient parameter fine-tuning: all parameters of the original model are frozen; a trainable low-rank matrix is ​​inserted into the model's Attention Layer; the LoRA rank is set to 8 and the scaling factor alpha is set to 16. The target module selects the query matrix q_proj, the key matrix k_proj, the value matrix v_proj, and the output matrix o_proj; Step 3: Establish a multi-dimensional evaluation system, including five evaluation modules: data loading and formatting, syntax consistency detection, execution accuracy detection, semantic similarity calculation, and teacher model guidance and scoring.

[0010] As a further preferred embodiment of the vertical large model fine-tuning method based on proprietary business database of the present invention, in step 1, the ratio of the Text-to-SQL corpus, the operation and maintenance knowledge question and answer corpus and the general knowledge question and answer corpus is 60%:25%:15%.

[0011] As a further preferred embodiment of the vertical large-scale model fine-tuning method based on a proprietary business database of the present invention, in step 1, data standardization processing is performed: all corpora are templated using a unified instruction format. {"instruction": "User's natural language input request or question", "output": "Standard answer or SQL statement"}.

[0012] As a further preferred embodiment of the vertical large model fine-tuning method based on a proprietary business database of the present invention, in step 3, the data loading and formatting module loads three types of test data from the evaluation sample set, including Text-to-SQL samples, operation and maintenance knowledge Q&A samples, and general knowledge Q&A samples, in Excel format, with fields including business category, business subcategory, requirement description, and standard answer; and performs SQL normalization processing on the Text-to-SQL samples, including capitalizing keywords, standardizing spaces, and adjusting the field order to eliminate format differences.

[0013] As a further preferred embodiment of the vertical large-model fine-tuning method based on proprietary business database of the present invention, in step 3, the syntax consistency detection module is applicable to Text-to-SQL tasks: Both the SQL generated by the model and the standard SQL are parsed into Abstract Syntax Trees (ASTs); a node comparison algorithm is used to calculate the syntax overlap: syntax matching degree = number of completely matched syntax nodes / total number of SQL nodes; when the syntax matching degree is 95%, the syntax is considered consistent; and the syntax accuracy index is output.

[0014] As a further preferred embodiment of the vertical large-model fine-tuning method based on a proprietary business database of the present invention, in step 3, the accuracy detection module is applied to Text-to-SQL tasks to verify the actual execution effect of SQL: Execute the model-generated SQL and standard SQL separately in an independent test environment using a proprietary database JDBC interface; compare the consistency of the execution result sets of the two; execution accuracy = number of successfully executed samples / total number of test samples; output the execution accuracy metric.

[0015] As a further preferred embodiment of the vertical large-scale model fine-tuning method based on a proprietary business database of the present invention, in step 3, the semantic similarity calculation module is applicable to the entire dataset to evaluate the quality of natural language question answering: The model output and the standard answer are vectorized based on the BERT-base-chinese model; Cosine similarity is used to measure semantic closeness: Similarity = (vector A · vector B) / (|vector A| × |vector B|); When the similarity is 0.85, the semantics are considered to be consistent; the semantic accuracy index is output.

[0016] As a further preferred embodiment of the vertical large-scale model fine-tuning method based on a proprietary business database of the present invention, in step 3, the teacher model-guided scoring module calls a large-scale teacher model to perform multi-dimensional comparison of the model output: The evaluation dimensions include: differences in syntactic structure, semantic consistency, matching of table and field names, and completeness of responses; the teacher model returns a structured score, which includes scores for each dimension and a final composite score; a portion of the samples are manually reviewed by experts from a proprietary database to confirm the accuracy of the teacher scores; The overall score is calculated based on the weights of each evaluation indicator. Overall score = grammatical accuracy 0.3 + execution accuracy 0.3 + semantic accuracy 0.25 + teacher score 0.15.

[0017] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects: 1. High parameter efficiency: This invention trains only 0.5% of the parameters using LoRA technology, significantly reducing training costs and hardware requirements; 2. Strong knowledge relevance: The proprietary corpus is built entirely for a specific database system, ensuring that the model has mastered proprietary grammar and operational knowledge; 3. Scientific evaluation system: Multi-dimensional evaluation methods comprehensively measure the model's capabilities in terms of syntax, execution, and semantics; 4. High feasibility: The finely tuned model can be directly deployed to the production environment to meet actual business needs. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the construction process of the FMDB proprietary database corpus of this invention; Figure 2 This is a flowchart of the LoRA-based model fine-tuning process of this invention; Figure 3 This is a flowchart of the multidimensional evaluation process of this invention; Figure 4 This is a flowchart of the comparative experiment of the model of this invention. Detailed Implementation

[0019] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings: The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments. The purpose and effects of the present invention will become clearer. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0020] This invention proposes a method for fine-tuning a large-scale vertical domain model based on a proprietary business database. By constructing a proprietary corpus, employing LoRa (Local Area Refinement) parameter tuning technology, and establishing a multi-dimensional evaluation system, it achieves effective transformation from a general model to a proprietary database-based vertical domain model. The overall technical solution includes the following core technical components: 1. Methods for constructing proprietary database corpora (1) Corpus sources and classification: A high-quality training corpus covering three major categories of data was constructed. The types and proportions of the corpus are shown in Table 1.

[0021] Table 1 type illustrate percentage Text-to-SQL Natural Language FMDB Proprietary SQL Mapping Sample 60% Operations and Maintenance Knowledge Q&A Fault diagnosis, performance optimization, configuration management, log analysis 25% General Knowledge Q&A Product introduction, function description, user guide 15% Among them, the Text-to-SQL corpus contains structured mapping samples of natural language descriptions and proprietary database SQL statements, accounting for 60%, covering proprietary syntax such as table creation statements, query statements, update statements, stored procedures, and triggers; Operations and maintenance knowledge Q&A corpus: covering professional Q&A pairs for operations and maintenance scenarios such as fault diagnosis, performance optimization, configuration management, and log analysis, accounting for 25%; General knowledge Q&A corpus: includes general content such as product introductions, function descriptions, and user guides, accounting for 15%.

[0022] Data acquisition method: Extract syntax specifications and usage examples from official documentation, technical manuals, and API documentation for proprietary databases; Collect real SQL statements and operation logs from the production environment; High-quality question-answer pairs are constructed through expert annotation; The initial corpus was generated using a general large model, and then manually reviewed and corrected.

[0023] (3) Data standardization processing: All corpora are templated using a unified instruction format: {"instruction": "User-input natural language request or question", "output": "Standard answer or SQL statement"}; To ensure consistency in training data format, the total number of training samples is approximately 50,000, and the validation set is 10,000.

[0024] 2. A LoRA-based method for efficient parameter fine-tuning: (1) Selection of the basic model: We chose Qwen3-8B as the base model. This model has 8 billion parameters, performs well in general natural language understanding tasks, and also has good Chinese language capabilities.

[0025] (2) Application of LoRA technology: Efficient parameter fine-tuning is achieved using Low-Rank Adaptation (LoRA) technology. Freeze all parameters of the original model; Insert a trainable low-rank matrix into the model's attention layer; Set the LoRA rank to 8 and the scaling factor alpha to 16; The target module selects the query matrix q_proj, the key matrix k_proj, the value matrix v_proj, and the output matrix o_proj; Only about 0.5% of the model parameters are trained, significantly reducing training costs and GPU memory usage.

[0026] (3) Training hyperparameter configuration Learning rate: 5 × 10⁻ 4 ; Learning rate scheduling strategy: Cosine strategy; Batch size: 2; Gradient accumulation steps: 8; Computational precision: FP16 mixed-precision training reduces GPU memory usage; Training epochs: 3; Maximum sequence length: 1024 tokens.

[0027] (4) Training process monitoring: During training, the loss function was monitored in real time to ensure model convergence. The training loss curve showed an L-shaped convergence trend, with a final average loss of approximately 0.0199, indicating that the model had fully learned the knowledge from the proprietary database.

[0028] 3. Multi-dimensional model evaluation system: A comprehensive evaluation methodology for vertical domain models of proprietary databases is established, as shown in Table 2, comprising five core evaluation modules: Table 2 Module Applicable Scenarios Core Methods Output metrics Data loading and formatting Three types of test data Excel import, SQL normalization Syntax consistency Text-to-SQL ANTLR4 AST, node matching EM Execution accuracy Text-to-SQL JDBC comparison execution EX semantic similarity All data BERT Vector Cosine S Teacher model scoring All data Qwen-32B Multidimensional Score Overall score Comprehensive weighted All data Weighting summary P0-P3 levels (1) Data loading and formatting module: Three types of test data (Text-to-SQL samples, Operations and Maintenance knowledge Q&A samples, and General knowledge Q&A samples) are loaded from the evaluation sample set. The data is in Excel format, with fields including major business category, minor business category, requirement description, and standard answer. The Text-to-SQL samples are then subjected to SQL normalization processing, including capitalizing keywords, standardizing spaces, and adjusting field order to eliminate formatting differences.

[0029] (2) Syntax consistency detection module: Suitable for Text-to-SQL tasks, the core method is as follows: call ANTLR4 SQL Parser to parse both the SQL generated by the model and the standard SQL into an Abstract Syntax Tree (AST); use a node comparison algorithm to calculate the syntax overlap: syntax matching degree = number of completely matching syntax nodes / total number of SQL nodes; when the syntax matching degree is 95%, the syntax is considered to be consistent. Output the syntax accuracy metric (Exact Match, EM).

[0030] (3) Execute the accuracy detection module Suitable for Text-to-SQL tasks, verifying the actual execution effect of SQL: Execute the model-generated SQL and standard SQL separately in an independent test environment through a proprietary database JDBC interface; compare whether the execution result sets of the two are consistent; execution accuracy = number of successful execution samples / total number of test samples; output the execution accuracy metric (EX).

[0031] (4) Semantic similarity calculation module: Applicable to all datasets, evaluating the quality of natural language question answering: The model output and standard answer are vectorized based on the BERT-base-chinese model; cosine similarity is used to measure semantic closeness. Similarity = (vector A · vector B) / (|vector A| × |vector B|); When the similarity is 0.85, the semantics are considered to be consistent; the semantic accuracy (S) metric is output.

[0032] (5) Teacher Model Guidance and Scoring Module: A large-scale teacher model (such as Qwen-32B) is invoked to perform multi-dimensional comparison of the model output: the evaluation dimensions include: differences in syntactic structure, semantic consistency, matching of table and field names, and completeness of responses; the teacher model returns a structured score, which includes scores for each dimension and a final comprehensive score; a portion of the samples are manually reviewed by experts from a proprietary database to confirm the accuracy of the teacher scores.

[0033] (6) Comprehensive weighted score: The comprehensive score of the model is calculated according to the weight of each evaluation indicator: comprehensive score = grammatical accuracy 0.3 + execution accuracy 0.3 + semantic accuracy 0.25 + teacher score 0.15; the model performance level is divided into (P0-P3) according to the comprehensive score, where P3 is the highest level (comprehensive score 95%).

[0034] This invention offers highly efficient parameters: by using LoRA technology, only 0.5% of the parameters are trained, significantly reducing training costs and hardware requirements; Highly targeted knowledge: The proprietary corpus is built entirely for a specific database system, ensuring that the model possesses proprietary grammar and operational knowledge; Scientific evaluation system: Multi-dimensional evaluation methods comprehensively measure the model's capabilities in terms of syntax, execution, and semantics; Highly deployable: The finely tuned model can be directly deployed to the production environment to meet actual business needs.

[0035] This invention aims to address the following core technical challenges in the application of existing general-purpose large language models in the domain of proprietary databases: how to construct a high-quality training corpus for proprietary databases, covering proprietary grammar, operational knowledge, and business knowledge; how to design effective model fine-tuning methods to enable general-purpose large models to accurately understand and generate customized grammar for proprietary databases; how to establish a scientific evaluation system to comprehensively measure the performance of the fine-tuned model in multiple dimensions such as grammatical correctness, execution accuracy, and semantic consistency; and how to realize the practical application of proprietary database vertical domain models to meet the actual needs of Text-to-SQL generation, operational problem diagnosis, and knowledge question answering in production environments.

[0036] The invention will be described in detail below with reference to four specific embodiments. This embodiment takes the self-developed proprietary database FMDB as the application object. By constructing a proprietary FMDB corpus, fine-tuning the Qwen3-8B model using LoRA technology, and establishing a multi-dimensional evaluation system, the FMDB vertical domain proprietary model qwen3-8B-fmdb is finally trained and compared with the un-fine-tuned general model.

[0037] Example 1: Construction of the FMDB proprietary database corpus: The FMDB proprietary database corpus construction process in this embodiment is referenced in the appendix. Figure 1The process involves corpus collection, data standardization, and dataset partitioning. The corpus collection stage acquires three types of data: Text-to-SQL, operational knowledge, and general question-and-answer data. After standardization, these are unified into JSON format. Finally, the dataset is divided into training, validation, and test sets according to a set ratio for model training and evaluation.

[0038] 1.1 Corpus Collection: (1) Text-to-SQL Corpus Collection: Extracting proprietary syntax examples from FMDB official documentation and technical manuals, including: FRC table (FMDB Real-time Cache Table) creation statements and query syntax; Real-time data entry syntax; stored procedure definition and calling syntax; timer creation and management syntax; partitioned table syntax; specific functions and operators (such as time window functions, aggregate functions, etc.).

[0039] Meanwhile, real SQL execution logs are collected from the production environment, frequently used SQL patterns are extracted, and after being anonymized, they are added to the training set.

[0040] (2) Collection of Operation and Maintenance Knowledge Q&A Corpus: Collect common questions in FMDB operation and maintenance scenarios, including: Slow query diagnostics: "Why does the query execution time exceed 10 seconds?" Analyze the execution plan, index usage, table scan methods, etc. Performance optimization: "How to optimize query performance for large tables?" Suggestions include creating appropriate indexes, adjusting partitioning strategies, and using cached tables. Troubleshooting: "How to handle database connection pool exhaustion?" Check connection configuration, troubleshoot slow SQL queries, optimize application connection management, etc. Parameter tuning: "How to configure FMDB memory parameters?" Provide configuration suggestions based on hardware resources and business characteristics.

[0041] These Q&As have been annotated and reviewed by senior FMDB operations and maintenance experts to ensure the professionalism and accuracy of the answers.

[0042] (3) Collection of general knowledge question and answer corpus: including general content such as FMDB product introduction, functional features, version evolution, and application cases. This part of the corpus helps the model understand the overall architecture and application scenarios of FMDB.

[0043] 1.2 Data Standardization Processing: All corpora were processed into JSON format: {"instruction": "Please generate a table creation statement that conforms to FMDB syntax. The table name is user_info, containing three fields: id, name, and age.", "output": "CREATE TABLE user_info (id INT PRIMARY KEY, name VARCHAR(100), age INT) PARTITIONED BY (id)"} 1.3 Dataset Partitioning The dataset is divided into training, validation, and test sets. The dataset division and sample statistics are shown in Table 3.

[0044] Table 3 gather Total Text-to-SQL Operations and maintenance General illustrate training set 50000 30000 12500 7500 Ratio 60 / 25 / 15 Validation set 10000 same proportion as training set test set 2658 1239 Approximately 700 Approximately 719 Final assessment Training set: 50,000 samples, including 30,000 Text-to-SQL corpus (60%), 12,500 operation and maintenance knowledge Q&A (25%), and 7,500 general knowledge Q&A (15%). Validation set: 10,000 samples, divided in the same proportion; Test set: 2658 samples, used for final evaluation, including 1239 Text-to-SQL samples, approximately 700 operation and maintenance knowledge Q&A samples, and approximately 719 general knowledge Q&A samples.

[0045] Example 2: Implementation process of LoRA-based model fine-tuning: This embodiment uses the LoRA method to perform lightweight fine-tuning of the Qwen3-8B model. The training process is completed in a GPU environment, lasting 3 epochs. The loss decreased from 2.5 to 0.0199, achieving efficient convergence and significant parameter compression. The LoRA-based model fine-tuning process can be found in the appendix. Figure 2 As shown.

[0046] 2.1 Hardware Environment Configuration: GPU: NVIDIA TITAN 24G 2 CPU: Intel Xeon Gold 6240 @ 2.60GHz Memory: 400GB Operating System: CentOS Linux 7.41708 CUDA version: 12.1 Python version: 3.10 2.2 Software Environment Configuration: Deep learning framework: PyTorch 2.0.1 Large model training library: Transformers 4.34.0 LoRA implementation library: PEFT 0.5.0 Training acceleration library: Accelerate 0.23.0 Basic model: Qwen3-8B 2.3 LoRA Fine-tuning Configuration Parameters: lora_rank:8 lora_alpha:16 lora_dropout:0.05 target_modules:["q_proj", "k_proj", "v_proj", "o_proj"] task_type: CAUSAL_LM (Causal Language Model) 2.4 Training Hyperparameter Settings: learning_rate:5e-4 lr_scheduler_type:cosine warmup_ratio: 0.1 per_device_train_batch_size:4 gradient_accumulation_steps:4 num_train_epochs:3 max_seq_length:2048 fp16: True (Mixed Precision Training) 2.5 Training Process: The training consisted of 3 epochs, with a total of approximately 37,500 steps. During training, the loss curve exhibited a typical L-shaped convergence trend. Epoch 1: The initial loss was about 2.5, which quickly dropped to about 0.3; Epoch 2: The loss gradually decreased from 0.3 to around 0.05; Epoch 3: The loss fluctuates around 0.02 and eventually converges to 0.0199.

[0047] The total training time is approximately 18 hours, and the model parameter file size is approximately 120MB (containing only LoRA weights), which significantly reduces storage and deployment costs compared to the full model (approximately 16GB).

[0048] Example 3: Implementation process of multi-dimensional model evaluation: This embodiment constructs a multi-dimensional model evaluation system, which can be referred to in the appendix. Figure 3The multi-dimensional evaluation process includes four indicators: syntactic consistency, execution accuracy, semantic similarity, and teacher model score. Through comprehensive evaluation using AST parsing, result set comparison, BERT semantic computation, and Qwen-32B scoring, the model achieved a comprehensive score of 98.64%, ranking as the highest level, P3, validating the model's high consistency and stability at the syntactic, semantic, and execution levels.

[0049] 3.1 Assessment Environment Setup: GPU server: NVIDIA TITAN 24GB; CUDA version: 12.1; Python version: 3.10; Database environment: FMDB test instance, independently deployed, version FMDB 2.1.0; 3.2 Test Data Preparation: 2658 real test samples were extracted from the business system, in Excel format, containing the following fields: Business Categories: Text-to-SQL / Operations and Maintenance Knowledge / General Knowledge; Business subcategories: table creation statements / query statements / performance optimization / fault diagnosis, etc.; Requirement description: User's natural language input; Standard answer: The correct answer or SQL statement marked by experts; Sample distribution: Text-to-SQL sample: 1239 records; Operations and maintenance knowledge Q&A sample: approximately 700 entries; General knowledge Q&A sample: approximately 719 items; 3.3 Implementation of Syntax Consistency Check: Step 1: SQL Normalization Preprocessing Standardize all SQL statements: convert keywords to uppercase (select SELECT), remove extra spaces, and standardize delimiter format.

[0050] Step 2: AST analysis: The ANTLR4 SQL Parser is used to parse the model to generate SQL and standard SQL respectively, and then an abstract syntax tree is constructed. For example: Enter SQL:SELECT id, name FROM user WHERE age > 18 AST structure: SELECT [COLUMN_LIST(id, name), FROM(user), WHERE(age > 18)] Step 3: Node matching calculation: Traverse both syntax trees, count the number of nodes that are perfectly matched, and calculate the matching degree.

[0051] Experimental results: Among 1239 Text-to-SQL test samples, 1225 were grammatically identical, with a syntax accuracy of EM = 98.87%.

[0052] 3.4 Implementation of accuracy testing: Step 1: Test Environment Preparation: Pre-create test tables and data in the FMDB test database and ensure that the SQL is executable.

[0053] Step 2: Parallel SQL execution: Execute the model-generated SQL and standard SQL separately through the FMDB JDBC interface, and capture the execution results: If the SQL syntax is incorrect, record it as an execution failure; if the SQL execution is successful, obtain the returned result set.

[0054] Step 3: Result set comparison: Compare whether the number of rows, columns, field names, and data values ​​of the two result sets are completely consistent.

[0055] Experimental results: Of the 1239 test samples, 1216 SQL statements were executed successfully and the results were consistent, with an execution accuracy of EX = 98.14%.

[0056] 3.5 Implementation of Semantic Similarity Calculation: Step 1: Text Vectorization: Encode the model output and the standard answer using a BERT-based-chinese pre-trained model: Embedding_model = BERT.encode(model output text) Embedding_standard = BERT.encode(standard answer text) Step 2: Cosine similarity calculation: Calculate the cosine similarity between two vectors: Similarity = cos(θ) = (A·B) / (|A|×|B|) When the similarity is not less than 0.85, the semantics are considered to be consistent.

[0057] Step 3: Statistical Analysis: The semantic similarity evaluation results for all 2658 test samples are shown in Table 4: the average similarity between Text-to-SQL is 98.7%, the similarity between Operation and Maintenance knowledge is 99.2%, the similarity between General knowledge and the overall average semantic similarity is 98.76%, and the semantic consistency is excellent.

[0058] Table 4 Type Sample Sample distribution Through samples Average similarity Text-to-SQL 1239 1223 98.7% Operations and Maintenance Knowledge Q&A 700 694 99.2% General Knowledge Q&A 719 707 98.4% total 2658 98.76% 3.6 Teacher Model-Guided Scoring Implementation: Step 1: Construct a scoring prompt: Design scoring instructions that require the teacher model (Qwen-32B) to evaluate the model output from multiple dimensions. Prompt example: "Please compare the quality of the following two texts, scoring them from three dimensions: grammatical structure (0-1 points), semantic consistency (0-1 points), and content matching (0-1 points), and give the final comprehensive score. Standard answer: [Standard text] Model output: [Text to be evaluated]" Step 2: Batch scoring: Call the Qwen-32B model API to score all test samples and return structured JSON results. {"syntax_score": 0.95, "semantic_score": 0.89, "content_match": 0.92, "final_score": 0.92}; Step 3: Expert review: FMDB operation and maintenance experts randomly selected 100 samples for manual review to verify the accuracy of teacher scores. The results showed that the consistency rate between teacher scores and expert scores was over 95%.

[0059] Experimental results: The average comprehensive score of the teacher model was 98.98%.

[0060] 3.7 Calculation of Comprehensive Weighted Score: The final comprehensive score is calculated based on the weights of each indicator, as shown in Table 5.

[0061] Table 5 Evaluation indicators Score Weight Weighted score Grammatical accuracy (EM) 98.87% 0.3 29.66 Execution accuracy (EX) 98.14% 0.3 29.44 Semantic similarity (S) 98.76% 0.25 24.69 Teacher model scoring 98.98% 0.15 14.84 Overall score 98.64% - Performance level P3 Calculation formula: Overall score = 98.87% * 0.3 + 98.14% * 0.3 + 98.76% * 0.25 + 98.98% * 0.15 = 98.64%; According to the scoring criteria, the overall score reaches 98.64%, and it is rated as P3 performance level (the highest level, requiring an overall score of 95%).

[0062] Example 4: Model Performance Comparison Experiment and Result Analysis: To verify the effectiveness of the method of this invention, this embodiment selects different versions of the QWEN series general model as the control group, and conducts a comprehensive comparative evaluation with the qwen3-8B-fmdb proprietary model (experimental group) trained based on the method of this invention. See appendix. Figure 4 Model comparison experiment procedure.

[0063] 4.1 Comparative Experiment Design: Experimental group: qwen3-8B-fmdb: A vertical domain model based on the Qwen3-8B model, using the LoRA fine-tuning method proposed in this invention, and trained using the FMDB proprietary corpus.

[0064] Control group (unadjusted general model): qwen3-8B: The Qwen3 series 8B parameter base model, without any fine-tuning of proprietary database corpora; qwen3-32B: The Qwen3 series 32B large-scale model with approximately 4 times the number of parameters of the 8B model; qwen3-Coder: The Qwen3 series is a model specifically optimized for code generation tasks, with strong SQL generation capabilities.

[0065] Test dataset: Same as in Example 3, containing 2658 test samples.

[0066] Evaluation metrics: grammatical accuracy, execution accuracy, semantic accuracy, teacher rating, overall score, and performance rating.

[0067] 4.2 Comparative Experimental Results: As can be seen from the comparative experimental results in Table 6, the qwen3-8B-fmdb model, finely tuned by the method of this invention, achieves excellent performance of over 98% in all indicators, which is a significant improvement compared to the untuned general model.

[0068] Table 6 Model version Grammar accuracy Execution accuracy semantic accuracy Teacher rating Overall score performance level qwen3-8B 36.97% 33.98% 28.29% 27.54% 32.49% P0 qwen3-32B 43.34% 41.32% 31.15% 31.15% 37.81% P0 qwen3-Coder 50.28% 41.49% 30.62% 29.46% 39.61% P0 qwen3-8B-fmdb 98.87% 98.14% 98.76% 98.98% 98.64% P3 4.3 Detailed Results Analysis: (1) Analysis of the improvement in grammatical accuracy and execution accuracy; Limitations of the general model: The untuned QWEN series models possess a certain ability to understand common SQL structures (such as standard statements like SELECT, INSERT, and UPDATE), thus maintaining a 30%-50% accuracy rate in basic syntax tests. Among them, qwen3-Coder, specifically optimized for code generation, achieves a syntax accuracy of 50.28%, outperforming other general-purpose models.

[0069] The challenges of proprietary grammar: The FMDB system contains a large amount of proprietary syntax, which is not within the scope of general SQL corpora, causing the basic large model to fail to parse and generate correctly. FRC table syntax: FMDB-specific syntax format, requiring the specification of a special storage engine and caching strategy; Real-time INSERT syntax differs from standard INSERT syntax. Stored procedure syntax: FMDB's stored procedure definition method differs from mainstream databases such as MySQL and Oracle; Timer syntax: The CREATE TIMER statement is unique to FMDB; this syntax has never been seen in the general model. Partitioned table syntax: FMDB's PARTITIONED BY clause syntax differs from systems like Hive.

[0070] Significant improvements after fine-tuning: By introducing custom FMDB syntax and execution logic during the fine-tuning phase, the qwen3-8B-fmdb model can: Accurately identify and generate FRC table creation statements; Use REALTIME INSERT correctly for real-time data writing; Define stored procedures and triggers according to the FMDB specification; Configure partitioning strategies and index structures appropriately.

[0071] Experimental data show that the grammatical accuracy improved from 50.28% (optimal general model) to 98.87%, an increase of 96.5%; and the execution accuracy improved from 41.49% to 98.14%, an increase of 136.5%. This proves that the method of this invention can effectively solve the problem of learning proprietary grammar.

[0072] (2) Analysis of the improvement of semantic accuracy and teacher scores: specialization of business knowledge understanding; Semantic and human-rating assessments include three types of tasks: Text-to-SQL conversion: requires the model to accurately understand the user's intent and generate SQL that conforms to the syntax; Operations and maintenance Q&A: covering professional scenarios such as fault diagnosis, performance optimization, and parameter tuning; General knowledge Q&A: including basic knowledge such as product introduction and function description.

[0073] Shortcomings of the general model: The untuned model still retains some accuracy in general knowledge areas (such as standard SQL usage instructions), but it exhibits significant shortcomings in FMDB-specific business scenarios: Operation and maintenance problem diagnosis: Unable to accurately analyze FMDB's unique log format and error codes, the proposed solutions are not specific; Anomaly analysis: Without understanding FMDB's internal execution mechanism, it is impossible to locate performance bottlenecks; Parameter optimization suggestions: There is a lack of understanding of FMDB's memory management, connection pool configuration, caching strategy and other parameters, so the suggestions are not feasible or applicable.

[0074] A leap in quality after fine-tuning: The qwen3-8B-fmdb model achieves the following by introducing FMDB operation and maintenance logs, parameter descriptions, and problem scenario corpora: Precise fault location: It can quickly identify the type of problem (such as connection timeout, memory overflow, index failure, etc.) based on the log content; Professional optimization suggestions: Provide targeted optimization solutions based on the characteristics of FMDB (such as suggesting the use of FRC tables to accelerate hot data queries); Consistent knowledge references: The answer content is consistent with the official FMDB documentation to avoid knowledge conflicts.

[0075] Experimental data show that the semantic accuracy improved from 31.15% (optimal general model) to 98.76%, an increase of 217.1%; and the teacher rating improved from 31.15% to 98.98%, an increase of 217.7%.

[0076] (3) Comprehensive performance and feasibility analysis Performance level comparison All untuned general models scored below 40% overall, achieving a performance level of P0 (unacceptable), and failed to meet the application requirements of the production system. In contrast, the qwen3-8B-fmdb model achieved a comprehensive score of 98.64%, a performance level of P3 (excellent), with all indicators meeting the standards.

[0077] Practical application capabilities: The finely tuned qwen3-8B-fmdb model achieved a performance improvement of approximately 2.5-3 times across all metrics, meeting the multi-dimensional needs of the database field. Syntax generation: It can accurately generate various SQL statements that conform to the FMDB specification; Execution verification: The generated SQL can be executed directly in the FMDB system without manual correction; Knowledge Q&A: Able to accurately answer FMDB-related operation, maintenance, and business questions.

[0078] These capabilities make the qwen3-8B-fmdb model feasible for direct deployment in production systems, and it can be applied to: Intelligent SQL Assistant: Helps developers quickly generate database operation statements; Operations and maintenance Q&A system: Provides 24 / 7 technical support for operations and maintenance personnel; Automated operation and maintenance tools: assist in completing tasks such as log analysis and performance diagnosis.

[0079] 4.4 Experimental Conclusions: Through comparative experiments and detailed analysis, the following conclusions can be drawn: (1) The importance of proprietary knowledge: The application of general large models in the field of proprietary databases has obvious limitations. Even large-scale models with 32-B parameters or Coder models specifically designed for code optimization cannot meet the needs of practical applications without proprietary knowledge.

[0080] (2) Effectiveness of the fine-tuning method: The fine-tuning method based on proprietary corpus and LoRA technology proposed in this invention can achieve a qualitative leap in multiple dimensions such as grammatical correctness, execution accuracy and semantic consistency while maintaining a small number of parameters (8B).

[0081] (3) Scientific nature of the evaluation system: The multi-dimensional evaluation method comprehensively measures the actual capabilities of the model in the field of proprietary databases, and the various indicators corroborate each other, resulting in high credibility.

[0082] (4) Feasibility of deployment: The FMDB proprietary large model trained by the method of this invention can not only correctly parse the customized syntax of the FMDB system at the syntax and execution level, but also significantly improve the accuracy and consistency at the semantic understanding and business question answering level. It realizes a complete mastery of the database proprietary knowledge system and high-quality output, showing significant model proprietary and industry adaptability capabilities, and has practical application value.

[0083] It will be understood by those skilled in the art that the above descriptions are merely preferred embodiments of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention should be included within the scope of protection of the invention. All technical features in this embodiment can be freely combined according to actual needs.

Claims

1. A method for fine-tuning a vertical large model based on a proprietary business database, characterized in that: By constructing a proprietary corpus, employing the LoRA (Local Area Relationship) technique for efficient parameter fine-tuning, and establishing a multi-dimensional evaluation system, the effective transformation from a general model to a proprietary database-based vertical domain model is achieved; specifically, the following steps are included. Step 1: Construct a proprietary database corpus, which includes: Text-to-SQL Corpus: Contains structured mapping samples of natural language descriptions and proprietary database SQL statements, covering table creation statements, query statements, update statements, stored procedures, and trigger syntax; Operations and maintenance knowledge Q&A corpus: covering professional Q&A pairs for operations and maintenance scenarios such as fault diagnosis, performance optimization, configuration management, and log analysis; General knowledge Q&A corpus: includes general content such as product introductions, function descriptions, and user guides; Step 2, efficient parameter fine-tuning based on LoRA technology, includes: The Low-Rank Adaptive LoRA technique is used for efficient parameter fine-tuning: all parameters of the original model are frozen; a trainable low-rank matrix is ​​inserted into the model's Attention Layer. Set the LoRA rank to 8 and the scaling factor alpha to 16; The target module selects the query matrix q_proj, the key matrix k_proj, the value matrix v_proj, and the output matrix o_proj; Step 3: Establish a multi-dimensional evaluation system, including five evaluation modules: data loading and formatting, syntax consistency detection, execution accuracy detection, semantic similarity calculation, and teacher model guidance and scoring.

2. The method for fine-tuning a vertical large model based on a proprietary business database according to claim 1, characterized in that: In step 1, the ratio of the Text-to-SQL corpus, the Operation and Maintenance Knowledge Q&A corpus, and the General Knowledge Q&A corpus is 60%:25%:15%.

3. The method for fine-tuning a vertical large model based on a proprietary business database according to claim 1, characterized in that: In step 1, data standardization is performed: all corpora are templated using a unified instruction format. {"instruction": "User's natural language input request or question", "output": "Standard answer or SQL statement"}.

4. The method for fine-tuning a vertical large model based on a proprietary business database according to claim 1, characterized in that: In step 3, the data loading and formatting module loads three types of test data from the evaluation sample set, including Text-to-SQL samples, operation and maintenance knowledge Q&A samples, and general knowledge Q&A samples. The format is an Excel file, and the fields include business category, business subcategory, requirement description, and standard answer. The Text-to-SQL samples are subjected to SQL normalization processing, including capitalizing keywords, standardizing spaces, and adjusting the field order to eliminate format differences.

5. The method for fine-tuning a vertical large model based on a proprietary business database according to claim 1, characterized in that: In step 3, the syntax consistency detection module is applicable to Text-to-SQL tasks: Both the SQL generated by the model and the standard SQL are parsed into Abstract Syntax Trees (ASTs); a node comparison algorithm is used to calculate the syntax overlap: syntax matching degree = number of completely matched syntax nodes / total number of SQL nodes; when the syntax matching degree is 95%, the syntax is considered consistent; and the syntax accuracy index is output.

6. The method for fine-tuning a vertical large model based on a proprietary business database according to claim 1, characterized in that: In step 3, the accuracy detection module is applied to Text-to-SQL tasks to verify the actual execution effect of the SQL: Execute the model-generated SQL and standard SQL separately in an independent test environment using a proprietary database JDBC interface; compare the consistency of the execution result sets of the two; execution accuracy = number of successfully executed samples / total number of test samples; output the execution accuracy metric.

7. The method for fine-tuning a vertical large model based on a proprietary business database according to claim 1, characterized in that: In step 3, the semantic similarity calculation module is applied to the entire dataset to evaluate the quality of natural language question answering. The model output and the standard answer are vectorized based on the BERT-base-chinese model; Cosine similarity is used to measure semantic closeness: Similarity = (vector A · vector B) / (|vector A| × |vector B|); When the similarity is 0.85, the semantics are considered to be consistent; the semantic accuracy index is output.

8. The method for fine-tuning a vertical large model based on a proprietary business database according to claim 1, characterized in that: In step 3, the teacher model guides the scoring module to call a large-scale teacher model to perform multi-dimensional comparisons of the model output: The evaluation dimensions include: differences in syntactic structure, semantic consistency, matching of table and field names, and completeness of responses; the teacher model returns a structured score, which includes scores for each dimension and a final composite score; a portion of the samples are manually reviewed by experts from a proprietary database to confirm the accuracy of the teacher scores; The overall score is calculated based on the weights of each evaluation indicator. Overall score = grammatical accuracy 0.3 + execution accuracy 0.3 + semantic accuracy 0.25 + teacher score 0.15.

Images