A low-resource vertical intelligent question-answering system construction method for a private field

Abstract

The application discloses a low-resource vertical field intelligent question and answer system construction method for a private field, and relates to the technical field of natural language processing. In view of the problems of high private data leakage risk, training data scarcity, insufficient knowledge injection and limited reasoning ability of small models of the existing vertical field question and answer system, the application adopts public field documents to construct a training data set, trains a local vertical field small model through a two-stage parameter efficient fine-tuning strategy of DoRA+LoRA, constructs a private knowledge base in a local physically isolated environment, obtains candidate contexts in combination with a hybrid retrieval and reordering mechanism, and constructs a large and small model collaborative reasoning framework to dynamically determine a reasoning strategy according to the complexity of a question. The application realizes that private data is not transmitted throughout the process, significantly improves the vertical field question and answer accuracy under the condition of low data and computing resources, and is suitable for medical, legal, financial and all other vertical fields with private data requirements.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing, artificial intelligence, and large language model application technology, specifically to a method for constructing a low-resource vertical domain intelligent question answering system for private domains. Background Technology

[0002] With the rapid development of large language model technology, general-purpose large language models have demonstrated strong versatility in tasks such as natural language understanding, text generation, logical reasoning, and question answering, and are gradually being applied to vertical fields such as healthcare, law, and finance. However, in the process of building intelligent question answering systems for private or professional vertical fields, general-purpose large language models and existing vertical question answering systems still have the following shortcomings.

[0003] First, constructing high-quality training data for vertical domains is costly. Existing technologies typically rely on manually organizing corpora, manually writing question-and-answer pairs, or simple text extraction to construct training data. There is a lack of a systematic method for automatically parsing, batch-generating, and cleaning training corpora and question-and-answer pair datasets from domain knowledge documents. This results in low efficiency in constructing training data for small vertical domain models, thus limiting training data resources. Furthermore, vertical domain applications often involve sensitive and private information, such as patient medical records, internal corporate documents, or legal case materials. This type of data is only permitted for internal use and cannot be shared externally, making it difficult to directly construct datasets for model training. This further restricts the application of small vertical domain models in private domains.

[0004] Secondly, the injection of domain knowledge is insufficient under computationally limited conditions. Existing training methods for vertical domain models often employ single-stage supervised fine-tuning strategies under such conditions, typically performing LoRA fine-tuning directly on the base model based on domain question-answering pairs. While this approach can improve the model's adaptability to specific question-answering tasks, it lacks sufficient injection of domain knowledge, easily leading to the model only learning the question-answering format without fully mastering the domain knowledge. Furthermore, it is difficult to balance the specialized capabilities and complex reasoning abilities of small vertical domain models. While domain-fine-tuned small vertical domain models typically excel in understanding specialized terminology and expressing domain knowledge, their capabilities are significantly insufficient for tasks such as complex logical reasoning and multi-step problem decomposition due to limitations in model parameter size, making it difficult to independently address the diverse needs of vertical domain question answering.

[0005] In summary, existing technologies urgently need a method for constructing a low-resource vertical domain intelligent question answering system for private domains, in order to achieve automated training data generation and effective injection of domain knowledge; and, while ensuring data privacy, combine it with a local private knowledge base for high-precision reasoning, thereby improving the overall performance of the vertical domain small model in terms of professional capabilities and complex logical reasoning capabilities. Summary of the Invention

[0006] To address the aforementioned issues, this invention aims to propose a method for constructing a low-resource vertical domain intelligent question answering system for private domains. This method systematically solves the problems of scarce training data, inability to directly train with private data, insufficient knowledge injection, and limited reasoning ability of small models under limited data and computing resources, thereby achieving efficient vertical domain question answering for private domains.

[0007] To achieve the above objectives, the technical solution of the present invention is implemented as follows: A method for constructing a low-resource vertical domain intelligent question answering system for private domains includes the following steps: S1. Construct a training dataset based on publicly available domain documents and non-sensitive domain materials. The training dataset includes a domain corpus dataset for domain knowledge injection and a domain question-answer pair dataset for task adaptation. The construction of the training dataset includes automatic parsing of domain knowledge documents, structured segmentation, multi-model question-answer pair generation, automatic cleaning of multi-rule data, and manual verification. The entire process does not involve any private domain documents, and all data processing can be completed locally. S2. A two-stage parameter efficient fine-tuning strategy is adopted to train a local vertical domain small model. In the first stage, based on the domain corpus dataset, the DoRA method is used to continuously pre-train the local base language model to obtain an intermediate model with basic domain knowledge. In the second stage, based on the domain question-answering pair dataset, the LoRA method is used to supervise and fine-tune the intermediate model to obtain a local vertical domain small model that has both domain knowledge understanding ability and question-answering task execution ability. S3. In a local computing environment physically isolated from external networks, a local private knowledge base is constructed based on private domain documents. Combined with a hybrid retrieval and re-ranking mechanism, candidate contexts and their semantic relevance scores are obtained from the local private knowledge base for user questions. These can be used as a reference for the local vertical domain small model to generate answers. The private domain documents do not participate in the construction of the training dataset and are not used to update any model parameters of the local vertical domain small model. S4. Construct a collaborative reasoning framework for large and small models. Dynamically determine the reasoning strategy based on the complexity of the user's question or the relevance score between the candidate context and the user's question. Prioritize calling the local vertical domain small model with low computational resource consumption to generate answers. Only in complex question scenarios should the local general large model be called to extract keywords in order to retrieve the local private knowledge base and assist in completing the context validity judgment.

[0008] Furthermore, in step S1, the process of generating the domain corpus dataset and the domain question-answer pair dataset specifically includes: S1.1 Obtain publicly available domain knowledge documents and non-sensitive domain materials from open academic literature databases, open book resource databases, and open online articles, and parse domain documents in different formats: For text documents in WORD, TXT, and HTML formats, extract the heading levels, body paragraphs, and table content, and convert them into Markdown format documents with hierarchical structure; for PDF format documents, use a deep learning-based multi-model collaborative parsing pipeline to process them, extract the text, formulas, tables, and heading level information, and convert the parsing results into Markdown format documents; S1.2. Using Markdown's special syntax, the Markdown document is divided into text blocks. Each section of the document is divided into a text block. If the content of a section exceeds the preset maximum character length threshold, it is truncated according to special symbols. The number of overlapping characters between adjacent text blocks is set to the preset overlap threshold. While effectively splitting the domain knowledge document, semantic integrity is preserved. Text blocks with a character count lower than the preset minimum character threshold are filtered to remove fragmented content with insufficient information. Regular expressions are used to delete hyperlinks and noise content unrelated to domain knowledge in the text blocks, resulting in text blocks that have undergone preliminary cleaning and structuring. S1.3. Based on the preliminary cleaning and structuring of the text blocks obtained in S1.2, prompt word engineering is used to clean the text blocks using a general large model, filtering out meaningless content, including garbled text and content irrelevant to the domain, thus obtaining the final text blocks. These final text blocks are used to form the domain corpus dataset for the first stage of fine-tuning training, where each text block serves as a training sample. Simultaneously, based on each final text block, prompt word engineering is used to extract questions using a general large model, with several questions extracted from each text block. Based on the current text block and the extracted questions, prompt word engineering is used to generate answers and thought chains using a general large model, thus forming the domain question-answering pair dataset. Based on the generated question-answering pair dataset, the general large model is used to evaluate and optimize the answers and thought chains, ultimately generating the domain corpus dataset and domain question-answering pair dataset required for the two-stage fine-tuning.

[0009] Furthermore, in S1.1, the multi-model collaborative parsing pipeline for PDF format documents specifically includes: The DocLayout-YOLO model is used to divide PDF document pages into regions and identify three core layout elements: text, tables, and formulas. The DB text detection algorithm and the CRNN+CTC text recognition algorithm are used to extract the text content of the text region; The YOLOv8 formula detection model and the UniMERNet formula recognition model were used to extract mathematical formulas and convert them into LaTeX format. The SLANet+ / UNet structure recognition model is used to reconstruct the row and column structure and cell content of the table; The text content extracted by text recognition, the mathematical formulas extracted by formula recognition, and the table content restored by table recognition are reorganized according to the heading level of the original document and output as a Markdown format document that preserves the semantic structure.

[0010] Furthermore, in S2, the weight update method for continuous pre-training using the DoRA method in the first stage is as follows: The pre-trained weight matrix of the local base language model The nth column is decomposed into two parts: amplitude and direction.

[0011] in Direction matrix The List, Amplitude vector The One element, Represents finding vectors Norm.

[0012] Initialization command , ; Amplitude vector Set as trainable parameters; direction matrix Keeping itself frozen, its updates are achieved by introducing low-rank increments. The complete weight update formula is:

[0013] in , , , Representation matrix The Columns; during initialization, Initialize according to the uniform distribution of Kaiming. Initialize it as a zero matrix to ensure that training starts at the beginning. ,Right now .

[0014] Furthermore, in S2, the set of trainable parameters during the training process is as follows: Let the domain corpus dataset be , among which samples For the first A text block, Given the total number of text blocks and a sample of domain-specific corpus data. Given the existing token sequence, the training objective of the model is to maximize the predicted probability of the next token, i.e., minimize the following negative log-likelihood loss:

[0015] in, Indicates the first The first sample One token, for The length of the token sequence, For the model in parameters The conditional probability under the given conditions.

[0016] Furthermore, in S2, the specific process of supervised fine-tuning using the LoRA method in the second stage is as follows: Based on a domain question-answering pair dataset, the LoRA method is used to supervise and fine-tune the intermediate model obtained in the first stage to achieve domain task adaptation. The LoRA method freezes the weights of the intermediate model and achieves efficient parameter fine-tuning by injecting a trainable low-rank factorization matrix. For the weight matrix in the intermediate model... LoRA updates its weights The constraints are in low-rank decomposition form:

[0017] in , , During initialization, Initialize according to the uniform distribution of Kaiming. Initialize it as a zero matrix to ensure that training starts at the beginning. During the training process Keep frozen, the set of trainable parameters is ; Let the domain question-answering pair dataset be... ,in For the first One question, for The corresponding thought chain, for The corresponding answer, The total number of samples; each sample is constructed as an input / output pair in instruction format. ,in A sequence of prompt words containing the question. The target output sequence contains the thought process and the answer; in the dataset When conducting oversight and fine-tuning, for the first Input sequence of samples The forward propagation of the model is as follows:

[0018] in This indicates the output of the specified network layer where the LoRA method was applied. The preset scaling constant, As a scaling factor to reduce rank The need to readjust hyperparameters when they change.

[0019] Furthermore, in S2, the loss function and final weight merging method for the second-stage supervised fine-tuning are as follows: Given a sequence of input questions, the training objective of the model is to maximize the predicted probability of each token in the thought chain and the answer, i.e., to minimize the following negative log-likelihood loss:

[0020] in Indicates the first The first sample target sequence One token, for The length of the token sequence, For the model in parameters The conditional probability is given below. After the second stage of training is completed, the low-rank matrix will be... With intermediate model weights The final weights are obtained by merging. :

[0021] Weight is The model is a small vertical model that combines domain knowledge and task adaptability.

[0022] Furthermore, in S3, the specific implementation of the hybrid retrieval and reordering mechanism is as follows: S3.1. In the local computing environment, the private domain document is parsed and the text is segmented to obtain multiple text blocks; the text blocks are converted into text block vectors using a text vectorization model, and the text blocks and text block vectors are stored in a vector database to build a local private knowledge base. S3.2. Based on the keywords in the question, a hybrid retrieval strategy combining vector retrieval and keyword retrieval is adopted to recall candidate text blocks from the local private knowledge base; wherein, vector retrieval recalls candidate text blocks based on the similarity between the query vector and the text block vector, and keyword retrieval recalls candidate text blocks based on the BM25 algorithm, and the recall results are merged to obtain a candidate context set. S3.3 Calculate the semantic relevance score between the user question and each candidate text block in the candidate context set using the reordering model, and select the candidate text block with the highest score according to the semantic relevance score for subsequent system processing.

[0023] Furthermore, in S4, the execution flow of the large and small model collaborative reasoning framework is as follows: S4.1. Based on the preset keyword extraction prompt template, call the local general large model to perform semantic parsing on the user question, extract the keywords and domain entities in the user question, and use the keywords and domain entities for the hybrid retrieval combining vector retrieval and keyword retrieval in step S3.2; S4.2. Based on the prompt word project, call the local general large model to identify the question type of the user question and determine whether the user question belongs to the domain basic knowledge category. When the user question belongs to the domain basic knowledge category, call the vertical domain small model to directly generate a reply. When the user question does not belong to the domain basic knowledge category, execute the hybrid retrieval and re-ranking mechanism in step S3 to obtain candidate reference context from the local private knowledge base. S4.3. Based on the prompt word, call the local general large model to determine whether the reference context can support answering the user's question; if the reference context can support answering the user's question, input the reference context into the vertical domain small model so that the vertical domain small model generates a response based on the reference context; if the reference context cannot support answering the user's question, generate a prompt that no valid evidence was found and refuse to answer.

[0024] Furthermore, the method also includes: The local vertical domain small model, the local general large model, and the local private knowledge base are deployed in the same physically isolated local computing cluster. All data interactions are completed through the internal local area network, and no data is transmitted with the external Internet. After the local vertical domain small model is trained, its weight file and private knowledge base are physically separated and stored. During the model inference process, only the text content of the private knowledge base is read, and no knowledge base data is modified or updated.

[0025] Beneficial effects: Absolute data privacy and security: Private domain documents are processed entirely in the local computing environment, do not participate in model training, do not update model parameters, and are not transmitted to any third-party platform, thus eliminating the risk of sensitive information leakage at the source.

[0026] Training costs are significantly reduced: By automatically completing document parsing, text segmentation, question-answer pair generation, and quality assessment through a general large model, a large amount of manual intervention is required, which significantly reduces the cost of building training datasets for vertical domains; at the same time, an efficient parameter fine-tuning strategy is adopted, training only about 0.02% of the model's parameters, and the computational resource requirement is as low as less than 1% of the full training, which can be completed on ordinary enterprise-level servers.

[0027] More comprehensive domain knowledge injection: A two-stage fine-tuning strategy is adopted. In the first stage, continuous pre-training is performed using the DoRA method to prioritize the injection of general domain knowledge into the model, which solves the problem that single-stage LoRA fine-tuning only learns the question-answering format but does not master domain knowledge. In the second stage, task adaptation is performed using the LoRA method, enabling the model to have both professional knowledge understanding ability and question-answering task execution ability.

[0028] Significantly improved reasoning accuracy: By combining a hybrid retrieval mechanism of vector retrieval and keyword retrieval with the fine ranking capability of the re-ranking model, relevant context can be obtained more accurately from the private knowledge base; at the same time, a collaborative reasoning framework of large and small models is constructed, which uses the complex semantic understanding capability of the general large model to assist the small model, which not only ensures reasoning speed, but also solves the problem of insufficient complex reasoning capability of the small model, and greatly reduces model illusion.

[0029] High versatility: This method is applicable to all vertical fields with private data needs, such as healthcare, law, finance, and corporate internal management, and can be quickly adapted to intelligent question-answering scenarios in different industries. Attached Figure Description

[0030] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart illustrating the method for constructing a low-resource vertical domain intelligent question-answering system for private domains as described in an embodiment of the present invention; Figure 2 This is a flowchart illustrating the construction of a vertical domain training dataset in the method for building a low-resource vertical domain intelligent question-answering system for private domains as described in this embodiment of the invention. Figure 3 This is a flowchart of the two-stage fine-tuning strategy for training a small vertical domain model in the method for constructing a low-resource vertical domain intelligent question answering system for private domains as described in the embodiments of the present invention. Figure 4 This is a schematic diagram of the hybrid retrieval and re-ranking mechanism in the method for constructing a low-resource vertical domain intelligent question answering system for private domains as described in the embodiments of the present invention. Detailed Implementation

[0031] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0032] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0033] Example 1 See Figure 1-4 A method for constructing a low-resource vertical domain intelligent question answering system for private domains includes the following steps: S1. Construct a training dataset based on publicly available domain documents and non-sensitive domain materials. The training dataset includes a domain corpus dataset for domain knowledge injection and a domain question-answer pair dataset for task adaptation. The construction of the training dataset includes automatic parsing of domain knowledge documents, structured segmentation, multi-model question-answer pair generation, automatic cleaning of multi-rule data, and manual verification. The entire process does not involve any private domain documents, and all data processing can be completed locally. S2. A two-stage parameter efficient fine-tuning strategy is adopted to train a local vertical domain small model. In the first stage, based on the domain corpus dataset, the DoRA method is used to continuously pre-train the local base language model to obtain an intermediate model with basic domain knowledge. In the second stage, based on the domain question-answering pair dataset, the LoRA method is used to supervise and fine-tune the intermediate model to obtain a local vertical domain small model that has both domain knowledge understanding ability and question-answering task execution ability. S3. In a local computing environment physically isolated from external networks, a local private knowledge base is constructed based on private domain documents. Combined with a hybrid retrieval and re-ranking mechanism, candidate contexts and their semantic relevance scores are obtained from the local private knowledge base for user questions. These can be used as a reference for the local vertical domain small model to generate answers. The private domain documents do not participate in the construction of the training dataset and are not used to update any model parameters of the local vertical domain small model. S4. Construct a collaborative reasoning framework for large and small models. Dynamically determine the reasoning strategy based on the complexity of the user's question or the relevance score between the candidate context and the user's question. Prioritize calling the local vertical domain small model with low computational resource consumption to generate answers. Only in complex question scenarios should the local general large model be called to extract keywords in order to retrieve the local private knowledge base and assist in completing the context validity judgment.

[0034] This embodiment forms a complete closed loop: training the model with public data → building a private knowledge base locally → collaborative reasoning between large and small models. S1 provides the training foundation for S2, the small model in the vertical domain trained by S2 provides the core reasoning capability for S4, the private knowledge base built by S3 provides private data support for S4, and the collaborative framework of S4 achieves a balance between performance and cost. In this embodiment, the private data only flows locally and does not participate in any model training process.

[0035] This embodiment addresses four major technical shortcomings of existing technologies: ① risk of private data leakage; ② scarcity of vertical domain training data; ③ insufficient knowledge injection for single-stage fine-tuning; and ④ limited reasoning ability of small models. As a result, it enables the construction of a high-accuracy and high-security vertical domain intelligent question-answering system under conditions of low data and low computing resources.

[0036] In a specific example, see Figure 2 The process of generating the domain corpus dataset and the domain question-answering pair dataset in S1 specifically includes: S1.1 Obtain publicly available domain knowledge documents and non-sensitive domain materials from open academic literature databases, open book resource databases, and open online articles, and parse domain documents in different formats: For text documents in WORD, TXT, and HTML formats, extract the heading levels, body paragraphs, and table content, and convert them into Markdown format documents with hierarchical structure; for PDF format documents, use a deep learning-based multi-model collaborative parsing pipeline to process them, extract the text, formulas, tables, and heading level information, and convert the parsing results into Markdown format documents; S1.2. Using Markdown's special syntax, the Markdown document is divided into text blocks. Each section of the document is divided into a text block. If the content of a section exceeds the preset maximum character length threshold (which can be set to 2000 as needed), it is truncated according to a special symbol (which can be a period or a semicolon). The number of overlapping characters between adjacent text blocks is set to the preset overlap threshold (set to 100). While effectively splitting the domain knowledge document, semantic integrity is preserved. Text blocks with a character count lower than the preset minimum character threshold (set to 100) are filtered to remove fragmented content with insufficient information. Regular expressions are used to delete hyperlinks and noise content unrelated to domain knowledge in the text blocks, resulting in text blocks that have undergone preliminary cleaning and structuring. S1.3: Based on the preliminarily cleaned and structured text blocks obtained in S1.2, prompt word engineering is used to clean the text blocks using a general-purpose large model (such as Google's Gemini), filtering out meaningless content, including garbled text and domain-irrelevant content, thus obtaining the final text blocks. These final text blocks are used to form the domain corpus dataset for the first stage of fine-tuning training, where each text block serves as a training sample. Simultaneously, based on each final text block, prompt word engineering is used to extract questions from it using a general-purpose large model, with several questions extracted from each text block. Based on the current text block and the extracted questions, prompt word engineering is used to generate answers and thought chains using a general-purpose large model, thus forming the domain question-answering pair dataset. Based on the generated question-answering pair dataset, the general-purpose large model is used to evaluate and optimize the answers and thought chains, ultimately generating the domain corpus dataset and domain question-answering pair dataset required for the two-stage fine-tuning.

[0037] In the specific implementation, each text block generates 2 to 3 questions. Then, the current text block and the questions generated based on it are input into a preset answer generation prompt template, and the DeepSeek general-purpose model is called to generate the answers and thought chains corresponding to the questions. After obtaining the initial domain question-answer pair dataset, the Claude general-purpose model is used to evaluate and optimize the quality of the initial domain question-answer pair dataset. For the answers, it is determined whether the answers are too short, inconsistent in facts, or have unfounded expansions. For answers with limited content but supported by text blocks, they are supplemented and expanded based on the effective information in the corresponding text blocks to make the answer content more sufficient, complete, and clear; question-answer pairs that are inconsistent with the text block content, cannot be supported by text blocks, or contain obvious errors are corrected or removed. For the thought chains, meaningless content such as "based on reference content..." appearing in the thought chains due to prompt engineering is deleted, finally obtaining the domain question-answer pair dataset for the second stage of supervised fine-tuning.

[0038] In this embodiment, S1.1 achieves unified parsing and structured conversion of documents in multiple formats; S1.2 ensures the semantic integrity and purity of text blocks through semantic-based text segmentation and multi-rule cleaning; S1.3 automatically completes secondary text cleaning, question-answer pair generation and quality optimization through multi-model collaboration, requiring only a small amount of manual verification to obtain high-quality training data.

[0039] This embodiment significantly reduces the manual cost of constructing the training dataset, solving the problem of scarce and costly manually labeled data in vertical domains. Simultaneously, structured data segmentation and multiple rounds of data cleaning ensure the quality of the training data, laying the foundation for efficient subsequent model training.

[0040] In a specific example, S1.1, the multi-model collaborative parsing pipeline for PDF format documents specifically includes: The DocLayout-YOLO model is used to divide PDF document pages into regions and identify three core layout elements: text, tables, and formulas. The DB text detection algorithm and the CRNN+CTC text recognition algorithm are used to extract the text content of the text region; The YOLOv8 formula detection model and the UniMERNet formula recognition model were used to extract mathematical formulas and convert them into LaTeX format. The SLANet+ / UNet structure recognition model is used to reconstruct the row and column structure and cell content of the table; The text content extracted by text recognition, the mathematical formulas extracted by formula recognition, and the table content restored by table recognition are reorganized according to the heading level of the original document and output as a Markdown format document that preserves the semantic structure.

[0041] This embodiment addresses the challenge of parsing PDF documents by designing a multi-model collaborative parsing pipeline. First, a layout detection model is used to segment different types of content regions. Then, a dedicated model is employed for precise identification of each region. Finally, the content is reconstructed according to the original document's hierarchical structure to fully restore the semantic information of the PDF document.

[0042] This embodiment solves the problem that traditional OCR methods cannot accurately recognize formulas and complex tables in PDF documents. The accuracy of PDF document parsing can be greatly improved, and the professional formulas, table structures and heading levels in the document can be completely preserved, ensuring the integrity and accuracy of the training data.

[0043] In a specific example, see Figure 3 In S2, the weight update method for continuous pre-training using the DoRA method in the first stage is as follows: The pre-trained weight matrix of the local base language model (which can be Qwen3-8B-Base) The nth column is decomposed into two parts: amplitude and direction.

[0044] in Direction matrix The List, Amplitude vector The One element, Represents finding vectors Norm.

[0045] Initialization command , ; Amplitude vector Set as trainable parameters; direction matrix Keeping itself frozen, its updates are achieved by introducing low-rank increments. The complete weight update formula is:

[0046] in , , , Representation matrix The Columns; during initialization, Initialize according to the uniform distribution of Kaiming. Initialize it as a zero matrix to ensure that training starts at the beginning. ,Right now .

[0047] It should be noted that Qwen3-8B-Base contains 36 Transformer decoder layers, each consisting of a self-attention sub-layer and a feedforward neural network sub-layer. The self-attention sub-layer contains the query projection matrix. Key projection matrix Value projection matrix and output projection matrix Four linear transformations; the MLP sublayer contains a gated projection matrix. Ascending projection matrix and descent projection matrix Three linear transformations. The DoRA method is applied to all 36 layers of linear transformations; the weight matrix of the model is denoted as... .

[0048] This embodiment defines the core weight update mechanism of the DoRA method. By decomposing the weight matrix into magnitude and direction, freezing the direction matrix representing general knowledge, and training only the magnitude vector and low-rank increment matrix representing domain knowledge, domain knowledge is efficiently injected without compromising the model's generality.

[0049] Compared to the traditional LoRA method, DoRA can more effectively inject domain knowledge into the model weights, solving the problem that single-stage LoRA fine-tuning only learns the question-answer format without mastering domain knowledge. At the same time, it maintains the advantage of high parameter efficiency, with trainable parameters accounting for a low proportion of the total model parameters.

[0050] In S2, the set of trainable parameters during the training process is as follows: Let the domain corpus dataset be , among which samples For the first A text block, Given the total number of text blocks and a sample of domain-specific corpus data. Given the existing token sequence, the training objective of the model is to maximize the predicted probability of the next token, i.e., minimize the following negative log-likelihood loss:

[0051] in, Indicates the first The first sample One token, for The length of the token sequence, For the model in parameters The conditional probability under the given conditions.

[0052] This embodiment defines the optimization objective for the first stage of continuous pre-training. By minimizing the negative log-likelihood loss of the next token prediction, the model learns the linguistic patterns and knowledge distribution of the domain corpus, thereby mastering the domain's professional terminology, basic concepts, and core logic.

[0053] This embodiment enables the model to quickly grasp fundamental domain knowledge under low data conditions, laying a solid foundation for task adaptation in the second stage. Experiments show that after this stage of training, the model's accuracy in understanding domain-specific terminology improves.

[0054] In S2, the specific process of supervised fine-tuning using the LoRA method in the second stage is as follows: Based on a domain question-answering pair dataset, the LoRA method is used to supervise and fine-tune the intermediate model obtained in the first stage to achieve domain task adaptation. The LoRA method freezes the weights of the intermediate model and achieves efficient parameter fine-tuning by injecting a trainable low-rank factorization matrix. For the weight matrix in the intermediate model... LoRA updates its weights The constraints are in low-rank decomposition form:

[0055] in , , During initialization, Initialize according to the uniform distribution of Kaiming. Initialize it as a zero matrix to ensure that training starts at the beginning. During the training process Keep frozen, the set of trainable parameters is ; Let the domain question-answering pair dataset be... ,in For the first One question, for The corresponding thought chain, for The corresponding answer, The total number of samples; each sample is constructed as an input / output pair in instruction format. ,in A sequence of prompt words containing the question. The target output sequence contains the thought process and the answer; in the dataset When conducting oversight and fine-tuning, for the first Input sequence of samples The forward propagation of the model is as follows:

[0056] in This indicates the output of the specified network layer where the LoRA method was applied. The preset scaling constant, As a scaling factor to reduce rank The need to readjust hyperparameters when they change.

[0057] This embodiment defines the forward propagation mechanism for the second-stage LoRA fine-tuning. By freezing the intermediate model weights obtained from the first-stage training, a new low-rank decomposition matrix is ​​injected to learn the specific patterns of the question-answering task, while a scaling factor ensures the stability of training under different rank parameters.

[0058] This embodiment rapidly optimizes the model's question-answering task performance without compromising the domain knowledge injected in the first stage. It features efficient parameters, fast training speed, and effectively avoids catastrophic forgetting problems.

[0059] In a specific example, in S2, the loss function and final weight merging method for the second-stage supervised fine-tuning are as follows: Given a sequence of input questions, the training objective of the model is to maximize the predicted probability of each token in the thought chain and the answer, i.e., to minimize the following negative log-likelihood loss:

[0060] in Indicates the first The first sample target sequence One token, for The length of the token sequence, For the model in parameters The conditional probability is given below. After the second stage of training is completed, the low-rank matrix will be... With intermediate model weights The final weights are obtained by merging. :

[0061] Weight is The model is a small vertical model that combines domain knowledge and task adaptability.

[0062] This embodiment defines the optimization objective and final model generation method for the second-stage supervised fine-tuning. By minimizing the joint prediction loss of the thought chain and the answer, the model learns to generate logically clear, accurate, and reliable answers based on the question. After training, the low-rank matrix is ​​merged with the original weights, eliminating the need for additional computational overhead in the inference process. This enables the model to simultaneously possess domain knowledge understanding and question-answering task execution capabilities, resulting in improved question-answering accuracy compared to single-stage fine-tuning. After weight merging, the model's inference speed is consistent with the original base model, requiring no additional inference resources.

[0063] It should be noted that since the first and second stages only train the magnitude vector and the low-rank increment matrix, without updating all parameters of the base model or intermediate model, it is possible to complete the training of small vertical models under the condition of limited GPU memory and computing resources.

[0064] In a specific instance, in S3, see [link to S3]. Figure 4 The specific implementation of the hybrid retrieval and re-ranking mechanism is as follows: S3.1. In the local computing environment, the private domain document is parsed and the text is segmented to obtain multiple text blocks; the text blocks are converted into text block vectors using a text vectorization model (bge-large-zh-v1.5 model can be selected), and the text blocks and text block vectors are stored in a vector database to build a local private knowledge base. S3.2. Based on the keywords in the question, a hybrid retrieval strategy combining vector retrieval and keyword retrieval is adopted to recall candidate text blocks from the local private knowledge base; wherein, vector retrieval recalls candidate text blocks based on the similarity between the query vector and the text block vector, and keyword retrieval recalls candidate text blocks based on the BM25 algorithm, and the recall results are merged to obtain a candidate context set. It should be noted that a hybrid retrieval strategy combining vector retrieval and keyword retrieval is employed during the retrieval phase. For vector retrieval, the bge-large-zh-v1.5 model is used to convert specified keywords into dense vectors, and cosine similarity is calculated between these vectors and the text block vectors stored in the vector database to recall the Top-K text blocks with the highest similarity. For keyword retrieval, the BM25 algorithm is used to perform a full-text search on the original text blocks in the vector database, recalling the Top-K text blocks with the highest keyword matching degree. Finally, the union of the two types of retrieval results is taken to obtain the candidate context set.

[0065] S3.3 Calculate the semantic relevance score between the user question and each candidate text block in the candidate context set using the reordering model, and select the candidate text block with the highest score according to the semantic relevance score for subsequent system processing.

[0066] During the re-ranking stage, the bge-reranker-v2-m3 re-ranking model is used to refine the candidate context set obtained in S3.2. The user question and each candidate text block are concatenated and input into the re-ranking model. The semantic relevance score between the two is calculated. The top-N text blocks are selected from high to low scores and used as the reference context for the final input into the vertical domain small model.

[0067] This embodiment defines the retrieval mechanism for the private knowledge base. First, it expands the recall scope by using vector retrieval and keyword retrieval in parallel, taking into account semantic similarity and keyword matching. Then, it refines the recall results through a re-ranking model, selecting the most relevant context for the model to generate answers.

[0068] Compared to a single retrieval method, hybrid retrieval improves recall; the re-ranking mechanism further enhances context relevance, effectively reducing the interference of irrelevant context on the model's generated answers and lowering the model illusion rate.

[0069] In a specific example, the execution flow of the large and small model collaborative reasoning framework in S4 is as follows: S4.1. Based on the preset keyword extraction prompt template, call the local general large model to perform semantic parsing on the user question, extract the keywords and domain entities in the user question, and use the keywords and domain entities for the hybrid retrieval combining vector retrieval and keyword retrieval in step S3.2; S4.2. Based on the prompt word project, call the local general large model to identify the question type of the user question and determine whether the user question belongs to the domain basic knowledge category. When the user question belongs to the domain basic knowledge category, call the vertical domain small model to directly generate a reply. When the user question does not belong to the domain basic knowledge category, execute the hybrid retrieval and re-ranking mechanism in step S3 to obtain candidate reference context from the local private knowledge base. S4.3. Based on the prompt word, call the local general large model to determine whether the reference context can support answering the user's question; if the reference context can support answering the user's question, input the reference context into the vertical domain small model so that the vertical domain small model generates a response based on the reference context; if the reference context cannot support answering the user's question, generate a prompt that no valid evidence was found and refuse to answer.

[0070] This embodiment details the specific process of collaborative reasoning between large and small models. Dynamic selection of the reasoning path is achieved through question type recognition: simple questions are answered quickly by the small model, while complex questions are assisted by the large model in semantic parsing and context verification. Finally, the small model generates the answer based on the valid context.

[0071] This embodiment fully leverages the advantages of small vertical models (fast response speed, low computational consumption) and large general models (strong semantic understanding capabilities). While ensuring answer accuracy, it reduces the average inference latency and establishes a strict answer verification mechanism to avoid generating false information.

[0072] In a specific instance, the method further includes: The local vertical domain small model, the local general large model, and the local private knowledge base are deployed in the same physically isolated local computing cluster. All data interactions are completed through the internal local area network, and no data is transmitted with the external Internet. After the local vertical domain small model is trained, its weight file and private knowledge base are physically separated and stored. During the model inference process, only the text content of the private knowledge base is read, and no knowledge base data is modified or updated.

[0073] This embodiment defines the system's privacy protection mechanism. By deploying in a physically isolated manner, data leakage is prevented at the network level; by physically separating the storage of weights and the knowledge base and setting read-only permissions, the system level is protected against accidental modification or export of private data by the model.

[0074] This embodiment achieves absolute security of private data from the source, completely solving the data leakage risks of traditional cloud-based large model solutions and solutions that require training models with private data, and meeting the compliance requirements of industries with extremely high data privacy requirements such as medical, legal, and financial industries.

[0075] Simulation verification and effect analysis: To verify the effectiveness of the low-resource vertical domain intelligent question-answering system construction method for private domains described in this invention, simulation comparison experiments were conducted in a local computing environment to verify the actual performance of the model fine-tuning scheme and the retrieval scheme, as detailed below: 1. Experimental setup Experimental Environment: The system used was Ubuntu 22.04. The model training program was built using Python 3.11, the PyTorch deep learning framework, and the Transformers library from HuggingFace. The LLaMA-Factory tool was used to implement a two-stage training process: LoRA fine-tuning and DoRA pre-training followed by LoRA fine-tuning. Peak GPU memory usage during training was recorded using the nvidia-smi tool. Training duration was automatically calculated from training logs. Question answering accuracy and domain relevance were obtained through statistical analysis of test set evaluation results.

[0076] Comparison scheme: Two control experiments were set up, one to verify the performance differences of different fine-tuning methods, and the other to verify the effect differences of different retrieval methods.

[0077] 2. Model fine-tuning performance comparison experiment For three schemes—single-stage LoRA fine-tuning, the DoRA pre-training + LoRA two-stage fine-tuning of this invention, and full-parameter fine-tuning—their factual accuracy, domain relevance, peak training memory usage, and training time were tested under the same experimental environment. The experimental results are shown in Table 1 below. Table 1

[0078] Note: In Table 1, OOM indicates that the experimental environment has insufficient video memory, and the model cannot complete training.

[0079] In the specific training process, a small-scale pre-training test was first conducted, and the model training status was observed through training logs and memory monitoring results. In the initial experiments, the rank settings for DoRA pre-training and LoRA fine-tuning were relatively small. When the memory usage during training was lower than the upper limit of the available memory on the device, or when the model's accuracy on the validation set was low, it indicated that the model's domain knowledge adaptation ability was insufficient. In subsequent experiments, the rank value was appropriately increased.

[0080] 3. Search performance comparison experiment In the retrieval validation experiment, the recall effects of BM25 keyword retrieval and vector retrieval were first tested separately. Based on the test results, the text block length, overlap length, and the number of candidate texts recalled were adjusted. After multiple rounds of testing, the results of vector retrieval and BM25 keyword retrieval were merged and deduplicated to form a hybrid retrieval candidate set. A re-ranking model was then added to rank the candidate texts by relevance. Finally, the Top-5 recall rate, Top-10 recall rate, and final answer accuracy for the four methods—BM25 keyword retrieval, vector retrieval, hybrid retrieval, and hybrid retrieval + re-ranking—were statistically analyzed. The experimental results are shown in Table 2 below. Table 2

[0081] 4. Analysis of Experimental Results The above experimental data demonstrates that the present invention possesses the following technical effects: Significantly improved domain knowledge injection effect: The two-stage scheme of DoRA pre-training + LoRA fine-tuning adopted in this invention improves the fact accuracy from 72.3% to 83.9% and the domain relevance from 80.1% to 86.1% compared with the single-stage LoRA fine-tuning. This proves that the two-stage step-by-step training can more fully inject domain knowledge into the model and solve the defects of single-stage fine-tuning that only learns the question-answer format and has insufficient knowledge injection.

[0082] The computational resource requirements are significantly reduced: full parameter fine-tuning cannot run in an environment with 24GB of video memory, while the peak training video memory of the two-stage fine-tuning of this invention is only 23.2GB. Training can be completed with a single consumer-grade GPU, and the training time is only 8.3 hours. This greatly reduces the training computing power threshold for vertical domain models and is suitable for local private deployment scenarios with low resources.

[0083] Simultaneous improvement in retrieval accuracy and question-answering quality: The hybrid retrieval scheme combining vector retrieval and BM25 improved the Top-5 recall rate by 16.3 percentage points compared to single keyword retrieval and by 7.3 percentage points compared to single vector retrieval; after adding re-ranking and fine-tuning, the final answer accuracy was further improved to 85.3%, proving that the hybrid retrieval and re-ranking mechanism can effectively improve the accuracy of contextual recall and reduce model illusion.

[0084] Full-process adaptation to private scenarios: All training and inference processes can be completed in a local, physically isolated environment. Private data is only used to build a local knowledge base, does not participate in model training, and is not transmitted externally, thus ensuring data privacy and security from a technical architecture perspective.

[0085] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for constructing a low-resource vertical domain intelligent question-answering system for private domains, characterized in that, Includes the following steps: S1. Construct a training dataset based on publicly available domain documents and non-sensitive domain materials. The training dataset includes a domain corpus dataset for domain knowledge injection and a domain question-answer pair dataset for task adaptation. The construction of the training dataset includes automatic parsing of domain knowledge documents, structured segmentation, multi-model question-answer pair generation, automatic cleaning of multi-rule data, and manual verification. The entire process does not involve any private domain documents, and all data processing can be completed locally. S2. A two-stage parameter efficient fine-tuning strategy is adopted to train a local vertical domain small model. In the first stage, based on the domain corpus dataset, the DoRA method is used to continuously pre-train the local base language model to obtain an intermediate model with basic domain knowledge. In the second stage, based on the domain question-answering pair dataset, the LoRA method is used to supervise and fine-tune the intermediate model to obtain a local vertical domain small model that has both domain knowledge understanding ability and question-answering task execution ability. S3. In a local computing environment physically isolated from external networks, a local private knowledge base is constructed based on private domain documents. Combined with a hybrid retrieval and re-ranking mechanism, candidate contexts and their semantic relevance scores are obtained from the local private knowledge base for user questions. These can be used as a reference for the local vertical domain small model to generate answers. The private domain documents do not participate in the construction of the training dataset and are not used to update any model parameters of the local vertical domain small model. S4. Construct a collaborative reasoning framework for large and small models. Dynamically determine the reasoning strategy based on the complexity of the user's question or the relevance score between the candidate context and the user's question. Prioritize calling the local vertical domain small model with low computational resource consumption to generate answers. Only call the local general large model for keyword extraction in complex question scenarios to facilitate the retrieval of the local private knowledge base and assist in completing the context validity judgment. In S4, the execution flow of the large and small model collaborative reasoning framework is as follows: S4.

1. Based on the preset keyword extraction prompt template, call the local general large model to perform semantic parsing on the user question, extract the keywords and domain entities in the user question, and use the keywords and domain entities for a hybrid retrieval that combines vector retrieval and keyword retrieval. S4.

2. Based on the prompt word project, call the local general large model to identify the question type of the user question and determine whether the user question belongs to the domain basic knowledge category. When the user question belongs to the domain basic knowledge category, call the vertical domain small model to directly generate a reply. When the user question does not belong to the domain basic knowledge category, execute the hybrid retrieval and re-ranking mechanism to obtain candidate reference context from the local private knowledge base. S4.

3. Based on the prompt word, call the local general large model to determine whether the reference context can support answering the user's question; if the reference context can support answering the user's question, input the reference context into the vertical domain small model so that the vertical domain small model generates a response based on the reference context; if the reference context cannot support answering the user's question, generate a prompt that no valid evidence was found and refuse to answer.

2. The method for constructing a low-resource vertical domain intelligent question-answering system for private domains according to claim 1, characterized in that, In step S1, the process of generating the domain corpus dataset and the domain question-answer pair dataset specifically includes: S1.1 Obtain publicly available domain knowledge documents and non-sensitive domain materials from open academic literature databases, open book resource databases, and open online articles, and parse domain documents in different formats: For text documents in WORD, TXT, and HTML formats, extract the heading levels, body paragraphs, and table content, and convert them into Markdown format documents with hierarchical structure; for PDF format documents, use a deep learning-based multi-model collaborative parsing pipeline to process them, extract the text, formulas, tables, and heading level information, and convert the parsing results into Markdown format documents; S1.

2. Using Markdown's special syntax, the Markdown document is divided into text blocks. Each section of the document is divided into a text block. If the content of a section exceeds the preset maximum character length threshold, it is truncated according to special symbols. The number of overlapping characters between adjacent text blocks is set to the preset overlap threshold. While effectively splitting the domain knowledge document, semantic integrity is preserved. Text blocks with a character count lower than the preset minimum character threshold are filtered to remove fragmented content with insufficient information. Regular expressions are used to delete hyperlinks and noise content unrelated to domain knowledge in the text blocks, resulting in text blocks that have undergone preliminary cleaning and structuring. S1.

3. Based on the preliminary cleaning and structuring of the text blocks obtained in S1.2, prompt word engineering is used to clean the text blocks using a general large model, filtering out meaningless content, including garbled text and content irrelevant to the domain, thus obtaining the final text blocks. These final text blocks are used to form the domain corpus dataset for the first stage of fine-tuning training, where each text block serves as a training sample. Simultaneously, based on each final text block, prompt word engineering is used to extract questions using a general large model, with several questions extracted from each text block. Based on the current text block and the extracted questions, prompt word engineering is used to generate answers and thought chains using a general large model, thus forming the domain question-answering pair dataset. Based on the generated question-answering pair dataset, the general large model is used to evaluate and optimize the answers and thought chains, ultimately generating the domain corpus dataset and domain question-answering pair dataset required for the two-stage fine-tuning.

3. The method for constructing a low-resource vertical domain intelligent question-answering system for private domains according to claim 2, characterized in that, In S1.1, the multi-model collaborative parsing pipeline for PDF format documents specifically includes: The DocLayout-YOLO model is used to divide PDF document pages into regions and identify three core layout elements: text, tables, and formulas. The DB text detection algorithm and the CRNN+CTC text recognition algorithm are used to extract the text content of the text region; The YOLOv8 formula detection model and the UniMERNet formula recognition model were used to extract mathematical formulas and convert them into LaTeX format. The SLANet+ / UNet structure recognition model is used to reconstruct the row and column structure and cell content of the table; The text content extracted by text recognition, the mathematical formulas extracted by formula recognition, and the table content restored by table recognition are reorganized according to the heading level of the original document and output as a Markdown format document that preserves the semantic structure.

4. The method for constructing a low-resource vertical domain intelligent question-answering system for private domains according to claim 1, characterized in that, In S2, the weight update method for continuous pre-training using the DoRA method in the first stage is as follows: The pre-trained weight matrix of the local base language model The nth column is decomposed into two parts: amplitude and direction. in Direction matrix The List, Amplitude vector The One element, Represents finding vectors Norm; Initialization command , ; Amplitude vector Set as trainable parameters; direction matrix Keeping itself frozen, its updates are achieved by introducing low-rank increments. The complete weight update formula is: in , , , Representation matrix The Columns; during initialization, Initialize according to the uniform distribution of Kaiming. Initialize it as a zero matrix to ensure that training starts at the beginning. ,Right now .

5. The method for constructing a low-resource vertical domain intelligent question-answering system for private domains according to claim 4, characterized in that, In S2, the set of trainable parameters during the training process is as follows: Let the domain corpus dataset be , among which samples For the first A text block, Given the total number of text blocks and a sample of domain-specific corpus data. Given the existing token sequence, the training objective of the model is to maximize the predicted probability of the next token, i.e., minimize the following negative log-likelihood loss: in, Indicates the first The first sample One token, for The length of the token sequence, For the model in parameters The conditional probability under the given conditions.

6. The method for constructing a low-resource vertical domain intelligent question-answering system for private domains according to claim 1, characterized in that, In S2, the specific process of supervised fine-tuning using the LoRA method in the second stage is as follows: Based on a domain question-answering pair dataset, the LoRA method is used to supervise and fine-tune the intermediate model obtained in the first stage to achieve domain task adaptation. The LoRA method freezes the weights of the intermediate model and achieves efficient parameter fine-tuning by injecting a trainable low-rank factorization matrix. For the weight matrix in the intermediate model... LoRA updates its weights The constraints are in low-rank decomposition form: in , , During initialization, Initialize according to the uniform distribution of Kaiming. Initialize it as a zero matrix to ensure that training starts at the beginning. During the training process Keep frozen, the set of trainable parameters is ; Let the domain question-answering pair dataset be... ,in For the first One question, for The corresponding thought chain, for The corresponding answer, The total number of samples; each sample is constructed as an input / output pair in instruction format. ,in A sequence of prompt words containing the question. The target output sequence contains the thought process and the answer; in the dataset When conducting oversight and fine-tuning, for the first Input sequence of samples The forward propagation of the model is as follows: in This indicates the output of the specified network layer where the LoRA method was applied. The preset scaling constant, As a scaling factor to reduce rank The need to readjust hyperparameters when they change.

7. The method for constructing a low-resource vertical domain intelligent question-answering system for private domains according to claim 6, characterized in that, In S2, the loss function and final weight merging method for the second-stage supervised fine-tuning are as follows: Given a sequence of input questions, the training objective of the model is to maximize the predicted probability of each token in the thought chain and the answer, i.e., to minimize the following negative log-likelihood loss: in Indicates the first The first sample target sequence One token, for The length of the token sequence, For the model in parameters The conditional probability is given below. After the second stage of training is completed, the low-rank matrix will be... With intermediate model weights The final weights are obtained by merging. : Weight is The model is a small vertical model that combines domain knowledge and task adaptability.

8. The method for constructing a low-resource vertical domain intelligent question-answering system for private domains according to claim 1, characterized in that, In S3, the specific implementation of the hybrid retrieval and reordering mechanism is as follows: S3.

1. In the local computing environment, the private domain document is parsed and the text is segmented to obtain multiple text blocks; the text blocks are converted into text block vectors using a text vectorization model, and the text blocks and text block vectors are stored in a vector database to build a local private knowledge base. S3.

2. Based on the keywords in the question, a hybrid retrieval strategy combining vector retrieval and keyword retrieval is adopted to recall candidate text blocks from the local private knowledge base; wherein, vector retrieval recalls candidate text blocks based on the similarity between the query vector and the text block vector, and keyword retrieval recalls candidate text blocks based on the BM25 algorithm, and the recall results are merged to obtain a candidate context set. S3.3 Calculate the semantic relevance score between the user question and each candidate text block in the candidate context set using the reordering model, and select the candidate text block with the highest score according to the semantic relevance score for subsequent system processing.

9. The method for constructing a low-resource vertical domain intelligent question-answering system for private domains according to claim 1, characterized in that, The method further includes: The local vertical domain small model, the local general large model, and the local private knowledge base are deployed in the same physically isolated local computing cluster. All data interactions are completed through the internal local area network, and no data is transmitted with the external Internet. After the local vertical domain small model is trained, its weight file and private knowledge base are physically separated and stored. During the model inference process, only the text content of the private knowledge base is read, and no knowledge base data is modified or updated.

Images