Large language model training method and system adaptable to vertical domain knowledge
By parallelizing domain information processing units and fusion attention mechanisms in a large language model, and combining them with a semi-frozen training strategy, the problem of low adaptation efficiency of general models in aero-engine fault repair is solved. This achieves high-precision, low-latency model adaptation, meeting the complex task requirements of aero-engine fault repair scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING INST OF TECH
- Filing Date
- 2025-12-01
- Publication Date
- 2026-05-12
AI Technical Summary
General-purpose large language models lack specific knowledge in specific vertical fields such as aircraft engine fault repair, resulting in low adaptation efficiency, insufficient ability to parse complex tasks, and limited real-time response and deployability.
By connecting a domain information processing unit in parallel outside the backbone of a large language model, and combining fusion attention mechanism, semi-frozen training strategy, chained reasoning enhancement, few-shot learning and transfer optimization, and lightweight distillation and quantization, rapid adaptation of aviation domain knowledge is achieved.
It achieves high-precision, interpretable, and low-latency model adaptation under low computing power conditions, improving training efficiency and adaptation costs, and meeting the complex task requirements of aero-engine fault repair scenarios.
Smart Images
Figure CN122021867A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a method and system for training large language models that can be adapted to vertical domain knowledge, belonging to the field of artificial intelligence and natural language processing technology, and applied to the scenario of aircraft engine fault repair. Background Technology
[0002] In recent years, Large Language Models (LLMs) have achieved remarkable results in general natural language understanding, text generation, question answering systems, and intelligent search. However, when faced with specific vertical domains, general-purpose large language models often suffer from limitations due to a lack of targeted knowledge and reasoning patterns, mainly exhibiting the following problems:
[0003] (1) Low efficiency of domain knowledge adaptation: When introducing specific domain knowledge, a large amount of labeled data and repeated training are required, resulting in a long adaptation cycle and high cost;
[0004] (2) Insufficient ability to analyze complex tasks: When dealing with complex tasks such as multi-round reasoning, long-link logical relationships, and multimodal data fusion, the existing models are not stable, interpretable, and have insufficient execution accuracy.
[0005] (3) Limited real-time response and deployability: Some application scenarios have high requirements for response latency, operating resources and edge deployment capabilities, while existing large models are difficult to implement directly under these conditions.
[0006] Therefore, improving the efficiency of adapting knowledge of engines in the aviation field has become an urgent problem to be solved. Summary of the Invention
[0007] The purpose of this invention is to address the technical problem of improving the efficiency of adapting aerospace engine knowledge in aero-engine fault repair scenarios. It proposes a large language model training method and system adaptable to vertical domain knowledge. This invention achieves rapid adaptation to any vertical domain by connecting a domain information processing unit in parallel outside the main body of the large language model, and employing techniques such as fusion attention mechanisms, semi-frozen training strategies, chained reasoning enhancement, few-shot learning and transfer optimization, and lightweight distillation and quantization. This results in a high-precision, interpretable, low-latency, and deployable large language model. This invention considers the needs of both the training and deployment phases and can operate stably under relatively low computing power conditions.
[0008] The objective of this invention is achieved through the following technical solution:
[0009] This invention discloses a method for training a large language model that can adapt to vertical domain knowledge, comprising the following steps:
[0010] Step 1: Perform data preprocessing on the multi-source corpus to form formatted data. Based on the formatted data, extract high-frequency terms, abbreviations, and special symbols in the domain and obtain an extended word segmenter that encodes domain concepts. At the same time, construct a task sample set and an error path comparison sample set that are expanded with rule templates and generative methods.
[0011] Step 1.1: Collect multi-source corpora from the target vertical domain, perform data preprocessing, and obtain formatted data for training;
[0012] Step 1.2: Use regular expressions and LLM models to extract high-frequency domain terms, abbreviations and special symbols from the formatted data. Inject terms with incomplete word segmentation or semantic errors in the model into the word segmenter vocabulary to obtain an extended word segmenter for domain concept encoding.
[0013] Step 1.3: Construct a task sample set and an error path comparison sample set, and expand the sample set using rule templates and generative methods;
[0014] Step 1.3.1: When sufficient data is available, construct a task sample set and an error path comparison sample set;
[0015] Step 1.3.2: When data is insufficient, expand the sample set using rule templates and generative methods;
[0016] Step 2: Construct a model backbone network with the domain information processing unit integrated into the decoder. After weighted normalization and fusion of the basic attention vector obtained by the attention mechanism of the backbone network and the domain latent vector obtained by the domain information processing unit through the bottleneck layer, the output is obtained through a multi-layer neural network.
[0017] Step 2.1: Use the Decoder-Only Transformer model as the backbone network;
[0018] Step 2.2: Connect the Domain Information Processing Unit (DIPU) to the backbone network in parallel to form a domain vertical network;
[0019] Step 2.3: Input the task sample set and the error path comparison sample set into the extended word segmenter of the domain concept encoding at the same time to obtain the latent vector;
[0020] Step 2.4: Input the latent vectors into the domain vertical network at the same time. The backbone network uses the attention mechanism to transform the latent vectors into basic attention vectors. The domain information processing unit generates domain latent vectors with query-key-value form through the bottleneck layer.
[0021] Step 2.5: Perform head-by-head weighted normalization and fusion of the basic attention vector and the domain latent vector to form a fused attention vector;
[0022] Step 2.6: Obtain the output by passing the fused attention vector through a multi-layer neural network;
[0023] Step 3: Using terminology samples and multi-step logical reasoning, conditional branching, and long contextual dependencies, the domain vertical network is trained by shallow freezing and deep unfreezing in sequence, and cross-entropy loss is used for network optimization; further, meta-learning and cross-task transfer mechanisms are used to improve the task adaptability of the domain vertical network in the aero-engine fault repair scenario.
[0024] Step 3.1: When training the domain vertical network, freeze the shallow Transformer Blocks in the domain vertical network and unfreeze the deep Transformer Blocks, domain information processing units and output layers in the domain vertical network.
[0025] Step 3.1.1: Perform partitioned management of the parameters of the domain vertical network; treat the first half of the Transformer Block in the domain vertical network as a shallow layer; treat the second half of the Transformer Block in the domain vertical network as a deep layer;
[0026] Step 3.1.2: Freeze the shallow Transformer Blocks in the domain vertical network;
[0027] Step 3.1.3: Unfreeze the deep Transformer Block, domain information processing unit, and output layer in the domain vertical network;
[0028] Step 3.2: Train the domain vertical network sequentially using terminology samples and multi-step logical reasoning, conditional branching, and long contextual dependencies, and further optimize the domain vertical network using cross-entropy loss;
[0029] Step 3.3: Employ meta-learning and cross-task transfer mechanisms to improve the task adaptability of domain-specific vertical networks in aero-engine fault repair scenarios;
[0030] Step 4: Input the test set into the trained domain vertical network to obtain evaluation accuracy, inference chain consistency, and interpretability metrics;
[0031] This invention discloses a large language model training system adaptable to vertical domain knowledge, used to implement the above-mentioned method. The large language model training system adaptable to vertical domain knowledge disclosed in this invention includes a domain-specific vertical network module and a training control module;
[0032] The domain vertical network module, consisting of a vocabulary expansion module, a domain information processing module, and a fusion attention module, is used to construct a model for aero-engine fault detection and will serve as the input to the training control module.
[0033] Furthermore, the vocabulary expansion module, used to expand the vocabulary and retrain the embedding layer to incorporate domain terms, will serve as input to the domain information processing module.
[0034] Furthermore, the domain information processing module is used to connect a bottleneck structure in parallel outside the Transformer backbone and perform multi-subspace projection to generate a domain-enhanced representation; this will serve as the input to the fusion attention module.
[0035] Furthermore, the fusion attention module is used to fuse the backbone attention result with the attention distribution of the domain enhancement representation to obtain a fusion attention vector;
[0036] The training control module is used for semi-frozen training of the domain vertical network and for optimizing the domain vertical network using meta-learning and transfer.
[0037] Compared with existing technologies, it has the following beneficial effects:
[0038] 1. This invention achieves the integration of general capabilities and domain knowledge through a domain information processing unit and a fusion attention mechanism; at the same time, it adopts a semi-frozen training strategy to significantly reduce adaptation costs and improve training efficiency.
[0039] 2. This invention introduces a chain-based reasoning enhancement mechanism to improve the interpretability and reasoning depth of the model in complex tasks; at the same time, it combines few-shot learning and transfer optimization to enable the model to still have high generalization performance in low-resource scenarios.
[0040] 3. This invention meets the deployment requirements in low-computing-power environments through knowledge distillation and quantification optimization, and has a wide range of applications. Attached Figure Description
[0041] Figure 1 A diagram illustrating the architecture of a large language model system that integrates domain information processing units;
[0042] Figure 2 This is a training strategy diagram for the large language model training method proposed in this invention. Detailed Implementation
[0043] To better illustrate the purpose and advantages of this invention, the invention will be further described below with reference to the accompanying drawings and examples. It should be noted that the implementation of this invention is not limited to the following embodiments, and any modifications or alterations made to this invention will fall within the scope of protection of this invention.
[0044] Example
[0045] like Figure 1 As shown in the figure, the specific implementation steps of a large language model training method adaptable to vertical domain knowledge in this embodiment are as follows:
[0046] Step 1: Perform data preprocessing on the multi-source corpus to form formatted data. Based on the formatted data, extract high-frequency terms, abbreviations, and special symbols in the domain and obtain an extended word segmenter that encodes domain concepts. At the same time, construct a task sample set and an error path comparison sample set that are expanded with rule templates and generative methods.
[0047] Step 1.1: Collect multi-source corpora from the target vertical domain, perform data preprocessing, and obtain formatted data for training;
[0048] In this embodiment, domain corpus collection and cleaning involves collecting multi-source corpora (standard specifications, maintenance work orders, logs and cases, expert Q&A, terminology lists, etc.) of the target vertical domain (including but not limited to aircraft engine fault repair), and performing deduplication, sentence segmentation, format standardization, abbreviation and symbol standardization, and sensitive information desensitization processing.
[0049] Step 1.2: Use regular expressions and LLM models to extract high-frequency domain terms, abbreviations and special symbols from the formatted data. Inject terms with incomplete word segmentation or semantic errors in the model into the word segmenter vocabulary to obtain an extended word segmenter for domain concept encoding.
[0050] In this embodiment, high-frequency terms, abbreviations, and special symbols in the domain are mined. For terms that are not fully segmented or have incorrect semantic segmentation in the general model, the tokenizer vocabulary is expanded to improve the accuracy of identification and encoding of core domain concepts in the input stage.
[0051] Step 1.3: Construct a task sample set and an error path comparison sample set, and expand the sample set using rule templates and generative methods;
[0052] Step 1.3.1: When sufficient data is available, construct a task sample set and an error path comparison sample set;
[0053] Step 1.3.2: When data is insufficient, expand the sample set using rule templates and generative methods;
[0054] In this embodiment, a task sample set is constructed, including instruction following, question answering or diagnosis, retrieval enhancement, and chain reasoning (CoT) samples; at the same time, error path comparison samples are constructed to explicitly record the multi-step reasoning path from question to answer to support reasoning enhancement training; in scenarios with insufficient data, rule / template / generative methods are used to expand the samples; the samples are divided according to the ratio of training set / validation set / test set to ensure consistent distribution and task coverage.
[0055] Step 2: Construct a model backbone network with the domain information processing unit integrated into the decoder. After weighted normalization and fusion of the basic attention vector obtained by the attention mechanism of the backbone network and the domain latent vector obtained by the domain information processing unit through the bottleneck layer, the output is obtained through a multi-layer neural network.
[0056] Step 2.1: Use the Decoder-Only Transformer model as the backbone network;
[0057] Step 2.2: Connect the Domain Information Processing Unit (DIPU) to the backbone network in parallel to form a domain vertical network;
[0058] Step 2.3: Input the task sample set and the error path comparison sample set into the extended word segmenter of the domain concept encoding at the same time to obtain the latent vector;
[0059] Step 2.4: Input the latent vectors into the domain vertical network at the same time. The backbone network uses the attention mechanism to transform the latent vectors into basic attention vectors. The domain information processing unit generates domain latent vectors with query-key-value form through the bottleneck layer.
[0060] Step 2.5: Perform head-by-head weighted normalization and fusion of the basic attention vector and the domain latent vector to form a fused attention vector;
[0061] Step 2.6: Obtain the output by passing the fused attention vector through a multi-layer neural network;
[0062] In the embodiments, such as Figure 2 As shown, a general Decoder-Only Transformer is used as the backbone network, retaining general language capabilities and long context processing capabilities. For the parallel Domain Information Processing Unit (DIPU), a new domain information processing unit is added in a bypass parallel manner outside the backbone, employing a lightweight bottleneck structure. Internally, it contains multiple layers of nonlinear activation and three sets of projection matrices to generate domain latent vectors in query (Q), key (K), and value (V) formats. First, bottleneck dimensionality reduction is performed to reduce computational cost, and then multi-subspace projection is used to capture domain semantic features from different perspectives. This unit maintains non-intrusive coupling with the backbone path. Within each domain subspace, the local attention distribution of the backbone attention output and the DIPU enhanced representation are calculated separately, then weighted and fused head-by-head and uniformly normalized to solve the feature alignment and weight allocation problems of the two information streams, improving the attention accuracy and coverage of domain-related segments. The output layer is configured according to the target task to ensure the model is directly applicable to the application scenario.
[0063] Step 3: Using terminology samples and multi-step logical reasoning, conditional branching, and long contextual dependencies, the domain vertical network is trained by shallow freezing and deep unfreezing in sequence, and cross-entropy loss is used for network optimization; further, meta-learning and cross-task transfer mechanisms are used to improve the task adaptability of the domain vertical network in the aero-engine fault repair scenario.
[0064] Step 3.1: When training the domain vertical network, freeze the shallow Transformer Blocks in the domain vertical network and unfreeze the deep Transformer Blocks, domain information processing units and output layers in the domain vertical network.
[0065] Step 3.1.1: Perform partitioned management of the parameters of the domain vertical network; treat the first half of the Transformer Block in the domain vertical network as a shallow layer; treat the second half of the Transformer Block in the domain vertical network as a deep layer;
[0066] Step 3.1.2: Freeze the shallow Transformer Blocks in the domain vertical network;
[0067] Step 3.1.3: Unfreeze the deep Transformer Block, domain information processing unit, and output layer in the domain vertical network;
[0068] Step 3.2: Train the domain vertical network sequentially using terminology samples and multi-step logical reasoning, conditional branching, and long contextual dependencies, and further optimize the domain vertical network using cross-entropy loss;
[0069] Step 3.3: Employ meta-learning and cross-task transfer mechanisms to improve the task adaptability of domain-specific vertical networks in aero-engine fault repair scenarios;
[0070] In this embodiment, parameters are managed by partitioning; the embedding layer and part of the shallow Transformer Block are frozen to retain general capabilities, while the deep structure, DIPU, and output layer are unfrozen, allowing them to be updated under domain data-driven conditions for efficient adaptation; starting with basic terminology and common task samples, the training gradually transitions to complex tasks involving multi-step logical reasoning, conditional branching, and long context dependencies, reducing training difficulty and improving convergence stability. Meta-learning and cross-task transfer mechanisms are employed to achieve rapid adaptation in data-scarce scenarios; and synthetic data augmentation is combined to improve generalization performance for unseen tasks.
[0071] Step 4: Input the test set into the trained domain vertical network to obtain evaluation accuracy, inference chain consistency, and interpretability metrics;
[0072] In this embodiment, accuracy, inference chain consistency, and interpretability metrics are evaluated on the test set, while resource metrics such as latency, throughput, and GPU memory usage are recorded.
[0073] This embodiment provides a large language model training system adaptable to vertical domain knowledge, used to implement the above method. This large language model training system adaptable to vertical domain knowledge includes a domain-specific vertical network module and a training control module.
[0074] The domain vertical network module, consisting of a vocabulary expansion module, a domain information processing module, and a fusion attention module, is used to construct a model for aero-engine fault detection and will serve as the input to the training control module.
[0075] Furthermore, the vocabulary expansion module, used to expand the vocabulary and retrain the embedding layer to incorporate domain terms, will serve as input to the domain information processing module.
[0076] Furthermore, the domain information processing module is used to connect a bottleneck structure in parallel outside the Transformer backbone and perform multi-subspace projection to generate a domain-enhanced representation; this will serve as the input to the fusion attention module.
[0077] Furthermore, the fusion attention module is used to fuse the backbone attention result with the attention distribution of the domain enhancement representation to obtain a fusion attention vector;
[0078] The training control module is used for semi-frozen training of the domain vertical network and for optimizing the domain vertical network using meta-learning and transfer.
[0079] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for training a large language model adaptable to vertical domain knowledge, characterized in that: Includes the following steps, Step 1: Perform data preprocessing on the multi-source corpus to form formatted data. Based on the formatted data, extract high-frequency terms, abbreviations, and special symbols in the domain and obtain an extended word segmenter that encodes domain concepts. At the same time, construct a task sample set and an error path comparison sample set that are expanded with rule templates and generative methods. Step 2: Construct a model backbone network with the domain information processing unit integrated into the decoder. After weighted normalization and fusion of the basic attention vector obtained by the attention mechanism of the backbone network and the domain latent vector obtained by the domain information processing unit through the bottleneck layer, the output is obtained through a multi-layer neural network. Step 2.1: Use the Decoder-Only Transformer model as the backbone network; Step 2.2: Connect the Domain Information Processing Unit (DIPU) to the backbone network in parallel to form a domain vertical network; Step 2.3: Input the task sample set and the error path comparison sample set into the extended word segmenter of the domain concept encoding at the same time to obtain the latent vector; Step 2.4: Input the latent vectors into the domain vertical network at the same time. The backbone network uses the attention mechanism to transform the latent vectors into basic attention vectors. The domain information processing unit generates domain latent vectors with query-key-value form through the bottleneck layer. Step 2.5: Perform head-by-head weighted normalization and fusion of the basic attention vector and the domain latent vector to form a fused attention vector; Step 2.6: Obtain the output by passing the fused attention vector through a multi-layer neural network; Step 3: Using terminology samples and multi-step logical reasoning, conditional branching, and long contextual dependencies, the domain vertical network is trained by shallow freezing and deep unfreezing in sequence, and cross-entropy loss is used for network optimization; further, meta-learning and cross-task transfer mechanisms are used to improve the task adaptability of the domain vertical network in the aero-engine fault repair scenario. Step 4: Input the test set into the trained domain vertical network to obtain evaluation metrics such as accuracy, inference chain consistency, and interpretability.
2. The method for training a large language model adaptable to vertical domain knowledge as described in claim 1, characterized in that: Step 1 is implemented as follows: Step 1.1: Collect multi-source corpora from the target vertical domain, perform data preprocessing, and obtain formatted data for training; Step 1.2: Use regular expressions and LLM models to extract high-frequency domain terms, abbreviations and special symbols from the formatted data. Inject terms with incomplete word segmentation or semantic errors in the model into the word segmenter vocabulary to obtain an extended word segmenter for domain concept encoding. Step 1.3: Construct a task sample set and an error path comparison sample set, and expand the sample set using rule templates and generative methods.
3. The method for training a large language model adaptable to vertical domain knowledge as described in claim 2, characterized in that: Step 1.3 is implemented as follows: Step 1.3.1: When sufficient data is available, construct a task sample set and an error path comparison sample set; Step 1.3.2: When data is insufficient, the sample set is expanded using rule templates and generative methods.
4. The method for training a large language model adaptable to vertical domain knowledge as described in claim 1, characterized in that: Step 3 is implemented as follows: Step 3.1: When training the domain vertical network, freeze the shallow Transformer Blocks in the domain vertical network and unfreeze the deep Transformer Blocks, domain information processing units and output layers in the domain vertical network. Step 3.2: Train the domain vertical network sequentially using terminology samples and multi-step logical reasoning, conditional branching, and long contextual dependencies, and further optimize the domain vertical network using cross-entropy loss; Step 3.3: Employ meta-learning and cross-task transfer mechanisms to improve the task adaptability of domain vertical networks in aero-engine fault repair scenarios.
5. The method for training a large language model adaptable to vertical domain knowledge as described in claim 1, characterized in that: Step 3.1 is implemented as follows: Step 3.1.1: Perform partitioned management of the parameters of the domain vertical network; treat the first half of the Transformer Block in the domain vertical network as a shallow layer; treat the second half of the Transformer Block in the domain vertical network as a deep layer; Step 3.1.2: Freeze the shallow Transformer Blocks in the domain vertical network; Step 3.1.3: Unfreeze the deep Transformer Block, domain information processing unit and output layer in the domain vertical network.
6. A large language model training system adaptable to vertical domain knowledge, implementing the method described in claim 1, characterized in that: Includes a domain-specific vertical network module and a training control module; The domain vertical network module, consisting of a vocabulary expansion module, a domain information processing module, and a fusion attention module, is used to construct a model for aero-engine fault detection and will serve as the input to the training control module. The training control module is used for semi-frozen training of the domain vertical network and for optimizing the domain vertical network using meta-learning and transfer.
7. The large language model training system adaptable to vertical domain knowledge as described in claim 6, characterized in that: The vocabulary expansion module is used to expand the vocabulary and retrain the embedding layer to incorporate domain terms; it will serve as input to the domain information processing module. The domain information processing module is used to connect a bottleneck structure in parallel outside the Transformer backbone and perform multi-subspace projection to generate a domain-enhanced representation; it will serve as the input to the fusion attention module. The fusion attention module is used to fuse the attention distribution of the backbone attention result and the domain-enhanced representation; Obtain the fused attention vector.