Vertical domain large model training system and method based on incremental proportioning and progressive corpus set

Abstract

This application relates to a training system and method for a large-scale vertical domain model based on incremental matching and progressive corpus. The method includes: preprocessing raw data from different sources to generate a corpus, which includes general corpus and target domain corpus; sampling the corpus based on an incremental matching algorithm to generate multiple incremental training corpus sets; performing a first-stage training on a general-based large-scale model using the multiple incremental training corpus sets, and generating a vertical-domain-based large-scale model through a multi-dimensional loss monitoring mechanism; expanding the corpus with multi-domain corpora of different lengths to generate a progressive corpus; and performing a second-stage training on the vertical-domain-based large-scale model using the progressive corpus combined with an annealing optimization strategy to generate a long-context vertical-domain large-scale model. This application can achieve smooth fusion of multi-source knowledge, significantly improving the domain adaptability, long text understanding ability, and training stability of large language models.

Description

Technical Field

[0001] This application relates to the field of computer information processing, and more specifically, to a vertical domain large model training system and method based on incremental matching and progressive corpus. Background Technology

[0002] As large language models demonstrate powerful capabilities in general tasks, how to efficiently adapt them to professional fields such as medicine and law has become a key research topic in the industry. However, existing technical solutions generally have significant limitations. Currently, most large model pre-training adopts a single-stage and uniform sampling method, that is, mixing general and domain-specific corpora in a fixed ratio during training. This extensive training mechanism is prone to catastrophic forgetting problems when injecting domain knowledge, causing the model to acquire professional capabilities while its original general language understanding and generation capabilities suffer a severe decline. At the same time, the static data ratio can also easily cause conflicts between knowledge from different sources, affecting the model's internalization of consistent world knowledge.

[0003] The aforementioned problems are particularly prominent in highly specialized and rigorous fields. Texts in specific domains are typically characterized by dense terminology, rigorous logic, and lengthy context (such as annual reports of listed companies and regulatory documents). Existing models not only struggle to strike a balance between general capabilities and domain-specific specialization, but their inherent limitations in understanding and modeling long texts further restrict their practicality.

[0004] Furthermore, traditional training methods lack effective dynamic scheduling and coordination mechanisms for fusing multi-source heterogeneous corpora (such as general web pages, domain-specific reports, code, and mathematical text). This directly leads to instability in the training process, slow model convergence, and ultimately limits the overall performance of the model in complex multi-task scenarios.

[0005] Therefore, a new training system and method for large-scale models in vertical domains based on incremental matching and progressive corpus is needed.

[0006] The information disclosed in the background section is only intended to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] In view of this, this application provides a vertical domain large model training system and method based on incremental matching and progressive corpus, which can achieve smooth fusion of multi-source knowledge and significantly improve the domain adaptability, long text understanding ability and training stability of large language models.

[0008] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0009] According to one aspect of this application, a vertical domain large-scale model training system based on incremental matching and progressive corpus is proposed. The system includes: a preprocessing module for preprocessing raw data from different sources to generate a corpus, the corpus including a general corpus and a target domain corpus; an incremental matching sampling module for sampling the corpus based on an incremental matching algorithm to generate multiple incremental training corpus sets; wherein the incremental matching algorithm is used to progressively increase the sampling ratio of the target domain corpus in each sequentially generated incremental training corpus set; a first-stage training module for performing a first-stage training on a general-based large-scale model using the multiple incremental training corpus sets, and generating a vertical domain-based large-scale model through a multi-dimensional loss monitoring mechanism; a progressive corpus module for expanding the corpus with multi-domain corpora of different length distributions to generate a progressive corpus; and a second-stage training module for performing a second-stage training on the vertical domain-based large-scale model using the progressive corpus combined with an annealing optimization strategy to generate a long-context vertical domain large-scale model.

[0010] Optionally, the preprocessing module includes: a cleaning unit for cleaning, filtering, and formatting general corpora and target domain corpora from different sources, wherein the general corpora include corpora from multiple different domains; a deduplication unit for deduplicating the general corpora and the target domain corpora based on paragraph hash values; and a quality unit for calculating the data quality of the target domain corpora using a corpus quality assessment model to remove low-quality target domain corpora.

[0011] Optionally, the incremental matching sampling module includes: a proxy model unit, used to determine the internal corpus category matching of the general corpus through a proxy model; a proportion unit, used to set the proportion of the initial general corpus and the initial target domain corpus; and an incremental corpus unit, used to gradually increase the sampling proportion of the target domain corpus using an incremental matching strategy to generate multiple incremental training corpus sets.

[0012] Optionally, the incremental corpus unit is further configured to divide the overall training process into N training sessions; configure the incremental matching strategy so that the sampling ratio of the target domain corpus increases linearly from the initial value of the first training session to the termination value of the Nth training session; and generate multiple incremental training corpus sets according to the incremental matching strategy.

[0013] Optionally, the first-stage training module includes: a configuration unit for configuring a corresponding incremental training corpus set for each training session; and a first unit for performing N training sessions on the general base model using the multiple incremental training corpus sets, controlling the training process through a multi-dimensional loss monitoring mechanism to obtain a vertical domain base model.

[0014] Optionally, the first unit is further configured to perform incremental training by employing gradient accumulation and distributed data parallel training strategies during the first stage of training; monitor the multi-dimensional loss changes of the general base model under different data sources and different corpus sources during the first stage of training; and dynamically adjust the optimizer parameters according to the multi-dimensional loss changes until training converges to obtain a vertical domain base model.

[0015] Optionally, the progressive corpus module includes: a domain unit, used to obtain the proportion of each domain corpus in the corpus; and an adding unit, used to add corpus samples of different length distributions to each domain based on the proportion of each domain corpus, thereby generating the progressive corpus.

[0016] Optionally, the second-stage training module includes: an equalization unit, used to perform equal sampling on the progressive corpus to generate an equalized training corpus; an instruction unit, used to construct an instruction dataset containing general instructions, target domain instructions, and preset domain instructions; and a second unit, used to perform second-stage training on the vertical domain base model by combining the equalized training corpus and instruction dataset using an annealing optimization strategy to obtain the long-context vertical domain large model.

[0017] Optionally, the second unit is also used to dynamically adjust the learning rate using a cosine annealing strategy during the second stage of training; to enhance the context length during the second stage of training; and to obtain the long context vertical domain large model after training is completed.

[0018] According to one aspect of this application, a method for training a large vertical domain model based on incremental matching and progressive corpus is proposed. This method includes: preprocessing raw data from different sources to generate a corpus, the corpus including a general corpus and a target domain corpus; sampling the corpus based on an incremental matching algorithm to generate multiple incremental training corpus sets; wherein the incremental matching algorithm is used to progressively increase the sampling ratio of the target domain corpus in each sequentially generated incremental training corpus set; performing a first-stage training on a general-based large model using the multiple incremental training corpus sets, and generating a vertical domain-based large model through a multi-dimensional loss monitoring mechanism; expanding the corpus with multi-domain corpora of different length distributions to generate a progressive corpus set; and performing a second-stage training on the vertical domain-based large model using the progressive corpus set combined with an annealing optimization strategy to generate a long-context vertical domain large model.

[0019] According to one aspect of this application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method as described above.

[0020] According to one aspect of this application, a computer-readable medium is provided having a computer program stored thereon that, when executed by a processor, implements the method described above.

[0021] According to one aspect of this application, a computer program product is provided, comprising: a computer program / instructions that, when executed by a processor, implement the method as described above.

[0022] According to the vertical domain large model training system and method based on incremental matching and progressive corpus of this application, a corpus is generated by preprocessing raw data from different sources. The corpus includes general corpus and target domain corpus. The corpus is sampled based on the incremental matching algorithm to generate multiple incremental training corpus sets. The incremental matching algorithm is used to gradually increase the sampling ratio of the target domain corpus in each incremental training corpus set generated sequentially. The general base large model is trained in the first stage using the multiple incremental training corpus sets, and a vertical domain base large model is generated through a multi-dimensional loss monitoring mechanism. The corpus is expanded by multi-domain corpus with different length distributions to generate a progressive corpus set. The vertical domain base large model is trained in the second stage using the progressive corpus set combined with an annealing optimization strategy to generate a long-context vertical domain large model. This method can achieve smooth fusion of multi-source knowledge and significantly improve the domain adaptability, long text understanding ability and training stability of the large language model.

[0023] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application. Attached Figure Description

[0024] The above and other objects, features, and advantages of this application will become more apparent from the detailed description of exemplary embodiments with reference to the accompanying drawings. The drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0025] Figure 1 This is a flowchart illustrating a vertical domain large model training method based on incremental matching and progressive corpus according to an exemplary embodiment.

[0026] Figure 2 This is a flowchart illustrating a vertical domain large model training method based on incremental matching and progressive corpus, according to another exemplary embodiment.

[0027] Figure 3 This is a flowchart illustrating a vertical domain large model training method based on incremental matching and progressive corpus, according to another exemplary embodiment.

[0028] Figure 4 This is a schematic diagram illustrating a vertical domain large model training method based on incremental matching and progressive corpus according to another exemplary embodiment.

[0029] Figure 5 This is a block diagram illustrating a vertical domain large model training system based on incremental matching and progressive corpus according to an exemplary embodiment.

[0030] Figure 6 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0031] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this application will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0032] Explanation of technical abbreviations used in this application:

[0033] General-purpose foundational large model: refers to a basic language model pre-trained on a large-scale general corpus, which has strong general language understanding and generation capabilities and serves as the foundation for training subsequent domain models.

[0034] Incremental pre-training: refers to continuing training on an existing model using new corpus, in order to inject new domain knowledge while retaining the original knowledge.

[0035] Annealing optimization strategy: refers to dynamically adjusting the learning rate or data sampling ratio during model training to ensure stable convergence of the model during the knowledge injection phase and avoid performance fluctuations.

[0036] Catastrophic forgetting refers to the phenomenon where neural networks forget their original knowledge after being trained or fine-tuned on a specific task using new data, resulting in a significant decline in the model's performance on older tasks.

[0037] Channel Loss monitoring mechanism: refers to calculating and monitoring the loss from different data sources (such as general corpora, information domain corpora, code corpora, etc.) during the training process to evaluate the learning status and quality of the model on knowledge in various domains in real time, thereby achieving dynamic balance and optimization of multi-source knowledge.

[0038] In view of the technical difficulties existing in the prior art, this application proposes a two-stage pre-training method for large-scale models applicable to specific domains, aiming to solve the problems of catastrophic forgetting, weak long text processing capabilities, and insufficient multi-task generalization ability in the training of general-purpose large-scale models in specialized domains. This method achieves efficient integration of general capabilities and target domain-specific capabilities through a structured two-stage training process, combined with dynamic corpus scheduling, knowledge alignment mechanisms, and long context optimization strategies. The following is a detailed description of this solution:

[0039] Figure 1 This is a flowchart illustrating a vertical domain large model training method based on incremental matching and progressive corpus according to an exemplary embodiment. The vertical domain large model training method 10 based on incremental matching and progressive corpus includes at least steps S102 to S110.

[0040] like Figure 1 As shown, in S102, the raw data from different sources are preprocessed to generate a corpus set, which includes general corpus and target domain corpus.

[0041] For example, data cleaning, filtering, and formatting can be performed on general corpora and target domain corpora from different sources, where the general corpora include corpora from multiple different domains; deduplication can be performed on the general corpora and the target domain corpora based on paragraph hash values; and the data quality of the target domain corpora can be calculated using a corpus quality assessment model to eliminate low-quality target domain corpora.

[0042] For example, data cleaning, filtering, and formatting can be performed on general corpora and target domain corpora from different sources. The general corpora include corpora from multiple different fields such as encyclopedias, news, books, web pages, code, and mathematical text. A paragraph hashing algorithm can then be used to deduplicate the general corpora and target domain corpora to eliminate duplicate samples.

[0043] The target domain corpus can be generated according to the actual application scenario. More specifically, it can be a corpus for the smart home field, or a corpus for the banking field, etc. This application is not limited to this.

[0044] Furthermore, a corpus quality assessment model is used to calculate data quality scores for both the general corpus and the target domain corpus, filtering out low-quality texts below a preset threshold to ensure the quality and reliability of the corpus. This step yields a structured, standardized, and high-quality corpus, providing a reliable data foundation for subsequent training stages.

[0045] In S104, the corpus is sampled based on the incremental matching algorithm to generate multiple incremental training corpus sets; wherein the incremental matching algorithm is used to gradually increase the sampling ratio of the target domain corpus in each incremental training corpus set generated sequentially.

[0046] In one embodiment, for example, the internal corpus category ratio of the general corpus can be determined by a proxy model; the initial proportion of the general corpus and the initial proportion of the target domain corpus can be set; and an incremental ratio strategy can be adopted to gradually increase the sampling proportion of the target domain corpus to generate multiple incremental training corpus sets.

[0047] More specifically, a proxy model (a small model with the same origin) with a similar structure to the large model but a smaller number of parameters can be selected. The general corpus can be tested on the proxy model and the data ratio can be adjusted. By training the loss, the optimal corpus ratio in fields such as code, mathematics, and web pages can be verified.

[0048] More specifically, the overall training process can be divided into N training sessions; the incremental matching strategy is configured so that the sampling ratio of the target domain corpus increases linearly from the initial value of the first training session to the termination value of the Nth training session; and multiple incremental training corpus sets are generated according to the incremental matching strategy.

[0049] In one embodiment, for example, the internal category ratio of the general corpus can be determined by a proxy model; the initial ratio of the general corpus to the initial target domain corpus can be set; and an incremental ratio strategy can be adopted to gradually increase the sampling ratio of the target domain corpus in order to generate multiple incremental training corpus sets.

[0050] The incremental allocation strategy can be expressed as:

[0051] in, For each training session The corresponding corpus sampling data, The corresponding total number of training sessions.

[0052] The internal ratio of the general corpus is taken from the optimal ratio obtained on the small model. In the initial setting, the proportion of general corpus is high (e.g., 80% general and 20% target domain). As training progresses, the proportion of target domain corpus is gradually increased, with the final upper limit being 50% each, thereby achieving a smooth injection of knowledge.

[0053] This process allows for the gradual infusion of target domain knowledge while preserving general knowledge, thus preventing catastrophic amnesia.

[0054] In S106, the general-purpose base model is trained in the first stage using the multiple incremental training corpora, and a vertical-domain base model is generated through a multi-dimensional loss monitoring mechanism. A corresponding incremental training corpus can be configured for each training iteration; the general-purpose base model is trained N times using the multiple incremental training corpora, with the multi-dimensional loss monitoring mechanism controlling the training process to obtain the vertical-domain base model.

[0055] For example, a corresponding incremental training corpus can be configured for each training iteration; the general base model can be trained N times sequentially using these multiple incremental training corpora to obtain a large base model for the vertical domain. During training, gradient accumulation and distributed data parallelism (DDP) strategies are employed to improve training efficiency and stability; simultaneously, multi-dimensional monitoring (Channel Loss monitoring mechanism) is performed on the losses from different data sources and corpus sources to determine the learning balance of the model across various corpora; and the optimizer parameters (such as learning rate, weight decay factor, etc.) are dynamically adjusted based on the monitoring results until the model training converges, outputting a base model with target domain knowledge.

[0056] In S108, the corpus is expanded by using multi-domain corpora with different length distributions to generate an incremental corpus. For example, the proportion of corpora in each domain in the corpus can be obtained; based on the proportion of corpora in each domain, corpus samples with different length distributions are added to each domain to generate the incremental corpus.

[0057] For example, one could obtain the proportion of corpora from each domain within the corpus; based on this proportion, add corpus samples of different lengths (e.g., 8k, 16k, 32k corpora) to each domain to enhance the model's ability to learn long texts; and then integrate the expanded multi-domain samples to form a progressive corpus. This step can further improve the model's ability to understand long contextual structures and its adaptability to multi-task scenarios.

[0058] In S110, the vertical domain foundation model is trained in the second stage using the progressive corpus combined with an annealing optimization strategy to generate a long-context vertical domain model. For example, the progressive corpus can be sampled evenly to generate an even training corpus; an instruction dataset containing general instructions, target domain instructions, and preset domain instructions can be constructed; and the long-context vertical domain model can be obtained by combining the annealing optimization strategy with the even training corpus and instruction dataset to perform the second stage of training.

[0059] In one embodiment, for example: the progressive corpus is sampled evenly to generate an even training corpus to ensure that the frequency of occurrence of samples from different tasks and domains is balanced; an instruction dataset is constructed that includes general instructions, target domain instructions, and preset specific domain instructions, the instruction dataset including multiple types of tasks such as question answering, summarizing, translation, financial statement analysis, code generation, and mathematical reasoning; the vertical domain base model is trained in the second stage using the even training corpus and the instruction dataset to achieve comprehensive optimization of the model in multi-task, multi-domain, and long text scenarios.

[0060] During training, the learning rate can be dynamically adjusted using a cosine annealing strategy, and the model context length can be extended to 32k to enhance its long text understanding and generation capabilities. After training, a large language model with target domain knowledge, general language capabilities, and long text understanding capabilities is obtained.

[0061] According to the vertical domain large model training method based on incremental matching and progressive corpus of this application, a corpus is generated by preprocessing raw data from different sources. The corpus includes general corpus and target domain corpus. The corpus is sampled based on the incremental matching algorithm to generate multiple incremental training corpus sets. The incremental matching algorithm is used to gradually increase the sampling ratio of the target domain corpus in each incremental training corpus set generated sequentially. The general base large model is trained in the first stage using the multiple incremental training corpus sets, and a vertical domain base large model is generated through a multi-dimensional loss monitoring mechanism. The corpus is expanded by multi-domain corpus with different length distributions to generate a progressive corpus set. The vertical domain base large model is trained in the second stage using the progressive corpus set combined with an annealing optimization strategy to generate a long-context vertical domain large model. This method can achieve smooth fusion of multi-source knowledge and significantly improve the domain adaptability, long text understanding ability and training stability of the large language model.

[0062] It should be clearly understood that this application describes how specific examples are formed and used, but the principles of this application are not limited to any details of these examples. Rather, based on the teachings of the disclosure of this application, these principles can be applied to many other embodiments.

[0063] Figure 2 This is a flowchart illustrating a vertical domain large model training method based on incremental matching and progressive corpus, according to another exemplary embodiment. Figure 2 The process shown in step 20 is... Figure 1 The flowchart shown describes in detail S106 "the general base model is trained in the first stage using the multiple incremental training corpora, and a vertical domain base model is generated through a multi-dimensional loss monitoring mechanism".

[0064] like Figure 2 As shown in S202, a corresponding incremental training corpus set is configured for each training session. Specifically, multiple incremental training corpus sets generated according to the incremental matching strategy are respectively configured into the training rounds, so that each training round uses a different corpus ratio.

[0065] In the initial training rounds, the proportion of general corpus is relatively high (e.g., 80% general corpus and 20% target domain corpus). As the training rounds increase, the proportion of target domain corpus is gradually increased until the final round reaches the preset upper limit (e.g., 50% general corpus and 50% target domain corpus).

[0066] By configuring an independent corpus for each training session, this application enables the model to smoothly absorb target domain knowledge throughout continuous training, a stark contrast to existing techniques that use a single, fixed-ratio corpus throughout the entire training cycle. By pre-planning a series of training sets with varying corpus proportions, this application lays the foundation for subsequent gradual knowledge injection, mitigating the risks of training instability and catastrophic forgetting caused by abrupt changes in data distribution. It also avoids gradient oscillations and catastrophic forgetting caused by sudden increases in domain data.

[0067] In S204, the general base model is trained N times in the first stage using the multiple incremental training corpora.

[0068] In this process, each training round uses the corresponding incremental training corpus as input to update the parameters of the general base model. The value of N can be dynamically set according to the corpus size and training stability, for example, 5 to 10 incremental rounds.

[0069] After each round of training, the model parameters can be cumulatively updated based on the previous round, thereby achieving gradual knowledge transfer and integration. This step, through a progressive incremental training strategy, enables a smooth transition from general knowledge to target domain knowledge.

[0070] In S206, during the first phase of training, incremental training is performed using gradient accumulation and distributed data parallel training strategies to monitor the multi-dimensional loss changes of the general base model under different data sources and different corpus sources.

[0071] Specifically, the Distributed Data Parallel (DDP) training framework can be used to distribute training tasks in parallel to multiple GPUs or multiple computing nodes to improve computational efficiency and throughput; gradient accumulation technology can be used to achieve large-batch training effects under limited GPU memory, thereby maintaining training stability.

[0072] Simultaneously, loss values ​​from different data sources (such as general corpora, target domain corpora, code corpora, and mathematical corpora) can be calculated separately in each training batch to form a multi-dimensional loss vector (i.e., Channel Loss). By monitoring the loss change trends of each channel in real time, the learning balance of the model on different corpus categories can be judged, and potential domain bias or overfitting phenomena in the model can be identified.

[0073] In S208, the optimizer parameters are dynamically adjusted according to the changes in the multi-dimensional loss until the training converges, resulting in a large vertical domain base model.

[0074] Specifically, based on the real-time monitoring results of Channel Loss, the key parameters of the optimizer (including learning rate, weight decay coefficient, momentum term, etc.) can be dynamically adjusted: when the loss value of the target domain channel decreases too quickly, the sampling ratio of the target domain corpus or the learning rate can be appropriately reduced to prevent the model from overfitting; when the loss of the general corpus channel increases significantly, the proportion of the general corpus can be appropriately increased or an annealing strategy can be used to smooth the change in the learning rate to prevent catastrophic forgetting.

[0075] The optimizer can use the AdamW or LAMB algorithm and converge gradually under dynamic learning rate scheduling.

[0076] When the overall training loss curve stabilizes and the loss of each channel reaches equilibrium, the model is considered to have converged, and the first-stage training result—the vertical domain foundation large model—is output.

[0077] Through this process, the model retains its general language capabilities while acquiring professional knowledge and semantic understanding of the target domain, providing a solid foundation for the subsequent second-stage progressive corpus training.

[0078] Figure 3 This is a flowchart illustrating a vertical domain large model training method based on incremental matching and progressive corpus, according to another exemplary embodiment. Figure 3 The process shown in step 30 is... Figure 1 The flowchart shown describes in detail S110 "the second stage training of the vertical domain base model is performed by combining the progressive corpus with the annealing optimization strategy to generate a long context vertical domain model".

[0079] like Figure 3 As shown, in S302, the vertical domain base model is trained in the second stage by combining the annealing optimization strategy to balance the training corpus and the instruction dataset.

[0080] Specifically, based on the progressive corpus generated in the previous stage, a balanced sampling strategy is first used to sample various types of corpora (including general corpora, target domain corpora, mathematical corpora, code corpora, etc.) so that the samples in each domain appear with basically the same frequency during training, thus avoiding overfitting of the model in a single domain.

[0081] Subsequently, a multi-task instruction dataset is constructed, which may include:

[0082] General instruction tasks: such as question answering, summarizing, translation, and text reasoning;

[0083] Target domain instruction tasks: such as financial statement analysis, risk modeling, and compliance checks;

[0084] Mathematical and coding instruction tasks: such as formula derivation, quantization algorithm writing, code generation, etc.

[0085] During training, a balanced corpus can be combined with an instruction dataset to improve the model's overall capabilities through multi-task hybrid training. Each training batch loads multi-task samples, and the model uses an adaptive weight learning mechanism to share gradients and transfer semantics across different tasks, thereby achieving cross-task knowledge fusion.

[0086] In S304, during the second phase of training, a cosine annealing strategy is used to dynamically adjust the learning rate and enhance the context length. Specifically, a higher learning rate is used in the early stages of training to accelerate parameter convergence; as training progresses, the learning rate gradually decreases according to the cosine function curve to achieve refined training and prevent overfitting.

[0087] Meanwhile, during this stage, the model's context window is gradually expanded to 32k tokens, and long context optimization strategies, such as RotaryPositionEmbedding and GradientClipping, are introduced into the data loading and attention mechanisms to ensure stable training and information association capabilities of the model in long text input scenarios.

[0088] In S306, the long-context vertical domain large model is obtained after training.

[0089] During training, the model's performance on multiple task dimensions (such as general question-answering accuracy, analytical consistency, and code generation accuracy) can be evaluated in real time using a validation set, for example, until the model reaches a preset threshold on each task. The model is considered to have converged when the training loss stabilizes and the validation performance no longer improves significantly.

[0090] The final output large language model has the following characteristics: it retains the broad semantic understanding capabilities of a general language model; it possesses professional knowledge and expressive capabilities for the target domain (such as IT, law, and medicine); and it shows significant performance improvements in long text understanding, complex reasoning, and multi-task collaborative scenarios.

[0091] Figure 4 This is a schematic diagram illustrating a vertical domain large model training method based on incremental matching and progressive corpus, according to another exemplary embodiment. Figure 4 The corresponding method 40 is as follows: Based on the general base model, target domain knowledge is injected to construct a domain enhancement model with the cognitive foundation of the target domain.

[0092] First, standardization preprocessing is performed on general corpora and target domain corpora from multiple sources. A document deduplication algorithm based on paragraph hashing can also be used to remove duplicate corpus samples to improve the quality of training data. A corpus quality assessment model based on deep learning is constructed to score the credibility of corpora in the target domain and automatically filter low-quality text based on the score results.

[0093] In one embodiment, a proxy model with a similar structure to the large model but a smaller number of parameters can also be used to optimize the proportions of multiple subclasses within the general corpus and determine the optimal proportions within the general corpus.

[0094] Subsequently, a dynamic corpus scheduling strategy can be implemented based on the optimal ratio: in the early stage of training, the proportion of general corpus is set to 80% and the proportion of target domain corpus is set to 20%; as the training progresses, the sampling ratio of target domain corpus is gradually increased, and finally the ratio of general corpus to target domain corpus reaches 50%:50% to achieve a smooth injection of knowledge.

[0095] In the specific training process, gradient accumulation and distributed data parallelism strategies can be used to train the model to improve training efficiency. During the training process, a multi-channel loss monitoring mechanism can be used to calculate the loss value of different sources of corpus, monitor the model's fitting status on data in various domains in real time, and dynamically adjust the optimizer parameters according to the loss curve to prevent overfitting or catastrophic forgetting.

[0096] More specifically, the training context length can be set to 8K, and the learning rate to 2e-5;

[0097] After training is completed, a large-scale vertical domain-based model with a cognitive foundation in the target domain is output.

[0098] To improve the model's ability to understand long text structures and enhance its generalization ability in various high-knowledge-density tasks such as target domains, mathematics, and code, a multi-scale progressive corpus can be constructed by adding corpus samples with different context length distributions of 8K, 16K, and 32K according to domain proportions to the existing corpus. This increases the proportion of high-knowledge-density corpora such as code, mathematics, and target domains, while correspondingly reducing the proportion of web pages and news corpora.

[0099] In one embodiment, a multi-type instruction dataset can be constructed, including general instructions, target domain instructions, and mathematical and code instructions. The general instructions include question-answering, summarizing, and translation tasks; the target domain instructions include financial statement analysis, risk modeling, and compliance inspection tasks; and the mathematical and code instructions include quantitative formula derivation and target domain code generation tasks.

[0100] A task-balanced sampling strategy can be adopted to uniformly sample various types of instruction data, ensuring that each task type appears in a balanced manner during training.

[0101] More specifically, a cosine annealing strategy can be used to dynamically adjust the learning rate, setting the initial learning rate to 2e-6 to support fine-grained training; the training context length can be increased to 32K to enhance the model's ability to process long texts; after training is completed, a large long-context vertical domain model with strong target domain expertise, long context understanding ability and multi-task generalization ability can be output.

[0102] This application uses the above two-stage training mechanism to systematically inject target domain knowledge into the general large model and further expand its long context understanding and multi-task adaptability, thereby effectively balancing the model's professionalism, versatility and practicality.

[0103] More specifically, this application can effectively mitigate catastrophic forgetting. Through dynamic corpus matching and progressive knowledge injection strategies, while introducing professional knowledge in the target domain, it controls the performance degradation of the model on general language understanding benchmarks (such as C-Eval) to within 2%, which is significantly better than the degradation of more than 10% of traditional methods.

[0104] This application can also significantly improve the long text processing capability. Through multi-length corpus training and annealing optimization strategy, it can improve the capability of 32k long text by 15%, and maintain better coherence and consistency when processing long contexts.

[0105] This application also enables efficient multi-source knowledge fusion. Based on the Channel Loss monitoring mechanism and task balancing sampling strategy, it effectively coordinates the learning process of various types of knowledge, including general, target domain, code, and mathematics, achieving significant performance improvements in multiple tasks such as target domain question answering, code generation, and mathematical reasoning.

[0106] This application also improves training efficiency and stability. The combination of dynamic scheduling strategy and annealing optimization makes the training process more stable, increases the convergence speed by about 20%, and reduces the training resource consumption by about 15%.

[0107] This application can also enhance the model's practicality and generalization ability. Through two-stage training and multi-task adaptation, the model not only excels at professional tasks in the target domain, but also maintains good general language ability and cross-domain transfer ability.

[0108] Those skilled in the art will understand that all or part of the steps of the above embodiments are implemented as a computer program executed by a CPU. When the computer program is executed by the CPU, it performs the functions defined by the method provided in this application. The program can be stored in a computer-readable storage medium, such as a read-only memory, a magnetic disk, or an optical disk.

[0109] Furthermore, it should be noted that the above figures are merely illustrative representations of the processes included in the method according to exemplary embodiments of this application, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously in multiple modules.

[0110] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0111] Figure 5 This is a block diagram illustrating a large-scale vertical domain model training system based on incremental matching and progressive corpus, according to another exemplary embodiment. Figure 5 As shown, the vertical domain large model training system 50 based on incremental matching and progressive corpus includes: a preprocessing module 502, an incremental matching sampling module 504, a first-stage training module 506, a progressive corpus module 508, and a second-stage training module 510.

[0112] The preprocessing module 502 is used to preprocess raw data from different sources to generate a corpus set, which includes general corpus and target domain corpus.

[0113] More specifically, the preprocessing module 502 includes: a cleaning unit for cleaning, filtering, and formatting general corpora and target domain corpora from different sources, wherein the general corpora include corpora from multiple different domains; a deduplication unit for deduplicating the general corpora and the target domain corpora based on paragraph hash values; and a quality unit for calculating the data quality of the target domain corpora using a corpus quality assessment model to remove low-quality target domain corpora.

[0114] The incremental matching sampling module 504 is used to sample the corpus based on the incremental matching algorithm to generate multiple incremental training corpus sets; wherein the incremental matching algorithm is used to gradually increase the sampling ratio of the target domain corpus in each incremental training corpus set generated sequentially;

[0115] More specifically, the incremental matching sampling module 504 includes: a proxy model unit, used to determine the internal corpus category ratio of the general corpus through a proxy model; a proportion unit, used to set the initial proportion of the general corpus and the initial target domain corpus; and an incremental corpus unit, used to gradually increase the sampling proportion of the target domain corpus using an incremental matching strategy to generate multiple incremental training corpus sets. The incremental corpus unit is also used to divide the overall training process into N training sessions; configure the incremental matching strategy so that the sampling proportion of the target domain corpus increases linearly from the initial value of the first training session to the termination value of the Nth training session; and generate multiple incremental training corpus sets according to the incremental matching strategy.

[0116] The first-stage training module 506 is used to perform the first-stage training of the general base model through the multiple incremental training corpora, and to generate a vertical domain base model through a multi-dimensional loss monitoring mechanism.

[0117] More specifically, the first-stage training module 506 includes: a configuration unit, used to configure a corresponding incremental training corpus set for each training iteration; and a first unit, used to perform N training iterations of the general base model using the multiple incremental training corpus sets, controlling the training process through a multi-dimensional loss monitoring mechanism to obtain a large-scale vertical domain base model. The first unit is also used to perform incremental training using a gradient accumulation and distributed data parallel training strategy during the first-stage training process; monitor the multi-dimensional loss changes of the general base model under different data sources and corpus sources during the first-stage training process; and dynamically adjust the optimizer parameters according to the multi-dimensional loss changes until training converges to obtain the large-scale vertical domain base model.

[0118] The progressive corpus module 508 is used to expand the corpus by using multi-domain corpora with different length distributions to generate a progressive corpus;

[0119] More specifically, the progressive corpus module 508 includes: a domain unit, used to obtain the proportion of each domain corpus in the corpus; and an adding unit, used to add corpus samples of different length distributions to each domain based on the proportion of each domain corpus, thereby generating the progressive corpus.

[0120] The second-stage training module 510 is used to perform a second-stage training on the vertical domain base model using the progressive corpus combined with an annealing optimization strategy, thereby generating a long-context vertical domain model.

[0121] More specifically, the second-stage training module 510 includes: an equalization unit, used to perform equal sampling on the progressive corpus to generate an equalized training corpus; an instruction unit, used to construct an instruction dataset containing general instructions, target domain instructions, and preset domain instructions; and a second unit, used to perform second-stage training on the vertical domain foundation model by combining an annealing optimization strategy with the equalized training corpus and instruction dataset to obtain the long-context vertical domain large model. The second unit is also used to dynamically adjust the learning rate using a cosine annealing strategy during the second-stage training; to enhance the context length during the second-stage training; and to obtain the long-context vertical domain large model after training.

[0122] According to the vertical domain large model training system based on incremental matching and progressive corpus of this application, a corpus is generated by preprocessing raw data from different sources. The corpus includes general corpus and target domain corpus. The corpus is sampled based on an incremental matching algorithm to generate multiple incremental training corpus sets. The incremental matching algorithm is used to gradually increase the sampling ratio of the target domain corpus in each incremental training corpus set generated sequentially. The general base large model is trained in the first stage using the multiple incremental training corpus sets, and a vertical domain base large model is generated through a multi-dimensional loss monitoring mechanism. The corpus is expanded by multi-domain corpus with different length distributions to generate a progressive corpus set. The vertical domain base large model is trained in the second stage using the progressive corpus set combined with an annealing optimization strategy to generate a long-context vertical domain large model. This method can achieve smooth fusion of multi-source knowledge and significantly improve the domain adaptability, long text understanding ability and training stability of the large language model.

[0123] like Figure 6 As shown, this application provides an electronic device 600, including a processor 610, a communication interface 620, a memory 630, and a bus 640, wherein the processor 610 and the memory 630 communicate with each other through the bus 640;

[0124] Memory 630 is used to store computer programs;

[0125] When the processor 610 executes the program stored in the memory 630, it implements the vertical domain large model training method based on incremental matching and progressive corpus of any of the above embodiments.

[0126] The communication interface 620 is used for communication between the above-mentioned electronic devices and other devices.

[0127] The memory 630 may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory 630 may also be at least one storage device located remotely from the aforementioned processor 610.

[0128] If the methods described in this application are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms.

[0129] This application provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the vertical domain large model training method based on incremental matching and progressive corpus sets according to any of the above embodiments. For example, raw data from different sources is preprocessed to generate a corpus set, which includes a general corpus and a target domain corpus; the corpus set is sampled based on an incremental matching algorithm to generate multiple incremental training corpus sets; wherein the incremental matching algorithm is used to progressively increase the sampling ratio of the target domain corpus in each sequentially generated incremental training corpus set; a first-stage training of a general base large model is performed using the multiple incremental training corpus sets, and a vertical domain base large model is generated through a multi-dimensional loss monitoring mechanism; the corpus set is expanded using multi-domain corpora of different lengths to generate a progressive corpus set; a second-stage training of the vertical domain base large model is performed using the progressive corpus set combined with an annealing optimization strategy to generate a long-context vertical domain large model.

[0130] Exemplary embodiments of this application have been specifically shown and described above. It should be understood that this application is not limited to the detailed structures, arrangements, or implementation methods described herein; rather, this application is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A vertical field large model training system based on incremental proportioning and progressive corpus set, characterized in that, include: The preprocessing module is used to preprocess raw data from different sources to generate a corpus set, which includes general corpus and target domain corpus. The incremental matching sampling module is used to gradually increase the sampling ratio of the target domain corpus to generate multiple incremental training corpus sets, wherein the incremental matching algorithm is used to divide the overall training process into N training sessions. Configure the incremental matching strategy so that the sampling ratio of the target domain corpus increases linearly from the initial value of the first training to the termination value of the Nth training stage. Multiple incremental training corpus sets are generated based on the incremental matching strategy; The first-stage training module is used to perform the first-stage training of the general-purpose base model using the multiple incremental training corpora, and to generate a vertical-domain base model through a multi-dimensional loss monitoring mechanism. The multi-dimensional loss monitoring mechanism is used to calculate the loss values ​​of different data sources to form a multi-dimensional loss vector. Based on the real-time monitoring results of the multi-dimensional loss vector, the optimizer parameters are dynamically adjusted until the training converges. Furthermore, the first stage of training adopts a gradient accumulation and distributed data parallel training strategy, which allocates training tasks in parallel through multiple GPUs or multiple computing nodes. The progressive corpus module is used to obtain the proportion of corpus in each domain in the corpus; based on the proportion of corpus in each domain, corpus samples with different length distributions of 8K, 16K and 32K are added to each domain respectively to generate a progressive corpus; The second-stage training module is used to perform second-stage training on the vertical domain base model by combining the progressive corpus with the annealing optimization strategy, thereby generating a long-context vertical domain model. The annealing optimization strategy is specifically a cosine annealing scheduling strategy. Furthermore, in the second stage of training, the progressive corpus is first sampled in a balanced manner to generate a balanced training corpus, and then an instruction dataset containing general instructions, target domain instructions, and preset domain instructions is constructed. The balanced training corpus and the instruction dataset are combined for multi-task hybrid training. During training, the model's context length is gradually expanded to 32k.

2. The vertical domain large model training system of claim 1, wherein, The preprocessing module includes: The cleaning unit is used to perform data cleaning, filtering, and formatting on general corpora and target domain corpora from different sources. The general corpora include corpora from multiple different domains. The deduplication unit is used to perform deduplication processing on the general corpus and the target domain corpus based on paragraph hash values; The quality unit is used to calculate the data quality of the target domain corpus through a corpus quality assessment model in order to eliminate low-quality target domain corpus.

3. The vertical domain large model training system of claim 1, wherein, The incremental ratio sampling module includes: The proxy model unit is used to determine the internal corpus category allocation of the general corpus through the proxy model; The proportion unit is used to set the proportion of the initial general corpus and the initial target domain corpus; Incremental corpus units are used to gradually increase the sampling ratio of the target domain corpus by adopting an incremental matching strategy to generate multiple incremental training corpus sets.

4. The vertical domain large model training system of claim 1, wherein, The first-stage training module includes: The configuration unit is used to configure a corresponding incremental training corpus set for each training session. The first unit is used to perform N training iterations of the general base model through the multiple incremental training corpora. During the training process, a multi-dimensional loss monitoring mechanism is used for control in order to obtain a large base model for the vertical domain.

5. The vertical domain large model training system of claim 4, wherein, The first unit is also used for In the first phase of training, incremental training is performed using gradient accumulation and distributed data parallel training strategies. During the first stage of training, the multi-dimensional loss changes of the general base model are monitored under different data sources and different corpus sources; the optimizer parameters are dynamically adjusted according to the multi-dimensional loss changes until the training converges, and the vertical domain base model is obtained.

6. The vertical domain large model training system of claim 1, wherein, The second-stage training module includes: The equalization unit is used to perform equalization sampling on the progressive corpus to generate an equalized training corpus; Instruction unit, used to construct instruction dataset containing general instructions, target domain instructions, and preset domain instructions; The second unit is used to perform a second-stage training of the vertical domain base model by combining an annealing optimization strategy to balance the training corpus and instruction dataset, thereby obtaining the long-context vertical domain model.

7. The vertical domain large model training system as described in claim 6, characterized in that, The second unit is also used for During the second phase of training, a cosine annealing strategy was used to dynamically adjust the learning rate. During the second phase of training, the context length is increased; After training, the large model with long context vertical domain is obtained.

8. A method for training large-scale models in vertical domains based on incremental matching and progressive corpus, characterized in that, include: Raw data from different sources are preprocessed to generate a corpus, which includes general corpus and target domain corpus; The corpus is sampled based on the incremental matching algorithm, and the proportion of target domain corpus sampling is gradually increased to generate multiple incremental training corpus sets. The incremental matching algorithm is used to divide the overall training process into N training sessions. Configure the incremental matching strategy so that the sampling ratio of the target domain corpus increases linearly from the initial value of the first training to the termination value of the Nth training stage. Multiple incremental training corpus sets are generated based on the incremental matching strategy; The general-purpose base model is trained in the first stage using the multiple incremental training corpora. A vertical domain base model is generated using a multi-dimensional loss monitoring mechanism. The multi-dimensional loss monitoring mechanism is used to calculate the loss values ​​from different data sources to form a multi-dimensional loss vector. The optimizer parameters are dynamically adjusted based on the real-time monitoring results of the multi-dimensional loss vector until the training converges. Furthermore, the first stage of training adopts a gradient accumulation and distributed data parallel training strategy, which allocates training tasks in parallel through multiple GPUs or multiple computing nodes. Obtain the proportion of each domain in the corpus; Based on the proportion of corpus data in each domain, corpus samples of different lengths (8K, 16K, and 32K) are added to each domain to generate a progressive corpus set. The second stage of training is performed on the vertical domain base model using the progressive corpus combined with the annealing optimization strategy to generate a long-context vertical domain model. The annealing optimization strategy is specifically a cosine annealing scheduling strategy. Furthermore, in the second stage of training, the progressive corpus is first sampled in a balanced manner to generate a balanced training corpus, and then an instruction dataset containing general instructions, target domain instructions, and preset domain instructions is constructed. The balanced training corpus and the instruction dataset are combined for multi-task hybrid training. During training, the model's context length is gradually expanded to 32k.

9. The method for training large models in a vertical domain as described in claim 8, characterized in that, Raw data from different sources are preprocessed to generate a corpus, including: Data cleaning, filtering, and formatting are performed on general corpora and target domain corpora from different sources. The general corpora include corpora from multiple different domains. The deduplication process for the general corpus and the target domain corpus is performed based on the paragraph hash value; The data quality of the target domain corpus is calculated using a corpus quality assessment model to eliminate low-quality target domain corpus.

10. The vertical domain large model training method as described in claim 8, characterized in that, The corpus is sampled based on an incremental matching algorithm to generate multiple incremental training corpus sets, including: The internal corpus category allocation of the general corpus is determined by using a proxy model; Set the initial proportions of general corpus and initial target domain corpus; An incremental matching strategy is adopted to gradually increase the sampling ratio of the target domain corpus in order to generate multiple incremental training corpus sets.

11. A computer-readable medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method as described in any one of claims 8-10.

Images