Geological domain large language model with continuous learning enhancement

Abstract

According to the geological field large language model with continuous learning enhancement, the performance of the model on downstream tasks in the geological field is remarkably improved through a method of construction of a professional data set, continuous pre-training with domain knowledge enhancement and efficient fine tuning of parameters. Through a TG-MoLE architecture, a cross-group and intra-group routing cooperation mechanism is innovatively introduced, and expert weight distribution is optimized through a dynamically adjusted soft routing strategy, so that the expert cooperation capability and task adaptability of the model are improved, the problem of disastrous forgetting can be more effectively relieved, and meanwhile, the performance of geological downstream tasks is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of seismic phase identification, and in particular to a large language model in the geological field with continuous learning enhancement. Background Art

[0002] In recent years, large language models (LLMs) have demonstrated remarkable capabilities in general-purpose tasks such as natural language understanding and generation. They are able to efficiently complete a variety of tasks based on natural language instructions without specialized training data. However, due to the lack of deep domain expertise, the performance of these models in specific specialized domains is often limited.

[0003] To overcome this limitation, a significant amount of research in recent years has been devoted to exploring the application of large language models in vertical fields, applying them to solve specialized problems within those fields, and has achieved remarkable results. Popular application areas include healthcare (such as HuatuoGPT, BianQue, and ZhongJing), law (such as ChatLaw, LawGPT, and InternLM-Law), and finance (such as XuanYuan, PIXIU, and FinGPT). Large language models in other fields also include Educhat in education, OceanGPT in the oceanography field, K2 in the earth sciences field, and ChatHome in the home improvement field.

[0004] Although vertical domain big language models have made progress in multiple industries, geology, as a highly specialized discipline, still faces unique challenges. Geological research involves complex scientific issues such as the earth's material, structure, and its evolution, and has its own unique terminology system and conceptual framework. This makes existing general LLMs often unable to understand and generate geology-related content, especially when dealing with subtle differences in professional terminology and concepts. In addition, due to the high difficulty of data collection and the difficulty of collecting information in the geological industry, no big language model specifically for the field of geology has been released to date, nor are there any training datasets and evaluation standards for this field. Therefore, how to build a big language model with deep understanding and generation capabilities in the field of geology is a challenging problem that is worth studying.

[0005] In addition, continuous learning capability is a key challenge when building large language models in vertical domains. Traditional continuous learning methods can be divided into three categories: data replay technology (such as iCaRL, DGR, GEM, etc.), regularization strategies (such as EWC, LwF, etc.), and architecture modification methods (such as PNN, DEN, PackNet, BatchEnsemble, etc.). However, these methods are mainly aimed at small to medium-sized models, and it is difficult to balance the stability and plasticity of the model when applied to large-scale language models. In particular, when adopting the "continued pre-training + fine-tuning" paradigm to build vertical domain LLMs, the model is prone to catastrophic forgetting, that is, the previously acquired general knowledge may be lost when adapting to new domain knowledge. This poses an important challenge to the domain adaptation and continuous learning capabilities of large geological language models. Summary of the Invention

[0006] In order to overcome the defects of the existing technology, the technical problem to be solved by the present invention is to provide a large language model in the geological field with continuous learning enhancement, which significantly improves the performance of the model on downstream tasks in the geological field, can more effectively alleviate the problem of catastrophic forgetting, and at the same time improve the performance on geological downstream tasks.

[0007] The technical solution of the present invention is: this large language model for the geological field with continuous learning enhancement includes the following steps:

[0008] (1) Cross-group routing is defined as a task type-guided gating function, which is used to assign greater weights to expert groups that belong to the same task type as the sample. The cross-group gating function CG(·) is used as a cross-group router. This router assigns weights to each expert group based on the characteristics of the data sample and softly adjusts the cross-group weight distribution based on the task type. With x as input and the task type as task_type, its value 0 indicates general knowledge and its value 1 indicates geological domain knowledge, the output of the cross-group gating routing is expressed as:

[0009] tg=one-hot(task_type)

[0010] CG(·) init =Softmax(W cg x)

[0011]

[0012] Where tg(task_guidance) represents the one-hot encoding of task type guidance, CG(·) init represents the initial cross-group routing weight, W cg represents the trainable weight matrix of router CG(·), CG(·) represents the adjusted cross-group routing weight, and β is a learnable parameter used to control the degree of task type guidance;

[0013] (2) All experts are divided into two groups with the same number of experts in each group. experts, and the intra-group gating function IG(·) is used as the intra-group router to assign weights to the intra-group experts according to the input x. The output of the intra-group gating routing is expressed as:

[0014] IG(·)=Softmax(W ig x)

[0015] W ig represents the trainable weight matrix of router IG(·);

[0016] (3) The forward propagation process of TG-MoLE is expressed as follows:

[0017]

[0018] in represents all experts in group j, Ω j represents the routing weight assigned by cross-group routing to the j-th group of experts, ω i It represents the routing weight assigned to the i-th expert by the intra-group routing.

[0019] This paper significantly improves the model's performance on downstream geological tasks by constructing specialized datasets, continuing pre-training with enhanced domain knowledge, and efficiently fine-tuning parameters. Through the TG-MoLE architecture, it innovatively introduces cross-group and intra-group routing collaboration mechanisms. This dynamically adjusted soft routing strategy optimizes expert weight distribution, thereby enhancing the model's expert collaboration capabilities and task adaptability. This can more effectively mitigate the problem of catastrophic forgetting while improving performance on downstream geological tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 The main architecture of the TG-MoLE method is shown. TG-MoLE's group routing module first globally assigns weights to expert groups and dynamically adjusts these weights based on task type, ensuring that the characteristics of different data samples and task requirements are adaptively addressed. Local weight normalization is then performed within the group to fully leverage the synergy of experts within the group.

[0021] Figure 2 The pre-training dataset construction process is shown. The text corpus is converted into a usable pre-training dataset through format pre-processing, data cleaning, quality filtering, and data deduplication.

[0022] Figure 3It is a comparative evaluation of baseline models on objective tasks in the GeoEval dataset. GeoKU and MinQA refer to geological knowledge understanding and mineral prospecting knowledge question answering, respectively.

[0023] Figure 4 This is a comparative evaluation of baseline models on subjective tasks in the GeoEval dataset. Perplexity, BERTScore, and Judge Acc. represent various model evaluation metrics. Arrows following the metrics indicate that larger or smaller values ​​indicate higher model evaluations. Judge Acc. represents the accuracy scores of each baseline model's answers as assessed by the LLM (i.e., Qwen2-72B-Instruct).

[0024] Figure 5 The manual evaluation results of the baseline model are shown. The evaluation indicators are rationality, correctness and consistency.

[0025] Figure 6 The performance of the model on geological downstream tasks and general tasks under different fine-tuning methods is shown.

[0026] Figure 7 The ablation experiment of the soft routing adjustment mechanism is shown to verify the effectiveness of the present invention.

[0027] Figure 8 A flowchart of a geological domain large language model enhanced by continuous learning according to the present invention is shown. DETAILED DESCRIPTION

[0028] like Figure 8 As shown in FIG, this continuous learning enhanced geological domain language model includes the following steps:

[0029] (1) Cross-group routing is defined as a task type-guided gating function, which is used to assign greater weights to expert groups that belong to the same task type as the sample. The cross-group gating function CG(·) is used as a cross-group router. This router assigns weights to each expert group based on the characteristics of the data sample and softly adjusts the cross-group weight distribution based on the task type. With x as input and the task type as task_type, its value 0 indicates general knowledge and its value 1 indicates geological domain knowledge, the output of the cross-group gating routing is expressed as:

[0030] tg=one-hot(task_type)

[0031] CG(·) init =Softmax(W cg x)

[0032]

[0033] Where tg(task_guidance) represents the one-hot encoding of task type guidance, CG(·) init represents the initial cross-group routing weight, W cg represents the trainable weight matrix of router CG(·), CG(·) represents the adjusted cross-group routing weight, and β is a learnable parameter used to control the degree of task type guidance;

[0034] (2) All experts are divided into two groups with the same number of experts in each group. experts, and the intra-group gating function IG(·) is used as the intra-group router to assign weights to the intra-group experts according to the input x. The output of the intra-group gating routing is expressed as:

[0035] 1G(·)=Softmax(W ig x)

[0036] W ig represents the trainable weight matrix of router IG(·);

[0037] (3) The forward propagation process of TG-MoLE is expressed as follows:

[0038]

[0039] in represents all experts in group j, Ω j represents the routing weight assigned by cross-group routing to the j-th group of experts, ω i It represents the routing weight assigned to the i-th expert by the intra-group routing.

[0040] This paper significantly improves the model's performance on downstream geological tasks by constructing specialized datasets, continuing pre-training with enhanced domain knowledge, and efficiently fine-tuning parameters. Through the TG-MoLE architecture, it innovatively introduces cross-group and intra-group routing collaboration mechanisms. This dynamically adjusted soft routing strategy optimizes expert weight distribution, thereby enhancing the model's expert collaboration capabilities and task adaptability. This can more effectively mitigate the problem of catastrophic forgetting while improving performance on downstream geological tasks.

[0041] Preferably, in step (3), a dense MoE structure is adopted to fuse the outputs of all experts through weighted fusion.

[0042] Preferably, the sources of the corpus of the large language model in the geological field enhanced by continuous learning include: geological literature, geological books, online geological resources, and geological professional documents.

[0043] Preferably, in the large language model of the geological field enhanced by continuous learning, a professional instruction dataset covering geology, mineralogy and geography is constructed.

[0044] Preferably, in the adaptive training process in the geological field, the open source large language model Qwen2-7B is selected as the model base, including continued pre-training and supervised instruction fine-tuning stages; in the continued pre-training stage, Qwen2-7B is initialized and trained on a geological field corpus of 181.66M tokens to obtain a domain-transferred GeoQwen model; in the supervised instruction fine-tuning stage, the earth science instruction dataset earth and the professional instruction dataset are used for fine-tuning.

[0045] Preferably, the supervised instruction fine-tuning stage adopts a fine-tuning data hybrid training paradigm, including: a hybrid training method and a staged training method. The staged training method is to first fine-tune the earth science instruction data and then deeply fine-tune the geological instruction data.

[0046] Preferably, in the supervised instruction fine-tuning stage, the PEFT method is used, the learning rate is set to 1e-5, the global batch size is 32, the micro-batch size is 1, and the gradient accumulation is 2; the LoRA method is used, lora_r is set to 8, lora_alpha is set to 16, and lora_target is set to all, and a total of 3 epochs are fine-tuned.

[0047] Preferably, in step (3), GeoQwen is used as a starting point for instruction fine-tuning.

[0048] The present invention is described in more detail below.

[0049] The present invention comprises the following steps:

[0050] 1. Model Method Architecture

[0051] TG-MoLE method Figure 1 As shown. TG-MoLE proposes an explicit routing grouping strategy that combines cross-group routing (Cross-Group Routing) and inside-group routing (Inside-Group Routing). In addition, TG-MoLE introduces a soft routing adjustment strategy, which achieves a better match between task types and expert groups by dynamically adjusting the weight distribution of each expert group. Specifically, TG-MoLE's group routing module first distributes weights to expert groups from a global perspective, and softly and dynamically adjusts the weight distribution of expert groups based on task types to ensure that the characteristics and task requirements of different data samples are adaptively addressed and processed. Local weight normalization is then performed within the group to give full play to the synergy of experts within the group.

[0052] Specifically, TG-MoLE makes modifications based on the feedforward network (FFN) module in the large language model because FFN stores diverse knowledge. For the FFN layer in each Transformer decoder block, a LoRA-like bypass is added, and N parallel LoRA modules are used as experts (N is an even number), as well as cross-group routing and intra-group routing modules for assigning weights to the corresponding expert groups and intra-group experts, respectively. During the implementation process, the weights of the backbone model are frozen, and only the additional routing and LoRA expert blocks are trained, so that the knowledge of the original model can be effectively retained. The TG-MoLE architecture is as follows: Figure 1 Next, the implementation details of TG-MoLE will be introduced in detail.

[0053] First, we define cross-group routing as a task-type-guided gating function that assigns greater weight to expert groups that share the same task type as the sample. Formally, we use the cross-group gating function CG(·) as a cross-group router, which assigns weights to each expert group based on the data sample characteristics and softly adjusts the cross-group weight distribution based on the task type. With x as input and the task type as task_type (a value of 0 indicates general knowledge and a value of 1 indicates geological domain knowledge), the output of the cross-group gating routing can be expressed as:

[0054] tg=one-hot(task_type)

[0055] CG(·) init =Softmax(W cg x)

[0056]

[0057] Where tg(task_guidance) represents the one-hot encoding of task type guidance, CG(·) init represents the initial cross-group routing weight, W cg represents the trainable weight matrix of the router CG(·), CG(·) represents the adjusted cross-group routing weight, and β is a learnable parameter used to control the degree of task type guidance.

[0058] according to Figure 1 It can be seen that all experts are divided into two groups with the same number of experts in each group. Experts. Formally, the intra-group gating function IG(·) is used as the intra-group router to assign weights to the experts in the group according to the input x. The output of the intra-group gating routing can be expressed as:

[0059] IG(·)=Softmax(W ig x)

[0060] W ig represents the trainable weight matrix of router IG(·).

[0061] In summary, the forward propagation process of TG-MoLE can be expressed as follows:

[0062]

[0063] in represents all experts in group j, Ω j represents the routing weight assigned by cross-group routing to the j-th group of experts, ω i It represents the routing weight assigned to the i-th expert by the intra-group routing.

[0064] 2. Task-driven explicit routing grouping strategy and soft routing adjustment mechanism

[0065] The core concepts of TG-MoLE consist of two aspects: a task-driven explicit routing grouping strategy and a soft routing adjustment mechanism. Specifically, experts are divided into two groups: one responsible for maintaining general knowledge and the other focusing on learning downstream tasks in the geological domain. Inspired by the application of the MoE architecture in multi-task learning, the goal is to route inputs with similar knowledge to the corresponding expert groups during training and inference, thereby reducing interference between different tasks and improving the model's generalization ability when processing new knowledge.

[0066] However, in practice, we've found that simply masking out expert groups whose task types differ from the input task type is not conducive to improving overall model performance. Therefore, we've designed a soft routing adjustment mechanism for cross-group routing. While this mechanism primarily routes to expert groups whose task types match, it still assigns a small amount of weight to expert groups that don't match the task type, thus avoiding the potential loss of model flexibility that could be caused by hard masking.

[0067] Due to computational resource constraints, the model uses a relatively small number of experts, thus adopting a dense MoE structure rather than the Top-K routing strategy used in a sparse MoE. While the sparse MoE design reduces computational cost by activating a small number of experts, it can pose engineering challenges due to uneven expert load or distributed communication overhead. In contrast, the dense MoE avoids the problem of uneven load by weightedly fusing the outputs of all experts while simplifying the routing logic. This strategy ensures that each expert can contribute knowledge while supporting the diversity and complexity of tasks. However, due to the high computational cost of activating all experts, this structure still presents a trade-off between model size and computational efficiency.

[0068] 3. Domain Adaptive Training Strategy and Implementation Details for Large Geological Models

[0069] In order to build a large language model in the field of geology, we first faced the challenge of building a large text corpus in the field of geology. At the same time, in order for the model to have strong instruction-following capabilities, it is necessary to build a sufficient amount of instruction data. Research has shown that large language models can acquire professional knowledge from corpora in specific fields, and the quality of training data is more important than the quantity. Therefore, a large amount of literature, books, reports, professional documents and other materials related to geology and related disciplines were collected extensively. To ensure the quality and diversity of the data, a systematic data collection and processing method was adopted. The main sources of the corpus include:

[0070] (1) Geological literature: We have collected a wide range of literature from geology-related journals, paper websites, and academic publications, covering multiple research fields to ensure the professionalism and diversity of the data.

[0071] (2) Geology books: We carefully selected authoritative books in the field of geology, including textbooks, professional manuals and scientific works. The content covers basic geological theories, technical methods and the latest research results, which can provide the model with both depth and breadth of knowledge.

[0072] (3) Online geological resources: Various data, including reports, dictionaries, etc., were collected from geological-related platforms and websites. These resources supplement the model with domain knowledge, professional terminology, and research results.

[0073] (4) Geological professional documents: We obtained some non-public geological prospecting-related professional documents to introduce unique knowledge reserves into the model and further enhance its application capabilities in the prospecting field.

[0074] This textual material was then processed through a self-built data preprocessing pipeline to form a dataset suitable for pre-training. The data for fine-tuning instructions primarily came from two sources: a self-built specialized instruction dataset in the field of geology and an open-source instruction dataset in the field of earth sciences. Due to the lack of existing open-source instruction data in the field of geology, we collaborated with experts in the field of geology and leveraged the generative capabilities of the open-source Large Language Model (LLM) in Chinese to construct a specialized instruction dataset covering disciplines such as geology, mineralogy, and geography. This specialized dataset enabled the model to more accurately understand and address the requirements of tasks in geology and related fields.

[0075] In the adaptive training process in the geological field, the open source large language model Qwen2-7B was selected as the model base, including two key stages: continued pre-training and supervised instruction fine-tuning.

[0076] In the pre-training phase, Qwen2-7B is initialized and trained on a geological corpus of 181.66M tokens. This step aims to infuse specialized geological domain knowledge into the general Qwen2-7B model.

[0077] This pre-training phase was conducted on a high-performance computing cluster based on the Hygon DCU architecture, integrated with the LLaMA-Factory training framework, using a multi-machine, multi-GPU distributed training model. Each node in the computing cluster was equipped with a 32-core CPU, 128GB of memory, and four DCU accelerator cards, each with 16GB of video memory. Each accelerator card boasts approximately 0.2 times the computing power of an NVIDIA A100 GPU. During training, repeated comparisons and performance testing revealed that increasing the number of nodes is not the only way to improve training efficiency. Inter-node communication overhead significantly impacts overall training performance, making finding the optimal number of nodes crucial. To this end, four computing nodes with a total of 16 DCU accelerator cards were selected for training, which lasted 160 hours. Parameter configuration included a learning rate of 1e-5, a global batch size of 256, a micro-batch size of 1, and gradient accumulation of 16. Training was conducted over three epochs, corresponding to 1755 training steps. At the same time, to reduce video memory usage during training, DeepSpeed ​​ZeRO3 technology was used for video memory optimization, further alleviating video memory pressure. Ultimately, the pre-trained model was obtained, which for better differentiation is called GeoQwen.

[0078] After completing the pre-training on the geological corpus, a domain-transferred GeoQwen model was obtained, and instruction fine-tuning was performed based on this model. In this stage, two types of instruction datasets were used for fine-tuning: one is the public earth science instruction dataset Earth, and the other is a self-constructed geological field professional instruction dataset. Since geology is a subfield of earth science, different fine-tuning data hybrid training paradigms were explored, including: (1) hybrid training method; (2) phased training method, that is, first fine-tuning the earth science instruction data, and then deeply fine-tuning the geological instruction data. The design of the second training method follows the laws of human cognition and adopts a learning method from basic to professional. In instruction fine-tuning, PEFT technology, especially the LoRA method, was used. This method can not only significantly reduce the consumption of computing resources, but also accurately adjust the model parameters. In some cases, it can achieve better results than full parameter fine-tuning. In this stage, the learning rate is set to 1e-5, the global batch size is 32, the micro-batch size is 1, and the gradient accumulation is 2. As for the LoRA settings, lora_r is set to 8, lora_alpha is set to 16, lora_target is set to all, and fine-tuning is performed for 3 epochs.

[0079] We then explored the TG-MoLE approach's ability to mitigate catastrophic forgetting during continuous fine-tuning of pre-trained models, comparing it to existing methods. As previously mentioned, the overall expert was divided into two groups: one responsible for maintaining general knowledge and the other focused on learning downstream tasks in the geological domain. In the experiment, GeoQwen was used as a starting point for instruction fine-tuning.

[0080] Baselines. To evaluate the performance of TG-MoLE, we compared it with LoRA and the latest LoRAMoE method.

[0081] Data Resources. We randomly sampled a portion of the high-quality, diverse, open-source instruction datasets, Alpaca-GPT4-zh and Belle, constructed using the Self-Instruct method, as general instruction data. We also used a geological instruction dataset as specialized instruction data. The ratio of general instruction data to geological instruction data was 1:1. This data was split into 90% for training and 10% for testing.

[0082] Parameter Settings. Due to resource constraints, four experts were used: the first two served as a general expert group, and the last two served as a geological domain expert group. Following the LoRAMoE parameter settings, the hyperparameters controlling the constraint strength β and the imbalance degree δ were both set to 0.1. For the LoRA settings, lora_rank and lora_alpha were set to 4 and 32, respectively, lora_dropout was set to 0.05, the maximum segment length was 512, and lora_target was set to "gate_proj, down_proj, up_proj."

[0083] Evaluation Benchmarks and Metrics. To investigate whether large language models in the geological domain experience catastrophic forgetting during continuous learning, and whether the TG-MoLE architecture can alleviate this problem while ensuring improvements in downstream geological tasks, the evaluation is divided into two parts: geological domain capability evaluation and general capability evaluation. The geological domain capability evaluation continues to use the GeoEval dataset, while the general capability evaluation utilizes evaluation datasets such as CMMLU, C-Eval, C3, NQ, and BoolQ. The general capability of the model is evaluated across multiple dimensions, including knowledge implication, reading comprehension, and common sense question answering.

[0084] To verify the superiority of the geological large model and the proposed method, an evaluation was carried out on subjective and objective tasks in the GeoEval evaluation dataset as well as on relevant general capability benchmarks.

[0085] First, for the objective tasks GeoKU and MinQA in GeoEval, appropriate prompts were given, ending with "The answer is" to guide the model to predict the next token. During the evaluation process, greedy decoding was used to obtain the model response, and a regular expression was used to extract one of the four options A, B, C, or D from the model response as the model-generated answer. Finally, the accuracy score was calculated by comparing the generated answer with the standard answer. The results show that the proposed model outperforms baseline models with similar parameter sizes on both objective tasks in GeoEval: geological knowledge comprehension (GeoKU) and prospecting knowledge question answering (MinQA), and also has certain advantages over Qwen1.5-14B, which has a larger number of parameters. Compared with the base model Qwen2-7B, GeoProspect improved by 2.96% on MinQA, which strongly confirms the effectiveness of the training method.

[0086] Secondly, in the subjective task evaluation of GeoEval, two automatic methods, BERTScore and perplexity, are mainly used to measure the output quality. In addition, the more powerful Qwen2-72B-Instruct is used to evaluate the baseline model, and accuracy is used as the scoring metric. Specifically, the output of each baseline model is collected and scored based on the answers generated by the model and the reference answers. Because the scores of large language models may fluctuate, each scoring process is called three times and the average is taken as the final scoring result.

[0087] In addition, in order to comprehensively evaluate the actual performance of the model, a manual evaluation was conducted. An evaluation set of 50 geological problems was constructed, covering basic geological knowledge and complex problems in prospecting applications. Ten geological practitioners were invited to evaluate the model output. The evaluation dimensions included: (1) rationality, whether the content generated by the model is logically reasonable; (2) correctness, whether the content generated by the model is consistent with the facts; (3) consistency, whether the content generated by the model is consistent with the requirements of the problem or task. The score is based on a percentage system and is divided into four levels: excellent, good, medium and poor. The manual evaluation results show that the proposed model is significantly better than the baseline model in all three dimensions mentioned above, which verifies the superiority of the model in understanding and applying geological knowledge.

[0088] Finally, the proposed TG-MoLE method was evaluated in terms of geological domain capabilities and general capabilities. The experimental results show that the TG-MoLE method outperforms the LoRAMoE method in both geological downstream tasks and multiple general capability evaluations. Specifically, in the evaluation of geological downstream tasks, the average accuracy of TG-MoLE reached 67.29%, which is significantly higher than that of the method using the local load balancing constraint (Llbc ) and the LoRAMoE version without this constraint, with improvements of 2.95% and 1.91% respectively. However, on Chinese comprehensive evaluation benchmarks such as CMMLU and C-Eval, the LoRA, LoRAMoE and TG-MoLE methods all showed a certain degree of forgetting, and their performance declined compared to the base model GeoQwen. In terms of specific performance, TG-MoLE was second only to LoRAMoE in the CMMLU benchmark evaluation, and second only to the LoRA method in the C-Eval benchmark evaluation. In addition, the three fine-tuning methods achieved performance improvements on the C3, NQ and BoolQ tasks, among which the TG-MoLE method achieved the most significant improvement. In terms of average performance indicators, TG-MoLE not only outperformed the LoRA and LoRAMoE methods, but also improved the performance by 8.08% compared to the base model GeoQwen. This experimental result strongly confirms the effectiveness and superiority of the TG-MoLE method in maintaining general task processing capabilities.

[0089] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are still within the scope of protection of the technical solution of the present invention.

Claims

1. A large language model for geology enhanced by continuous learning, characterized by: The method comprises the following steps: (1) Cross-group routing is defined as a task type-guided gating function, which is used to assign greater weights to expert groups that belong to the same task type as the sample. The cross-group gating function CG(·) is used as a cross-group router. This router assigns weights to each expert group based on the characteristics of the data sample and softly adjusts the cross-group weight distribution based on the task type. With x as input and the task type as task_type, its value 0 indicates general knowledge and its value 1 indicates geological domain knowledge, the output of the cross-group gating routing is expressed as: tg=one-hot(task_type) CG(·) init =Softmax(W cg x) Where tg(task_guidance) represents the one-hot encoding of task type guidance, CG(·) init represents the initial cross-group routing weight, W cg represents the trainable weight matrix of router CG(·), CG(·) represents the adjusted cross-group routing weight, and β is a learnable parameter used to control the degree of task type guidance; (2) All experts are divided into two groups with the same number of experts in each group. experts, and the intra-group gating function IG(·) is used as the intra-group router to assign weights to the intra-group experts according to the input x. The output of the intra-group gating routing is expressed as: <h2 style=";text-align:left;direction:ltr">IG(·) = Softmax(W<h2 style=";text-align:left;direction:ltr"> ig <h2 style=";text-align:left;direction:ltr"> x) W ig represents the trainable weight matrix of router IG(·); (3) The forward propagation process of TG-MoLE is expressed as follows: in represents all experts in group j, Ω j represents the routing weight assigned by cross-group routing to the j-th group of experts, ω i It represents the routing weight assigned to the i-th expert by the intra-group routing.

2. The continuously learning enhanced geological domain large language model according to claim 1, characterized in that: In step (3), a dense MoE structure is adopted to perform weighted fusion of the outputs of all experts.

3. The continuously learning enhanced geological domain large language model according to claim 2, characterized in that: The sources of the corpus of the large language model in the geological field enhanced by continuous learning include: geological literature, geological books, online geological resources, and geological professional documents.

4. The continuously learning enhanced geological domain large language model according to claim 3 is characterized by: In the large language model in the geological field enhanced by continuous learning, a professional instruction dataset covering geology, mineralogy and geography is constructed.

5. The continuously learning enhanced large language model for geological fields according to claim 4, characterized in that: In the adaptive training process in the geological domain, the open source large language model Qwen2-7B was selected as the model base, including continued pre-training and supervised instruction fine-tuning stages; in the continued pre-training stage, Qwen2-7B was initialized and trained on a geological domain corpus of 181.66M tokens to obtain a domain-transferred GeoQwen model; in the supervised instruction fine-tuning stage, the earth science instruction dataset Earth and professional instruction datasets were used for fine-tuning.

6. The continuously learning enhanced large language model for geological fields according to claim 5, characterized in that: The supervised instruction fine-tuning phase adopts a fine-tuning data hybrid training paradigm, including: There are two methods: hybrid training and phased training. The phased training method is to first fine-tune the earth science instruction data, and then deeply fine-tune the geological instruction data.

7. The continuously learning enhanced geological domain large language model according to claim 6, characterized in that: In the supervised instruction fine-tuning phase, the PEFT method is used, the learning rate is set to 1e-5, the global batch size is 32, the micro-batch size is 1, and the gradient accumulation is 2; the LoRA method is used, lora_r is set to 8, lora_alpha is set to 16, and lora_target is set to all, and a total of 3 epochs are fine-tuned.

8. The continuously learning enhanced large language model for geological fields according to claim 7, characterized in that: In step (3), GeoQwen is used as a starting point for instruction fine-tuning.

Images