Geological domain named entity recognition and classification method based on thinking chain and hybrid experts

Abstract

The invention discloses a geological domain named entity accurate recognition and classification method based on thinking chain enhancement and hybrid expert architecture, which is characterized by comprising the following steps: firstly, extracting geological document text data through an OCR (Optical Character Recognition) technology, and extracting structured entity data by utilizing a locally deployed large language model; then calling a local large model to generate a diversified sentence pattern template according to language styles in the geological field, filling the template with the extracted professional entities, and constructing an instruction fine tuning data set; further constructing a thinking chain (CoT) enhanced data set on the basis, and explicitly simulating an expert reasoning process; efficient fine tuning is carried out on the large model by innovatively combining a low-rank adaptation (DoRA) technology and a hybrid expert (MoE) architecture, the DoRA technology carries out dimension reduction decomposition and orthogonal transformation on weight matrixes of a decoder layer and a multi-layer perceptron, and the MoE architecture constructs a plurality of special sub-networks to enhance the multi-task processing capability; and finally, performing entity extraction on the geological document by using the fine-tuned model, outputting an identification result containing a reasoning process, and filtering and perfecting the result through a rule matching mechanism. According to the method, the problems of fuzzy boundary, indefinite semantics, difficulty in classification and the like of the named entities in the geological field are effectively solved, the recognition and classification accuracy of the named entities in the geological field is remarkably improved, and key technical support is provided for downstream applications such as geological resource exploration and mineral evaluation.

Description

Technical Field

[0001] The present invention mainly belongs to the fields of geology, geophysics, natural language processing, and artificial intelligence applications. Specifically, it relates to a method for identifying and classifying geological domain named entities based on the chain of thought and mixture of experts. Background Art

[0002] The identification and classification of geological domain named entities is an important task in geological research and applications. Geological named entities include rock types, geological ages, tectonic units, etc. The accurate identification and classification of these entities are of great significance for geological academic tasks, geological data management, etc. In geological research and production practice, accurate named entity recognition can help geologists better understand geological phenomena, predict resource distribution, and formulate reasonable development strategies, thereby improving resource exploration efficiency and geological disaster prevention and control capabilities. However, the text data in the geological field has complex professional terms, diverse language structures, and rich semantic levels, making it difficult for traditional named entity recognition methods to meet actual needs.

[0003] Traditional deep learning methods use recurrent neural networks, which are inefficient and unable to understand deep text semantics, and are difficult to handle complex structures and dynamic changes. In recent years, with the development of artificial intelligence technology, large language models in the field of natural language processing have made significant progress, providing new ideas for the identification of geological domain named entities. Pretrained models trained on large text datasets can understand text semantics at a deeper level. On this basis, fine-tuning the large model for different needs enables the large model to adapt to various specific domain requirements and achieve good performance. Summary of the Invention

[0004] In view of the above, the purpose of the present invention is to provide a method for identifying and classifying geological domain named entities based on the chain of thought and mixture of experts, which can efficiently and accurately identify and classify geological named entities, and at the same time has good flexibility and adaptability, reducing the cost and burden of extracting geological domain named entities to meet the complex and changeable named entity recognition requirements in the geological field.

[0005] A. Structured entity data extraction and construction of instruction fine-tuning dataset:

[0006] Optical character recognition technology is used to extract geological document data, and prompt engineering is implemented through local deployment of a large model to extract structured entity data; collect geological object descriptive documents containing structural features such as tables and paragraph entries; use the local large model to generate diverse sentence templates according to the language characteristics of the geological field; apply the entity filling algorithm to map the extracted geological domain professional entities into the sentence templates to construct a domain-specific instruction fine-tuning dataset;

[0007] B. Construction of the chain of thought reasoning process and generation of enhanced dataset:

[0008] Implement chain-of-thought enhancement for the instruction fine-tuning dataset. By adjusting the prompt instruction structure, explicitly construct intermediate reasoning steps including syntactic analysis, entity boundary recognition, semantic annotation, etc.; according to geological expert knowledge documents and the classification characteristics of different geological entities, generate a complete reasoning link for each recognition task, construct a chain-of-thought enhanced dataset, and improve the model's recognition ability for complex geological named entities and the interpretability of the reasoning process;

[0009] C. Low-rank parameter decomposition and weight orthogonal adaptation fine-tuning:

[0010] Select the Qwen-2.5 pre-trained model as the base model and implement DoRA parameter-efficient fine-tuning; decompose the weight matrices of the model decoder and the multi-layer perceptron into two orthogonal components, namely the amplitude vector and the direction matrix; enhance the model's representation ability for geological specific terms through amplitude vector optimization; adopt low-rank orthogonal transformation technology to sparsely update the direction matrix and capture the hierarchical relationship between geological entities; achieve parameter subspace optimization, significantly reduce the computational complexity and maintain the model performance;

[0011] D. Conditional activation of multi-expert architecture and balance strategy implementation:

[0012] Integrate the MoE (Mixture of Experts) expert module into the fine-tuning model architecture and deploy multiple dedicated parameter matrices in each network layer; implement an adaptive gating mechanism, use the layer input features as the conditional signal, and dynamically calculate the expert activation weights; achieve conditional calculation through the gating mask and perform feature fusion on the outputs of each expert matrix; introduce a balance loss function and a load balancing algorithm to optimize the expert activation distribution and improve the model's recognition and classification ability for multiple types of geological entities;

[0013] E. Structural consistency verification and semantic filtering processing:

[0014] Use a rule-based program to filter and enhance the results generated by the large model. First, judge whether the results generated by the large model conform to the target structure in the instruction fine-tuning. If the structure does not match, use the infix expression matching algorithm to repair the structure. Then, check whether the labels of the entities in the results belong to the pre-defined categories in the geological field, and delete the mis-extracted data that do not belong. Through this processing, ensure that the output results have a standardized structure and accurate semantics, and improve the reliability and practicality of named entity recognition and classification in the geological field. Brief Description of the Drawings

[0015] Figure 1 is the model structure diagram of the present invention Detailed Embodiments

[0016] The technical solutions of the present invention will be described in detail below in combination with specific embodiments.

[0017] A. Structured Entity Data Extraction and Instruction Fine-tuning Dataset Construction

[0018] (1) Collect a large number of descriptive documents in the geological field, and extract the text data in the documents by combining optical character recognition technology with scripts.

[0019] (2) Use a locally deployed large language model through prompt engineering to extract structured data with target text features from the text obtained above, and classify them at the same time. Features such as:

[0020] Code Name Description 1301 Bohai Bay Basin Also known as the Bohai Basin

[0021] (3) Use the descriptive documents in the geological field collected above as a local corpus, and use a local large language model through prompt engineering to generate template sentences, such as:

[0022] The <tectonic unit> in the <basin> has an important influence on the formation of <oil and gas field>.

[0023] At the same time, generate robust data without target vocabulary.

[0024] (4) Fill in entities for each sentence pattern with the existing data. For example, in the above example, select the basin, tectonic unit, and oil and gas field for filling. First, randomly shuffle the order of the data within these three categories, and then fill in the entities in the order of the list to generate sentence patterns. Stop when each entity in all the lists has been filled in at least once, and the category with fewer entities is filled in cyclically.

[0025] (5) Use a script to convert the obtained training text data into the format of instruction fine-tuning, and the format is as follows:

[0026]

[0027] {"entity_text": "North Margin Uplift Area", "entity_label": "tectonic unit"},

[0028] {"entity_text": "Block 12", "entity_label": "well number"},

[0029] {"entity_text": "Kabashiqikike Formation", "entity_label": "horizon"}],

[0030] "text": "Drilling in Block 12 of the North Margin Uplift Area of the Tarim Basin revealed the mudstone channel facies of the Kabashiqikike Formation."}

[0031] B. Construction of the Chain of Thought Reasoning Process and Generation of an Enhanced Dataset:

[0032] (1) Implement chain-of-thought enhancement for the instruction fine-tuning dataset. By adjusting the prompt instruction structure, explicitly construct intermediate reasoning steps including syntactic analysis, entity boundary determination, semantic category judgment, etc. For the oil and gas field domain, the constructed specialized chain-of-thought reasoning template is as follows:

[0033] Input: The test results of Well K1 drilled in the Beiduanshan Dome are good.

[0034] Reasoning steps:

[0035] 1. Syntactic parsing: Analyze that "Beiduanshan Dome" is the subject part, "drilled" is the predicate, "Well K1" is the object, and "the test results are good" is additional information.

[0036] 2. Entity candidate recognition and boundary determination:

[0037] - "Beiduanshan Dome": A geological structure noun phrase, a complete structural unit entity

[0038] - "K1": A combination of Chinese characters and numbers, conforming to the well number naming rule

[0039] 3. Semantic category judgment:

[0040] - "Beiduanshan Dome": Contains the structural term "dome", representing a specific geological structure, belonging to the "structural unit" category

[0041] - "K1": The combination format of Chinese characters and numbers conforms to the well number standard, the prefix "K" conforms to the rule of "starting with a Chinese character of the regional name", and the number part represents the well serial number, belonging to the "well number" category

[0042] 4. Result verification: Check the rationality of the relationship between entities, whether the entity categories conform to the predefined scope, and the accuracy of boundary division.

[0043] Output:

[0044] [{"entity_text":"Beiduanshan Dome","entity_label":"structural unit"},

[0045] {"entity_text":"K1","entity_label":"well number"}]

[0046] (2) According to the expert knowledge document in the oil and gas field, formulate professional entity discrimination rules for the recognition task, such as: Well number discrimination rule: Usually composed of a combination of Chinese characters and numbers, such as "K1";

[0047] (3) Construct a chain-of-thought enhanced dataset to improve the model's recognition ability of complex named entities in the oil and gas field and the interpretability of the reasoning process. Each record in the enhanced dataset contains:

[0048] ①Original oil and gas exploration and development texts

[0049] ②Detailed reasoning steps (including syntactic analysis, entity boundary recognition, semantic annotation)

[0050] ③Final structured entity annotation results

[0051] Chain-of-thought enhancement enables the model to handle complex boundary determination problems, such as distinguishing nested entities in "Ledong Uplift, Bohai Bay Basin", improving the accuracy and reliability of named entity recognition in the professional field.

[0052] C. Low-rank parameter decomposition and weight orthogonal adaptive fine-tuning:

[0053] (1) For a model based on a mixture-of-experts architecture, implement DoRA (Decomposed Orthogonal Rank Adaptation) parameter-efficient fine-tuning. The DoRA technique is particularly suitable for handling professional terms and complex entities in the oil and gas field, and optimizes the parameter update process through orthogonal decomposition techniques;

[0054] (2) Decompose the weight matrices of the model decoder and the multi-layer perceptron into two orthogonal components: an amplitude vector and a direction matrix. The original parameter matrix of the model is DoRA decomposes W into:

[0055] W0 = s × A

[0056] where, is the amplitude vector, is the direction matrix, and for each row i, ||A i ||2 = 1 (i.e., each row of A is a unit vector). This decomposition method is particularly suitable for handling the characterization optimization problems of a large number of professional terms in the oil and gas field;

[0057] (3) Enhance the model's characterization ability for oil and gas specific terms through amplitude vector optimization. During fine-tuning, the amplitude vector s is updated by gradient descent:

[0058]

[0059] where η is the learning rate and L is the loss function. This update mechanism enables the model to precisely adjust the characterization importance of different types of entities such as "horizon" and "lithology";

[0060] (4) Adopt low-rank orthogonal transformation technology to sparsely update the direction matrix and capture the hierarchical relationship between entities in the oil and gas field. The update of the direction matrix A is achieved through low-rank decomposition:

[0061] ΔA = UV T

[0062] Among them, U, V, and r are low-rank factors. For the oil and gas field, this decomposition is particularly beneficial for capturing hierarchical nested relationships such as "basin - tectonic unit - oil and gas field - well location".

[0063] (5) Implement parameter subspace optimization to significantly reduce the computational complexity and maintain the model performance.

[0064] The complete DoRA update process includes:

[0065] · Forward propagation: Calculate the model output using the current s and A;

[0066] · Loss calculation: Calculate the loss function L in combination with the characteristics of the oil and gas entity recognition task;

[0067] · Gradient calculation: and

[0068] · Magnitude vector update:

[0069] · Low-rank update of the direction matrix: ΔA = UV T ;

[0070] · Normalized update of the direction matrix: A (t+1) = normalize(A t + ΔA)

[0071] This fine-tuning strategy is particularly suitable for building professional models in the oil and gas field in resource-constrained environments, and can obtain high named entity recognition accuracy at a low computational cost.

[0072] D. Implementation of conditional activation multi-expert architecture and balance strategy

[0073] In the geological named entity recognition task, introducing a hybrid expert architecture is a key step to ensure that the large model can adapt to chain-of-thought multi-task processing. The present invention selects a hybrid expert architecture with gated routing, and inserts the hybrid expert architecture into the attention layer of the pre-trained large prediction model Qwen2.5-Instruct to implement model fine-tuning. The specific process is as follows:

[0074] (1) Selection and transformation of the pre-trained large model

[0075] Qwen2.5 is an advanced pre-trained large language model with powerful language understanding and generation capabilities. It has learned rich language knowledge and semantic information through unsupervised pre-training on a large-scale text corpus. Qwen2.5-Instruct is a pre-trained model specialized for instruction fine-tuning, which can better meet the requirements of the structured output of the present invention.

[0076] First, extract the attention layer and multi-layer perceptron layer from the Qwen 2.5 model, attach the fine-tuning layer constructed based on the mixture-of-experts architecture to the layer where the target fine-tuning parameters are located, and transform its feed-forward function into:

[0077] o = W0x + M(x)

[0078] where W0 is the parameter of the original linear layer, and M(x) is the output of the fine-tuning layer. The result is fine-tuned by adding the output of the fine-tuning layer constructed based on the mixture-of-experts architecture to the output of the original linear layer.

[0079] Finally, freeze the parameter W0 in the transformed linear layer, and only update the parameters of the fine-tuning layer M(x) in the subsequent fine-tuning training.

[0080] (2) Construct the mixture-of-experts fine-tuning layer

[0081] Use the input dimension d of the target fine-tuning linear layer in and the output dimension d out as parameters to construct the basic fine-tuning layer, which is constructed using the DoRA method in the present invention and is called a single expert. The model creates multiple experts, and the number depends on the hyperparameter en of the model output. And use the input dimension of the target linear layer as the input dimension and the number of experts as the output dimension to construct the gating routing linear layer. Calculate the output of the gating routing linear layer, and use the softmax function to process the output result. The calculation process is as follows:

[0082] G(x) = Softmax(x × W g )

[0083]

[0084] G(x) is the weight of en intervals between 0 and 1, {w1, w2, w3, …, w en}, take the first n values of w with the largest values i and their subscripts j to obtain the selected gating weights {w1, w2, w3, …, w n} and their indices {I1, I2, I3, …, I n}. Then normalize the obtained gating weights, and the calculation formula is as follows:

[0085]

[0086] Obtain the corresponding experts according to the index and calculate the output results of these experts, and perform weighted summation of the output results according to the gating weights. The calculation process is as follows:

[0087]

[0088] where is the output result of the expert with index I i .

[0089] (3) Construct the local balance constraint loss function

[0090] If the mixture of experts architecture is directly fine-tuned without any constraints, the gating function usually converges to the situation where a few experts have the right to speak. Therefore, a local balance constraint loss function needs to be constructed to avoid this situation.

[0091] First, define an importance matrix Q for each fine-tuning layer. Q n,m represents the importance of the m-th sample in a batch of training data on the n-th expert, and the calculation formula is as follows:

[0092]

[0093] where T m is the total number of tokens of the m-th sample, x i is the word embedding vector of the i-th token in the sample, and G(x) n is the n-th weight in the weight vector output by the gating function. τ is a model hyperparameter. Use the variance and mean of the importance matrix to construct the loss function:

[0094]

[0095] The final loss is the original loss of this model plus the sum of the balance constraint losses L lbc of all fine-tuning layers. The latter is multiplied by a weight β, which is a hyperparameter controlling the strength of the local balance constraint. The calculation formula is:

[0096]

[0097] E. Structural consistency verification and semantic filtering processing

[0098] Sometimes there are some structural or result problems in the output results of the large model. By performing structural verification and semantic filtering on the output results through a program, the output of the large model can be effectively normalized, and the entity extraction results irrelevant to the geological field can be filtered out, thereby improving the accuracy and reliability of the results.

[0099] (1) Verify whether the output result of the large model is the target json structure of the above geological field instruction fine-tuning data. If it cannot be recognized, try to repair it using the method of infix expression matching. If it cannot be repaired, directly output the result.

[0100] (2) Check whether the label of each entity in the output result belongs to the predefined categories in the geological field. If the output result is not within the existing detection types, delete this piece of data.

Claims

1. A method for accurate identification and classification of named entities in the geological field based on chain-of-thought enhancement and mixture-of-experts architecture, characterized in that Including the following steps: A. Structured entity data extraction and instruction fine-tuning dataset construction: Optical character recognition technology is used to extract geological document data, and prompt engineering is implemented through locally deployed large models to extract structured entity data; collect geological object descriptive documents containing structural features such as tables and paragraph entries; Use the local large model to generate diverse sentence templates according to the language characteristics of the geological field; apply the entity filling algorithm to map the extracted geological field professional entities into the sentence templates to construct a domain-specific instruction fine-tuning dataset; B. Construction of the chain of thought reasoning process and generation of an enhanced dataset: Implement chain of thought enhancement on the instruction fine-tuning dataset. By adjusting the prompt instruction structure, explicitly construct intermediate reasoning steps including syntactic analysis, entity boundary recognition, semantic annotation, etc.; according to the geological expert knowledge document and the classification characteristics of different geological entities, generate a complete reasoning link for each recognition task, construct a chain of thought enhanced dataset, and improve the model's recognition ability for complex geological named entities and the interpretability of the reasoning process; C. Low-rank parameter decomposition and weight orthogonal adaptation fine-tuning: Select the Qwen-2.5 pre-trained model as the base model and implement DoRA parameter-efficient fine-tuning; decompose the weight matrices of the model decoder and the multi-layer perceptron into two orthogonal components, namely the amplitude vector and the direction matrix; enhance the model's representation ability for geological specific terms through amplitude vector optimization; use low-rank orthogonal transformation technology to sparsely update the direction matrix to capture the hierarchical relationship between geological entities; achieve parameter subspace optimization, significantly reduce the computational complexity and maintain the model performance; D. Conditional activation of the multi-expert architecture and implementation of the balance strategy: Integrate the MoE (Mixture of Experts) expert module into the fine-tuning model architecture and deploy multiple dedicated parameter matrices in each network layer; implement an adaptive gating mechanism, use the layer input features as conditional signals, and dynamically calculate the expert activation weights; achieve conditional calculation through the gating mask and fuse the features of the outputs of each expert matrix; Introduce a balance loss function and a load balancing algorithm to optimize the expert activation distribution and improve the model's recognition and classification ability for multiple types of geological entities; E. Structural consistency verification and semantic filtering processing: Use a rule-based program to filter and enhance the results generated by the large model. First, judge whether the results generated by the large model conform to the target structure in the instruction fine-tuning. If the structure does not match, use the infix expression matching algorithm to repair the structure. Then, check whether the labels of the entities in the results belong to the predefined categories in the geological field, and delete the mis-extracted data that does not belong. Through this processing, ensure that the output results have a standardized structure and accurate semantics, and improve the reliability and practicality of geological field named entity recognition and classification.

Images