Multi-feature fusion ancient language named entity recognition method based on large language model

Abstract

The invention discloses a multi-feature fusion ancient language named entity recognition method based on a large language model. The method comprises the following steps: extracting sentence semantic vectors by utilizing a pre-trained ancient language model GuwenBERT; sanza is adopted for dependency syntactic analysis, and structural features such as subject-called dependency, centering dependency and syntactic tree depth are extracted; the semantic features and the structural features are spliced to form fusion representation, and a similarity index is constructed based on FAISS to retrieve a small number of annotation examples; a large language model is guided to complete clue recognition, reasoning judgment and entity labeling in sequence through structured reasoning Prompt, and automatic recognition of personal names, place names, official names and time names in ancient texts is achieved. According to the method, the accuracy and robustness of ancient language named entity recognition are remarkably improved under the small sample condition, and an efficient technical scheme is provided for ancient book digitization and knowledge graph construction.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing and artificial intelligence technology, specifically involving a multi-feature fusion method for ancient Chinese named entity recognition based on a large language model, which is applicable to application scenarios such as ancient book digitization, knowledge graph construction, and intelligent analysis of historical documents. Background Technology

[0002] With the development of digital humanities research, the digitization of large-scale classical Chinese corpora has become a research hotspot. Named Entity Recognition (NER) is one of the core tasks of classical Chinese information extraction, aiming to automatically identify semantic entities such as personal names, place names, official titles, and time names from ancient Chinese texts. However, classical Chinese is characterized by complex grammatical structures, widespread omissions, and high lexical ambiguity, making traditional rule-based or statistical learning-based NER methods difficult to achieve ideal results. Although deep learning and Transformer models have performed well on modern Chinese NER tasks in recent years, their transfer effects are limited due to the scarcity of annotated classical Chinese corpora and significant differences in semantic and syntactic features.

[0003] Existing technologies mainly suffer from the following problems: lack of semantic modeling mechanisms for classical Chinese features; strong sample dependence. Supervised learning models require a large amount of labeled data, while classical Chinese NER has high labeling costs and a limited number of samples; the prompting mechanism is simple: existing large language models lack structured reasoning ability in zero-sample or few-sample classical Chinese NER tasks, resulting in low recognition accuracy.

[0004] Therefore, it is necessary to design an ancient Chinese NER method that can integrate semantic features, syntactic features, and contextual priors under low resource conditions. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention proposes a multi-feature fusion method for ancient Chinese named entity recognition (GuNER-SR) based on a large language model. This method combines a pre-trained semantic model with syntactic feature extraction tools, and achieves accurate recognition of ancient Chinese entities by constructing multi-source feature representations and structured reasoning prompts. The method includes the following steps: (1) Basic semantic feature extraction: The basic semantic vector representation of the input text is obtained using a pre-trained GuwenBERT model; (2) Extraction of syntactic structural features: The Stanza dependency parser is used to extract syntactic structural features such as subject-predicate relations, attributive-head relations and syntactic tree depth in classical Chinese texts; (3) Feature fusion and retrieval enhancement: Semantic vectors and syntactic features are concatenated to form a multi-feature representation. A similarity index library is built through FAISS, and the most relevant examples are quickly retrieved using the inverted file index (IndexIVFFlat). (4) Structured prompt generation: Based on the search results, construct a three-stage structured reasoning prompt (clue recognition, reasoning, judgment, and label generation) to guide the large language model to complete entity annotation; (5) Large language model reasoning and output: The large language model outputs annotation results in the format @@entity|category## based on the prompts, and generates the final named entity recognition result.

[0006] Compared with existing methods, this invention has achieved significant performance improvements on multiple publicly available ancient Chinese entity recognition datasets, with an F1 score consistently exceeding 88%, enhancing the understanding and reasoning capabilities of large models in ancient Chinese scenarios. Attached Figure Description

[0007] Figure 1 This is a detailed flowchart of the implementation of the present invention. Figure 2 This is a model framework diagram of the present invention. Detailed Implementation 1. Basic semantic feature vector extraction First, the original classical Chinese text is segmented by character, and special markers "[CLS]" are added at the beginning and "[SEP]" at the end, resulting in the sequence S to be processed. Then, sequence S is input into a pre-trained GuwenBERT model, which uses word embedding layers and multi-layer Transformer encoders to obtain the initial hidden vector for each position. H(0) The main purpose of this step is to map discrete textual information into continuous vector representations, which facilitates subsequent processing by deep models.

[0008] To enhance the modeling ability for long-distance dependencies and complex grammatical structures in classical Chinese, a multi-head self-attention mechanism (MHSA) is introduced. Specifically, H(0) is mapped to... Query, Key, Value: Subsequently, for each attention head i Calculate the attention output: By concatenating the outputs of all heads and applying a linear mapping, the multi-head attention result can be obtained. To ensure stable training, they also... The residual is added to the original input, and the original information is preserved through residual connections and layer normalization (LayerNorm), thus mitigating the gradient vanishing problem. Finally, the basic semantic vector P is obtained through a feedforward neural network and a second residual normalization.

[0009] This step enables the model to automatically focus on keywords in different positions within a sentence and capture long-distance dependencies. Here, LayerNorm() is the layer normalization function, FFN() is the feedforward neural network, W1 and W2 are weight matrices, and b1 and b2 are biasable terms.

[0010] 2. Extraction of syntactic structural features To accurately capture the inherent structural complexity of text, this module focuses on extracting a series of core features at the syntactic level. We first process the input text x using syntactic analysis tools and then define four full-sentence-level structural features based on dependency parsing trees.

[0011] We use the Stanza tool to perform syntactic analysis on the input sentence x, thereby constructing a labeled directed tree. .in: This represents the set of words or character nodes in a sentence, where n is the total number of nodes. It is a set of dependent arcs (edges), representing the dependency relationships between nodes.

[0012] Each dependent arc Accompanied by a dependency tag (like nsubj , amod , root wait).

[0013] Each node It has a part-of-speech tag ,like VERB , NOUN wait.

[0014] 3. Feature Fusion and Retrieval Enhancement Module 3.1 Feature Fusion To eliminate the dominant effect caused by differences in various features, we used the training set (support set) S Estimate the mean of each dimension on ) and standard deviation The feature vectors are then Z-score standardized. Here, is a very small positive number, used to avoid division by zero. The standardized eigenvector is denoted as . To further balance the importance of each structural feature, we introduce a learnable, dimension-wise weight matrix W, which is applied to the standardized structural feature vector to obtain the syntactic feature vector representation.Z : Syntactic feature vectors Z This is the final syntactic structure feature representation. Finally, feature fusion is used to establish the relationships between features. The basic semantic feature vector generated by Guwenbert is then used... P Syntactic Dependency Feature Vectors Z Perform a concatenation operation to obtain the comprehensive feature vector. : .

[0015] 3.2 Search Enhancement To achieve efficient retrieval, the input sentence x first needs to be transformed into a unified comprehensive feature vector. For the training sample set, we map each sample xi to a d-dimensional feature vector. Subsequently, to support efficient K-nearest neighbor (KNN) search on large-scale corpora, we use the FAISS (FacebookAISimilaritySearch) library to construct an index structure for the feature vectors. To optimize retrieval performance, this study adopts an index structure based on the inverted file index (IVF). The construction process of this index includes two main steps: (1) Vector space partitioning: First, the training vectors are partitioned... X conduct k- means Clustering, generation The central vector ,Will d The dimensional feature space is divided into Subspace. (2) Inverted list creation: each central vector Corresponding to an inverted list Store all items belonging to this subspace (i.e., those outside the subspace). The vector index of the most recent (recent) vector.

[0016] For each query sample to be predicted We combine its feature vector with the features of all training samples in the FAISS index. A comparison is performed to determine the similarity. This paper uses Euclidean distance (L2 distance) as the similarity measure. in q To query the feature vector of a sample, For the first in the training set i The feature vector of each sample d The dimension of the feature vector. This is calculated... We retrieve the vectors that are closest to the query vector from the training samples. k There are 10 samples. Assume the returned set of sample indices is 10. The set of top-k similar samples is as shown in the equation. 4. Prompt for building modules For the input text sample x The prompt constructor generates a composite prompt (CompositePrompt). P ( x The prompt consists of three parts: task instructions, clue identification prompts, and reasoning annotation prompts. Where: I(x): Task instruction, which defines the task objective, constraints and final output format; C(x): Clue identification prompt; R(x): Reasoning & Label Prompt.

[0017] Few-shot example provided by the retrieval enhancement module After integration, the complete input sent to the large language model can be formalized as: .

[0018] 5. Large Language Model Inference Module The LLM inference module receives the complete structured data generated by the prompting constructor module. The goal is to calculate the conditional generation probability and solve for the optimal output sequence. : Here are the parameters of the large language model. This is the optimal output sequence generated. Because... It contains retrieved Few-shot examples. The conditional generation probability of the model is actually based on retrieval augmentation. Autoregressive decoding mechanism: The DeepSeek model employs an autoregressive language modeling mechanism based on the Transformer architecture. Given a complete... Then, the model generates the target text step by step through the decoder. y Each step of the generation depends on the previously generated token sequence. y <t : Through this conditional probability chain, the model's output sequence y+ It has a recursive causal constraint structure to ensure that the output logic of cue recognition (CLUES) and diagnostic reasoning (REASONING) affects the final labeled output (LABEL) generation result.

[0019] 6. Experimental Setup and Training Details To address the need for small-sample experiments, the original corpus was preprocessed: only a few subsets with relatively balanced sample sizes for each category were selected from the dataset of over 4000 entries, ensuring that each entity class had a representative distribution in the training samples. Furthermore, to verify the generalization ability of the proposed method under low-annotation conditions, this paper constructed a small-scale Few-shot dataset containing 5 training samples (for KNN retrieval) and 5 test samples (for model evaluation).

[0020] The training environment used an NVIDIA GeForce RTX 4090 GPU and software based on Python 3.8.1, PyTorch 2.2.1, and CUDA 12.4. The retrieval algorithm was IndexFlatL2; KNN Top-K: k=3; the generation model was DeepSeek-14B; the maximum generation length was 512; and the generation temperature was 0.2.

[0021] 7. Results of the Example Experiments were conducted on the publicly available ancient text NER evaluation dataset GuNER2023. The results show that the method of this invention achieves an F1 score of 88% with only 5 training examples, significantly outperforming other baseline models with fewer samples. It demonstrates strong robustness and generalization ability in low-resource ancient text NER scenarios, validating the effectiveness and innovation of the proposed method.

[0022] The above description is only a preferred embodiment of the present invention. Those skilled in the art can make equivalent changes or improvements without departing from the principle of the present invention, and all such changes should be covered within the protection scope of the present invention.

Claims

1. A method for ancient text named entity recognition based on large language model multi-feature fusion, characterized in that, Includes the following steps: S1. Corpus Input and Preprocessing: Standardize the classical Chinese text by unifying simplified and traditional characters, segmenting sentences and removing irrelevant symbols to form an input corpus suitable for model processing. S2. Basic semantic feature extraction: The input sentence is encoded using the pre-trained GuwenBERT semantic model to obtain the semantic vector representation of the sentence, where the semantic vector is the deep representation of the sentence in the high-dimensional semantic space. S3. Syntactic Structure Feature Extraction: The Stanza dependency parsing tool is used to perform dependency parsing on the classical Chinese sentences, and syntactic feature vectors containing subject-verb dependency number, noun-head dependency number, core verb depth, and maximum syntactic tree depth are extracted. S4. Construction of Multi-Feature Fusion Representation and Similar Sample Retrieval Library: The semantic vector obtained in step S2 is concatenated with the syntactic structure feature vector extracted in step S3 to form a multi-feature vector representation that fuses semantics and structure, which is used for subsequent similarity retrieval; a sample index is built based on the FAISS vector retrieval library to index the multi-feature vectors of a small number of labeled samples in the training set. S5. Construction of Structured Prompts: Based on the retrieved examples, construct structured reasoning prompts, which include three stages: Clue Identification, which guides the model to identify possible entity cues; Diagnostic Reasoning, which requires the model to infer entity categories by combining context and syntactic features; The results output stage (Structured Labeling) outputs the recognition results in a specified labeling format; S6. Large Language Model Inference and Output: Input the structured prompts constructed in step S5 into the Large Language Model (LLM), guide the model to complete the clue extraction, inference decision and label generation in sequence, and output the ancient Chinese named entity recognition results.

2. The method according to claim 1, characterized in that: In step S2, the specific steps for extracting the basic semantic feature vector of the sentence are as follows: First, the ancient Chinese text is converted into ancient Chinese vectors using the pre-trained model GuwenBERT; then, a multi-head self-attention mechanism is introduced to capture richer contextual information in the ancient Chinese vectors; finally, residual connections, layer normalization (Add&Norm), and a feed-forward network (FFN) are used to obtain the basic semantic feature vector representation in order to capture the contextual semantics and the unique grammatical features of ancient Chinese.

3. The method according to claim 1, characterized in that: In step S3, the specific steps for extracting a series of core features at the syntactic level are as follows: (1) Subject-verb dependency count (f1(x), nsubj count): This feature calculates the number of instances of all nsubj (noun subject) dependency relations in a sentence, used to measure the number of core events or subject complexity of the sentence. f1(x)=#{(i→j)∈E|rel(i→j)=nsubj} (2) Adjective Modifier Count (f2(x), amod count): This feature counts the number of instances of all amod (adjective modifier) ​​dependencies in a sentence, reflecting the adjectival modifiers in the sentence. f2(x)=#{(i→j)∈E|rel(i→j)=amod} (3) Root verb count (f3(x)) In the Universal Dependencies (UD) specification, the syntactic tree points to the core head node of the sentence through a special virtual root node; we define the root verb set R(x) as the set of all nodes pointed to by the root relation and whose part of speech is verb or auxiliary verb: The number of root verbs, f3(x), is defined as the cardinality of the set and can be used to identify compound or multi-center sentence structures. f3(x)=|R(x)| (4) Maximum Dependency Tree Depth (f4(x)) This feature measures the maximum depth of the dependency syntax tree and is a commonly used indicator of sentence structure complexity and nesting degree. First, define the set of child nodes of a node as ch(i) = {j|(i→j)∈E}; the recursive depth of a node, depth(i), is defined as follows: The maximum dependency tree depth f4(x) is the recursion depth of the syntactic core head node r, which is uniquely determined by the root relation: f4(x) = depth(r).

4. According to the syntactic structure feature extraction step described in claim 3, the final step is to construct a feature vector and standardize it, combining the above four scalar features into a syntactic structure feature vector.

5. The method according to claim 1, characterized in that: In few-shot learning scenarios, traditional deep learning models are often difficult to train sufficiently due to data scarcity. Therefore, this study introduces a retrieval enhancement module (Retriever), which uses the K-nearest neighbor (KNN) retrieval mechanism to guide the model's inference process by using similar samples retrieved from a small amount of training data, thereby significantly improving the model's robustness and generalization ability under low-resource conditions.

6. The method according to claim 1, characterized in that, The Prompt Construction Module is the core component of this method for achieving few-shot named entity recognition, and its design is based on the Clue And Reasoning Prompting (CARP) theory. This module guides the Large Language Model (LLM) to simulate the human cognitive process through explicit step-by-step reasoning prompts, thereby achieving logically interpretable and reasoning-consistent named entity recognition.

7. The method according to claim 1, characterized in that, The Large Language Model-Based Inference Module (LLM-Based Inference Module) is the core execution unit of this system for completing the named entity recognition task. Based on the DeepSeek large model, this module adopts a generative thinking chain reasoning paradigm and completes knowledge representation and entity discrimination through language generation.

Images