A hierarchical context-aware key phrase extraction method and system with functional explainability

Abstract

The application discloses a hierarchical context-aware keyword extraction method with functional explainability, adopts a two-stage cascade architecture of supervised extraction enhanced by hierarchical context and functional semantic classification, and arranges a pre-extraction module to determine candidate keywords, arranges a post-lightweight student model to perform classification within the range of the candidate keywords, and reduces the risk of free generation of keywords outside the original text by a generative large model; an explicit modeling hierarchical context structure feature of "document-sentence-word element" is generated, and a word element-by-word element modulation parameter is generated according to the hierarchical context structure feature to modulate sequence modeling features, so that the model has a global structure perception ability for a long document; the keywords can be accurately mapped to five dimensions of "method-task-data-index-domain", and the extraction result has clear explanation value, and can be directly used for construction of a downstream academic knowledge graph.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing and text mining technology, and relates to a hierarchical context-aware keyword extraction method, specifically a hierarchical context-aware keyword extraction method and system with functional interpretability. Background Technology

[0002] Keyword extraction is a core step in connecting unstructured scientific and technological texts with structured knowledge intelligence, and it is widely used in downstream tasks such as scientific and technological intelligence mining, technology trend analysis, and information retrieval.

[0003] Deep learning-based supervised sequence labeling is currently the mainstream paradigm for high-precision keyword extraction. It utilizes pre-trained language models (neural networks pre-trained on large-scale corpora capable of extracting deep semantic representations of tokens, such as BERT and DeBERTa) to obtain deep contextual representations of tokens (the basic language units obtained after word segmentation). This is combined with Mixture-of-Experts (MoE) models (neural networks consisting of gated routing and multiple expert subnetworks, dynamically activating some experts based on input) or Conditional Random Fields (CRF). Field, CRF (a sequence labeling statistical model that can model the transition relationship between adjacent tags) captures specific noun phrases and syntactic structures, and models keyword extraction as a BIO sequence labeling task (i.e., labeling each lexical with three types of tags: B-Begin, I-Inside, and O-Outside), which to some extent alleviates the problem of insufficient semantic understanding ability of traditional statistical methods; however, such solutions essentially rely on local context windows, making it difficult to model the hierarchical relationship of "document-sentence-lexical" from a macro perspective. As a result, the model often can only extract locally significant words, while missing global core concepts that run through the whole text but are not prominent locally, making it difficult to accurately assess the global importance of long-tail words in the overall logic of the text.

[0004] Another approach alleviates the keyword omission problem in long texts by explicitly modeling the hierarchical structure of documents, which to some extent solves the problem of missing global core concepts under the traditional flat sequence perspective. However, such approaches mostly adopt an unsupervised paradigm, which cannot identify the specific functional roles of words in the article's logic (such as "core methods", "research tasks", "experimental datasets", etc.). The extracted keywords are only discrete symbols, which are difficult to use directly to build high-quality heterogeneous knowledge graphs. Moreover, the effectiveness of the approach depends on specific unsupervised assumptions and is difficult to stably adapt to industrial applications.

[0005] With the emergence of large language models (LLMs) such as GPT (Gross Language Platforms), which are giant neural network models trained on massive amounts of text and possess powerful language understanding and generation capabilities, some solutions use prompts (instruction text input to the language model) to allow the large model to directly read documents and generate explanatory keyword lists. However, existing solutions that directly call large language models with extremely large parameters for extraction not only suffer from a serious "illusion" phenomenon (i.e., the model generates seemingly plausible but actually non-existent terms in the original text), but also have unacceptable computational costs and inference delays of tens of seconds when dealing with databases of millions of academic papers. Furthermore, the random generation characteristics of large models lead to unstable output formats, making them difficult for automated scripts to parse accurately.

[0006] In summary, existing technical solutions have failed to simultaneously address the three technical problems of insufficient global structure perception of long texts, lack of functional semantic roles in the extraction results, and high cost and susceptibility to illusion when directly applying large language models. These are the technical problems that this invention aims to solve. Summary of the Invention

[0007] In order to overcome the defects of the existing technology, the present invention aims to provide a hierarchical context-aware keyword extraction method and system with functional interpretability, so as to solve at least one of the technical problems of difficulty in extracting long-tail keywords from long texts, lack of semantic roles in the extraction results, and high cost and easy illusion of directly applying large language models.

[0008] To solve the above problems, the present invention adopts the following technical solution: Firstly, a hierarchical context-aware keyword extraction method with functional interpretability includes the following steps: S1. The document to be extracted is segmented into words and encoded into initial context vector representations of word elements; S2. Perform dual-channel context feature enhancement on the initial context vector representation in parallel to obtain local feature vectors and hierarchical context structure features; S3. Based on the local feature vector and the hierarchical context structure features, a candidate keyword set is generated through sequence modeling, word-by-word modulation and sequence labeling decoding. S4. Construct a supervised fine-tuning dataset, and perform model fine-tuning on the lightweight student model based on the supervised fine-tuning dataset; S5. Concatenate the candidate keywords with the document to be extracted to form prompt words. Input these prompt words into the lightweight student model after fine-tuning in S4. Perform functional semantic classification and output a structured functional keyword knowledge dictionary.

[0009] Furthermore, S1 specifically involves performing word segmentation on the document to be extracted, generating sentence numbers based on sentence boundaries, and generating an attention mask to block padding bits to obtain the input word sequence; using a pre-trained language model as the basic encoder to encode the input word sequence, and taking the last hidden state as the initial context vector representation of each word.

[0010] Furthermore, the pre-trained language model is one of DeBERTa-v3-base, RoBERTa, BERT, or SciBERT; the word segmentation process is performed using a word segmenter that is compatible with the pre-trained language model.

[0011] Furthermore, the dual channels include a local pattern capture channel and a global hierarchical information evaluation channel. The local pattern capture channel is implemented by a local pattern hybrid expert module, which is used to capture local fine-grained patterns, including noun phrases and proper nouns, to obtain local feature vectors. The global hierarchical information evaluation channel is implemented by a hierarchical context scoring module, which calculates an explicit hierarchical context score for each lexical unit to obtain hierarchical context structure features.

[0012] Furthermore, the local pattern hybrid expert module includes a routing network and several parallel expert networks. The routing network includes two linear projections: the first linear projection maps the initial context vector of the word to obtain a routing score, and the second linear projection generates noise parameters. During the training phase, Gaussian noise is superimposed on the routing scores to perform NoisyTop-k routing; during the inference phase, no noise is superimposed. For each word, the K experts with the highest routing scores are selected, and the routing scores of unselected experts are set to negative infinity. After softmax normalization, a sparse routing weight vector is obtained. The expert network is a two-layer nonlinear feedforward network. Each expert network processes the words routed to it in parallel, and the outputs are weighted and summed according to the sparse routing weight vector to obtain the local feature vector of the word.

[0013] Furthermore, the hierarchical context scoring module outputs two results simultaneously for each word: the first is the hierarchical context score, which is obtained by calculating the geometric mean of four types of hierarchical context structure features: topic features, hierarchical features, novelty features, and positional prior features, and then performing z-score standardization and truncation; the second is the hierarchical context structure feature vector, which is composed of three parts: positional features, semantic features, and sentence boundary features. The four types of hierarchical contextual structure features are constructed by combining word-level representation vectors with their corresponding sentence-level representation vectors, sentence-level representation vectors with document-level representation vectors, and word-level representation vectors with the average direction of all valid word representation vectors in the document, along with word position information. They are represented as follows:

[0014]

[0015]

[0016]

[0017] Where cos(·,·) represents the cosine similarity. Let be the word-level representation vector, that is, the initial context vector representation of the t-th word. This is a sentence-level representation vector. For document-level representation vectors, The average direction of all valid tokens in the document. Let be the position index of the t-th term in the document. The effective length of the document. As a thematic feature, it measures the representativeness of lexical units in the sentence. As a hierarchical feature, it measures the degree to which a sentence carries the central idea of ​​the document; This novelty feature highlights the unique semantic information carried by lexical units; As a priori feature of position, it characterizes the positional salience of lexical units.

[0018] Further, S3 specifically involves: first, inputting the local feature vector into a bidirectional recurrent neural network for sequence modeling; then, inputting the hierarchical context structure feature vector into a feature encoding network after layer normalization, sequentially performing linear mapping to 32 dimensions, GELU activation, layer normalization, and then generating scaling and translation parameters for each word through two linear projections; the weights and biases of the two linear projections are initialized to zero, making the initial modulation state an identity mapping; the output features of the bidirectional recurrent neural network are modulated using the scaling and translation parameters, and the modulation only takes effect at the effective word positions; the modulated feature vector is input into a linear classification head and projected onto... The sequence label space generates the probability distribution of each label in the entire document. The optimal label sequence is determined by a span-based decoding strategy. After decoding, the candidate keywords are stemmed and deduplicated. They are then sorted in descending order by confidence and frequency of occurrence. The first few keywords are truncated and output as the candidate keyword set.

[0019] Furthermore, in step S4, when constructing the training dataset, task labels and domain labels of the literature are obtained from the domain knowledge base. Data sets and evaluation indicator entities are extracted from the literature text through rule matching. The core method information of the literature is extracted based on strict prompt word reasoning using a large language model. The results are fused to obtain instruction-input-output triple training data. The reasoning ability of the teacher's large language model is transferred to the lightweight student model through knowledge distillation. The labeled data generated by the teacher model is used as a supervision signal to supervise and fine-tune the student model. During training, a low-rank adaptive LoRA parameter efficient fine-tuning method is adopted. The cross-entropy loss of autoregressive language modeling is used as the target. The loss mask is applied to the word units of the prompt words, and the loss is calculated only for the word units of the model output.

[0020] Furthermore, S5 specifically involves concatenating the title and abstract of the document to be extracted with the candidate keyword set output in step S3 according to a strict prompt word template to form prompt words. This prompt word is then input into the lightweight student model, which has been fine-tuned in step S4. The lightweight student model categorizes each candidate keyword by functional dimension, classifying them into five dimensions: task, method, domain, dataset, and indicator. It also identifies interfering keywords that cannot be classified into any dimension. Under structured output constraints, the model generates classification results and extracts valid fields according to structured format rules through a preset parsing engine. Redundant explanatory text is removed, and finally, a structured functional keyword knowledge dictionary is output.

[0021] Secondly, the present invention also provides a hierarchical context-aware keyword extraction system with functional interpretability. The system is used to implement the method described in any of the first aspects, comprising a basic encoding module, a dual-channel context feature enhancement module, a feature fusion and decoding module, and a lightweight student model functional classification module. The basic encoding module converts the document to be extracted into a high-dimensional context vector sequence suitable for subsequent processing, and consists of a word segmentation preprocessing submodule and a pre-training encoding submodule. The dual-channel context feature enhancement module performs local fine-grained pattern capture and global hierarchical structure information evaluation in parallel, and consists of a local pattern mixing expert submodule and a hierarchical context classification submodule. The lightweight student model functional classification module constructs training data and supervises the fine-tuning of the lightweight student model, performs classification of the candidate keyword set across five functional dimensions and identifies interference items, outputting a structured functional keyword knowledge dictionary, and consists of a data construction submodule, a supervised fine-tuning submodule, and an online inference submodule.

[0022] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention employs a two-stage cascaded architecture of hierarchical context-enhanced supervised extraction and functional semantic classification. First, the extraction module narrows down the range of candidate keywords, and then the subsequent model performs classification on the given candidate keywords. This architecture avoids the content distortion problem that easily occurs when traditional large models directly generate keywords, effectively improving the accuracy and interpretability of the document keyword extraction results.

[0023] 2. This invention designs a complementary data construction and model distillation scheme. It relies on a rule engine to ensure entity extraction accuracy, combines the logical reasoning capabilities of a large language model to synthesize training data, and fine-tunes a lightweight student model based on the synthesized high-quality data. While significantly reducing computational overhead, this allows the lightweight model to possess the functional semantic classification performance of a large language model, achieving low-cost, high-precision keyword classification.

[0024] 3. This invention employs a post-processing mechanism combining prompt word constraints and structured output parsing. By specifying the output format of the model's classification results through structured output standards, a dedicated parsing engine completes the result parsing process. This effectively solves the problem of messy and uncontrollable output formats in generative models, ensuring the stable and efficient operation of the downstream automated parsing process.

[0025] 4. This invention can achieve precise mapping between keywords and five dimensions: methods, tasks, data, indicators, and domains, giving the keyword extraction results clear semantic dimension information and explanatory value, and can be directly adapted to the construction, sorting and application scenarios of downstream academic knowledge graphs.

[0026] 5. This invention distills the deep reasoning capabilities of large language models into lightweight student models with parameter scales of less than one billion, significantly reducing the memory usage and computational power consumption of model reasoning, and reducing the average processing time of a single document in the complete pipeline to about 1.7 seconds (calculated based on 1706.35 seconds for 1000 documents), which can meet the needs of large-scale batch processing of documents.

[0027] 6. The lightweight student model of this invention only filters and semantically classifies candidate keywords extracted from the original text, abandoning the work mode of free generation. By limiting the candidate range, it reduces the risk of the model generating words that do not exist in the original text, and reduces content errors and information security risks in the keyword processing process. Attached Figure Description

[0028] Figure 1 This is an overall architecture diagram of a hierarchical context-aware keyword extraction method with functional interpretability according to the present invention. It shows the correspondence between the two-stage process and processing steps of hierarchical context-enhanced supervised extraction (steps S1-S3) and functional semantic classification based on a lightweight student model (steps S4-S5).

[0029] Figure 2 The diagram illustrates the calculation principle of the hierarchical context scoring module, showing the process of obtaining document-level, sentence-level, and word-level representation vectors, calculating hierarchical context structure features, and generating word-by-word modulation parameters.

[0030] Figure 3 This diagram illustrates the instruction data structure for Supervised Fine-Tuning (SFT) of large models, showcasing the complete construction process of the "instruction-input-output" triple dataset: the three sources are domain knowledge base metadata, rule matching, and large language model inference, providing task / domain, dataset / metric, and method class labels respectively; the triple formed after merging consists of three parts: a functional classification prompt template (instruction), input text containing a title, summary, and candidate keyword set, and functional dimension labels and distractor annotations for each candidate keyword (output). Detailed Implementation

[0031] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. It should be noted that the following embodiments are only for explaining the present invention and are not intended to limit the scope of protection of the present invention; the technical features in the embodiments of the present invention can be combined with each other without conflict.

[0032] See Figure 1 This diagram illustrates the overall architecture of a hierarchical context-aware keyword extraction method with functional interpretability, showcasing the cascaded process and corresponding steps of hierarchical context-enhanced supervised extraction and functional semantic classification: S1, segmenting the document to be extracted and encoding it into initial context vector representations; S2, performing parallel dual-channel context feature enhancement on the initial context vector representations to obtain local feature vectors and hierarchical context structure features; S3, generating a candidate keyword set based on dual-channel features through sequence modeling, word-by-word modulation, and sequence labeling decoding; S4, constructing supervised fine-tuning data and performing knowledge distillation fine-tuning on a lightweight student model; S5, concatenating the candidate keywords with the document to be extracted to form prompt words, inputting them into the lightweight student model fine-tuned in S4, performing functional semantic classification, and outputting a structured functional keyword knowledge dictionary. Specifically, S1 to S3 constitute the supervised extraction stage, which can be used for offline training and online candidate keyword extraction; S4 constitutes the training stage of the functional classification model; and S5 constitutes the online functional semantic classification stage.

[0033] The specific implementation steps of the method of the present invention will be described in detail below: Step S1: Perform word segmentation on the document to be extracted, and use a pre-trained language model as the basic encoder to extract the initial context vector representation of each word.

[0034] The titles and abstracts of the scientific documents to be extracted are used as the documents to be extracted and processed into word segments to obtain the input word sequence. During word segmentation, sentence numbers (sentence_map) are generated based on sentence boundaries, and attention masks (attention_mask) are generated to block padding. A pre-trained language model is used as the basic encoder to encode the input word sequence. The last hidden state is taken as the initial context vector representation of each word, which also serves as the input for subsequent dual-channel context feature enhancement.

[0035] In this embodiment, word segmentation is performed using a pre-trained language model-compatible word segmenter: a space is added before each word to avoid cross-language segmentation ambiguity, dividing the text into sub-word sequences. A maximum sequence length is set; in this embodiment, it is set to 256 tokens. Excess lengths are truncated, and insufficient lengths are padded. The original word position of each sub-word is recorded, ensuring that subsequent labeling only applies to the first sub-word of each word. Simultaneously, the document is divided into sentences using sentence boundary markers such as periods, question marks, and exclamation marks, generating sentence numbers and sentence boundary markers for global hierarchical information evaluation. The pre-trained language model used in this embodiment is DeBERTa-v3-base, containing a 12-layer Transformer encoder with a hidden layer dimension of 768 and 12 attention heads; relative position encoding and de-entanglement attention mechanisms are employed. The core calculations in this embodiment are as follows:

[0036]

[0037]

[0038]

[0039] in, Given a sequence of words as input, Let t be the t-th word, and n be the length of the input word sequence; Encoder(·) is the basic encoder. The context representation matrix obtained by encoding, where d is the hidden dimension. Represent the space of real matrices with n rows and d columns; Let be the initial context vector representation of the t-th word.

[0040] In this embodiment, the basic encoder is fine-tuned using Low-Rank Adaptive (LoRA) mode, updating only a small number of low-rank increment parameters. The rank r = 16, scaling factor alpha = 16, and dropout rate 0.1 are set to apply to the query projection and value projection of the attention mechanism. The weight update method is as follows:

[0041] in, For the pre-trained weight matrix, , It is a low-rank factor matrix. Scaling factor For rank. This is the weight matrix after low-rank adaptive updating, i.e., the weights actually used in the forward computation after fine-tuning. In other embodiments, other pre-trained language models based on the Transformer architecture, such as RoBERTa, BERT, and SciBERT, can also be selected; the hyperparameters in the hierarchical context structure feature calculation can be determined by optimization on the validation set based on downstream tasks.

[0042] Step S2: Perform dual-channel context feature enhancement on the initial context vector representation in parallel: the local pattern capture channel captures local fine-grained patterns such as noun phrases and proper nouns; the global hierarchical information evaluation channel evaluates the global hierarchical importance of each word, and obtains local feature vectors and hierarchical context structure features respectively.

[0043] S2.1 The local pattern capture channel is implemented by the Local Pattern Hybrid Expert (MoE) module, which includes a routing network (Gating Network) and several parallel expert networks. The routing network consists of two linear projections: the first linear projection represents the initial context vector of each word. The route score is mapped to a length equal to the number of experts, and a second linear projection generates noise parameters. During the training phase, Gaussian noise with a standard deviation of softplus (noise parameters) is superimposed on the route score (Noisy Top-k routes). During the inference phase, this is not superimposed. Therefore, the effective route score for the t-th term is:

[0044]

[0045] in, Let t be the initial context vector representation of the t-th word (i.e., the output of step S1). , These are the learnable route projection matrix and the noise projection matrix, respectively. Let N(0,1) be a noise vector that follows a standard normal distribution, and softplus(·) be a smooth activation function. This indicates element-wise multiplication.

[0046] Subsequently, for each term, select the K experts with the highest routing scores (in this embodiment, the total number of experts is set to 4, and K is 2). Set the routing scores of the unselected experts to negative infinity, and then normalize using softmax to obtain the sparse routing weights of the selected experts:

[0047] in, (·) indicates that the K highest scores are retained and the remaining components are set to zero. The truncation operation, softmax(·) is the normalized exponential function. Let be the sparse routing weight vector for the t-th term. This vector is arranged in order of activation experts, i.e. ,in The routing weight assigned to the j-th expert for the t-th term.

[0048] Each expert network is a two-layer nonlinear feedforward network: first, the input features are linearly mapped to 4 times the hidden dimension, activated by ReLU, then mapped back to the original hidden dimension, and a dropout operation is applied with a dropout rate of 0.1. Each expert network processes the terms assigned to it by the routing in parallel, and the final output is weighted and summed according to the corresponding sparse routing weights. The calculation expression is as follows:

[0049] in, (·) represents the j-th expert network. The routing weight assigned to the j-th expert for the t-th word. Let be the local feature vector of the t-th word.

[0050] S2.2, the global hierarchical information evaluation channel is implemented by the Hierarchical ContextScoring module, which calculates an explicit hierarchical context score for each lexical unit. For example... Figure 2 As shown, firstly, the effective word set is determined based on the attention mask and sentence number, denoted as... By aggregating effective lexical units, document-level and sentence-level representation vectors are obtained:

[0051]

[0052] in, The set of valid terms within the document. Let be the set of valid lexical units in the s-th sentence. Let be the word-level representation vector, that is, the initial context vector representation of the t-th word. This is a sentence-level representation vector. First, we calculate the document-level representation vector. Next, we calculate the cosine similarity between the word-level representation vector and its corresponding sentence-level representation vector, the cosine similarity between the sentence-level representation vector and the document-level representation vector, and the cosine similarity between the word-level representation vector and the average direction of all valid word representations within the document.

[0053] Furthermore, based on the aforementioned similarity and combined with word position information, four types of hierarchical contextual structure features are constructed (all mapped to the (0,1] interval):

[0054]

[0055]

[0056]

[0057] Where cos(·,·) is the cosine similarity, and the average direction of all valid token representations in the document is denoted as . , Let be the position index of the t-th term in the document. This is the effective length of the document. Subject-specific characteristics. A measure of the representativeness and hierarchical features of lexical units in a sentence. Novelty features are used to measure the degree to which a sentence carries the central idea of ​​a document. Highlighting the unique semantic information carried by lexical units, location-prior features Characterize the positional salience of lexical units.

[0058] Finally, the geometric mean of the four structural features is taken, and then the geometric mean is standardized by z-score (i.e., standardized according to the mean and standard deviation within the document) and truncated to [the specified value]. To suppress the influence of outliers, the hierarchical context score of the word is finally obtained:

[0059] Where F = {topicality, hierarchy, novelty, position_prior} is a set of four types of structural features. Let z be any structural feature of the t-th term, zscore(·) be the standardization based on the document mean and standard deviation, and clip(·, ) is to be cut off to interval, This is the hierarchical context score for the t-th word.

[0060] The hierarchical context scoring module outputs two results for each lexical unit: one is the hierarchical context score obtained by normalizing and truncating the geometric mean of the four structural features mentioned above using z-score. The first is used for the k-best span rearrangement in step S3.3; the second is the hierarchical context structure feature vector. This is used for word-by-word meta-modulation in step S3.2. The hierarchical context structure feature vector is composed of three parts: positional features, semantic features, and sentence boundary features, specifically represented as follows:

[0061]

[0062]

[0063]

[0064] In the formula , , These are the 0-based indices of the tokens within the document, the sentence they belong to, and within the sentence, respectively. , , The document's effective length, number of sentences, and sentence length are all normalized to [0,1]. Structural feature vector. The definition is as follows.

[0065]

[0066] in Indicates the normalized position , , The Fourier positional encoding is calculated using the following formula:

[0067]

[0068]

[0069] In the formula = (k=0,1,2,3) represents the Fourier basis frequencies; each normalized position is expanded into 9 dimensions by the 1-dimensional original value and 4 sets of sine / cosine basis expansions, and then concatenated in three ways. Dimensional location features.

[0070]

[0071] In the formula , For sentence boundary indicator variables taking values ​​in {0,1}: = 1 indicates that the lexical t is the first lexical of the sentence it belongs to; otherwise, it is 0. Similarly, the final word unit is represented, with each word unit being 2-dimensional. After concatenating the three... There are 32 dimensions in total.

[0072] It should be noted that location prior features Only participate in hierarchical context scoring The calculation does not enter the structural feature vector. ; Is with Another parallel positional coding approach, with different responsibilities: the former is positional embedding for FiLM modulation, while the latter is attenuation prior for rearrangement.

[0073] Step S3: Based on dual-channel features, a set of candidate keywords is generated through sequence modeling, word-by-word modulation, and sequence labeling decoding.

[0074] S3.1 Sequence Modeling. First, the local feature vectors output from the local pattern capture channel in step S2 are input into a bidirectional recurrent neural network (BiLSTM) for sequence modeling:

[0075] in, This indicates that the bidirectional recurrent neural network is in the first... The sequence hidden state vector output at each lexical position; This represents the operation of a bidirectional recurrent neural network; The input feature of the t-th word is the local feature vector output in step S2.1; and These represent the hidden states of the previous and next time steps in the sequence, respectively.

[0076] In this embodiment, the bidirectional recurrent neural network is a two-layer bidirectional long short-term memory network, with the hidden dimension in each direction consistent with the encoder's hidden dimension, i.e., 768 dimensions. The input is packed into a sequence according to the effective length of each sample before being fed into the network to ignore the influence of padding on context modeling. The bidirectional output is summed at corresponding positions and residually connected to the input, i.e., a cross-layer connection where the network output and input are added element-wise, and a dropout rate of 0.3 is applied between layers to prevent overfitting. The bidirectional recurrent neural network is not limited to BiLSTM; in other embodiments, it can also be replaced by a bidirectional gated recurrent unit (BiGRU) or an additional Transformer encoder layer.

[0077] S3.2 Utilize the structural feature vector output by the hierarchical context scoring module in step S2.2 Based on this vector, scaling and translation parameters are generated for each word. The features after sequence modeling are modulated using the two sets of parameters. The label sequence is output after word-by-word classification, and a candidate keyword set is generated by decoding according to the span.

[0078] Specifically, the structural feature vector After layer normalization (LayerNorm), the input is fed into the feature encoding network, sequentially linearly mapped to 32 dimensions, activated by GELU, and normalized again. Then, two linear projections are used to generate scaling and translation parameters for each word. The weights and biases of both linear projections are initialized to zero, ensuring an identity mapping in the initial modulation state, meaning the original features are not initially altered. This avoids interference with the word-level classification objective in the early stages of training.

[0079]

[0080] Where LN(·) is layer normalization, and MLP(·) is feature encoding network. , These are linear projections (both initialized to zero) that generate scaling and translation parameters, respectively. , These are the scaling and translation parameters for the t-th word, respectively.

[0081] During modulation, scaling and translation parameters are applied to the output of the bidirectional recurrent neural network, and only at valid word positions:

[0082] in, Let be the sequence hidden state vector output by the bidirectional recurrent neural network at the t-th word position. The modulated feature vector; , Let be the scaling and translation parameters for the t-th word, respectively, initially... = =0, modulation degenerates into an identity mapping. This mechanism injects global hierarchical structure information into the local sequence modeling process without introducing additional supervised loss, thereby strengthening lexical features that are highly important in terms of global structure and semantics.

[0083] Finally, the modulated feature vector Input is fed into a linear classification head and projected onto... Sequence label space, generated based on the entire document probability distribution of each label ,in Let t be the label of the t-th word.

[0084] An optimal label sequence is determined using a span-based decoding strategy, which includes three options: Viterbi decoding, k-best span rearrangement, and semi-Markov span decoding. Viterbi decoding is a dynamic programming algorithm for finding the globally optimal label sequence; semi-Markov span decoding is a method that labels consecutive word segments as a whole; and k-best span rearrangement reorders the top k optimal span candidates. During the decoding phase, for each word, its normalized score on the three labels B (Begin), I (Inside), and O (Outside) is output as the emission score for the decoding phase, i.e., the probability distribution described above. The probability value of the corresponding tag is used in the decoding stage; legal transition constraints that conform to the BIO structure are adopted, prohibiting direct transition from the O tag to the I tag, or from the I tag to another B tag, so as to ensure that the decoded sequence conforms to the BIO structure of the keyword.

[0085] The Viterbi decoding, k-best span rearrangement, and semi-Markov span decoding strategies described above are optional top-level alternative decoding schemes. After verification and configuration through the decoder, one of these strategies is selected as the top-level decoding strategy. These three strategies are not executed sequentially. Specifically, when k-best span rearrangement is selected as the top-level strategy, this strategy internally calls semi-Markov decoding to obtain the first k span candidates as rearrangement input. This semi-Markov decoding is merely an internal implementation detail of k-best rearrangement and is not used as a separate top-level output strategy.

[0086] For Viterbi decoding, this invention, given the lexical emission score and tag transition score matrix (i.e., the constraint matrix of legal transitions between BIO tags), employs the Viterbi dynamic programming algorithm to solve for the globally optimal tag sequence that maximizes the joint score of the tag sequence, i.e., the sum of the emission score at each position and the transition score at adjacent positions, rather than independently finding the local maximum for each lexical. Therefore, it effectively eliminates tag conflicts and illegal segments caused by local greedy decoding. The time complexity of this decoding process is O(n log n). , where n is the length of the word sequence and L is the number of tags, in this embodiment L=3.

[0087] For semi-Markov span decoding, also known as segment-level decoding, this invention assigns labels to consecutive lexical units ("spans") instead of individual lexical units: the segment-level score of a span is defined as the sum of the emission scores of each lexical unit within that span plus the span-level label transition score. Then, through dynamic programming, it searches for the span division that maximizes the total score across all possible starting positions, lengths, and label combinations, directly outputting the complete keyword phrase boundary. This helps reduce the misalignment problem of BIO word-by-word labeling on the boundaries of longer multi-word phrases, and allows for the explicit introduction of span length priors during the decoding stage. In this embodiment, the maximum span length is set to 8.

[0088] For k-best span rearrangement, this invention, based on the top k optimal span candidate sets obtained from semi-Markov decoding, combines the global importance information of lexical units calculated by the hierarchical context scoring module in step S2, i.e. Each candidate span is weighted and ranked, and low-frequency long-tail concepts with high global structural importance and strong semantic uniqueness are retained first. In this embodiment, k is 8, so as to further improve the balance between recall and precision of the candidate keyword set without significantly increasing the computational overhead.

[0089] After decoding, stemming and deduplication are performed on the candidate keywords, retaining the original spelling. The keywords with the smallest position and the largest confidence are selected, and their frequency is accumulated. They are then sorted in descending order by weighted score based on confidence and frequency, and the top 10 keywords are truncated and output as the candidate keyword set. The candidate keyword set is processed by the functional classification module. The input range of the post-functional semantic classification stage is limited and constrained: the classification model selects and classifies only within the set, and the output keywords can all be traced back to the original text, which enhances the traceability and parsability of the extraction and classification results from a mechanism perspective.

[0090] Step S4, Training Data Construction and Knowledge Distillation: Construct "instruction-input-output" triple training data from three sources: domain knowledge base, rule matching, and large language model inference. Then, perform low-rank adaptive (LoRA) fine-tuning on the lightweight student model to enable it to inherit the functional semantic classification capabilities of the large language model.

[0091] S4.1, Training data construction. For example... Figure 3As shown, task and domain labels are taken from domain knowledge bases, such as PaperswithCode (PwC), which is a public knowledge base that includes structured metadata of types such as tasks and domains. For entities with fixed patterns, such as "dataset" and "evaluation metrics," rule matching is used to accurately extract them from the literature; for complex implicit semantics, such as "core methods," large language models are called to extract them based on strict prompt word reasoning. This embodiment uses public large language models such as the GLM series. The results of the three are merged to construct a "command-input-output" triple dataset. The merged triple consists of three parts: a functional classification prompt template (command), an input text containing a title, abstract, and candidate keyword set, and functional dimension labels and interference annotations for each candidate keyword (output). The above "command-input-output" triple is the training sample for instruction fine-tuning (SFT). Each sample is equivalent to a "question and standard answer" pair, and its actual form is shown in the following example: Instruction: Determine the functional role of the following candidate keywords in the literature (TASK / METHOD / FIELD / DATASET / METRIC; output NONE if not belonging to any functional dimension). The definitions of each category are as follows: TASK (the scientific problem or technological challenge studied in the literature), METHOD (the proposed or adopted model, algorithm, and implementation method), FIELD (the broader research field to which the literature belongs), DATASET (the dataset used in the experiment or a benchmark), METRIC (the evaluation metric reported in the experiment), and distractors (NONE) that do not belong to any dimension.

[0092] Input: The title of this paper is "Knowledge Graph Construction via Neural Relation Extraction", and the abstract is "Extracting entities and relations from scientific literature is fundamental to knowledge graph construction. We propose an aneural framework that combines sequence labeling with distant supervision." The candidate keywords are: hierarchical context-aware, keyword extraction, BiLSTM, and F1-score.

[0093] Output: hierarchical context-aware: METHOD; keyword extraction: TASK; BiLSTM: METHOD; F1-score: METRIC.

[0094] Regarding data format, the system prompts the model to play the role of a scientific literature analysis expert and has built-in judgment rules for each functional dimension; the input text is organized according to a fixed template and includes three parts: paper title, abstract and candidate keyword list. The candidate keywords are listed in a line-by-line list format, and placeholders are used when the candidate set is empty.

[0095] S4.2 Knowledge Distillation Fine-Tuning. Knowledge distillation refers to transferring the reasoning ability of the teacher's large language model to a lightweight student model: the student model is trained using labeled data generated by the teacher model as a supervised signal, enabling it to reproduce the output of the teacher model. This embodiment achieves knowledge distillation through instruction fine-tuning (SFT), using "instruction-input-output" triplet samples to perform supervised fine-tuning on the model, enabling it to learn to follow instructions to complete tasks: Step S4.1 constructs triplet samples using the labels generated by the teacher's large language model. Each sample is equivalent to a "question and standard answer" pair. This step uses these samples to perform supervised fine-tuning on the student model, enabling it to learn to classify candidate keywords into various functional dimensions according to instructions, thereby inheriting the classification reasoning ability of the teacher model. This embodiment uses Qwen3.5-0.8B as a lightweight student model. Its supervised fine-tuning employs a low-rank adaptive parameter efficient fine-tuning method: a low-rank increment matrix is ​​added to the frozen pre-trained weights, with a rank of 8, a scaling factor of 16, and a low-rank dropout rate of 0.05. The low-rank increment is simultaneously applied to various projection matrices of the multi-head attention module, including query, key, value, and output projections, while also covering linear transformations such as gating, up projection, and down projection in the feedforward fully connected layer. The fine-tuning target is the standard autoregressive language modeling cross-entropy loss. During training, loss masks are applied to the lexical components of the prompt words (instructions and input text), and the loss is only calculated for the lexical components of the expected functional classification result, allowing the lightweight student model to inherit the reasoning capabilities of large language models while strictly aligning with the structured output format. The optimizer is AdamW, with a learning rate set to... The training process, employing cosine annealing scheduling and a 3% linear warm-up, is conducted over three epochs. During training, the batch size per card is set to 1, combined with an 8-step gradient accumulation step, resulting in an equivalent batch size of 8. The random seed is fixed at 42. The total length of the concatenated instructions, inputs, and expected outputs in the training samples does not exceed 4096 tokens; any exceeding this limit is handled with a strategy of prioritizing output retention and truncating input. This lightweight student model can also be used for other lightweight language models with instruction fine-tuning capabilities, such as Llama and Gemma.

[0096] Step S5, Functional Semantic Classification: The title and abstract of the document to be extracted are concatenated with the candidate keyword set output in Step S3 according to a fixed template to form prompt words. These prompt words are then input into the lightweight student model that has been fine-tuned in Step S4. The candidate keywords are assigned to five functional dimensions: task, method, domain, dataset, and indicator, and interference items that do not belong to any dimension are identified. The model outputs the classification results under structured output constraints. The results are extracted by the parsing engine, irrelevant explanatory text is discarded, and a structured functional keyword knowledge dictionary is output.

[0097] In step S5, the prompt words consist of system prompts and user input, consistent with step S4. The inference decoding uses greedy decoding with a temperature parameter of 0; top_p is set to 1.0, representing the cumulative probability threshold for kernel sampling; the maximum number of newly generated lexical units is 768, and the model's thought chain output (i.e., the intermediate text of the model's step-by-step inference) is disabled, generating only structured classification results. The generated results are processed by the parsing engine: first, the Markdown code fences are removed (i.e., the tags enclosed in three backticks), then parsed according to JSON syntax. If parsing fails, the content between the first and last curly braces is used as a fallback. Subsequently, functional labels are aligned one by one according to the candidate keyword list, missing items are judged as interference (NONE) by default, and explanatory text unrelated to classification is discarded. Finally, a structured functional keyword knowledge dictionary is output according to the five functional dimensions. This five-dimensional label system and interference discrimination are the core mechanisms of this invention that explicitly endow the extracted results with semantic interpretation value, making the system output no longer an unstructured list of words, but a knowledge structure organized according to functional dimensions. The aforementioned processes, such as removing code fences, providing fallback for JSON parsing, and aligning by candidate list, further ensure that even if the model output occasionally deviates, the system can still recover a complete and alignable structured result, serving as the last line of defense for the stability of downstream automated parsing.

[0098] In accordance with the aforementioned method, this invention further provides a deployable implementation system. This system encapsulates all the logic of the above method into a computer program, and can be deployed on electronic devices with computing and storage resources, such as servers or workstations equipped with graphics processors.

[0099] The system comprises four main functional modules: a basic encoding module, a dual-channel contextual feature enhancement module, a feature fusion and decoding module, and a lightweight student model functional classification module. It is divided into two interconnected data pipelines: offline training and online inference. The offline training pipeline trains the extraction model and performs supervised fine-tuning of the functional classification model. The online inference pipeline uses the trained model to perform end-to-end processing of "candidate keyword extraction—functional semantic classification" on the scientific documents to be extracted. Each module corresponds to a processing step in the method of this invention, and data is transferred between modules through a unified intermediate data structure. Their correspondence and execution order are as follows.

[0100] M1, Basic Encoding Module (corresponding to step S1) The basic encoding module is used to convert the document to be extracted into a high-dimensional context vector sequence that can be processed subsequently. It consists of a word segmentation preprocessing submodule and a pre-trained encoding submodule. Its output initial context vector representation serves as the input to both channels of the dual-channel context feature enhancement module. The specific implementation details of word segmentation and encoding are the same as in step S1, and will not be repeated here.

[0101] M2, Dual-channel contextual feature enhancement module (corresponding to step S2) The dual-channel context feature enhancement module performs local fine-grained pattern capture and global hierarchical structure information evaluation in parallel. It consists of a local pattern mixing expert submodule and a hierarchical context segmentation submodule, which respectively output the local feature vector and hierarchical context structure features of each word. Its specific implementation details are the same as those in step S2, and will not be repeated here.

[0102] M3, Feature Fusion and Decoding Module (corresponding to step S3) The feature fusion and decoding module performs feature fusion, sequence modeling, word-by-word modulation and sequence decoding. It consists of a sequence modeling submodule, a modulation submodule, a label classification submodule and a decoding submodule, and finally generates a set of candidate keywords for use by the lightweight student model functional classification module. Its specific implementation details are the same as those in step S3, and will not be repeated here.

[0103] M4, Lightweight Student Model Function Classification Module (corresponding to steps S4-S5) The lightweight student model functional classification module constructs training data and supervises the fine-tuning of the lightweight student model. It performs classification on the candidate keyword set across five functional dimensions and identifies interference items, outputting a structured functional keyword knowledge dictionary. It consists of a data construction submodule, a supervised fine-tuning submodule, and an online inference submodule. Its specific implementation details are the same as steps S4-S5, and will not be repeated here.

[0104] Module collaboration and data flow In the offline training pipeline: the basic encoding module, the dual-channel contextual feature enhancement module, and the feature fusion and decoding module constitute the extraction model. This model is jointly trained end-to-end on labeled corpora (training data obtained by splitting the validation sets of each dataset into 80 / 20 segments). The training objective is a weighted sum of the focus loss and the load-balancing auxiliary loss for label classification. The data construction submodule and the supervised fine-tuning submodule of the lightweight student model functional classification module independently train the functional classification model. In the online inference pipeline: the documents to be extracted sequentially pass through the basic encoding module, the dual-channel contextual feature enhancement module, and the feature fusion and decoding module to generate a candidate keyword set. The lightweight student model functional classification module then outputs a structured functional keyword knowledge dictionary. Both pipelines share the intermediate representation definitions of the basic encoding module and the dual-channel contextual feature enhancement module. Model parameters are persistently stored in files and loaded during the inference phase. Data transfer between modules uses a standardized structured format, enabling each module to be trained, replaced, and deployed independently.

[0105] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0106] Experimental Setup and Evaluation Protocol This invention evaluates the results on the commonly used Keyword Extraction (KPE) benchmark in academia. Specifically, the validation set of each dataset is split into a training set and a model selection set at an 80 / 20 ratio. This validation set protocol, used to select the validation subset for model hyperparameters and checkpoints, is used for model fine-tuning and selection. The results are evaluated on the publicly available standard test sets Inspec, KP20k, Krapivin, and openKP. The evaluation metrics are F1@5 and F1@10, which calculate the harmonic mean of precision and recall using full-matching for the top 5 and top 10 keywords predicted by the model, respectively. Results are reported for each dataset based on the actual number of runs: Inspec, Krapivin, and openKP report the mean results from 5 runs with random seeds (seeds 13 / 21 / 42 / 87 / 100). The sample standard deviation, KP20k, represents a single run (seed-42, marked †), to demonstrate the stability and reproducibility of the results.

[0107] The training protocol uses a validation set for evaluation: the validation set of the dataset is shuffled with a fixed random seed of 42 and then split into a training set and a model selection set at 80 / 20. The data split is shared for five runs, with only the random seed of the model being changed. The subject-supervised extraction model (steps S1-S3) uses microsoft / deberta-v3-base as the pre-trained encoder, which includes a 12-layer Transformer with a hidden dimension of 768 and an attention head of 12. It enables a hybrid expert module (4 experts, top-2 activated each time) and two layers of bidirectional LSTM (768 hidden dimensions per direction) for sequence modeling. The encoder is fine-tuned using LoRA low-rank adaptive (rank r=16, scaling factor alpha=16, dropout rate 0.1). The main training uses the AdamW optimizer (an adaptive learning rate optimization algorithm with weight decay), with an initial learning rate of 2e-4 and weight decay of 1e-2, combined with cosine annealing scheduling (learning rate decays according to a cosine curve) and 10% linear warm-up (learning rate increases linearly in the early stage of training). The gradient clipping threshold is 1.0 (limiting the gradient norm to prevent gradient explosion). The loss function is Focal Loss (a loss function to alleviate class imbalance, gamma=2). The maximum input length is 256, the batch size is 8, the maximum number of training epochs is 20, and training is terminated early when the validation set metric no longer improves. The patience value is 5, the minimum improvement threshold is 0.005, the data seed is 42, and the model random seed is 13 / 21 / 42 / 87 / 100. The validation set F1@10 is used as the metric for early stopping and selecting the optimal checkpoint. The decoding stage uses Viterbi dynamic programming for decoding, selecting the top 10 candidate keywords based on confidence. Knowledge distillation of the lightweight student model (step S4) uses a large language model (GLM series) as the teacher and Qwen3.5-0.8B as the lightweight student model. The lightweight student model is fine-tuned using LoRA low-rank adaptive tuning; rank r=8, scaling factor alpha=16, dropout rate 0.05, and q / k / v / o projection (query / key / value / output projection) and gate / up / down projection (gating / dimensionality increase / dimensionality decrease projection) applied to the attention mechanism. Distillation training is carried out with a learning rate of 2e-4, warmup ratio of 0.03, gradient accumulation steps of 8, maximum length of 4096, random seed of 42, and cosine annealing scheduling for a total of 3 rounds.

[0108] Effect verification All experiments were conducted in a unified hardware and software environment, with a fixed random seed to ensure reproducibility. To verify the effectiveness and superiority of the proposed solution, three decoding variants under the improved SEKE architecture—Viterbi, k-best span rearrangement, and semi-Markov—were compared with existing mainstream algorithms on a benchmark dataset. Multiple repeated runs of the experiments, using the mean ± standard deviation, were employed to quantify the differences in operational stability among the different decoding variants. Detailed comparison results of the F1 scores for each group of algorithms are shown in Table 1.

[0109] Table 1: Comparison of F1 scores of our method with existing mainstream methods on benchmark datasets (unit: percentage points; † indicates single-seed result, mean ± standard deviation indicates multiple runs; "this work" refers to three decoding variants of the improved SEKE method of this invention). TF-IDF (Unsupervised) Unsupervised baseline 16 24.4 9.4 9.3 19.6 TextRank (unsupervised) Unsupervised baseline 28.6 33.9 15 16 7.9 GPT-4o-mini (Large Model) Large Language Model Baseline 34.4 42.1 19.8 20.5 26.8 JointKPE (Supervised) Supervised baseline 35.2 35 34.4 29.2 33.8 Diff-KPE (Supervised) Supervised baseline 32.3 35 34.3 31.4 32.7 DeBERTa MoE BiLSTM (already available as a state-of-the-art device) There is already a full oversight of the State of the Origin (SOTA). 49.4±1.0 58.8±0.4 36.5±0.3 34.7±0.9 42.9±0.5 Improved SEKE + Viterbi decoding (this work) This method 57.791±0.487 63.149±0.471 37.106† 34.600±1.107 41.804±0.581 Improved SEKE + k-best span rearrangement (this work) This method 57.964† 63.785† 37.815† 37.082±1.703 42.899† Improved SEKE + semi-Markov decoding (this work) This method 58.213† 64.225† 38.300† 38.017±0.647 41.051† Note: The three rows in this method represent the results under the validation set protocol; those marked with † are the results of a single run (fixed seed-42), and the rest are the average of 5 runs with random seeds. Sample standard deviation; each dataset is presented according to the actual reporting standards (Inspec reports F1@5 and F1@10, KP20k and Krapivin report F1@10, openKP reports F1@5), and other cutoffs calculated at the same time in the table (F1@5, F1@1 / F1@3) are not listed one by one.

[0110] Regarding keyword extraction performance, our method was comprehensively compared with unsupervised methods, zero-shot large language models, and existing supervised methods on four public benchmark datasets. The results are shown in Table 1. In the unsupervised and zero-shot baselines, TextRank's F1@5 on Inspec is only 28.6, while the zero-shot large language model GPT-4o-mini achieves 34.4. In the supervised baselines, representative methods JointKPE and Diff-KPE have F1@5 scores of 35.2 and 32.3 on Inspec, respectively, and F1@10 scores of 34.4 and 34.3 on KP20k, respectively. In contrast, the three decoding variants of this invention achieve F1@5 scores above 57.79 and F1@10 scores above 63.1 on Inspec, representing improvements of approximately 23 and 29 percentage points respectively compared to the strongest supervised baseline, and approximately 24 percentage points compared to the zero-shot large language model baseline. Even compared to the strongest existing supervised method trained with all training data (DeBERTaMoE BiLSTM, Inspec F1@5 is...), this method surpasses the performance of the previous method. F1@10 is Compared to other methods, this invention only uses small-scale data obtained by splitting the validation set of each dataset into 80 / 20 segments under the validation set protocol. On Inspec, F1@5 and F1@10 are still improved by about 8.8 and 5.4 percentage points, respectively, and on KP20k, F1@10 is improved from 36.5 to 38.3. This shows that this invention has a significant data efficiency advantage while achieving higher performance. Among the three decoding variants, semi-Markov span decoding achieved the best overall performance, with Inspec achieving an F1@5 of 58.213 and an F1@10 of 64.225, while KP20k achieved an F1@10 of 38.300. k-best span rearrangement followed, and Viterbi decoding came in third. Specifically, on Krapivin, span decoding and k-best rearrangement improved the F1@10 of Viterbi decoding from 34.6 to 38.0 and 37.1 respectively, representing improvements of approximately 3.4 and 2.5 percentage points. This confirms the advantage of segment-level decoding over word-level BIO annotation in characterizing long words, nested words, and multi-word phrase boundaries. On OpenKP, this invention performed roughly on par with the fully trained DeBERTa MoE BiLSTM, with an F1@5 of 41.1–42.9 vs. This indicates that the present invention also achieves the best performance in short abstract documents.

[0111] Regarding deployment costs, a comparison was made between a complete pipeline for processing 1,000 documents (two stages: extraction and functional classification) and a solution that directly calls the API of a cutting-edge large model. The results are shown in Table 2: This method completes functional classification with a local lightweight student model with fewer than 1 billion parameters, without external API fees, and can be deployed locally on a single consumer-grade GPU, significantly reducing memory usage, computing power consumption, and inference latency.

[0112] Table 2: Comparison of Deployment Efficiency and External API Costs for Processing 1000 Documents This method: HC-SEKE + Qwen3.5-0.8B-PwC-SFT-robust 1706.35 3.53 none Local reasoning; robust SFT; no-thinking; batch=16 DeepSeek-V4-Pro API 2481.39 N / A $0.60 Remote API; thinking disabled; max_workers=4 MiMo-V2.5-Pro ​​API 1246.88 N / A $1.87 Remote API; max_workers=4 Gemini-3.1-Pro-Preview API 2400.00 N / A $5.49 Estimated from 100 subsets; rate limited to 25 times / minute In the functional semantic classification stage (steps S4-S5), a test set (containing 1906 documents and 48881 candidate keyword instances, of which about one-third are interference items not belonging to any functional dimension) was constructed and evaluated. The results are shown in Table 3: The lightweight student model after knowledge distillation and low-rank adaptive fine-tuning achieved an accuracy of 91.32% and a macro-average F1 of 94.11% (with a rejection F1 of 95.65% for interference items), which is significantly higher than the API baseline of the un-fine-tuned state-of-the-art large model.

[0113] Table 3: Comparison of Functional Semantic Classification Performance with API Baseline of State-of-the-Art Large Models Qwen3.5-0.8B-PwC-SFT-robust (this method) Full 1906 0.9132 0.9411 0.9984 DeepSeek-V4-Pro Full 1906 0.7053 0.5904 1.0000 MiMo-V2.5-Pro Full 1906 0.6701 0.5734 0.9995 Gemini-3.1-Pro-Preview Sanity 100 0.6870 0.6115 1.0000 For S3.2 and word-by-word modulation parameters, pairwise validation was performed on a long document dataset. The experiment adopted target domain fine-tuning and protocol-based point selection settings. Points were selected through the validation set, and the test set was evaluated only once. The results are shown in Table 4. After introducing the modulation, the F1@10 of KP20k increased from 38.866 to 40.394 (F1@5 was 40.198), and the F1@10 of KPTimes increased from 51.216 to 51.856. The gain was mainly driven by the recall improvement, that is, the recall of low-frequency long-tail keywords was improved without sacrificing precision. Moreover, the modulation was based on the identity mapping as the initial state and did not have a negative impact on training stability.

[0114] Table 4: Paired validation results of S3.2 word-by-word meta-modulation parameters on long document datasets (F1@10, seed-42) KP20k Full text of the long paper 38.866 40.394 +1.528 KPTimes China-Long News 51.216 51.856 +0.640 In summary, this invention focuses on two main lines: hierarchical context-aware keyword extraction and functional semantic classification, achieving systematic improvements in extraction quality, functional classification quality, and deployment efficiency. In the extraction phase, using only small-scale training data under the validation set protocol, the three decoding variants outperformed existing supervised methods using full training data on Inspec and KP20k. On Krapivin, the k-best and semi-Markov variants exceeded the baseline, while the Viterbi variant approached it. On openKP, the k-best variant was roughly on par with the baseline, while the other two variants fell short. The semi-Markov span decoding achieved the best performance in four of the five metrics listed in Table 1. Paired validation further demonstrates that the hierarchical context modulation mechanism delivers stable gains on long documents, improving KP20k by 1.528 percentage points and KPTimes by 0.640 percentage points. This gain originates from improved recall of low-frequency long-tail keywords, validating the effectiveness of the mechanism. In the functional semantic classification stage, the lightweight student model with fewer than 1 billion parameters, after knowledge distillation and low-rank adaptive fine-tuning, achieved a classification accuracy of 91.32% and a macro-average F1 score of 94.11% on a test set of 1906 documents and 48881 candidate keyword instances, significantly outperforming the baseline of the un-fine-tuned state-of-the-art large-scale model API. Combined with localized inference, the complete pipeline for processing 1000 documents took approximately 1706 seconds with a peak GPU memory consumption of approximately 3.53GB, without external API fees, demonstrating significant advantages in cost and data security compared to remote API solutions. Overall, this invention, with its modular, lightweight, and functionally interpretable architecture, significantly lowers the deployment threshold while maintaining high performance, providing an end-to-end, implementable engineering solution for keyword extraction and knowledge organization in scientific and technological literature.

Claims

1. A hierarchical context-aware keyword extraction method with functional interpretability, characterized in that, Includes the following steps: S1. The document to be extracted is segmented into words and encoded into initial context vector representations of word elements; S2. Perform dual-channel context feature enhancement on the initial context vector representation in parallel to obtain local feature vectors and hierarchical context structure features; S3. Based on the local feature vector and the hierarchical context structure features, a candidate keyword set is generated through sequence modeling, word-by-word modulation and sequence labeling decoding. S4. Construct a supervised fine-tuning dataset, and perform model fine-tuning on the lightweight student model based on the supervised fine-tuning dataset; S5. Concatenate the candidate keywords with the document to be extracted to form prompt words. Input these prompt words into the lightweight student model after fine-tuning in S4. Perform functional semantic classification and output a structured functional keyword knowledge dictionary.

2. The method according to claim 1, characterized in that, Specifically, S1 involves segmenting the document to be extracted into words, generating sentence numbers based on sentence boundaries, and generating an attention mask to block padding bits to obtain the input word sequence; using a pre-trained language model as the basic encoder to encode the input word sequence, and taking the last hidden state as the initial context vector representation of each word.

3. The method according to claim 2, characterized in that, The pre-trained language model is one of DeBERTa-v3-base, RoBERTa, BERT, or SciBERT; the word segmentation process is performed using a word segmenter that is compatible with the pre-trained language model.

4. The method according to any one of claims 1-3, characterized in that, The dual channels include a local pattern capture channel and a global hierarchical information evaluation channel. The local pattern capture channel is implemented by a local pattern hybrid expert module, which is used to capture local fine-grained patterns, including noun phrases and proper nouns, to obtain local feature vectors. The global hierarchical information evaluation channel is implemented by a hierarchical context scoring module, which calculates an explicit hierarchical context score for each word to obtain hierarchical context structure features.

5. The method according to claim 4, characterized in that, The local pattern hybrid expert module includes a routing network and several parallel expert networks. The routing network includes two linear projections: the first linear projection maps the initial context vector of the word to obtain the routing score, and the second linear projection generates noise parameters. During the training phase, Gaussian noise is superimposed on the routing scores to perform NoisyTop-k routing, while no noise is superimposed during the inference phase. For each term, the K experts with the highest routing scores are selected, and the routing scores of unselected experts are set to negative infinity. The sparse routing weight vector is obtained by softmax normalization. The expert network is a two-layer nonlinear feedforward network. Each expert network processes the terms routed to it in parallel, and the output is weighted and summed according to the sparse routing weight vector to obtain the local feature vector of the term.

6. The method according to claim 5, characterized in that, The hierarchical context scoring module outputs two results for each word simultaneously: the first is the hierarchical context score, which is obtained by calculating the geometric mean of four types of hierarchical context structure features: topic features, hierarchical features, novelty features, and positional prior features, and then performing z-score standardization and truncation; the second is the hierarchical context structure feature vector, which is composed of three parts: positional features, semantic features, and sentence boundary features. The four types of hierarchical contextual structure features are constructed by combining word-level representation vectors with their corresponding sentence-level representation vectors, sentence-level representation vectors with document-level representation vectors, and word-level representation vectors with the average direction of all valid word representation vectors in the document, along with word position information. They are represented as follows: Where cos(·,·) represents the cosine similarity. Let be the word-level representation vector, that is, the initial context vector representation of the t-th word. This is a sentence-level representation vector. For document-level representation vectors, The average direction of all valid tokens in the document. Let be the position index of the t-th term in the document. The effective length of the document. As a thematic feature, it measures the representativeness of lexical units in the sentence. As a hierarchical feature, it measures the degree to which a sentence carries the central idea of ​​the document; This novelty feature highlights the unique semantic information carried by lexical units; As a positional prior feature, it characterizes the positional salience of lexical units.

7. The method according to claim 1, characterized in that, Specifically, S3 involves: first, inputting the local feature vector into a bidirectional recurrent neural network for sequence modeling; then, inputting the hierarchical context structure feature vector into a feature encoding network after layer normalization, sequentially performing linear mapping to 32 dimensions, GELU activation, layer normalization, and then generating scaling and translation parameters for each word through two linear projections; initializing the weights and biases of the two linear projections to zero, making the initial modulation state an identity mapping; modulating the output features of the bidirectional recurrent neural network using the scaling and translation parameters, with modulation only taking effect at valid word positions; and inputting the modulated feature vector into a linear classification head, projecting it onto... The sequence label space generates the probability distribution of each label in the entire document. The optimal label sequence is determined by a span-based decoding strategy. After decoding, the candidate keywords are stemmed and deduplicated. They are then sorted in descending order by confidence and frequency of occurrence. The first few keywords are truncated and output as the candidate keyword set.

8. The method according to claim 1, characterized in that, When constructing the training dataset in step S4, the task labels and domain labels of the literature are obtained from the domain knowledge base. The dataset and evaluation indicator entities are extracted from the literature text through rule matching. The core method information of the literature is extracted based on strict prompt word reasoning using a large language model and fused to obtain instruction-input-output triple training data. The reasoning ability of the teacher's large language model is transferred to the lightweight student model through knowledge distillation. The labeled data generated by the teacher model is used as the supervision signal to supervise and fine-tune the student model. During training, a low-rank adaptive LoRA parameter efficient fine-tuning method is adopted. The cross-entropy loss of autoregressive language modeling is used as the target, and loss masking is applied to the word units of the prompt word. The loss is calculated only for the word units of the model output.

9. The method according to claim 1, characterized in that, Specifically, S5 involves concatenating the title and abstract of the document to be extracted with the candidate keyword set output in step S3 according to a strict prompt word template to form prompt words. This prompt word is then input into a lightweight student model fine-tuned in step S4. The lightweight student model categorizes each candidate keyword by functional dimension, classifying them into five dimensions: task, method, domain, dataset, and indicator. It also identifies interfering keywords that cannot be classified into any dimension. Under structured output constraints, the model generates classification results and extracts valid fields according to structured format rules through a preset parsing engine. Redundant explanatory text is removed, and finally, a structured functional keyword knowledge dictionary is output.

10. A hierarchical context-aware keyword extraction system with functional interpretability, characterized in that, The system is used to implement the method of any one of claims 1-9, comprising a basic encoding module, a dual-channel context feature enhancement module, a feature fusion and decoding module, and a lightweight student model functional classification module; wherein, the basic encoding module is used to convert the document to be extracted into a high-dimensional context vector sequence that can be processed subsequently, and consists of a word segmentation preprocessing submodule and a pre-training encoding submodule; the dual-channel context feature enhancement module performs local fine-grained pattern capture and global hierarchical structure information evaluation in parallel, and consists of a local pattern mixing expert submodule and a hierarchical context segmentation submodule; the lightweight student model functional classification module is used to construct training data and supervise the fine-tuning of the lightweight student model, perform classification of the candidate keyword set in five functional dimensions and identify interference items, and output a structured functional keyword knowledge dictionary, and consists of a data construction submodule, a supervised fine-tuning submodule, and an online inference submodule.

Images