Entity relationship identification method based on rotation position coding and global pointer network

Abstract

The invention discloses an entity relationship recognition method based on rotation position coding and a global pointer network, which comprises the following steps of: coding an input Chinese threat intelligence text by utilizing a pre-training language model fused with the rotation position coding, and generating context semantic representation with enhanced position perception capability; based on the context semantic representation, decoding all possible entity spans and types thereof in parallel in a two-dimensional grid space through a global pointer network to generate an entity set; for a target entity pair in the entity set, constructing a structured input sequence containing entity position information; processing the structured input sequence by using a double-attention coding mechanism; and based on the output of the double-attention coding mechanism, determining the relationship type between the target entity pairs through a relationship classification module. According to the method, precise decoding of nested entities and robust recognition of cross-language terms are realized through geometric space mapping and a dynamic boundary optimization mechanism.

Description

Technical Field

[0001] This invention relates to the field of network security technology, and in particular to an entity relationship identification method based on rotational position encoding and a global pointer network. Background Technology

[0002] Against the backdrop of the accelerating evolution of the digital age, cyberspace security faces an increasingly complex threat landscape. New attack models such as Advanced Persistent Threats (APTs) utilize supply chain penetration and zero-day exploitation to form multi-stage, cross-platform attack chains, posing a severe challenge to critical infrastructure protection systems. Cybersecurity threat intelligence (CTI), as a core element of the defense system, carries crucial information such as attack characteristics and vulnerability exploitation patterns. However, most published CTI data exists in unstructured text format, with Chinese threat reports facing unique analytical challenges due to language characteristics and differences in the security ecosystem.

[0003] Current research on threat intelligence knowledge graph construction largely focuses on English contexts. A representative work is the Threat Intelligence Knowledge Graph (TiKG) framework proposed by Mouiche and Saad (2024), which achieves a 90% F1 score on the DNRTI dataset through joint modeling of SecureBERT and the attention mechanism BiLSTM. However, its reliance on Latin-based grammatical structures limits its cross-linguistic applicability (Mouiche I, Saad S. Entity and relation extractions for threat intelligence knowledge graphs[J]. Computers&Security, 2025, 148: 104120. DOI:10.1016 / j.cose.2024.104120.). There are three core challenges in Chinese CTI text: (1) the prevalence of nested entities, such as the need to identify both the organizational entity and the attack event in "SolarWinds supply chain attack incident"; (2) the high proportion of mixed Chinese and English terms, which leads to the accumulation of boundary judgment errors in traditional word segmentation models; and (3) the average number of characters for long-span Chinese entities far exceeds the length distribution of English entities. Although the first Chinese threat intelligence dataset CDTier constructed by Zhou et al. (2023) provides a benchmark for field research, the existing pipeline model has an entity recognition F1 score of only 77.99% on this dataset (Zhou YH, Ren YT, Yi M, et al. CDTier: A Chinese dataset of threat intelligence entityrelationships[J]. IEEE Transactions on Sustainable Computing, 2023, 8(4):627-638. DOI: 10.1109 / TSUSC.2023.3240411.), highlighting the shortcomings of existing methods in Chinese semantic representation and entity boundary modeling.

[0004] Early research on entity extraction primarily relied on rule-based and statistical learning methods. Relation extraction methods have evolved from rule-driven to semantic understanding-based approaches. Existing methods are largely geared towards English contexts and lack adaptation to the semantic structure and threat entity distribution characteristics of Chinese. Joint extraction methods effectively mitigate the error propagation problem of traditional pipeline methods through shared representation spaces and synchronous parameter optimization mechanisms. Pipeline methods employ a staged processing flow, which, while structurally clear, carries the risk of error accumulation. However, existing methods are generally limited to small-scale customized datasets (such as Vulcan, which contains only 6 entity classes), and general language models struggle to capture the semantic features of security terms like "watering hole attacks." The cumulative effect of entity recognition errors in the concatenated architecture on the relation extraction stage leads to significant average performance loss.

[0005] Current research has made significant progress in the construction of knowledge graphs for cyber threat intelligence, but the unique challenges of Chinese threat intelligence remain unresolved. While joint extraction frameworks demonstrate advantages in English contexts through parameter sharing mechanisms, their direct application to Chinese CTI texts is limited by differences in syntactic structure (such as nested entities in Chinese) and entity distribution characteristics (Chinese entities are on average longer than English entities). Pipeline methods, despite their modularity, struggle to address the error accumulation problem caused by mixed English and Chinese terminology in Chinese CTI.

[0006] Existing technologies for entity and relation extraction in Chinese cybersecurity threat intelligence have inherent limitations: Traditional entity recognition relies on sequence labeling architectures, requiring models to predict entity boundary labels word by word through linear decoding. This mechanism necessitates the pre-defined mutual exclusion of entities to avoid label conflicts. When dealing with the nested entity structures prevalent in Chinese CTI, the system is forced to decompose overlapping entities into sequential prediction tasks, leading to the loss of semantic-level information and causing boundary judgment contradictions. Simultaneously, relation extraction models employ general positional encoding and attention mechanisms, requiring the implicit integration of entity positional information into contextual representations. This process necessitates indirect learning of positional associations through multi-layered networks. When facing long-distance cross-sentence dependencies unique to Chinese, positional signals continuously attenuate during deep transmission, forcing models to consume additional computational resources to compensate for positional awareness. These architectural constraints result in two core defects: semantic ambiguity accumulation in entity recognition due to forced decoupling of nested structures, and reduced ability to capture cross-segment threat associations in relation extraction due to inefficient positional modeling. Ultimately, this significantly limits the accuracy and robustness of traditional methods in Chinese CTI scenarios. Summary of the Invention

[0007] This invention addresses the challenges of traditional entity relation extraction methods in automated processing of Chinese cybersecurity threat intelligence (CTI). These methods struggle to effectively resolve semantic ambiguity in nested entities, boundary conflicts in mixed Chinese and English terms, and the coupling of long-span entity recognition, resulting in poor performance on Chinese CTI datasets. This invention proposes an entity relation recognition method based on rotational position encoding and a global pointer network. Specifically, it presents a pipeline-style joint entity relation extraction framework (RoGloZER-RE-Net) that integrates rotational position encoding and a global pointer network. Through geometric space mapping and dynamic boundary optimization mechanisms, it achieves accurate decoding of nested entities and robust recognition of cross-linguistic terms. To adapt the entity relation extraction technology to the characteristics of Chinese cybersecurity threat intelligence, this invention builds upon the semantic representation advantages of pre-trained language models, proposing a collaborative optimization architecture based on rotational position encoding and a global pointer network, and designing a hierarchical position awareness and dynamic boundary optimization mechanism. Based on the framework constructed according to this invention, the system can simultaneously resolve semantic boundaries in nested entity structures and cross-linguistic terminology scenarios.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: An entity relationship recognition method based on rotational position encoding and a global pointer network includes: Entity extraction: The input Chinese threat intelligence text is encoded using a pre-trained language model that incorporates rotational position encoding to generate a contextual semantic representation with enhanced positional awareness. Based on the contextual semantic representation, all possible entity spans and their types are decoded in parallel in a two-dimensional grid space through a global pointer network to generate an entity set. The global pointer network uses a rotational position encoding-enhanced attention mechanism to jointly predict entity boundaries and types. Relation extraction: For target entity pairs in the entity set, a structured input sequence containing entity location information is constructed; the structured input sequence is processed using a dual attention encoding mechanism, which includes a global attention stage for capturing global context semantics and an entity perception attention stage for focusing on the interaction features of the target entity pairs; based on the output of the dual attention encoding mechanism, the relationship type between the target entity pairs is determined by the relationship classification module.

[0009] Furthermore, the pre-trained language model that incorporates rotational position encoding is a RoBERTa model that employs rotational position encoding; The injection process of the rotational position encoding achieves a position-aware attention mechanism through rotational transformations in geometric space, including: For the position in the sequence is query vector and position as key vector Through rotation matrix and Perform the transformation: In the formula, in and These are the transformed query vector and key vector, respectively, with rotation angles. , i∈[0,d / 2−1], where d is the dimension of the hidden layer. This represents the direct sum of a block diagonal matrix; The transformed attention score is: in Let m be the attention score of position m to position n, and T denote the matrix transpose.

[0010] Furthermore, the injection process encoded at the rotational position: It also includes a sub-word position compensation mechanism for Chinese text: For the sub-word units generated by the word segmenter, the following position-weighted strategy is adopted: in This represents the set of consecutive sub-word units from the i-th sub-word unit to the j-th sub-word unit. This represents the positional encoding of the k-th sub-word unit. This represents the positional encoding from the i-th sub-word unit to the j-th sub-word unit; It also includes a position truncation mechanism: Set maximum relative distance For any two positions m and n in the sequence, if their relative position difference is... The rotation angle is then adjusted using linear interpolation. in This is the adjusted rotation angle; A hybrid position representation strategy is adopted: in This represents the hidden state at position i in layer l. RoPE(⋅) indicates that rotational position encoding is performed. For learnable parameters, This is the learnable absolute position code for the i-th position.

[0011] Furthermore, in the encoding stage of entity extraction, a hierarchical feature extraction strategy is adopted to construct a multi-level semantic representation. This strategy focuses on positional and semantic information at different granularities through different network layers of the Transformer architecture. Shallow networks focus on local lexical patterns and capture character-level and word-level syntactic combination patterns through an attention mechanism enhanced by rotational positional encoding. The mid-layer network expands the receptive field by introducing a dynamic sparse attention mechanism to model semantic dependencies across clauses; Deep networks leverage the long-range association modeling capabilities of rotational position encoding to integrate document-level semantics, enabling global modeling of complex attack chains.

[0012] Furthermore, the operation of the global pointer network includes: The hidden state output by the encoder is projected through a type-sensitive projection matrix to generate query vectors and key vectors; Apply a rotational position encoding transformation to the query vector and the key vector, for positions where query vector and position as key vector The transformed query vector is obtained. and key vector ,based on and Obtain the rotation-enhanced attention score: Where Re(.) denotes taking the real part of the complex number, The attention score represents the presence of an entity with a starting position of i and an ending position of j, and T represents the matrix transpose. Multi-granular semantic features are captured through a parallelized multi-head design, and the score of head h is correlated with the scalar weight of type t. The output is integrated into an entity matrix E through a gating fusion mechanism.

[0013] Furthermore, the entity extraction process also includes entity boundary determination of the entity matrix, incorporating multi-level positional constraints during the determination process: Using the triangular mask matrix Apply rigid geometric constraints, where: in Maximum span threshold; Incorporate a learnable relative position bias term into the attention score. The length distribution characteristics of Chinese entities are encoded using learnable parameters: in span length The bias parameters, To pass Adjusted attention score; In the entity existence determination phase, define the entity existence determination function: In the formula, in This represents the existence determination function for an entity of type n, starting at position i and ending at position j. This represents the attention score for an entity of type n, with a starting position of i and an ending position of j, adjusted by the positional bias term. For activation function, For entities of type n, the adaptive dynamic threshold is... Let n be the average length of an entity of type n. It's a hyperparameter.

[0014] Furthermore, in the entity extraction, an improved ZLPR loss function is used for model training: in Represents the ZLPR loss function. and These represent the sets of positive and negative samples, respectively. Indicates the score of the positive sample; By introducing prior knowledge of entity distribution, a frequency-domain adaptive loss adjustment mechanism is constructed to... Adjustments will be made: in This represents the number of times the nth type of entity appears in the training set. T is the smoothing factor, representing the total number of entity types; Automatically identify Top-K difficult negative samples in each round of training. : in The threshold for positive sample scores. This represents the true label corresponding to the candidate span (i,j). This indicates that there is no entity with boundary (i,j) in the training data; Adjustments were made using a cosine annealing strategy: in This represents the training iteration number t corresponding to N represents the total number of training iterations; In the early stages of training, set up The focus is on correcting erroneous samples; as training progresses, the error rate is gradually reduced to [a lower threshold]. To capture difficult samples with blurred boundaries.

[0015] Furthermore, in the relation extraction, the encoding process of the structured input sequence is as follows: in Represents a structured input sequence. For the original text, For the target entity, This indicates a splicing operation. It is a classification label. It is a separator marker; To model the relative positions between entities, calculate the relative positional difference between entity pairs in the text. ,in Indicates the starting position of the entity in the text, and Mapped to a 128-dimensional vector space via a sine function; A learnable type embedding matrix is ​​established to map entity types to feature vectors, which are then concatenated with text encodings. Finally, the concatenated features are fused into the hidden representation of BERT through a linear transformation.

[0016] Furthermore, in the dual-attention encoding mechanism, the global attention stage employs standard multi-head self-attention to capture document-level semantic associations, while the entity-aware attention stage enhances attention to semantic cues surrounding entities by injecting relative positional biases. The relative position offset Calculate as follows: in h and Indicates the combination of location distances. Indicates the starting position of the entity in the text. For learnable parameters, Let be the dimension of the k-th attention head.

[0017] Furthermore, the relation classification module, based on the contextual representation of the deep semantic encoding layer, constructs a multi-granularity feature fusion and hierarchical decision architecture, specifically implementing the following steps: Multi-granularity feature fusion: generating fused feature vectors This vector is obtained by concatenating the following three types of features: Global context features: Extract the [CLS] marker vector at the first position of the encoder. As a global context feature; Entity interaction features: the hidden state vectors of two target entities E1 and E2 and After average pooling, element-wise multiplication is performed to obtain entity interaction features. : in Indicates the starting position of the entity in the text. Representing entities The hidden state vector of the token corresponding to the starting position in the encoder's 12th layer; Structured location features: Encoding the relative distance of entities Embedded with entity type By splicing the data, a structured positional representation is obtained; The multi-granularity feature fusion is represented as follows: in This represents a vector concatenation operation; Hierarchical decision architecture design: for the fusion features A gradual process is employed, including: Feature normalization: for By implementing standardization and nonlinear activation, we obtain : in For a dimension reduction matrix, Representation layer normalization, Represents the activation function of the Gaussian error linear unit; Nonlinear projection: Constructing a two-layer residual network to extract discriminative features : in These are projection parameters; Category-aware decoding: Employs a parameter isolation strategy to achieve adaptive classification decisions based on relation types, generating the final relation type probability distribution. : in An embedding matrix is ​​created for each relation type, with each row corresponding to a decision plane for a specific relation. The classification boundary is dynamically adjusted based on category frequency. This represents the total number of relation types.

[0018] Compared with the prior art, the present invention has the following beneficial effects: This invention overcomes the bottleneck of Chinese CTI parsing through a triple innovative mechanism: First, it introduces a RoBERTa model enhanced with rotational position encoding to solve the semantic ambiguity problem of nested entities and cross-linguistic terms; second, it uses a GlobalPointer network to achieve joint decoding of entity start and end points in a two-dimensional grid space, improving entity recognition accuracy to 84%; finally, it develops a zero-boundary Log-Sum-Exp loss function (ZLPR) to solve the long-tail effect of threat entity category distribution. Experiments show that this invention achieves an entity recognition F1 score of 84% on the CDTier dataset, a 6 percentage point improvement over the best baseline, while achieving a relation extraction accuracy of 92%, providing new methodological support for dynamic threat graph construction.

[0019] The main contributions of this invention are: (1) constructing the first rotational position-aware model for Chinese CTI, solving the boundary conflict problem of nested entities and mixed terms through a geometric space mapping mechanism; (2) proposing a decoding architecture that integrates global pointer networks and statistical learning, improving the recall rate of long-span entities while maintaining linear time complexity; and (3) designing a dynamically optimized multi-label loss function (optimized ZLPR), significantly improving the robustness of entity recognition in class imbalance scenarios. These innovations provide a new technical path for the automated processing of Chinese threat intelligence and promote the development of multilingual network security knowledge graph research. Attached Figure Description

[0020] Figure 1 This is a schematic flowchart of an entity relationship recognition method based on rotational position encoding and a global pointer network, provided in an embodiment of the present invention. Detailed Implementation

[0021] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments: like Figure 1 As shown, an entity relationship recognition method based on rotational position encoding and a global pointer network includes: Entity extraction: The input Chinese threat intelligence text is encoded using a pre-trained language model that incorporates rotational position encoding to generate a contextual semantic representation with enhanced positional awareness. Based on the contextual semantic representation, all possible entity spans and their types are decoded in parallel in a two-dimensional grid space through a global pointer network to generate an entity set. The global pointer network uses a rotational position encoding-enhanced attention mechanism to jointly predict entity boundaries and types. Relation extraction: For target entity pairs in the entity set, a structured input sequence containing entity location information is constructed; the structured input sequence is processed using a dual attention encoding mechanism, which includes a global attention stage for capturing global context semantics and an entity perception attention stage for focusing on the interaction features of the target entity pairs; based on the output of the dual attention encoding mechanism, the relationship type between the target entity pairs is determined by the relationship classification module.

[0022] Furthermore, the method specifically includes: 1 Entity Extraction Model This section proposes RoGloZER, a joint entity extraction framework for Chinese network threat intelligence. This model overcomes the technical bottlenecks of nested entity recognition and cross-lingual terminology processing through a triple innovative mechanism. The architecture consists of three core components: (1) a RoBERTa deep semantic encoder enhanced with Rotary Position Embedding (RoPE) to solve the problem of nested entities and mixed terminology ambiguity in Chinese CTI text; (2) a GlobalPointer network to achieve joint decoding of entity boundaries in two-dimensional grid space, significantly improving the performance of long-span entity recognition; and (3) ZLPR to alleviate the long-tail distribution problem of threat entity categories through dynamic marginal adjustment.

[0023] Compared to traditional sequence labeling models, RoGloZER's innovation lies in three dimensions: First, it transforms entity recognition into a Cartesian product matching problem of head and tail pointers through a geometric space mapping mechanism, reducing the decoding complexity of nested entities from O(T²) to a linear level; second, it integrates a hybrid encoding strategy that combines relative position awareness and absolute position benchmarks, effectively capturing the cross-paragraph entity relationships unique to Chinese; and finally, it constructs multi-granularity feature distillation channels to capture local lexical patterns in the lower-level network and establish document-level semantic relationships in the deeper network. This framework achieves an F1 score of 84.41% on the CDTier dataset, improving upon the existing state-of-the-art methods by 6.42 percentage points, laying a high-quality foundation for subsequent relation extraction and entity recognition.

[0024] 1.1 RoBERTa Encoding Optimization In Chinese cyber threat intelligence entity identification tasks, the choice of semantic encoder directly affects the modeling ability of technical terms and complex grammatical structures. This invention adopts the RoBERTa-wwm-ext-large pre-trained model as the basic encoding architecture, which has dual advantages in Chinese text processing: the whole word masking mechanism effectively alleviates the semantic fragmentation problem of traditional sub-word segmentation for complex Chinese technical terms (such as "supply chain attack") through a whole-word level masking strategy; the introduction of RoPE enables the model to explicitly capture the relative positional relationships between characters, which plays a key role in identifying long-distance entity associations in CTI text (such as APT attack phase descriptions). Through these two core optimizations, the encoder constructs a deep semantic representation space adapted to the characteristics of Chinese CTI, providing high-quality contextual features for subsequent entity detection modules.

[0025] 1.1.1 Input Vectorization (Embedding Layer) Threat entity identification begins with vectorizing the input, that is, transforming unstructured raw text into a numerical representation that can be processed by deep neural networks. For example, given Chinese threat intelligence text... Sub-word segmentation is performed using the RoBERTa-wwm-ext pre-trained word segmenter: It includes three key stages: 1) Whole-word masking segmentation: Employs a hybrid segmentation mechanism based on Unicode encoding to convert continuous character sequences into sets of sub-word units. This word segmenter performs maximum matching at the character level while maintaining the integrity of technical terms. This approach avoids the erroneous segmentation of technical terms by traditional Chinese word segmentation tools while retaining the ability of BERT-like models to handle out-of-vocabulary words.

[0026] 2) Context-sensitive encoding: using pre-trained embedding matrices Map subwords to initial vector representations , d=768 represents the size of the Chinese RoBERTa vocabulary and the hidden layer dimension.

[0027] 3) Position-aware enhancement: Injecting position information into the output of the embedding layer. Position encoding Position awareness is enhanced by learnable absolute position encoding, and relative positional relationships are implicitly captured by combining a self-attention mechanism. This vectorization process ultimately generates a vector with a dimension of [missing value]. The initial hidden state is used as the input for subsequent Transformer layers.

[0028] 1.1.2 Rotational Position Encoding Injection This module constructs a deep semantic coding system for Chinese CTI, achieving progressive feature extraction from basic position awareness to advanced semantic reasoning through seamless integration of positional encoding injection and hierarchical semantic abstraction. The coding process follows a progressive logic of "position enhancement → local modeling → global integration → domain distillation," with each stage achieving a natural transition through feature transfer and transformation.

[0029] a) Location-encoded injection Injecting rotational position information before self-attention computation can solve the problem of long-distance dependencies: In the Transformer architecture, the design of positional encoding is crucial for capturing the relative positional relationships of sequence elements. Traditional positional encoding methods (such as absolute positional encoding) face two core challenges in long text processing: (1) absolute positional indexes cannot effectively model the relative positional relationships between distant elements; (2) fixed-dimensional positional vectors exhibit positional information decay in ultra-long sequences (such as CTI reports with more than 512 tokens). Therefore, before self-attention calculation, rotational positional information is introduced, and a position-aware attention mechanism is achieved through rotational transformation in geometric space, significantly improving the model's ability to model long-distance dependencies in Chinese CTI texts.

[0030] The core concept of RoPE is to map positional information into rotation operations in the complex domain. For a dimension index of... The position encoding component, whose rotation angle function is defined as: in This represents the hidden layer dimension. The function follows a geometric decay law, ensuring that different dimensions focus on positional patterns of different granularities. For a sequence at position... query vector and position as key vector Its rotational operation can be decomposed into: in This represents the direct sum of the block diagonal matrices. This block rotation mechanism enables position-sensitive vector space transformation, and the transformed attention score is calculated as follows: in Let m be the attention score of position m to position n, and T denote the matrix transpose.

[0031] b) Advantages of RoPE processing In Chinese CTI text, attack chain descriptions often span multiple paragraphs (such as the reconnaissance, penetration, and lateral movement stages of an APT attack). Traditional positional encoding struggles to maintain relative positional sensitivity across paragraphs. RoPE improves upon this through three design improvements: 1. Relative position invariance: for arbitrary positional offsets... Attention weights satisfy: This allows the model to model sequence dependencies of arbitrary length without explicitly storing all position combinations. 2. Decay Mechanism: The exponential decay design with adaptive rotation angle enables different attention heads to automatically focus on different distance ranges: small (Low-frequency dimension) Focus on long-range dependencies (such as entity relationships across paragraphs); large (High-frequency dimension) Capture local syntactic structures (such as entity modifiers). 3. Zero initialization compatibility when... In this case, the rotation matrix degenerates into the identity matrix, which naturally supports attention-focused position alignment, especially important for the "entity-modifier" structure in Chinese (such as "spear phishing attacks against government agencies").

[0032] c) RoPE optimization and improvement To address the character-level processing requirements of Chinese CTI text, improvements have been made to the standard RoPE implementation.

[0033] 1. Sub-word position compensation: For sub-word units generated by the word segmenter (such as "##attack"), a position-weighted strategy is adopted to avoid sub-word segmentation from disrupting positional continuity. in The positional encoding of the k-th sub-word unit is generated by a learnable embedding matrix with dimension d=768. This represents the set of consecutive sub-word units from the i-th sub-word unit to the j-th sub-word unit. This represents the positional encoding from the i-th sub-word unit to the j-th sub-word unit.

[0034] 2. Set the maximum relative distance for position truncation. For ultra-long distances (i.e.) Linear interpolation is used: in The relative positional distance between entity pairs, the positional truncation setting can prevent gradient explosion while maintaining the ability to model long-range dependencies.

[0035] 3. Position encoding retains absolute position encoding at the underlying Transformer layer, forming a hybrid position representation: in These are learnable parameters, and different layers are independent of each other. It uses standard learnable absolute position encoding with the same dimensions as the hidden layers. This design gradually transitions to a relative position-dominated representation in deeper networks. Then, through RoPE vectors, it retains the traditional Transformer's ability to focus on local semantics while also possessing the potential to model cross-segment entity relationships, creating ideal conditions for the subsequent extraction of multi-level semantic features.

[0036] 1.1.3 Hierarchical Feature Extraction During the encoding phase, a hierarchical Transformer architecture is used to construct multi-level semantic representations. A progressive feature extraction strategy is employed, with shallow layers focusing on local lexical patterns (such as technical terminology structures), intermediate layers modeling cross-clause dependency relationships, and deep layers utilizing long associations from rotational encoding to integrate document-level semantics, achieving a global representation of complex concepts. This hierarchical architecture effectively coordinates the modeling needs of local details and global semantics, providing multi-granular semantic support for fine-grained entity recognition of Chinese CTI text.

[0037] ① Local syntax modeling Shallow networks focus on character-level and word-level grammatical combination patterns, capturing the compositional regularities of specialized terms through a multi-head attention mechanism. For the l-th layer (l∈[1,4]), its computation process can be formalized as follows: in They represent the first The query, key, and value matrices of the layer are output from the previous layer through linear transformation. generate. This represents the attention head dimension, used to scale the dot product attention score. This represents an offset matrix containing rotational position encoding, used to inject relative position information. This represents the attention weight matrix, which indicates the correlation between tokens. The representation layer normalization operation stabilizes the training process. For the projection matrix, The concatenated multi-head output is mapped back to the hidden layer dimension. Through local grammatical modeling, compound word structures unique to the Chinese cybersecurity field, such as "watering hole attack" and "zero-day vulnerability," can be effectively identified.

[0038] ② Global semantic modeling The middle-layer network gradually expands its perceptual field of view, establishing semantic dependencies across clauses. A dynamic sparse attention mechanism is introduced at this stage. in Indicates the first The first in the layer The token is paired with the first... Attention weights for each token. The unnormalized attention score is calculated via query-key dot product. It is a learnable relative position offset.

[0039] Indicates the first The neighborhood set of each token is dynamically determined using a learnable gating function: They represent the first and the The hidden state of each token. This represents the trainable parameter matrix of the gate function. This represents the sigmoid activation function, with an output range of [0,1], used for binary decision-making. This represents a vector concatenation operation. This adaptive neighborhood selection mechanism, while maintaining computational efficiency, can effectively model the cross-sentence association of "attack method-target" common in Chinese CTI text.

[0040] ③ Deep Domain Characteristic Distillation Deep neural networks establish document-level semantic relationships through RoPE, enabling global modeling of APT attack chains. Specifically, a feature fusion strategy is employed during the generation of the final hidden state. in This represents the hidden state after final fusion, which serves as input for subsequent tasks. Indicates the first Dynamic fusion weights for layer features. Indicates the first The hidden state output of the layer. The fusion coefficient is dynamically calculated through an attention mechanism: in This represents the trainable weight matrix, used to calculate the importance of each layer. Indicates the first The representation of the first token output by the layer (usually [CLS]). Indicates the first The representation of the last token output by the layer. This indicates a vector concatenation operation.

[0041] This mechanism endows the model with the ability to adaptively focus on key semantic levels. For example, when processing the description of "supply chain attack", deep networks can effectively integrate the following features: low-level features: product name recognition; mid-level features: attack vector modeling; high-level features: attack impact inference.

[0042] This hierarchical semantic abstraction process enables progressive feature extraction, achieving multi-granularity entity recognition on Chinese datasets: the bottom layer captures basic named entities, the middle layer parses complex threat behaviors, and the deep layer establishes cross-sentence semantic relationships. This hierarchical abstraction capability effectively supports the accurate identification of diverse threat entities in Chinese CTI text. During the encoding stage, based on the RoBERTa Chinese CTI semantic encoding framework, it integrates a full-word masking strategy with RoPE, and achieves multi-granularity feature extraction through a hierarchical Transformer architecture. The generated contextual representation provides input to the GlobalPointer module, supporting end-to-end entity boundary detection.

[0043] 1.2 Entity Recognition Framework Based on GlobalPointer This section proposes a global pointer entity recognition framework for Chinese CTI, which overcomes the performance bottleneck of traditional sequence labeling models by modeling the start and end positions and type information of entities. Based on the contextual representation output by the RoBERTa encoder, a global pointer network is used to simultaneously predict entity boundaries and categories in a two-dimensional grid space, effectively solving the challenges of nested entity recognition and cross-language terminology processing in Chinese CTI text.

[0044] The innovation of the framework is reflected in three aspects: (1) Integrating rotational position encoding to enhance relative position perception and effectively model long-range entity associations across paragraphs; (2) Designing type-sensitive projection matrices to achieve inter-class decoupling of entity boundaries and eliminate semantic interference from multiple categories; (3) Improving ZLPR to optimize the distribution of long-tail entities through dynamic marginal adjustment.

[0045] 1.2.1 Multi-head pointer detection architecture Based on the context representation generated by the deep semantic encoder, GlobalPointer is used to achieve global detection of entity boundaries. This module transforms entity recognition into a multi-label classification problem on a two-dimensional grid by constructing a Cartesian product space of head and tail pointers. The specific implementation includes the following key steps: 1. Decoupling Projection and Parameters. Based on the hidden states output by the encoder, a type-sensitive linear projection strategy is implemented. This involves projecting the hidden states output by the encoder... ( For batch size Generate query-key-value pairs through type-sensitive linear transformations: in For each dimension of the attention head, cross-category feature independence is achieved by decoupling the projection parameters of different entity types. , It is a type-sensitive independent projection matrix. Such feature uniqueness allows each entity type to have an independent feature space to avoid semantic interference between types, and allows attention heads of different entity types to focus on specific syntactic patterns to enhance interpretability.

[0046] 2. Position-enhanced attention computation. Inheriting the RoPE advantages of the coding layer, geometric space transformation is introduced into the pointer detection process. For positions where... query vector and position as key vector The rotation-enhanced attention computation can be formalized as follows: in This is a block-diagonal rotation matrix. Implement position encoding with geometrical decay. Mapping the attention score in the complex space back to the real domain preserves physical interpretability. The attention score represents the presence of an entity with starting position i and ending position j, and T represents the matrix transpose. This architecture exhibits three key characteristics in Chinese CTI scenarios: First, it is sensitive to long spans; for entities spanning multiple paragraphs (such as "initial intrusion → lateral movement → data theft" in APT attack descriptions), rotation encoding maintains the invariance of relative positional relationships. Second, it is precise in terms of boundaries; through geometrical attenuation of positional differences, it enhances the positional association between adjacent characters. Finally, it provides nested parsing; the rotation phase differences between different entity types can distinguish nested structures.

[0047] 3. The attention mechanism captures multi-granular semantic features through a parallelized multi-head design: in Head The score is related to the type scalar weights. For the first The attention score matrix for each head, with each head focusing on different types of boundary features (such as entity prefix modification, suffix constraint, etc.), is shown in Table 1 below: Table 1. Effects of different attention heads The output is then integrated into an entity matrix through a gating fusion mechanism. : Among them, the gating weight By dynamically adjusting the contribution of each attention point, this adaptive mechanism enables the model to automatically adjust the feature attention pattern when processing different types of CTI reports (such as event analysis reports vs. vulnerability announcements). Each element of the matrix Contains batch ,type Entity Start Termination of matching probability (hereinafter referred to as) However, there exists in two-dimensional grid space. Given the sheer size of the candidate pool, effectively filtering out entity boundaries that conform to grammatical rules and semantic logic still requires the development of a systematic set of judgment criteria.

[0048] 1.2.2 Entity Scoring and Boundary Determination In entity matrix Based on geometric space mapping, the entity scoring and boundary determination stages achieve precise localization through a combination of structured constraints and statistical learning, allowing arbitrary candidate entities to be precisely located. The final score calculation is broken down as follows: First, through the triangular mask matrix Implement rigid geometric constraints to exclude illegal combinations with excessively long spans or reverse directions; secondly, utilize relative position offset terms. The probability distribution characteristics of the injected Chinese entity length; finally, a type-related threshold is applied. It enables dynamic judgment and solves the decision bias problem caused by class imbalance.

[0049] 1. Mask Constraints: In Chinese CTI text, entity spans need to satisfy strict syntactic structure and semantic coherence; therefore, a strip mask matrix is ​​introduced. The legitimacy of entity boundaries is ensured through dual constraints: Set maximum span threshold This dynamic truncation mechanism can cover complex long entities such as those that use the Cobalt Strike post-penetration framework to perform lateral movement, while avoiding computational redundancy caused by dealing with ultra-long pseudo-entities.

[0050] 2. Positional Bias Term: A learnable relative positional bias term is added to the attention score. The length distribution characteristics of Chinese entities are encoded using learnable parameters: in Different bias parameters for different span lengths can enhance the model's prior knowledge of common entity lengths, flexibly adapt to the complex entity distribution patterns in Chinese CTI text, and provide a high-confidence candidate set for subsequent threshold determination. span length The bias parameters, To pass Adjusted attention score.

[0051] 3. Relevant threshold mechanism: In the entity existence determination phase, define an entity existence determination function: in This represents the existence determination function for an entity of type n, starting at position i and ending at position j. This represents the attention score for an entity of type n, with a starting position of i and an ending position of j, adjusted by the positional bias term. For activation function, The type-adaptive dynamic threshold is determined by statistically analyzing the average length of various entities in the training set. Automatic adjustment: hyperparameters The boundary determination was achieved through grid search. A type-related threshold mechanism dynamically adjusts the decision boundaries for different entity types, addressing the recognition bias caused by differences in entity length distribution in Chinese CTI text. Through the triple synergy of the aforementioned triangular mask constraint, relative position bias term, and type-related threshold mechanism, an entity boundary determination system conforming to the characteristics of Chinese CTI text was constructed. However, when faced with the class imbalance phenomenon exhibited by the CDTier dataset, relying solely on geometric constraints and statistical bias is still insufficient to achieve stable convergence across classes. Therefore, a new loss optimization paradigm needs to be designed to mitigate model bias caused by long-tail distribution while ensuring boundary determination accuracy.

[0052] 1.2.3 Multi-label optimization strategy To address the long-tailed entity distribution problem in Chinese CTI text, this invention proposes an improved ZLPR to enhance class balance: (1) Zero-boundary constraints suppress gradient dominance in easily distinguishable samples and focus on samples with ambiguous boundaries; (2) Adaptive frequency weights alleviate gradient decay in low-frequency entities; (3) Dynamic hard sample mining optimizes the decision boundary. This loss function works synergistically with the geometric constraint mechanism to effectively balance the class distribution while ensuring the accuracy of entity boundaries.

[0053] 1. ZLPR reconstructs the gradient distribution characteristics of the loss space by reparameterizing the decision boundary: in and Let represent the sets of positive and negative samples, respectively. By excluding the gradients of easily separable samples and focusing on optimizing difficult samples, gradient suppression of easily separable samples is achieved: when the positive sample scores... When it is much greater than 0, its corresponding exponent term As the gradient approaches zero, its contribution is automatically decayed; similarly, when the score of a negative sample is much less than zero, the gradient of the corresponding term is also suppressed. This characteristic allows the model to concentrate its limited optimization capabilities on difficult samples with ambiguous boundaries, such as distinguishing semantically similar entities like "phishing emails" and "spear phishing attacks" in Chinese CTI.

[0054] 2. Category-Sensitive Weight Allocation. By introducing prior knowledge of entity distribution, a frequency-domain adaptive loss adjustment mechanism is constructed: in This represents the number of times the nth type of entity appears in the training set. This is a smoothing factor. This function exhibits sublinear response characteristics in the frequency domain, and compared to traditional inverse frequency weighting methods, it can more gently adjust the gradient weights of low-frequency categories. In Chinese CTI scenarios, this design effectively mitigates optimization biases caused by extreme imbalances, such as gradient competition between "supply chain attacks" (low frequency) and "malicious software" (high frequency). Specifically, when... When the value drops from 100 to 10, The increase was only This avoids model oscillations caused by aggressive weighting.

[0055] 3. Dynamic Hard Sample Mining. Through an adaptive sample selection strategy, the training process is progressively optimized, automatically identifying Top-K hard negative samples in each training round: in The threshold for positive sample scores. This represents the true label corresponding to the candidate span (i,j). This indicates that there is no entity with boundary (i,j) in the training data.

[0056] threshold A cosine annealing strategy is used for adjustment, initially focusing on obvious errors and later capturing samples with blurred boundaries. in This represents the training iteration number t corresponding to N represents the total number of training iterations.

[0057] In the early stages of training (number of training iterations) Set a higher threshold. The focus is on correcting significantly misclassified samples; as training progresses, the error rate is gradually reduced to [a lower threshold]. This mechanism captures difficult samples with ambiguous boundaries. It demonstrates unique value in Chinese CTI text: in the early stages, it quickly corrects obvious misjudgments such as "APT28→APT29," and in the later stages, it handles subtle semantic differences such as "vulnerability exploitation→vulnerability attack." Through the implementation of the aforementioned multi-label optimization strategy, the model gradually establishes an accurate mapping capability between Chinese CTI entity boundaries and categories during training. With the help of ZLPR and the dynamic difficult sample mining mechanism, the prediction bias caused by class imbalance is effectively mitigated.

[0058] The proposed RoGloZER joint entity recognition framework for Chinese CTI text enables accurate extraction of threat entities from Chinese CTI text. Entity boundaries are determined directly through a global pointer network decoding the matrix positions in a two-dimensional grid space: the start position (i) and end position (j) of each candidate entity are jointly determined by attention scores in the matrix coordinate system, forming a type-sensitive entity mapping matrix. This mechanism overcomes the sequential decoding limitations of traditional sequence labeling models, enabling the synchronous parsing of the start-end positions of nested entities and long-span entities through matrix operations. This significantly improves the entity recognition accuracy of Chinese threat intelligence, provides high-quality entity boundary features for subsequent relationship extraction tasks, and lays a key technological foundation for constructing dynamic cybersecurity knowledge graphs.

[0059] 2 Relationship Extraction Model As cyberattack techniques become increasingly complex, entity relationship parsing in threat intelligence faces core challenges such as cross-sentence dependencies and semantic ambiguity. Existing relationship extraction methods have three limitations in Chinese CTI scenarios: 1) Insufficient modeling of entity location information leads to misjudgment of attack chain temporal relationships; 2) Ignoring the long-distance dependency structure unique to Chinese, making it difficult to capture cross-paragraph threat associations; 3) Lack of domain-adaptive feature fusion mechanisms, which restricts the reasoning ability for complex threat patterns.

[0060] This invention proposes the RE-Net relation extraction framework, overcoming the aforementioned bottlenecks through a triple innovative mechanism. First, a hierarchical input structure of [CLS]-entity pairs is designed to explicitly encode entity location and contextual semantics. Second, a dual-attention fusion module is constructed, introducing a global attention mechanism for entity awareness based on a pre-trained language model. Finally, a dynamic relative position encoding strategy is proposed to effectively model the cross-sentence dependency characteristics of Chinese CTI text. Experiments show that this framework achieves 92% relation extraction accuracy on the CDTier dataset, a 3 percentage point improvement over the BERT baseline, providing reliable relational semantic support for threat knowledge graph construction.

[0061] 2.1 Input Representation and Feature Engineering Text Structured Encoding: Considering the characteristics of sparse entity distribution and significant cross-sentence dependencies in Chinese CTI texts, a structured splicing of the original text and target entity pairs is designed. For the input triple {"text": "The 360 Threat Intelligence Center of the Fantasy Bear roaming in Eastern Europe and Central Asia...", "ent1": "Fantasy Bear", "ent2": "Eastern Europe"}, the encoding process is specifically as follows: Among them is the original text, is the target entity, represents the splicing operation, is the classification marker, is the separator marker. This strategy explicitly marks the entity positions through special separators, enabling the model to maintain the integrity of entity information under the maximum sequence length limit.

[0062] Position-Sensitive Feature Enhancement: To model the long-distance dependencies and nested entity structures unique to Chinese CTI, dynamic relative position embeddings are introduced based on the absolute position encoding of the traditional BERT, forming a hybrid position perception system: 1. Entity Relative Distance Encoding: Calculate the relative position difference of the entity pair in the text , and map it to a 128-dimensional vector space through the sine function.

[0063] This encoding enables the model to explicitly perceive the exponential decay law of entity spacing and effectively capture cross-paragraph relationships such as "attack vector → scope of influence" in Chinese CTI.

[0064] 2. Entity Type Embedding: Establish a learnable type embedding matrix , map the entity type (such as attack organization, attack means, etc.) to a feature vector, and splice it with the text encoding. Then, fuse the above features into the hidden representation of BERT through a linear transformation, achieving the collaborative representation of position and semantic information.

[0065] 2.2 Encoding Layer Based on the structured input representation, this module constructs a multi-level semantic abstraction network, aiming to effectively fuse the global semantic features and entity interaction information of threat intelligence texts. It adopts an architecture that combines a pre-trained language model and a dual attention mechanism, and breaks through the semantic modeling bottleneck of traditional relation extraction models in the Chinese CTI scenario through two major stages of feature enhancement and information fusion.

[0066] Pre-trained Model Adaptation: Use bert-base-chinese as the basic encoder. This model obtains general semantic representation capabilities through pre-training on 120G of Chinese corpus. Its 12-layer Transformer structure generates 768-dimensional context representations: Last layer output As a deep semantic feature, among For batch size, The sequence length is given.

[0067] Two-stage attention mechanism: Based on the native BERT attention mechanism, a two-stage attention architecture for entity perception is designed to strengthen the explicit modeling of threat entity interaction features, construct a global-local collaborative two-stage attention architecture, and achieve multi-level semantic modeling. Phase 1: Global semantic modeling, using standard multi-head self-attention to capture document-level semantic associations.

[0068] in Here is the global attention parameter matrix. This stage is used to fill in the mask. It models the entire sentence's contextual dependencies and identifies macroscopic patterns of threat behavior.

[0069] Phase Two: Entity Perception Attention. This phase involves designing entity-centric local attention to focus on the interaction features of target entity pairs. It consists of the following three steps.

[0070] 1. Entity feature projection, for the target entity Dynamically project the hidden state: By splicing The representation of ( Generate entity-aware query vectors.

[0071] 2. Relative position offset, injecting relative position information between entities: in and Indicates the combination of location distances. These are learnable parameters. This bias term guides the model to focus on semantic cues surrounding entities. The dimension of each attention head is set to 64.

[0072] 3. Gated attention calculation: Two-stage attention outputs are fused via residual connections: Gating factor The contribution of global and local features is dynamically adjusted. Experiments show that the architecture achieves a relation classification accuracy of 92% on the CDTier test set, which is 7% higher than a single attention baseline, validating the effectiveness of hierarchical attention modeling.

[0073] 2.3 Relationship Classification Module This module, based on contextual representations from deep semantic encoding layers, constructs a multi-granularity feature fusion and hierarchical decision architecture, enabling accurate relation classification of Chinese CTI text. Through three-step feature abstraction and two-level classification decoding, it effectively integrates global semantics, entity interaction, and location features, significantly improving the ability to discriminate complex threat relationships.

[0074] Multi-granularity feature fusion: To comprehensively capture the relational semantic cues of Chinese CTI, a sentence-level feature extraction method is adopted, fusing the following three types of features: 1. [CLS] Global Representation: Extract the [CLS] marker vector at the first position of the encoder. As a global contextual feature, this vector aggregates global sentence semantic information through a self-attention mechanism, making it particularly suitable for modeling cross-sentence threat associations.

[0075] 2. Entity Interaction Features: Perform the Hadamard product operation on the position vectors of two entities to capture fine-grained interaction features. in Representing entities The token corresponding to the starting position is the hidden state vector output by the encoder at layer 12. This operation strengthens the cooperative semantics of entity pairs (such as the technical attribution relationship of "exploitation → attack organization") through element-wise multiplication, while suppressing irrelevant noise.

[0076] 3. Encode the relative distance of entities By embedding and splicing with entity types, a structured location representation is ultimately formed; The characteristics after fusion are: Classifier Design: The classifier achieves accurate mapping from features to relationships through a hierarchical architecture. Its design follows a progressive optimization path of "feature normalization → nonlinear projection → category-aware decoding," employing a progressive feature processing architecture to gradually extract discriminative features. 1. Feature Normalization: Standardize and apply nonlinear activation to the fused features to enhance the separability of the feature space. in, To reduce the dimensionality of the matrix, the concatenated 1792-dimensional features are compressed to 768 dimensions aligned with the BERT hidden layers. LayerNorm eliminates the dimensional differences among multiple feature sources (global semantics, entity interactions, and positional encoding). The GELU activation function enhances the separability of fuzzy relationships by smoothing nonlinearities.

[0077] 2. Nonlinear projection layer: Construct a two-layer residual network to further refine discriminative features: This design uses a bottleneck structure to first expand the dimensions to 3072 to capture fine-grained interaction patterns, then compresses them back to 768 dimensions to filter out noise features, and the residual connections preserve the integrity of the original features.

[0078] 3. Category-aware decoding: Employing a parameter isolation strategy to achieve adaptive classification decisions based on relationship types. in An embedding matrix is ​​used for relation types, with each row corresponding to a decision plane for a specific relation. A parameter isolation strategy is used to avoid high-frequency categories dominating model training. The classification boundary is dynamically adjusted based on category frequency to mitigate the impact of long-tail distribution. This represents the total number of relation types. The combined effect of both allows the model to simultaneously capture global semantic relationships (through...). The projection space) and local distribution characteristics (through (bias correction), ultimately through Complete the determination of the relationship type.

[0079] This invention proposes RE-Net, a relation extraction framework for Chinese CTI (Content-Based Text) texts. Through a dual-attention mechanism combining hierarchical input encoding and dynamic positional embedding, it effectively addresses the long-distance dependencies and semantic ambiguity issues in Chinese CTI texts. Experiments show that the framework achieves 92% accuracy on the CDTier dataset. By enhancing entity position modeling with structured input representation, fusing global and local features through a dual-attention mechanism, and capturing cross-sentence dependencies with dynamic positional encoding, it provides a reliable foundation for constructing Chinese CTI knowledge graphs.

[0080] In summary, this invention: 1. Rotational Position Encoding Optimization Mechanism for Chinese CTI To address the boundary conflicts between nested entities and cross-linguistic terms in Chinese cybersecurity threat intelligence (CTI), this invention innovatively improves the application of Rotated Position Encoding (RoPE). By employing a geometric space mapping strategy, it models character-level relative positional relationships in the complex domain, effectively resolving semantic ambiguity in mixed Chinese and English terms. A dynamic position truncation technique is used to balance long-range dependencies and computational efficiency, and a hybrid positional representation mechanism is designed, preserving absolute positional references in the lower-level network while transitioning to relative position dominance in the deeper layers. This technology significantly improves the recognition accuracy of long-span entities (such as APT attack chain descriptions) in Chinese CTI text.

[0081] 2. Type-sensitive global pointer entity identification architecture Breaking through the limitations of traditional sequence labeling models, a joint decoding framework based on a two-dimensional grid space is proposed. By transforming entity recognition into Cartesian product matching of head and tail pointers, synchronous detection of nested entities is achieved. An innovative strip mask constraint matrix is ​​designed to retain only legal entity span combinations conforming to Chinese grammar rules, and a dynamic type threshold mechanism is introduced to adaptively adjust the judgment boundary based on the entity length distribution. This architecture, combined with a type-sensitive projection matrix, eliminates multi-class semantic interference, significantly improving the recognition accuracy of nested structures such as "attack organization - attack event".

[0082] 3. Optimization of Zero-Boundary Log-Sum-Exp Loss Function To alleviate the long-tail distribution problem of Chinese CTI entity categories, an improved ZLPR loss function is proposed. Zero-boundary constraints automatically suppress the gradient contribution of easily separable samples, focusing on optimizing difficult samples with ambiguous boundaries. A frequency-domain adaptive weighting strategy is designed to dynamically adjust the loss weights based on the entity's occurrence frequency, preventing low-frequency entities from being dominated by high-frequency categories. Combined with a dynamic difficult sample mining mechanism, a cosine annealing strategy is used to progressively optimize the decision boundary. This technique effectively balances the robustness of identifying entities such as "supply chain attacks" (low frequency) and "malicious software" (high frequency).

[0083] 4. Dual Attention Relationship Extraction Framework To address the cross-sentence dependency characteristics of Chinese CTI text, a hierarchical input structure and a dual-attention fusion mechanism are constructed. Entity positions are explicitly marked using special delimiters to enhance the entity pair's ability to locate within long texts. A two-stage attention mechanism for entity perception is designed: global attention captures document-level threat behavior patterns, while local attention focuses on entity interaction features. A dynamic relative position encoding is innovatively introduced, mapping entity spacing to exponentially decaying vectors to enhance the modeling ability of cross-paragraph relationships such as "attack vector → scope of influence".

[0084] This invention addresses three core challenges in Chinese cybersecurity threat intelligence (CTI) parsing: difficulty in identifying nested entities, blurred boundaries between cross-language terms, and imbalanced distribution of long-tail entities. It proposes innovative solutions. Traditional methods, relying on sequence labeling architectures and static loss functions, struggle to effectively handle the unique "APT organization-attack event" nested structure and low-frequency long entities such as "supply chain attacks" characteristic of Chinese CTI. By introducing a geometric mapping mechanism with rotational position encoding, the accuracy of nested entity identification is significantly improved. A two-dimensional joint decoding strategy using a global pointer network is designed to simultaneously capture the start and end positions of long-span entities. Furthermore, a dynamically optimized multi-label loss function adaptively balances the recognition weights of high-frequency threat terms (such as "malware") and low-frequency attack patterns (such as "zero-day vulnerabilities"). Ultimately, robust entity relationship extraction is achieved on the Chinese CTI dataset, providing crucial technical support for the automated construction of highly reliable threat knowledge graphs.

[0085] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for entity relationship recognition based on rotational position encoding and a global pointer network, characterized in that, include: Entity extraction: The input Chinese threat intelligence text is encoded using a pre-trained language model that incorporates rotational position encoding to generate a contextual semantic representation with enhanced positional awareness. Based on the contextual semantic representation, all possible entity spans and their types are decoded in parallel in a two-dimensional grid space through a global pointer network to generate an entity set. The global pointer network uses a rotational position encoding-enhanced attention mechanism to jointly predict entity boundaries and types. Relation extraction: For target entity pairs in the entity set, a structured input sequence containing entity location information is constructed; the structured input sequence is processed using a dual attention encoding mechanism, which includes a global attention stage for capturing global context semantics and an entity perception attention stage for focusing on the interaction features of the target entity pairs; based on the output of the dual attention encoding mechanism, the relationship type between the target entity pairs is determined by the relationship classification module.

2. The entity relationship recognition method based on rotational position encoding and global pointer network according to claim 1, characterized in that, The pre-trained language model that incorporates rotational position encoding is a RoBERTa model that uses rotational position encoding; The injection process of the rotational position encoding achieves a position-aware attention mechanism through rotational transformations in geometric space, including: For the position in the sequence is query vector and position as key vector Through rotation matrix and Perform the transformation: In the formula, in and These are the transformed query vector and key vector, respectively, with rotation angles. , i∈[0,d / 2−1], where d is the dimension of the hidden layer. This represents the direct sum of a block diagonal matrix; The transformed attention score is: in Let m be the attention score of position m to position n, and T denote the matrix transpose.

3. The entity relationship recognition method based on rotational position encoding and global pointer network according to claim 2, characterized in that, The injection process encoded at the rotational position: It also includes a sub-word position compensation mechanism for Chinese text: For the sub-word units generated by the word segmenter, the following position-weighted strategy is adopted: in This represents the set of consecutive sub-word units from the i-th sub-word unit to the j-th sub-word unit. This represents the positional encoding of the k-th sub-word unit. This represents the positional encoding from the i-th sub-word unit to the j-th sub-word unit; It also includes a position truncation mechanism: Set maximum relative distance For any two positions m and n in the sequence, if their relative position difference is... The rotation angle is then adjusted using linear interpolation. in This is the adjusted rotation angle; A hybrid position representation strategy is adopted: in This represents the hidden state at position i in layer l. RoPE(⋅) indicates that rotational position encoding is performed. For learnable parameters, This is the learnable absolute position code for the i-th position.

4. The entity relationship recognition method based on rotational position encoding and global pointer network according to claim 1, characterized in that, In the encoding stage of entity extraction, a hierarchical feature extraction strategy is adopted to construct a multi-level semantic representation. This strategy focuses on positional and semantic information at different granularities through different network layers of the Transformer architecture. Shallow networks focus on local lexical patterns and capture character-level and word-level syntactic combination patterns through an attention mechanism enhanced by rotational positional encoding. The mid-layer network expands the receptive field by introducing a dynamic sparse attention mechanism to model semantic dependencies across clauses; Deep networks leverage the long-range association modeling capabilities of rotational position encoding to integrate document-level semantics, enabling global modeling of complex attack chains.

5. The entity relationship recognition method based on rotational position encoding and global pointer network according to claim 1, characterized in that, The operations of the global pointer network include: The hidden state output by the encoder is projected through a type-sensitive projection matrix to generate query vectors and key vectors; Apply a rotational position encoding transformation to the query vector and the key vector, for positions where query vector and position as key vector The transformed query vector is obtained. and key vector ,based on and Obtain the rotation-enhanced attention score: Where Re(.) denotes taking the real part of the complex number, The attention score represents the presence of an entity with a starting position of i and an ending position of j, and T represents the matrix transpose. Multi-granular semantic features are captured through a parallelized multi-head design, and the score of head h is correlated with the scalar weight of type t. The output is integrated into an entity matrix E through a gating fusion mechanism.

6. The entity relationship recognition method based on rotational position encoding and global pointer network according to claim 5, characterized in that, The entity extraction process also includes determining the entity boundaries of the entity matrix, incorporating multi-level positional constraints during the determination process. Using the triangular mask matrix Apply rigid geometric constraints, where: in The maximum span threshold; Incorporate a learnable relative position bias term into the attention score. The length distribution characteristics of Chinese entities are encoded using learnable parameters: in span length The bias parameters, To pass Adjusted attention score; In the entity existence determination phase, define the entity existence determination function: In the formula, in This represents the existence determination function for an entity of type n, starting at position i and ending at position j. This represents the attention score for an entity of type n, with a starting position of i and an ending position of j, adjusted by the positional bias term. For activation function, For entities of type n, the adaptive dynamic threshold is... Let n be the average length of an entity of type n. It's a hyperparameter.

7. The entity relationship recognition method based on rotational position encoding and global pointer network according to claim 1, characterized in that, In the entity extraction, an improved ZLPR loss function is used for model training: in Represents the ZLPR loss function. and These represent the sets of positive and negative samples, respectively. Indicates the score of the positive sample; By introducing prior knowledge of entity distribution, a frequency-domain adaptive loss adjustment mechanism is constructed to... Adjustments will be made: in This represents the number of times the nth type of entity appears in the training set. T is the smoothing factor, representing the total number of entity types; Automatically identify Top-K difficult negative samples in each round of training. : in The threshold for positive sample scores. This represents the true label corresponding to the candidate span (i,j). This indicates that there is no entity with boundary (i,j) in the training data; Adjustments were made using a cosine annealing strategy: in This represents the training iteration number t corresponding to N represents the total number of training iterations; In the early stages of training, set up The focus is on correcting erroneous samples; as training progresses, the error rate is gradually reduced to [a lower threshold]. To capture difficult samples with blurred boundaries.

8. The entity relationship recognition method based on rotational position encoding and global pointer network according to claim 1, characterized in that, In the relation extraction, the encoding process of the structured input sequence is as follows: in Represents a structured input sequence. For the original text, For the target entity, This indicates a splicing operation. It is a classification label. It is a separator marker; To model the relative positions between entities, calculate the relative positional difference between entity pairs in the text. ,in Indicates the starting position of the entity in the text, and Mapped to a 128-dimensional vector space via a sine function; A learnable type embedding matrix is ​​established to map entity types to feature vectors, which are then concatenated with text encodings. Finally, the concatenated features are fused into the hidden representation of BERT through a linear transformation.

9. The entity relationship recognition method based on rotational position encoding and global pointer network according to claim 1, characterized in that, In the dual-attention encoding mechanism, the global attention stage uses standard multi-head self-attention to capture document-level semantic associations, and the entity-aware attention stage strengthens attention to semantic cues around entities by injecting relative positional biases. The relative position offset Calculate as follows: in h and Indicates the combination of location distances. Indicates the starting position of the entity in the text. For learnable parameters, Let be the dimension of the k-th attention head.

10. The entity relationship recognition method based on rotational position encoding and global pointer network according to claim 9, characterized in that, The relation classification module, based on the contextual representation of the deep semantic coding layer, constructs a multi-granularity feature fusion and hierarchical decision architecture, specifically implementing the following steps: Multi-granularity feature fusion: generating fused feature vectors This vector is obtained by concatenating the following three types of features: Global context features: Extract the [CLS] marker vector at the first position of the encoder. As a global context feature; Entity interaction features: the hidden state vectors of two target entities E1 and E2 and After average pooling, element-wise multiplication is performed to obtain entity interaction features. : in Indicates the starting position of the entity in the text. Representing entities The hidden state vector of the token corresponding to the starting position in the encoder's 12th layer; Structured location features: Encoding the relative distance of entities Embedded with entity type By splicing the data, a structured positional representation is obtained; The multi-granularity feature fusion is represented as follows: in This represents a vector concatenation operation; Hierarchical decision architecture design: for the fusion features A gradual process is employed, including: Feature normalization: for By implementing standardization and nonlinear activation, we obtain : in For a dimension reduction matrix, Representation layer normalization, Represents the activation function of the Gaussian error linear unit; Nonlinear projection: Constructing a two-layer residual network to extract discriminative features : in These are projection parameters; Category-aware decoding: Employs a parameter isolation strategy to achieve adaptive classification decisions based on relation types, generating the final relation type probability distribution. : in An embedding matrix is ​​created for each relation type, with each row corresponding to a decision plane for a specific relation. The classification boundary is dynamically adjusted based on category frequency. This represents the total number of relation types.

Images