A method to distinguish the difficulty of named entity recognition in network security data

Abstract

The present invention discloses a method for distinguishing the difficulty of named entity recognition of network security data, and belongs to the field of network data security technology. It solves the problem that the traditional named entity recognition method in the prior art is difficult to effectively distinguish the difficulty of data; the present invention includes the following steps: S1. Constructing a rule-based discriminator, inputting a data set, evaluating the complexity of entities in sentences through difficulty indicators, obtaining a total difficulty score, classifying the data in the data set, and obtaining a classification result; S2. Constructing a pre-trained model, setting a data discriminator based on the pre-trained model to verify the classification result, and obtaining the final named entity recognition classification result. The present invention effectively improves the overall accuracy of identifying challenging instances in a data set, can distinguish data difficulty, enhances the robustness of data classification results, and reduces the specific deviations of the pre-trained model, and can be applied to network security data processing.

Description

Technical Field

[0001] The invention relates to a method for distinguishing the difficulty of identifying network security data named entities, and belongs to the technical field of network data security. Background Art

[0002] Named entity recognition (NER) is an important task in natural language processing (NLP), which aims to identify entities with specific meanings from text, such as names of people, places, and organizations. Its application in the field of network security is crucial.

[0003] Due to the complexity of the data, the difficulty of identifying named entities may vary significantly. In the prior art, the named entity recognition methods are analyzed as follows: (1) Rule-based methods. Early NER systems mostly relied on manually written rules and dictionaries. The rules were based on linguistic knowledge, such as the part of speech and context of words. However, the disadvantages were that the cost of writing and maintaining rules was high, and they were difficult to adapt to new entities and language changes. (2) Statistical learning methods. For example, hidden Markov models (HMM) and conditional random fields (CRF) automatically learn entity recognition patterns through training data. They do not rely on predefined rules and have better adaptability and generalization capabilities, but their recognition efficiency is low. (3) Transformer architectures in deep learning methods, including BERT, GPT, RoBERTa and other models, capture contextual relationships through a self-attention mechanism, which improves the accuracy of NER. However, the constructed models are large in scale and have low recognition efficiency.

[0004] In summary, there is a need for a named entity recognition method that can effectively distinguish the difficulty of data and improve the accuracy and efficiency of network security data processing. Summary of the invention

[0005] A brief overview of the present invention is provided below in order to provide a basic understanding of certain aspects of the present invention. It should be understood that this overview is not an exhaustive overview of the present invention. It is not intended to identify key or important parts of the present invention, nor is it intended to limit the scope of the present invention. Its purpose is merely to present certain concepts in a simplified form as a prelude to a more detailed description discussed later.

[0006] In view of this, in order to solve the problem that the traditional named entity recognition method in the prior art is difficult to effectively distinguish the difficulty of data, the present invention provides a method for distinguishing the difficulty of named entity recognition of network security data.

[0007] The technical solution is as follows: A method for distinguishing the difficulty of identifying named entities in network security data, comprising the following steps:

[0008] S1. Build a rule-based discriminator, input the data set, evaluate the complexity of entities in the sentence through the difficulty index, obtain the total difficulty score, classify the data in the data set, and obtain the classification result;

[0009] S2. Build a pre-trained model, set a data discriminator based on the pre-trained model to verify the classification results, and obtain the final named entity recognition classification results.

[0010] Furthermore, in said S1, the difficulty index includes entity length difficulty, vocabulary rarity and entity type diversity;

[0011] The entity length difficulty is evaluated by calculating the total length of all entities and subtracting the number of entities to obtain the entity length complexity score D. length ;

[0012] Entity length complexity score D length It is expressed as:

[0013]

[0014] Where N is the number of entities, L i is the length of the i-th entity;

[0015] Perform vocabulary rarity evaluation, and judge each word in the entity whose frequency of occurrence is lower than the specified threshold as a rare word, and obtain the entity word rarity score D rarity ;

[0016] Entity word rarity score D rarity It is expressed as:

[0017]

[0018] Among them, R(w ij ) is the rarity of the jth word in the i-th entity, f(w ij ) is the word w ij The frequency of occurrence of , F is the total number of occurrences of all words in the corpus;

[0019] Conduct entity type diversity difficulty assessment, construct a type set to summarize the types of all entities in the sentence, and obtain the entity diversity difficulty score based on the size of the type set, that is, the number of different entity types;

[0020] Entity Diversity Difficulty Rating D diversity It is expressed as:

[0021] D diversity =|{T i |i=1,2,...,N}|

[0022] Among them, Ti is the type of the i-th entity, {T i} is the set of all entity types, || represents the size of the set;

[0023] Score D for entity length complexity length , entity word rarity score D rarity and entity diversity difficulty score D diversity The total difficulty score is obtained by summing up, and the total difficulty score is compared with the preset difficulty threshold. The classification results include difficult data, simple data and non-entity data. Non-entity data refers to sentences in which there is no entity in the data set. If the total difficulty score is greater than the preset difficulty threshold, the corresponding entity sentence is classified as difficult data. If the total difficulty score is greater than the preset difficulty threshold, the corresponding entity sentence is classified as simple data. If the non-entity data is adjacent to difficult data or simple data, the non-entity data adjacent to the difficult data is merged into difficult data, and the non-entity data adjacent to the simple data is merged into simple data. If the non-entity data is not adjacent to difficult data or simple data, it is discarded.

[0024] Furthermore, the step S2 specifically includes the following steps:

[0025] S21. Given an input sequence, generate an original annotation sequence according to the annotation rules;

[0026] S22. Use the fine-tuned BERT model, ALBERT model, and MobileBERT model to build a pre-training model, use the input sequence as the input of the pre-training model, and obtain the label score of the i-th word;

[0027] S23. According to the label score of the i-th word, the pre-trained model is trained to obtain the maximum conditional probability;

[0028] S24. According to the maximized conditional probability, a cross entropy loss function is used during the pre-training model training process to measure the difference between the predicted label corresponding to the maximized conditional probability and the actual label of the original annotation sequence, the classification result is verified, and the final named entity recognition classification result is obtained by integration;

[0029] In S21, given an input sequence X, X=(x1, x2, ……, x n ), x i For the i-th word, i=1,2,…,n, generate the original tag sequence Y, Y=(y1,y2,…y n ), y i For word x i The actual label of

[0030] In S22, the pre-trained model G consists of an input layer, a multi-layer self-attention mechanism, a linear layer and an output layer. The input sequence X is input into the input layer, and three embeddings are automatically generated, namely, word embedding e i , position embedding p i and segment embedding d i , embed the word into i , position embedding p i and segment embedding d i Add together to get the final input h of the word i ;

[0031] The final input of the word h i It is expressed as:

[0032] h i =e i +p i +d i

[0033] The multi-layer self-attention mechanism has a total of L layers, according to the final input h of the word i , the output of each layer of the multi-layer self-attention mechanism is expressed as Where, l = 1, 2, ..., L;

[0034] The self-attention mechanism Attention(Q, K, V) in each layer is expressed as:

[0035]

[0036] Among them, Q is the query matrix, K is the key matrix, V is the value matrix, and W Q is the learnable weight matrix of the query matrix, W K is the learnable weight matrix of the key matrix, W V is the learnable weight matrix of the value matrix, d k is the scaling factor;

[0037] At the end of the word h i After passing through the multi-layer self-attention mechanism, the output h of the Lth layer of the multi-layer self-attention mechanism is converted through a linear layer. i (L) Mapped to the label space, we get the label score S of the i-th word i ;

[0038] The label score S of the i-th word i It is expressed as:

[0039]

[0040] Where W is the first parameter to be learned, and b is the second parameter to be learned;

[0041] In S23, the maximization conditional probability of the training process of the pre-training model G is expressed as:

[0042]

[0043] Among them, P(y i =k|X) represents the probability that the i-th word is predicted as label k given the input sequence X, and s i,k represents the score of the i-th word corresponding to label k calculated by the linear layer, and exp is an exponential function;

[0044] The i-th word x i The score s corresponding to label k i，k It is expressed as:

[0045]

[0046] Among them, b k is the bias term of the linear layer, which corresponds to the bias of label k;

[0047] In S24, the cross entropy loss of the i-th word It is expressed as:

[0048]

[0049] in, is the true label of the i-th word;

[0050] The total loss is obtained by summing the cross entropy losses of all words

[0051] Total loss It is expressed as:

[0052]

[0053] The total loss Minimize and get the final NER model;

[0054] Use the final NER model to generate the final annotation sequence Final annotation sequence It includes three new annotation sequences and special marks. The elements corresponding to the three new annotation sequences are respectively recorded as the first element a i , the second element b i and the third element c i , check the consistency of the above elements and special marks to get the final annotation sequence Each predicted label in

[0055] The final annotation sequence Each predicted label in It is expressed as:

[0056]

[0057] Using the final annotation sequence Compared with the original annotation sequence Y, if the predicted label generated by the final NER model and the actual label y i If they are inconsistent, the corresponding word x i It is recorded as difficult data, otherwise it is recorded as simple data.

[0058] The beneficial effects of the present invention are as follows: the present invention realizes accurate prediction of entity labels by setting a linear layer, and provides a concise and effective method for named entity recognition using a large-scale language model; the present invention utilizes the complementary advantages of different NER models, namely the fine-tuned BERT model, the ALBERT model and the MobileBERT model, to enhance the robustness of data classification results, reduce the specific bias of the pre-trained model, and thus improve the overall accuracy of identifying truly challenging instances in the data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0060] Figure 1 A flowchart of a method for distinguishing the difficulty of named entity recognition of network security data is provided;

[0061] Figure 2 It is a schematic diagram of the structure of the rule-based discriminator;

[0062] Figure 3 Schematic diagram of the structure of the data discriminator based on the pre-trained model. DETAILED DESCRIPTION

[0063] In order to make the technical solutions and advantages of the embodiments of the present invention more clearly understood, the exemplary embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than an exhaustive list of all the embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0064] refer to Figure 1-Figure 3 The present embodiment is described in detail, a method for distinguishing the difficulty of named entity recognition of network security data, specifically comprising the following steps:

[0065] S1. Build a rule-based discriminator, input the data set, evaluate the complexity of entities in the sentence through the difficulty index, obtain the total difficulty score, classify the data in the data set, and obtain the classification result;

[0066] S2. Build a pre-trained model, set a data discriminator based on the pre-trained model to verify the classification results, and obtain the final named entity recognition classification results.

[0067] Furthermore, in said S1, the difficulty index includes entity length difficulty, vocabulary rarity and entity type diversity;

[0068] The entity length difficulty is evaluated by calculating the total length of all entities and subtracting the number of entities to obtain the entity length complexity score D. length ;

[0069] Entity length complexity score D length It is expressed as:

[0070]

[0071] Among them, N is the number of entities in a sentence, L i is the length of the i-th entity (including the number of words);

[0072] Perform vocabulary rarity evaluation, and judge each word in the entity whose frequency of occurrence is lower than the specified threshold as a rare word, and obtain the entity word rarity score D rarity ;

[0073] Entity word rarity score D rarity It is expressed as:

[0074]

[0075] Among them, R(w ij ) is the rarity of the jth word in the i-th entity, f(w ij ) is the word w ij The frequency of occurrence of a word (i.e., the number of times the word appears in the corpus), and F is the total number of occurrences of all words in the corpus (i.e., the number of occurrences of all words is counted);

[0076] Conduct entity type diversity difficulty assessment, construct a type set to summarize the types of all entities in the sentence, and obtain the entity diversity difficulty score based on the size of the type set, that is, the number of different entity types;

[0077] Entity Diversity Difficulty Rating D diversity It is expressed as:

[0078] D diversity =|{Ti |i=1,2,...,N}|

[0079] Among them, T i is the type of the i-th entity, {T i} is the set of all entity types, || represents the size of the set (the number of different entity types);

[0080] Score D for entity length complexity length , entity word rarity score D rarity and entity diversity difficulty score D diversity The total difficulty score is obtained by summing up, and the total difficulty score is compared with the preset difficulty threshold. The classification results include difficult data, simple data and non-entity data. Non-entity data refers to sentences in which there is no entity in the data set. If the total difficulty score is greater than the preset difficulty threshold, the corresponding entity sentence is classified as difficult data. If the total difficulty score is greater than the preset difficulty threshold, the corresponding entity sentence is classified as simple data. If the non-entity data is adjacent to the difficult data or the simple data, the non-entity data adjacent to the difficult data is merged into the difficult data, and the non-entity data adjacent to the simple data is merged into the simple data. If the non-entity data is not adjacent to the difficult data or the simple data, it is discarded;

[0081] Specifically, refer to Figure 2 The rule-based difficult and easy data discriminator evaluates the complexity of entities in sentences by calculating three different difficulty indicators, where Dataset is the dataset, Entity Length Complexity Analysis is the entity length difficulty, Rare Vocabulary Difficulty Assessment is the vocabulary rarity difficulty, Entity Type Diversity Examination is the entity type diversity difficulty, Entity Recognition Difficulty Discriminator is the rule-based discriminator, Non-Entity Data is non-entity data, Hard Data is difficult data, and Easy Data is easy data. The following details the difficulty indicators:

[0082] (1) Entity length difficulty: Consider the number of words contained in the entity. Entities containing more words may be more challenging to identify due to their complex contextual relationships or structural changes. Entity length difficulty prioritizes the case where a single entity contains multiple words, rather than simply counting the total number of entities.

[0083] (2) Lexical rarity difficulty: Focus on the frequency of occurrence of rare words in an entity. Entities containing rare words may be more difficult to accurately identify due to insufficient contextual information or the model’s unfamiliarity with these words. At the same time, in order to determine lexical rarity, it is necessary to check the frequency of occurrence of each word in the entity.

[0084] (3) Difficulty of entity type diversity: Evaluate the diversity of entity types in a sentence. Entity type refers to the classification after the prefix “B-” or “I-” in the entity label, such as “Person” in “B-Person”. Sentences containing multiple entity types may indicate that their structure is more complex, thus increasing the difficulty of entity recognition.

[0085] Furthermore, the step S2 specifically includes the following steps:

[0086] S21. Given an input sequence, generate an original annotation sequence according to the annotation rules;

[0087] S22. Use the fine-tuned BERT model, ALBERT model, and MobileBERT model to build a pre-training model, use the input sequence as the input of the pre-training model, and obtain the label score of the i-th word;

[0088] S23. According to the label score of the i-th word, the pre-trained model is trained to obtain the maximum conditional probability;

[0089] S24. According to the maximized conditional probability, a cross entropy loss function is used during the pre-training model training process to measure the difference between the predicted label corresponding to the maximized conditional probability and the actual label of the original annotation sequence, the classification result is verified, and the final named entity recognition classification result is obtained by integration;

[0090] In S21, given an input sequence X, X=(x1, x2, ……, x n ), x i For the i-th word, i=1,2,…,n, generate the original tag sequence Y, Y=(y1,y2,…y n ), y i For word x i The actual label of

[0091] In S22, the pre-trained model G consists of an input layer, a multi-layer self-attention mechanism, a linear layer and an output layer. The input sequence X is input into the input layer, and three embeddings are automatically generated, namely, word embedding e i , position embedding p i and segment embedding d i , embed the word into i , position embedding p i and segment embedding d iAdd together to get the final input h of the word i ;

[0092] The final input of the word h i It is expressed as:

[0093] h i =e i +p i +d i

[0094] The multi-layer self-attention mechanism has a total of L layers, according to the final input h of the word i , the output of each layer of the multi-layer self-attention mechanism is expressed as Where, l = 1, 2, ..., L;

[0095] The self-attention mechanism Attention(Q, K, V) in each layer is expressed as:

[0096]

[0097] Among them, Q is the query matrix, K is the key matrix, and V is the value matrix. The query matrix Q (Query) is used to query which parts of the input sequence need to be paid attention to. It can be regarded as a way to find relevant information in the current step. Each query vector of the query matrix Q corresponds to an element in the input sequence, indicating the question raised by the element when looking for the relationship with other elements. The key matrix K (Key) contains the information label related to each element of the input sequence. Each key vector corresponds to an element in the input sequence and represents the characteristics of the element. When calculating the attention weight, the query vector will be dot-producted with the key vector to measure the similarity or correlation between the two. The value matrix v (Value) contains the actual value or information of the input sequence. Each value vector of the value matrix v corresponds to an element in the input sequence, indicating the actual information content of the element. After the attention weight is calculated, it is applied to the value matrix V to obtain the attention-weighted output, W Q is the learnable weight matrix of the query matrix, W K is the learnable weight matrix of the key matrix, W V is the learnable weight matrix of the value matrix, d k is the scaling factor;

[0098] At the end of the word h i After passing through the multi-layer self-attention mechanism, the output h of the Lth layer of the multi-layer self-attention mechanism is converted through a linear layer. i (L) Mapped to the label space, we get the label score S of the i-th word i ;

[0099] The label score S of the i-th word iIt is expressed as:

[0100]

[0101] Where W is the first parameter to be learned, and b is the second parameter to be learned;

[0102] In S23, the maximization conditional probability of the training process of the pre-training model G is expressed as:

[0103]

[0104] Among them, P(y i =k|X) represents the probability that the i-th word is predicted as label k given the input sequence X, and s i，k represents the score of the i-th word corresponding to label k calculated by the linear layer, and exp is an exponential function;

[0105] The i-th word x i The score s corresponding to label k i，k It is expressed as:

[0106]

[0107] Among them, b k is the bias term of the linear layer, which corresponds to the bias of label k;

[0108] In S24, the cross entropy loss of the i-th word It is expressed as:

[0109]

[0110] in, is the true label of the i-th word;

[0111] The total loss is obtained by summing the cross entropy losses of all words

[0112] Total loss It is expressed as:

[0113]

[0114] The total loss Minimize and get the final NER model;

[0115] Use the final NER model to generate the final annotation sequence Final annotation sequence It includes three new annotation sequences and special marks. The elements corresponding to the three new annotation sequences are respectively recorded as the first element a i , the second element b i and the third element ci , check the consistency of the above elements and special marks to get the final annotation sequence Each predicted label in

[0116] The final annotation sequence Each predicted label in It is expressed as:

[0117]

[0118] Using the final annotation sequence Compared with the original annotation sequence Y, if the predicted label generated by the final NER model and the actual label y i Inconsistency (special_marker appears to represent the original label y i The comparison is necessarily inconsistent), then the corresponding word x i Record as difficult data, otherwise as simple data;

[0119] Specifically, refer to Figure 3 , where Results is the classification result, Consistency Check is the consistency check, Label+Data is the label and data, Linear layer is the linear layer, Embeddings is the embedding, ENCODER is the decoder, and Data is the data;

[0120] After the initial difficulty index evaluation, in order to ensure the accuracy of the classification results, three NER models of different scales were trained to form a data discriminator based on the pre-trained model to further filter the difficult data identified in the previous step. The three NER models of different scales selected the BERT model, ALBERT model and MobileBERT model for analysis. The model introduction is as follows:

[0121] (1) BERT (Bidirectional Encoder Representations from Transformers) model: Developed by Google, it is a breakthrough approach in the field of natural language processing. It introduces the concept of pre-training a deep bidirectional Transformer on a large text corpus and then fine-tuning it on a specific downstream task. The architecture of the BERT model allows for a large number of parameters. For example, BERT-Base has 110 million parameters, making it a large and powerful model that can achieve state-of-the-art results on a wide range of NLP (natural language processing) tasks.

[0122] (2) MobileBERT model: The MobileBERT model is a compact variant of the BERT model, optimized for mobile devices with limited processing power and storage. It aims to maintain a similar performance level to the BERT model while significantly reducing model size and latency, making it an ideal choice for device-side applications. The training of the MobileBERT model involves knowledge transfer from a specially designed teacher BERT model with an expanded number of layers and hidden units. The unique training method combines bottleneck layers and layer-by-layer optimization techniques to achieve a balance between compactness and performance. It can provide a smaller model size than the standard BERT model with minimal functional loss.

[0123] (3) ALBERT (ALite BERT) model: The ALBERT model is a variant of the BERT model, which aims to reduce the model size and speed up training without affecting performance. The present invention introduces two key modifications, namely, factorized embedding parameterization and cross-layer parameter sharing, to achieve the goal. The former reduces the size of the embedding layer by factorizing it into smaller matrices, while the latter involves sharing parameters across all layers of the model, significantly reducing the number of parameters of the ALBERT model. The basic version has only 12 million parameters, a fraction of the size of the BERT model, and allows faster training time. Despite the reduction in size, the ALBERT model still maintains competitive performance on various NLP benchmarks, demonstrating the effectiveness of its design optimization in achieving model training and deployment efficiency and scalability.

[0124] After extracting embeddings from BERT, ALBERT, and MobileBERT models, the present invention applies a linear layer for the named entity recognition task. The linear layer acts as a classifier to map detailed embeddings to predefined NER categories. After training on a dataset of annotated named entity recognition results, the language understanding of the NER model is refined.

[0125] After obtaining the three fine-tuned NER models, the difficult data identified by the aforementioned rule-based classifier are further verified. The verification process involves using three different models for entity recognition and checking the consistency of the entity labeling results. Specifically, if two or more models make consistent errors when labeling an instance, the instance is ultimately classified as difficult data. Conversely, if all three fine-tuned NER models successfully identify an instance, it is reclassified as easy data.

[0126] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments may be envisioned within the scope of the invention thus described. In addition, it should be noted that the language used in this specification is selected primarily for readability and teaching purposes, rather than for explaining or defining the subject matter of the present invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is illustrative, not restrictive, with respect to the scope of the present invention, which is defined by the appended claims.

Claims

1. A method for distinguishing the difficulty of named entity recognition of network security data, characterized in that: The following steps are involved: S1. Build a rule-based discriminator, input the data set, evaluate the complexity of entities in the sentence through the difficulty index, obtain the total difficulty score, classify the data in the data set, and obtain the classification result; S2. Build a pre-trained model, set a data discriminator based on the pre-trained model to verify the classification results, and obtain the final named entity recognition classification results; The S2 specifically includes the following steps: S21. Given an input sequence, generate an original annotation sequence according to the annotation rules; S22. Use the fine-tuned BERT model, ALBERT model, and MobileBERT model to build a pre-training model, use the input sequence as the input of the pre-training model, and obtain the label score of the i-th word; S23. According to the label score of the i-th word, the pre-trained model is trained to obtain the maximum conditional probability; S24. According to the maximized conditional probability, a cross entropy loss function is used during the pre-training model training process to measure the difference between the predicted label corresponding to the maximized conditional probability and the actual label of the original annotation sequence, the classification result is verified, and the final named entity recognition classification result is obtained by integration; In S21, given an input sequence X, X=(x1, x2, . . . , x n ), x i For the i-th word, i=1,2,...,n, generate the original tag sequence Y, Y=(y1,y2,...y n ), y i For word x i The actual label of In S22, the pre-trained model G consists of an input layer, a multi-layer self-attention mechanism, a linear layer and an output layer. The input sequence X is input into the input layer, and three embeddings are automatically generated, namely, word embedding e i , position embedding p i and segment embedding d i , embed the word into i , position embedding p i and segment embedding d i Add together to get the final input h of the word i ; The final input of the word h i It is expressed as: h i =e i +p i +d i The multi-layer self-attention mechanism has a total of L layers, according to the final input h of the word i , the output of each layer of the multi-layer self-attention mechanism is expressed as Where, l = 1, 2, ..., L; The self-attention mechanism Attention(Q, K, V) in each layer is expressed as: Where Q is the query matrix, K is the key matrix, V is the value matrix, WQ is the learnable weight matrix of the query matrix, and W K is the learnable weight matrix of the key matrix, W V is the learnable weight matrix of the value matrix, d k is the scaling factor; At the end of the word h i After passing through the multi-layer self-attention mechanism, the output h of the Lth layer of the multi-layer self-attention mechanism is converted through a linear layer. i (L) Mapped to the label space, we get the label score S of the i-th word i ; The label score S of the i-th word i It is expressed as: Where W is the first parameter to be learned, and b is the second parameter to be learned; In S23, the maximization conditional probability of the training process of the pre-training model G is expressed as: Among them, P(y i =k|X) represents the probability that the i-th word is predicted as label k given the input sequence X, and s i，k represents the score of the i-th word corresponding to label k calculated by the linear layer, and exp is an exponential function; The i-th word x i The score s corresponding to label k i,k It is expressed as: Among them, b k is the bias term of the linear layer, which corresponds to the bias of label k; In S24, the cross entropy loss of the i-th word It is expressed as: in, is the true label of the i-th word; The total loss is obtained by summing the cross entropy losses of all words Total loss It is expressed as: The total loss Minimize and get the final NER model; Use the final NER model to generate the final annotation sequence Final annotation sequence It includes three new annotation sequences and special marks. The elements corresponding to the three new annotation sequences are respectively recorded as the first element a i , the second element b i and the third element c i , check the consistency of the above elements and special marks to get the final annotation sequence Each predicted label in The final annotation sequence Each predicted label in It is expressed as: Using the final annotation sequence Compared with the original annotation sequence Y, if the predicted label generated by the final NER model and the actual label y i If they are inconsistent, the corresponding word x i It is recorded as difficult data, otherwise it is recorded as simple data.

2. According to claim 1, a method for distinguishing the difficulty of named entity recognition of network security data is characterized in that: In S1, the difficulty indicators include entity length difficulty, vocabulary rarity, and entity type diversity; The entity length difficulty is evaluated by calculating the total length of all entities and subtracting the number of entities to obtain the entity length complexity score D. length ; Entity length complexity score D length It is expressed as: Where N is the number of entities, L i is the length of the i-th entity Perform vocabulary rarity evaluation, and judge each word in the entity whose frequency of occurrence is lower than the specified threshold as a rare word, and obtain the entity word rarity score D rarity ; Entity word rarity score D rarity It is expressed as: Among them, R(w ij ) is the rarity of the jth word in the i-th entity, f(w ij ) is the word w ij The frequency of occurrence of , F is the total number of occurrences of all words in the corpus; Conduct entity type diversity difficulty assessment, construct a type set to summarize the types of all entities in the sentence, and obtain the entity diversity difficulty score based on the size of the type set, that is, the number of different entity types; Entity Diversity Difficulty Rating D diversity It is expressed as: D diversity =|{T i |i=1,2,...,N}| Among them, T i is the type of the ith entity, {T i } is the set of all entity types, || represents the size of the set; Score D for entity length complexity length , entity word rarity score D rarity and entity diversity difficulty score D diversity The total difficulty score is obtained by summing up, and the total difficulty score is compared with the preset difficulty threshold. The classification results include difficult data, simple data and non-entity data. Non-entity data refers to sentences in which there is no entity in the data set. If the total difficulty score is greater than the preset difficulty threshold, the corresponding entity sentence is classified as difficult data. If the total difficulty score is greater than the preset difficulty threshold, the corresponding entity sentence is classified as simple data. If the non-entity data is adjacent to difficult data or simple data, the non-entity data adjacent to the difficult data is merged into difficult data, and the non-entity data adjacent to the simple data is merged into simple data. If the non-entity data is not adjacent to difficult data or simple data, it is discarded.

Images