A CPE identification system based on AI pre-trained large model empowerment

Abstract

The present invention discloses a CPE identification system based on AI pre-training large model empowerment, which relates to the technical field of software security vulnerability analysis, including a bidirectional Transformer module, a bidirectional long short-term memory network module, and a conditional random field module. The bidirectional Transformer module receives text data, captures the context information of the text data, and obtains the high-dimensional feature representation of the text data; the high-dimensional feature representation is input into the bidirectional long short-term memory network module, the long-term dependency on the time series of the high-dimensional feature representation is captured, and the bidirectional feature representation of the time series is obtained; the conditional random field module performs sequence annotation on the bidirectional feature representation through a preset CRF model, and calculates the optimal label sequence. The present invention improves the accuracy and efficiency of CPE vulnerability information identification.

Description

Technical Field

[0001] The present invention relates to the technical field of software security vulnerability analysis, and more specifically to a CPE identification system enabled by an AI pre-trained large model. Background Art

[0002] In the digital age, the software industry has experienced unprecedented expansion. These software products are largely built on open source code bases and components. In order to ensure security, many organizations have invested heavily in monitoring potential security issues in open source software. The National Vulnerability Database (NVD) supported by the National Institute of Standards and Technology (NIST) is an important resource widely recognized around the world. Every year, the database records thousands of new security vulnerabilities, and the number is still rising. Given that many organizations rely on NVD to update their security vulnerability databases, it is extremely critical for every organization to quickly analyze and respond to security vulnerabilities in open source software.

[0003] However, studies have shown that when CVE (Common Vulnerabilities and Exposures) publishes newly discovered vulnerabilities on NVD, key metadata that vulnerability management relies on, such as CPE (Common Platform Enumeration) applicability statements, takes nearly 35 days to prepare. This delay affects the real-time availability of critical vulnerability information for organizations, which can expose organizations to zero-day attacks and put them at a disadvantage when defending. To address these challenges, named entity recognition (NER) technology is proposed to automatically extract CPE data from CVE reports, allowing researchers to quickly determine which systems and software components may be affected by specific CVEs. Although a variety of NER technologies based on expertise, machine learning, and deep learning have been developed historically, they usually only optimize the most common CPE products and vendors, while ignoring those less common CPE categories. The limitation of this approach is that it performs poorly in identifying uncommon CPE entities in new CVE reports, which limits the applicability of the model. In addition, previous studies have encountered obstacles in creating and annotating large-scale corpora, which limits the learning ability and parameter scale of the model, which in turn affects the model's expressiveness and recognition effect. Therefore, studying how to combine advanced AI technology, especially pre-trained large models, with NER technology is critical to improving the accuracy of CPE information extraction. This fusion can not only enhance the model's ability to identify uncommon CPE entities, but also improve the model's generalization ability, enabling it to better adapt to evolving security threats. With this approach, researchers can more efficiently extract key CPE information from CVE reports to provide support for security analysis and vulnerability management.

[0004] Therefore, in order to address the technical problems in the existing vulnerability security analysis field, a CPE identification system based on AI pre-trained large models is needed. Summary of the invention

[0005] In view of this, the present invention provides a CPE identification system based on AI pre-trained large model empowerment to improve the accuracy and efficiency of CPE vulnerability information identification.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A CPE identification system based on AI pre-trained large model empowerment includes: a bidirectional Transformer module, a bidirectional long short-term memory network module, and a conditional random field module. The bidirectional Transformer module receives text data, captures context information of the text data, and obtains a high-dimensional feature representation of the text data; the high-dimensional feature representation is input into the bidirectional long short-term memory network module, the long-term dependency on the time series of the high-dimensional feature representation is captured, and the bidirectional feature representation of the time series is obtained; the conditional random field module performs sequence annotation on the bidirectional feature representation through a preset CRF model, and calculates the optimal label sequence.

[0008] Preferably, the bidirectional Transformer module includes: an input embedding layer, a position encoding layer, a linear layer and several Transformer encoder layers, the Transformer encoder layer includes a multi-head attention mechanism layer, a feedforward neural network layer, a residual & regularization layer; the input embedding layer, the position encoding layer, several Transformer encoder layers, and the linear layer are connected in sequence; the multi-head attention mechanism layer, the residual & regularization layer, and the feedforward neural network layer are connected in sequence.

[0009] Preferably, the input embedding layer includes: a Token Embeddings layer, which converts each Token into a basic vector; a Segment Embeddings layer, which receives the original vulnerability description text data and distinguishes different parts of the text data; a Position Embeddings layer, which provides a position vector for each Token; the results of the Token Embeddings layer, the Segment Embeddings layer and the Position Embeddings layer are added to obtain a high-dimensional feature representation of the text data; the Token is the basic unit of text data.

[0010] Preferably, the bidirectional long short-term memory network module specifically includes two LSTM layers and a splicing layer. The forward LSTM layer processes the sequence from front to back, and the reverse LSTM layer processes the sequence from back to front. The splicing layer splices the output vectors of the forward and reverse LSTM layers to obtain a bidirectional feature representation of the time series. Each of the LSTM layers includes a forget gate, an input gate, an output gate and a unit state. The forget gate, input gate and output gate are connected in sequence, and the unit state stores the data of the forget gate, input gate and output gate.

[0011] Preferably, the forget gate specifically includes:

[0012] f t =σ(W f ·[h t-1 ,x t ]+b f );

[0013] Among them, f t is the output of the forget gate, W f is the forget gate weight matrix, h t-1 is the hidden state at step t-1, x t is the high-dimensional feature representation of the current time step t, b f is the bias, and σ is the sigmoid activation function.

[0014] Preferably, the input gate includes a sigmoid layer and a tanh layer, specifically:

[0015] Sigmoid layer: i t =σ(W xi x t +W hi h t-1 +b i );

[0016] Among them, σ represents the sigmoid activation function, W xi and W hi is the weight matrix, b i is the bias term, i t is the input gate output;

[0017] Calculate candidate cell status The calculation formula is:

[0018] Tanh layer:

[0019] Among them, tanh is the hyperbolic tangent activation function, W xc and W hc is the weight matrix, b c is the bias term. Finally, the output of the input gate and the candidate cell state are combined to update the cell state Ct , the update formula is:

[0020]

[0021] h t =o t ⊙tanh(C t );

[0022] Among them, f t is the output of the forget gate, ⊙ represents element-by-element multiplication, C t-1 is the cell state at time step t-1, h t is the hidden state at the current time step t, o t is the sigmoid output of the output gate.

[0023] Preferably, the conditional random field module includes an evaluation score layer, a transfer score layer and a decoder, wherein the evaluation score layer calculates a label score for each time step in a time series represented by a bidirectional feature; the transfer score layer calculates a label transfer probability based on the label score; and the decoder calculates an optimal label sequence based on the label score and the label transfer probability.

[0024] Preferably, the evaluation score layer specifically includes:

[0025]

[0026] Among them, w i For the characteristic f i The relevant weight parameter, S t Given an observation m at time step t t and label y t The scores under the conditions are converted into probability distributions and the softmax activation function is applied:

[0027]

[0028] Among them, y′ represents the set of all possible labels, and the emission score Emission(m t ,y t ) is calculated by the following linear combination and activation function:

[0029] Emission(m t ,y t ) = softmax(W·F(X)+b);

[0030] Among them, W is the weight matrix, b is the bias term, and the softmax function ensures that the output of each row is 1. During the training process, the weight W and bias b are optimized by maximizing the conditional probability of the observation sequence:

[0031] L(W,b)=-logP(Y|X;W,b);

[0032] Where L is the loss function and P is the probability of generating the label sequence Y from the observation sequence X given the weight matrix W and the bias term b.

[0033] Preferably, the transfer score hierarchy specifically includes:

[0034] s t-1→t (y t-1 ,y t )=Score(y t-1 ,y t );

[0035] Among them, (y t-1 ,y t ) represents the label pair in adjacent time steps t-1 and t, Score(y t-1 ,y t ) is used to calculate the label pair (y t-1 ,y t ) between the transfer scores, s t-1→t (y t-1 ,y t ) represents the transition score from time step t-1 to time step t.

[0036] Preferably, the decoder specifically comprises:

[0037]

[0038] Among them, V t (y t ) is the label y at time step t t The maximum cumulative score, Score(y t-1 →y t ) is the transfer score, Emission(x t ,y t ) is the launch score;

[0039] While calculating the cumulative score for each label, record the previous label y that led to that score t-1 , at the last time step of the sequence, the cumulative score V of all labels T (y T ) have been calculated, select the label with the maximum cumulative score as the final label of the sequence, start from the last time step of the sequence, use the pointer matrix to backtrack to find the optimal path, and the backtracking process uses the following formula:

[0040]

[0041] in, is the optimal label at time step t-1, the optimal label sequence Obtained from backtracking. argmax yt (·) is the maximization mathematical operator, V T (y T ) is the label y at the last time step T of the sequence T The cumulative score, Score(y t →y T ) is the label y from time step t t The label y at time step T T transfer score.

[0042] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a CPE identification system based on AI pre-trained large models. The CPE-BERT-BLCRF model of the present invention supports end-to-end vulnerability information learning, that is, from the original text to the final entity label sequence, without the need for complex feature engineering or multi-stage processing procedures, which simplifies the model training and deployment process and helps improve efficiency. In addition, the BERT model can capture the contextual information of each word in the text through its bidirectional Transformer structure, providing the model with rich semantic understanding capabilities and the ability to handle complex CPE identification scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0044] Figure 1 A schematic diagram of the structure of the AI-enabled CPE identification system provided by the present invention;

[0045] Figure 2 A schematic diagram of the bidirectional Transformer layer structure based on BERT provided by the present invention;

[0046] Figure 3 A schematic diagram of a bidirectional long short-term memory network BiLSTM module provided by the present invention;

[0047] Figure 4 Schematic diagram of the conditional random field (CRF) layer provided by the present invention; wherein (a) is a linear chain conditional random field structure, and (b) is a special linear chain CRF structure;

[0048] Figure 5This is an implementation flow chart of the software vulnerability CPE information identification system provided by the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] The embodiment of the present invention discloses a CPE identification system based on AI pre-trained large model empowerment, such as Figure 1 As shown, it includes: a bidirectional Transformer module, a bidirectional long short-term memory network module, and a conditional random field module. The bidirectional Transformer module receives text data, captures the context information of the text data, and obtains the high-dimensional feature representation of the text data; the high-dimensional feature representation is input into the bidirectional long short-term memory network module to capture the long-term dependency of the time series of the high-dimensional feature representation and obtain the bidirectional feature representation of the time series; the conditional random field module performs sequence annotation on the bidirectional feature representation through a preset CRF model and calculates the optimal label sequence.

[0051] Among them, the connection relationship between the three modules is mainly reflected in the fact that the modular design of the system allows efficient collaboration between the modules to achieve an accurate recognition process from input data to output results. The output of the BERT-based bidirectional Transformer module is directly used as the input of the BiLSTM module. This connection method enables the context information captured by BERT to be further processed by BiLSTM. The output of the BiLSTM module is used as the input of the CRF module, and the CRF module uses these feature representations to predict the final label sequence. The input and output of each module are closely connected to form a complete CPE recognition process. The BERT module provides rich contextual features, the BiLSTM module captures the temporal dependency of the sequence data, and the CRF module implements accurate sequence labeling on this basis. This modular design not only improves the recognition ability of the model, but also enhances the interpretability and flexibility of the model.

[0052] The BERT-based bidirectional Transformer module is the first-layer processing unit of the system, responsible for processing the input text data to capture the contextual information. This module uses the pre-training capability of the BERT model to deeply analyze the deep semantic relationship of words in sentences and provide rich feature representations for subsequent CPE identification tasks. Specifically, this module receives CPE-related vulnerability descriptions processed by word segmentation and embedding representation as input, and outputs high-dimensional feature representations of CPE text data, which contains CPE words, phrases, and contextual information of the entire sentence.

[0053] The BiLSTM module follows the BERT module, such as Figure 3 As shown in the figure, the sequence data is processed by two LSTM networks, forward and backward, to further capture the long-term dependencies in the time series. The bidirectional structure of this module enables the model to consider past information (forward) and future information (backward) at the same time, thereby enhancing the ability to understand sequence data. The input of the BiLSTM module is the output of the BERT module, that is, the high-dimensional feature representation of the text data, and its output is the bidirectional feature representation of the sequence data, which will be used for the next step of label prediction.

[0054] The conditional random field (CRF) module, as the linear layer of the system, is a discriminative model used to sequence the output of the BiLSTM module, that is, to determine the best label for each time step. By considering the transition probability between labels, the CRF module can find the optimal label path for the entire sequence and achieve accurate CPE identification. The input of this module is the output of the BiLSTM module, that is, the bidirectional feature representation of the sequence data, and the output is the optimal label path for the sequence, including key labels such as software vendor, product, and version number.

[0055] In a specific embodiment, Figure 2 As shown, the bidirectional Transformer module includes: an input embedding layer, a position encoding layer, a linear layer and several Transformer encoder layers, and the Transformer encoder layer includes a multi-head attention mechanism layer, a feedforward neural network layer, and a residual & regularization layer; the input embedding layer, the position encoding layer, several Transformer encoder layers, and the linear layer are connected in sequence; the multi-head attention mechanism layer, the residual & regularization layer, and the feedforward neural network layer are connected in sequence.

[0056] The connection order of each layer within the system is:

[0057] First, the input of the input embedding layer is the original vulnerability text data. This layer is used to convert the discrete vulnerability text data into a continuous vector representation so that the CPE recognition model can process it, and the corresponding output is the embedding vector. Second, the input of the position encoding layer is the embedding vector. This layer is used to add position information to the embedding vector so that the model can recognize the order of words in the vulnerability sequence. The corresponding output is the embedding vector with position information. Third, the input of the multi-head attention mechanism layer is the embedding vector with position encoding. This layer allows the CPE recognition model to focus on different parts of the input vulnerability sequence in different representation subspaces. The corresponding output is the attention-weighted representation. Fourth, the input of the residual & regularization layer is the attention-weighted representation. This layer is used to stabilize the CPE recognition model training process and improve the recognition accuracy performance of the model. The corresponding output is the normalized output. Fifth, the input of the feedforward neural network layer is the output after residual connection and layer normalization. This layer is used to perform further nonlinear transformation on the representation of each position. The corresponding output is the output of the feedforward network. Sixth, the input of the residual connection and layer normalization again is the output of the feedforward neural network. This layer applies residual connection and layer normalization again to stabilize the training, and the corresponding output is the final normalized output. Seventh, the above structure from the attention mechanism layer to the normalization layer is repeated many times to form multiple Transformer encoder layers. Eighth, the input of the linear layer is the final output high-dimensional abstract features after passing through the multi-layer Transformer encoder. This layer is used to convert the output of the Transformer into the prediction result of a specific CPE recognition task, and the corresponding output is the final CPE recognition prediction value.

[0058] Through this internal layer-by-layer connection design, each layer can not only learn more complex feature representations, but also ensure the stable transmission of information between layers.

[0059] The input embedding layer includes: Token Embeddings layer, which converts each Token into a basic vector; Segment Embeddings layer, which receives the original vulnerability description text data and distinguishes different parts of the text data; Position Embeddings layer, which provides a position vector for each Token; the results of the Token Embeddings layer, Segment Embeddings layer and Position Embeddings layer are added together to obtain a high-dimensional feature representation of the text data; Token is the basic unit of text data; the vulnerability information description text is converted into a numerical vector representation that can be processed by the model.

[0060] The Transformer encoder layer is composed of multiple Transformer encoder layers stacked together. Each encoder layer contains a self-attention mechanism and a feedforward neural network, allowing the model to capture complex dependencies in the input sequence.

[0061] In the self-attention mechanism layer, the model is allowed to pay attention to word tokens at different positions when processing sequences and calculate the attention weights between word tokens.

[0062] In the feedforward neural network layer, the output of the self-attention mechanism is further transformed to extract higher-level features.

[0063] Residual connections and normalization layers are used to improve the training stability and effect of the model.

[0064] In the linear layer, it is customized according to the specific task and used for the linear output of the sequence labeling layer.

[0065] In the input embedding layer, the input embedding layer is where the model receives raw data. In the BERT model, the input data is usually a sequence of text, which consists of a series of tokens, such as words or subwords of software vulnerability information. The purpose of the input embedding layer is to convert the text into a numerical form that the model can understand.

[0066] In the Embeddings layer, the discrete tokens of the input embedding layer are converted into continuous vector representations. This layer consists of three parts: Token Embeddings: Provides a basic vector representation for each token. Segment Embeddings: Used to distinguish different parts in the input sequence, such as distinguishing two sentences in the sentence pair task. Position Embeddings: Provides a position vector for each token to preserve the order information in the sequence. These three embedding vectors are added together to form the final embedding representation of each token.

[0067] In the Transformer encoder layer, the BERT model is composed of multiple Transformer encoder layers stacked together. Each encoder layer is responsible for processing the output of the Embeddings layer to capture the dependencies between tokens in the sequence.

[0068] In the self-attention layer, which is the core of the Transformer encoder, it allows the model to assign different attention weights between different positions in the sequence, thereby capturing long-range dependencies. In BERT, this is achieved through Multi-Head Self-Attention, which performs multiple self-attention operations in parallel, each focusing on a different part of the sequence.

[0069] In the feedforward neural network layer, the feedforward neural network layer is located after the self-attention mechanism layer. It is a fully connected neural network used to perform further nonlinear transformations on the output of the self-attention layer. This layer contains multiple linear transformations and a ReLU activation function to increase the expressiveness of the model.

[0070] In residual connections and normalization layers, residual connections add the input directly to the output of the sublayer to avoid the vanishing gradient problem in deep network training. In each Transformer encoder layer, the output of each sublayer (such as the self-attention layer and the feedforward neural network layer) is added to the input to form a residual connection, which aims to solve the gradient vanishing and gradient exploding problems in deep networks, and to deal with the phenomenon of network degradation, that is, the problem that the performance decreases as the number of network layers increases; the normalization layer is used to stabilize the training process, accelerate convergence, and improve the generalization ability of the model. The normalization layer is applied after each sublayer to stabilize the training process and improve model performance. It solves the problem of internal covariate shift by normalizing the output of each layer, that is, the problem that the parameter update of the previous layer causes the input distribution of the subsequent layer to change. The normalized output is scaled and translated by learnable parameters (such as beta and gamma) to retain the flexibility of the model and adapt to the characteristics of the data.

[0071] The linear layer, the last layer of the model, is used to transform the output of the Transformer encoder layer into the final prediction result.

[0072] In a specific embodiment, the Embeddings layer includes: a Token Embeddings layer, which converts each Token into a basic vector; a Segment Embeddings layer, which distinguishes different parts of the text data; a Position Embeddings layer, which provides a position vector for each Token; the results of the Token Embeddings layer, the Segment Embeddings layer and the Position Embeddings layer are added to obtain a high-dimensional feature representation of the text data; Token is the basic unit of text data.

[0073] In a specific embodiment, the bidirectional long short-term memory network module specifically includes two LSTM layers and a splicing layer. The forward LSTM layer processes the sequence from front to back, and the reverse LSTM layer processes the sequence from back to front. The splicing layer splices the output vectors of the forward and reverse LSTM layers to obtain a bidirectional feature representation of the time series. Each LSTM layer includes a forget gate, an input gate, an output gate and a unit state. The forget gate, input gate and output gate are connected in sequence, and the unit state stores the data of the forget gate, input gate and output gate.

[0074] In the forget gate, it is used to decide which information should be forgotten or retained from the cell state. It helps the model decide which old information is no longer important and can be discarded before new input arrives.

[0075] The input gate is used to determine the amount of information in the new input. It consists of two parts: a sigmoid layer that determines which values ​​need to be updated, and a tanh layer that creates a new candidate value vector that represents the values ​​to be updated.

[0076] The output gate is used to determine the value of the next hidden state, which controls the flow of information about the cell state into the next stage of the model.

[0077] The cell state, which is the core of the LSTM network, carries information about the observed vulnerability description sequence and is continuously updated throughout the sequence processing. The cell state is regarded as a "memory" of information, ensuring that the LSTM remembers long-term information.

[0078] The concatenation layer is used to concatenate the output vectors of the forward and backward LSTM layers to form a unified representation containing bidirectional information.

[0079] In BiLSTM, the forward LSTM and reverse LSTM at each time step will independently update their respective forget gates, input gates, output gates, and cell states. Finally, the forward and reverse LSTM outputs at each time step will be concatenated to form the output of the BiLSTM layer, which contains the bidirectional context information of the entire sequence. This structure makes BiLSTM particularly suitable for tasks that require understanding the context of each element in the sequence.

[0080] In a specific embodiment, in the forget gate, the calculation of the forget gate involves the input x of the current time step t and the hidden state h at the previous time step t-1 These inputs are first summed with the weight matrix W of the forget gate f Multiply and then add the bias b f , the calculation process is:

[0081] f t =σ(W f ·[h t-1 ,x t ]+b f );

[0082] Among them, f t is the output of the forget gate, W f is the forget gate weight matrix, h t-1 is the hidden state at step t-1, x t is the high-dimensional feature representation of the current time step t, b fis the bias, and σ is the sigmoid activation function, which compresses the output value between 0 and 1, indicating the retention probability of each unit state value. Once the output f of the forget gate is calculated t , which will be consistent with the cell state C at the previous time step t-1 Element-by-element multiplication (Hadamard product) is performed to determine which values ​​need to be forgotten. Specifically, the part of the forget gate output close to 0 will cause the corresponding cell state value to be forgotten, while the part of the output close to 1 will retain these values. The output of the forget gate actually controls the retention and forgetting of the cell state. If the output of the forget gate is 0, the corresponding cell state value will be completely forgotten; if the output is 1, the cell state value will be completely retained. In order to retain the maximum amount of information, the operation of the forget gate works closely with the input gate. The input gate determines the degree of addition of new information, while the forget gate determines the degree of retention of old information. The two work together to update the cell state C t .

[0083] In a specific embodiment, the input gate includes a sigmoid layer and a tanh layer, which together determine the extent to which new information is updated to the cell state, specifically:

[0084] Sigmoid layer: i t =σ(W xi x t +W hi h t-1 +b i );

[0085] Among them, σ represents the sigmoid activation function, W xi and W hi is the weight matrix, b i is the bias term, i t is the input gate output;

[0086] Calculate candidate cell status It represents the candidate value of new information and is calculated as:

[0087] Tanh layer:

[0088] Among them, tanh is the hyperbolic tangent activation function, W xc and W hc is the weight matrix, b c is the bias term. Finally, the output of the input gate and the candidate cell state are combined to update the cell state C t , the calculation of the output gate starts by receiving the hidden state h of the previous time step t-1 and the input x at the current time step tThis information is fed into a neural network layer with a sigmoid activation function, which outputs a value between 0 and 1 indicating the importance of each unit state value. Simultaneously with the output gate, the input xt at the current time step and the hidden state h at the previous time step are t-1 It is also input into another neural network layer, which uses the tanh activation function and outputs a vector of candidate cell states The values ​​of this vector range from -1 to 1, indicating new information that may be updated to the cell state. t The update involves the cell state C of the previous time step t-1 , the output f of the forget gate t and the output i of the input gate t The update formula is:

[0089]

[0090] h t =o t ⊙tanh(C t );

[0091] Among them, f t is the output of the forget gate, ⊙ represents element-wise multiplication (Hadamard product), C t-1 is the cell state at time step t-1. Finally, the sigmoid output of the output gate and the cell state C at the current time step t The hidden state h of the current time step is calculated by combining element-by-element multiplication and then using the tanh activation function. t ,h t is the hidden state at the current time step t, o t is the sigmoid output of the output gate.

[0092] In the bidirectional LSTM training phase, the weight matrices and bias items of the forget gate, input gate, output gate, and unit state are updated according to the loss function through the back-propagation algorithm and the gradient descent method to minimize the prediction error.

[0093] like Figure 4 The schematic diagram of the conditional random field CRF module of the present invention is shown in FIG. Figure 4 (a) shows the structure of a linear chain conditional random field, which is the most common form of CRF. The linear chain CRF predicts the optimal label path of the entire sequence by maximizing the conditional probability P(Y|X), which is achieved by considering the transition probability between labels and the emission probability between labels and input features. In this model, each node represents a random variable, which is connected in a linear chain to represent the order of elements in the sequence. irepresents an output variable in the sequence. In the CPE named entity recognition task, it represents the entity label of each Token. The edge represents the dependency relationship between nodes, that is, the output of the current node depends not only on the current input X i , but also depends on the output Y of the previous node i-1 .

[0094] Figure 4 (b) shows a special linear chain CRF, where the input sequence X and the output sequence Y have the same graph structure. This model structure is suitable for tasks where the output sequence corresponds one-to-one with the input sequence. In the CPE named entity recognition task, each input token directly corresponds to an output label. This means that each output variable Y i Directly corresponds to an input variable X i , and each Y i Only depends on its corresponding X i Each node Y i It also represents an output variable in the sequence, and each X i represents a feature vector in the input sequence. The edge only represents the output variable Y i The corresponding input variable X i The dependencies between them are not considered, and the transfer between output variables is not considered.

[0095] In a specific embodiment, the conditional random field module includes an evaluation score layer, a transfer score layer and a decoder. The evaluation score layer calculates a label score for each time step in the time series represented by the bidirectional feature; the transfer score layer calculates the label transfer probability based on the label score; and the decoder calculates the optimal label sequence based on the label score and the label transfer probability.

[0096] The evaluation score is used to reflect the degree of match between the observation sequence and the label sequence, and is usually composed of the matching score between the feature vector provided by the BERT and BiLSTM feature extraction models and the label set.

[0097] The transition score is used to reflect the probability of one label being converted to another label in the label sequence. The transition score is usually determined by a pre-trained transition matrix, which is learned during the training process.

[0098] In the decoder, after the model training is completed, the decoder is used to find the most likely label sequence given the observation sequence. The commonly used algorithm in the decoding process is the Viterbi algorithm, which is a dynamic programming technique that can find the global optimal label sequence in polynomial time.

[0099] In a specific embodiment, for each time step in the sequence, the CRF model calculates a score for each possible label. This is usually done by multiplying the extracted features with the weight parameters learned by the model. This score reflects the degree of match between a given feature and a label. For each time step m in the sequence t and for each possible label y t , the evaluation score layers specifically include:

[0100]

[0101] Among them, w i For the characteristic f i The relevant weight parameter, S t Given an observation m at time step t t and label y t The scores under the conditions are converted into probability distributions and the softmax activation function is applied:

[0102]

[0103] Among them, y′ represents the set of all possible labels, and the emission score Emission(m t ,y t ) is calculated by the following linear combination and activation function:

[0104] Emission(m t ,y t ) = softmax(W·F(X)+b);

[0105] Among them, W is the weight matrix, b is the bias term, and the softmax function ensures that the output of each row is 1. During the training process, the weight W and bias b are optimized by maximizing the conditional probability of the observation sequence:

[0106] L(W,b)=-logP(Y∣X; W,b);

[0107] Where L is the loss function and P is the probability of generating the label sequence Y from the observation sequence X given the weight matrix W and the bias term b.

[0108] In a specific embodiment, it reflects the probability of transitioning from one label to another in a label sequence given an observation sequence. This is calculated using the transfer matrix in the CRF layer, which is a parameter learned during model training. For a pair of labels (y t-1 ,y t ), the transfer score calculation formula is:

[0109] s t-1→t (yt-1 ,y t )=Score(y t-1 ,y t );

[0110] Among them, (y t-1 ,y t ) represents the label pairs in adjacent time steps t-1 and t. In sequence tagging or sequence prediction tasks, these label pairs are used to construct the path of the sequence, and their transfer scores are used to determine the optimal path. t-1 ,yt) is used to calculate the label pairs of two adjacent time steps (y t-1 ,y t ), which is usually learned by the model during training and reflects the relative probability or cost of label transfer. t-1→t (y t-1 ,y t ) represents the transfer score from time step t-1 to time step t, which is for label y t-1 to label y t conversion.

[0111] At each time step of sequence processing, the transition score is calculated using a dynamic programming algorithm to ensure that the optimal path can be efficiently calculated when considering all possible label paths. During the dynamic programming process, a temporary variable is maintained to store the cumulative scores of all possible paths to each label state. During the training phase, the parameters of the transition score (the values ​​in the transition matrix) are optimized by minimizing the loss function, usually through gradient descent.

[0112] In a specific embodiment, it is used to ensure that the globally optimal label sequence is found without considering all possible label combinations, which makes the decoding process acceptable in terms of time complexity. The Viterbi algorithm is used in decoding to find the label sequence with the highest probability. Viterbi decoding is a key step in the sequence labeling task. It uses the emission score of the observed sequence and the transition score between labels to find the most likely label sequence. The Viterbi algorithm is a dynamic programming algorithm that can find the optimal path in polynomial time without enumerating all possible paths. For each time step t and each label y, the cumulative score of each label at that time step is calculated. The cumulative score consists of the optimal path score of the previous time step, the transition score from the previous label to the current label, and the emission score of the current time step. The mathematical formula is:

[0113]

[0114] Among them, V t (y t) is the label y at time step t t The maximum cumulative score, Score(y t-1 →y t ) is the transfer score, Emission(x t ,y t ) is the launch score;

[0115] While calculating the cumulative score for each label, record the previous label y that led to that score t-1 , at the last time step of the sequence, the cumulative score V of all labels T (y T ) have been calculated, select the label with the maximum cumulative score as the final label of the sequence, start from the last time step of the sequence, use the pointer matrix to backtrack to find the optimal path, and the backtracking process uses the following formula:

[0116]

[0117] in, is the optimal label at time step t-1, the optimal label sequence Obtained from backtracking. argmax yt (·) is the maximization mathematical operator, V T (y T ) is the label y at the last time step T of the sequence T The cumulative score, Score(y t →y T ) is the label y from time step t t The label y at time step T T transfer score.

[0118] The above describes the structure of the CPE identification system based on AI pre-trained large model empowerment of the present invention. Figure 5 As shown, the offline model training and online testing process of the CPE identification system enabled by the AI ​​pre-trained large model of the present invention is explained below.

[0119] In the offline model training process, a large amount of CPE-related data is first collected, including software versions, vulnerability descriptions, etc., and the data is cleaned, labeled, and formatted to facilitate model training. Then, a suitable CPE information recognition model architecture is designed, and a suitable pre-trained large model, such as BERT, is selected to determine how it is integrated with components such as BiLSTM and CRF. The model is further trained using the prepared dataset, and the model parameters are optimized using algorithms such as backpropagation and gradient descent. The performance of the model is evaluated using the validation set, and the model structure and parameters are adjusted. The model is tuned based on the evaluation results, which may include adjusting the learning rate, increasing regularization, changing the network structure, etc. Finally, the trained model is exported into a format suitable for deployment.

[0120] Test the identification process online, deploy the offline trained model to the online environment, and configure model services, such as API interfaces, so that users can submit data and receive prediction results. Ensure that the data flow can be passed from the front end to the model by integrating the model into the existing system, ensure that data can flow into and out of the model smoothly, and return prediction results. The model begins to receive online data, make real-time predictions, and return CPE identification results. Ensure that the deployment and operation of the model meet security and privacy protection standards. Regularly iterate and optimize the model based on online performance and user feedback, and retrain the model with newly collected data to adapt to new CPE data and vulnerability information.

[0121] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0122] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A CPE identification system based on AI pre-trained large model empowerment, characterized in that: include: A bidirectional Transformer module, a bidirectional long short-term memory network module, and a conditional random field module. The bidirectional Transformer module receives text data, captures context information of the text data, and obtains a high-dimensional feature representation of the text data; the high-dimensional feature representation is input into the bidirectional long short-term memory network module, the long-term dependency of the high-dimensional feature representation on the time series is captured, and the bidirectional feature representation of the time series is obtained; the conditional random field module performs sequence annotation on the bidirectional feature representation through a preset CRF model, and calculates the optimal label sequence.

2. According to claim 1, a CPE identification system based on AI pre-trained large model empowerment is characterized in that: The bidirectional Transformer module includes: an input embedding layer, a position encoding layer, a linear layer and several Transformer encoder layers, and the Transformer encoder layer includes a multi-head attention mechanism layer, a feedforward neural network layer, and a residual & regularization layer; the input embedding layer, the position encoding layer, several Transformer encoder layers, and the linear layer are connected in sequence; the multi-head attention mechanism layer, the residual & regularization layer, and the feedforward neural network layer are connected in sequence.

3. According to claim 2, a CPE identification system based on AI pre-trained large model empowerment is characterized in that: The input embedding layer includes: a Token Embeddings layer, which converts each Token into a basic vector; a SegmentEmbeddings layer, which receives the original vulnerability description text data and distinguishes different parts of the text data; a PositionEmbeddings layer, which provides a position vector for each Token; the results of the Token Embeddings layer, the Segment Embeddings layer and the Position Embeddings layer are added to obtain a high-dimensional feature representation of the text data; the Token is the basic unit of text data.

4. According to claim 1, a CPE identification system based on AI pre-trained large model empowerment is characterized in that: The bidirectional long short-term memory network module specifically includes two LSTM layers and a splicing layer. The forward LSTM layer processes the sequence from front to back, and the reverse LSTM layer processes the sequence from back to front. The splicing layer splices the output vectors of the forward and reverse LSTM layers to obtain a bidirectional feature representation of the time series. Each of the LSTM layers includes a forget gate, an input gate, an output gate and a unit state. The forget gate, input gate and output gate are connected in sequence, and the unit state stores the data of the forget gate, input gate and output gate.

5. According to claim 4, a CPE identification system based on AI pre-trained large model empowerment is characterized in that: The forget gate specifically includes: f t =σ(W f ·[h t-1 ,x t ]+b f ); Among them, f t is the output of the forget gate, W f is the forget gate weight matrix, h t-1 is the hidden state at step t-1, x t is the high-dimensional feature representation of the current time step t, b f is the bias, and σ is the sigmoid activation function.

6. According to claim 4, a CPE identification system based on AI pre-trained large model empowerment is characterized in that: The input gate includes a sigmoid layer and a tanh layer, specifically: Sigmoid layer: i t = σ(W xi x t + W hi h t-1 + b i ); Among them, σ represents the sigmoid activation function, W xi and W hi is the weight matrix, b i is the bias term, i t is the input gate output; Calculate candidate cell status The calculation formula is: Tanh layer: Among them, tanh is the hyperbolic tangent activation function, W xc and W hc is the weight matrix, b c is the bias term. Finally, the output of the input gate and the candidate cell state are combined to update the cell state C t , the update formula is: h t =o t ⊙tanh(C t ); Among them, f t is the output of the forget gate, ⊙ represents element-by-element multiplication, C t-1 is the cell state at time step t-1, h t is the hidden state at the current time step t, o t is the sigmoid output of the output gate.

7. According to claim 1, a CPE identification system based on AI pre-trained large model empowerment is characterized in that: The conditional random field module includes an evaluation score layer, a transfer score layer and a decoder. The evaluation score layer calculates a label score for each time step in the time series represented by the bidirectional feature; the transfer score layer calculates a label transfer probability based on the label score; and the decoder calculates an optimal label sequence based on the label score and the label transfer probability.

8. According to claim 7, a CPE identification system based on AI pre-trained large model empowerment is characterized in that: The evaluation score layers specifically include: Among them, w i For the characteristic f i The relevant weight parameter, S t Given an observation m at time step t t and label y t The scores under the conditions are converted into probability distributions and the softmax activation function is applied: Among them, y′ represents the set of all possible labels, and the emission score Emission(m t ,y t ) is calculated by the following linear combination and activation function: Emission(m t ,y t )=softmax(W·F(X)+b); Where W is the weight matrix, b is the bias term, the softmax function ensures that the output of each row sums to 1, and F(X) represents the result of the observation sequence X after the feature extraction function F(·). During the training process, the weight W and bias b are optimized by maximizing the conditional probability of the observation sequence: L(W,b)=-logP(Y|X;W,b); Where L is the loss function and P is the probability of generating the label sequence Y from the observation sequence X given the weight matrix W and the bias term b.

9. According to claim 7, a CPE identification system based on AI pre-trained large model empowerment is characterized in that: The transfer score hierarchy specifically includes: s t-1→t (and t-1 ,and t )=Score(y t-1 ,and t ); Among them, (y t-1 ,y t ) represents the label pair in adjacent time steps t-1 and t, Score(y t-1 ,y t ) is used to calculate the label pair (y t-1 ,y t ) between the transfer scores, s t-1→t (y t-1 ,y t ) represents the transition score from time step t-1 to time step t.

10. The CPE identification system based on AI pre-trained large model empowerment according to claim 7 is characterized in that: The decoder specifically comprises: Among them, V t (y t ) is the label y at time step t t The maximum cumulative score, Score(y t-1 →y t ) is the transfer score, Emission(x t ,y t ) is the launch score; While calculating the cumulative score for each label, record the previous label y that led to that score t-1 , at the last time step of the sequence, the cumulative score V of all labels T (y T ) have been calculated, select the label with the maximum cumulative score as the final label of the sequence, start from the last time step of the sequence, use the pointer matrix to backtrack to find the optimal path, and the backtracking process uses the following formula: in, is the optimal label at time step t-1, the optimal label sequence Obtained from backtracking. argmax yt (·) is the maximization mathematical operator, V T (y T ) is the label y at the last time step T of the sequence T The cumulative score, Score(y t →y T ) is the label y from time step t t The label y at time step T T transfer score.

Images