A BiLSTM-CRF-based HTTP message keyword extraction method

Abstract

This invention discloses a keyword extraction method for HTTP messages based on BiLSTM-CRF. This invention uses a BiLSTM-CRF model to extract effective and valuable keywords from HTTP messages, and mainly includes three parts: data preprocessing, building a BiLSTM-CRF model, and outputting model testing results. This invention can intelligently extract effective network information, terminal features, and other content from HTTP messages, and while ensuring the extraction of existing effective and valuable keywords, it further enhances the ability to discover and mine new keywords.

Description

Technical Field

[0001] This invention belongs to the field of network data artificial intelligence analysis technology, and in particular relates to a method for extracting HTTP message keywords based on BiLSTM-CRF. Background Technology

[0002] With the rapid development of the internet, applications such as web pages, blogs, social networks, and instant messaging software have become widespread, generating a massive amount of HTTP (Hypertext Transfer Protocol) message data. Extracting valuable information from these messages has become a key research focus for many internet companies. Currently, the publicly available engineered approaches to keyword extraction from HTTP messages mainly fall into two categories: rule-based methods, which rely on manually defined rules, such as regular expression matching and domain-specific entity dictionary matching; and methods that build keyword feature libraries from existing HTTP messages and extract matching key information based on these feature libraries.

[0003] While rule-based or feature-based methods can extract valuable information from HTTP messages, they often fail to extract keywords that have not appeared in HTTP messages before or that violate manually defined rules, resulting in the loss of some useful information. Summary of the Invention

[0004] To address the problems of existing technologies and to solve the problem of insufficient ability of existing technologies to discover and mine newly emerging keywords or keywords that break through manually defined rules, this invention provides an HTTP message keyword extraction method based on BiLSTM-CRF by training a large number of effective HTTP messages to extract keywords (also known as named entities), so as to automatically detect effective and valuable keyword information in messages.

[0005] To solve the above technical problems, the present invention adopts the following technical solution: a method for extracting keywords from HTTP messages based on BiLSTM-CRF, comprising the following steps:

[0006] Step S1, establish a keyword feature library: use existing messages to establish a keyword feature library, store keyword tags, keyword extraction rules and original message full-text address information in the database, so that the Bi-LSTM-CRF model can extract data from the database as the source of model training samples and test samples; the keywords in the message appear in key-value pairs, and the keyword must have both a key value and a value value at the same time.

[0007] Step S2, Data Preprocessing: The data of the keyword feature library established in Step S1 is preprocessed sequentially, including: filtering dirty data in the database, segmenting the original HTTP message according to the "carriage return + newline" rule, converting special characters of keywords containing special characters, labeling keywords according to the retrieved keyword tags and extraction rules, and using part of the preprocessed data as training samples and the remaining part as test samples.

[0008] Step S3: Establish the Bi-LSTM-CRF model

[0009] (1) The characters after the message sample is segmented enter the look-up layer, and the i-th character is transformed using the word2ver method. i Mapped to an m-dimensional character embedding vector x i The character embedding vector x i The dimensions are preset;

[0010] (2) Embed the character into the vector x i The forward hidden state serves as the input to the BiLSTM layer at time t. and backward hidden state Directly concatenating the hidden states yields the complete m-dimensional hidden state at that moment. The hidden state h is obtained by using the linear layer formula (1). t The predicted probability p mapped from m dimensions to k dimensions i The k-dimensional dimension is the total number of pre-set keyword tags;

[0011] p i =W h ·h t +b h (1)

[0012] In the formula, W h To connect the weight matrix, b h It is the bias vector;

[0013] (3) For the character sequence w = {w1, w2, ..., w...} i ,…,w n The label sequence y = {y1, y2, ..., y} is obtained by labeling. i ,…,y n Then, based on the predicted probability p output by the BiLSTM layer... i The optimal label y is obtained by solving the state transition probability matrix A (A is a (k+2)×(k+2) matrix with initial and final states added) of the CRF layer using the Viterbi algorithm (2). * ;

[0014]

[0015] In the formula, y * This represents the optimal labeling output by the model. Indicates label y i To tag y i+1 The transition probability, Represents the i-th character w i Predicted label y i The probability; n represents the number of characters in the character sequence w;

[0016] Step S4: Using the valid message samples in the test set generated by the data preprocessing in step S2 as the object, substitute them into the BiLSTM-CRF model established in step S3 to obtain the model prediction results. Compare the prediction results with the labeled keywords in the test samples obtained in step S2, and output the evaluation results of the model test.

[0017] As a preferred embodiment, the specific process of data preprocessing in step S2 is as follows:

[0018] (1) Filter dirty data in the database, which includes keywords that do not conform to the definition and requirements;

[0019] (2) Based on the characteristics of the original HTTP message, the original HTTP message is segmented according to the rule of "carriage return + newline character" to form multiple messages, namely request line, request content, response header and response content;

[0020] (3) For keywords containing special characters, the special characters are converted into ordinary characters through escape processing. The special characters include \n and \r.

[0021] (4) Based on the keyword tags and extraction rules extracted from the database, the keywords in the message are labeled with keywords, and the keywords in the message appear in key-value pairs;

[0022] (5) When converting the message sample into a training or test sample suitable for model input, while ensuring the keyword order dependency, the message sample obtained in step (4) that exceeds a certain length is truncated and divided into multiple new samples. The segmentation conditions are: the length reaches a certain threshold, and the label of the keyword matched for the first time in reverse search at the truncated point cannot be the "_KEY" label, and if there are characters after the truncated point, the next character label must be a label without specific meaning.

[0023] As a preferred option, in step (4), the keywords are labeled with keywords, the keyword key value is labeled as “label value_KEY”, the keyword value is labeled as “label value_VALUE”, and each character in the segmented sample is labeled with the BIO labeling system. B indicates that the character is the beginning of the keyword key value or value value, that is, the character is labeled as “B-label value_KEY” or “B-label value_VALUE”, I indicates that the character is the middle or end of the keyword key value or value value, that is, the character is labeled as “I-label value_KEY” or “I-label value_VALUE”, and O indicates that the character has no specific meaning, that is, the character is labeled as “O”.

[0024] Preferably, in step S3, the m-dimensional character embedding vector x obtained in step (1) i Dropout processing is also required to prevent overfitting; step (2) concatenates the complete m-dimensional hidden states h at that moment. t Dropout processing is also required.

[0025] As a preferred option, in the process of solving for the optimal annotation in step (2) of step S3, the following constraints are introduced:

[0026] (1) The label of the first character of the original input data or keywords of the model cannot be "I-";

[0027] (2) “B-label1 I-label2 I-label3…”, in this mode, label1, label2, and label3 must be the same keyword label;

[0028] (3) Message keywords must appear in key-value pairs, and the "_KEY" label of the keyword must appear before the "_VALUE" label.

[0029] Beneficial effects: Compared with existing technologies, this invention can intelligently extract effective network information, terminal characteristics and other content from HTTP messages, and further enhance the ability to discover and mine new keywords while ensuring that existing effective and valuable keywords can be extracted. Attached Figure Description

[0030] Figure 1 This is a general block diagram illustrating the principle of the HTTP message keyword extraction method based on BiLSTM-CRF as described in this invention.

[0031] Figure 2 This is a schematic diagram of the BiLSTM-CRF model described in this invention;

[0032] Figure 3This is a schematic diagram of the LSTM model described in this invention. Detailed Implementation

[0033] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art fall within the scope defined by the appended claims.

[0034] This invention relates to a keyword extraction method for HTTP messages based on BiLSTM-CRF. It uses a BiLSTM-CRF model to extract effective and valuable keywords from HTTP messages, automatically detecting valuable keyword information within the messages. The system mainly consists of three parts: data preprocessing, building a BiLSTM-CRF model, and outputting model testing results. Figure 1 As shown. The specific process of the system method is as follows:

[0035] 1) Data preprocessing

[0036] This invention utilizes existing message data to establish a keyword feature library, storing information such as keyword tags, keyword extraction rules, and the full-text address of the original message in the database. The data retrieved from the database serves as the model training sample source. Keywords in the message appear in key-value pairs. When establishing the feature library, only the keyword key value is extracted. Extraction method information (the position of the keyword key value in the message, the decoding method during keyword extraction, etc.) is stored in the database as a keyword extraction rule field. When retrieving the original HTTP message from the full-text database, it may be necessary to decode or decrypt the message (e.g., URL decoding, base64 decoding, gzip decoding). The specific encoding method can be determined based on the specific data format before decoding.

[0037] Step 1: Regarding the keyword extraction method when building the feature library, there may be keyword values ​​in the database that do not conform to the definition and requirements (for example, domestic mobile phone numbers without international area codes should be eleven pure numbers; if a keyword's tag is a domestic mobile phone number, but the keyword value is not an eleven pure number, then it is considered dirty data that does not conform to the definition). Therefore, it is necessary to filter the dirty data in the database according to general rules.

[0038] The second step is to segment the original HTTP message according to the characteristics of the original HTTP message and the rule of "carriage return + newline character" to form multiple messages such as request line, request header, request content, response header, and response content. The reason for the segmentation is that, from the data content level, the segmented messages have no inherent semantic connection and do not affect the model detection effect. At the same time, it reduces the length of the model input parameters, thereby reducing the memory consumption of the algorithm.

[0039] Step 3: Some keywords contain special characters such as \n and \r, which requires escaping these characters during data preprocessing to reduce their impact on the model's detection performance.

[0040] Step 4: Based on the keyword tags and extraction rules retrieved from the database, the keywords are labeled. In this invention, the keyword key value is labeled as "tag value_KEY", and the keyword value value is labeled as "tag value_VALUE" (for example, the request line of the original message contains GET / ?user_id=20170425&XXX=XXX). According to HTTP / 1.1, based on the query results from the PG database, "user_id" and "20170425" are the keyword key and value, respectively. The keyword tag is USERID, so "user_id" and "20170425" are labeled as "USERID_KEY" and "USERID_VALUE", respectively. The keyword tag adopts the BIO tagging system (B indicates that this character is the beginning of the keyword key or value, that is, this character is labeled as "B-tag_key" or "B-tag_VALUE", I indicates that this character is the middle or end of the keyword key or value, that is, this character is labeled as "I-tag_key" or "I-tag_VALUE", and O indicates that this character has no specific meaning, that is, this character is labeled as "O").

[0041] Step 5: When converting message samples into training or testing samples suitable for model input, while ensuring keyword order dependency, message samples exceeding a certain length are truncated and divided into multiple new samples. The segmentation conditions are: the length reaches a certain threshold, the first matched keyword tag found in reverse search at the truncated point cannot be the "_KEY" tag, and if there are characters after the truncated point, the next character tag must be a keyword tag without specific meaning.

[0042] In response to the presence of Chinese and English characters in HTTP messages, this invention adopts character-based segmentation to convert messages into training and testing samples for models. This preserves the inherent relationships between characters, which is beneficial for models to discover and utilize long-term dependencies in the data.

[0043] 2) Establish a BiLSTM-CRF model

[0044] Figure 2A character-based BiLSTM-CRF model is presented, where BiLSTM is composed of a forward LSTM and a backward LSTM. LSTM is suitable for classification, processing, and prediction based on time series data and can solve long-term dependency problems. However, for keyword extraction from HTTP messages, it is necessary to consider not only the preceding text but also the following text to make judgments based on context. Therefore, BiLSTM provides a new approach for this invention, as the output at the current moment may be related to both previous and future states. Figure 3 As shown, for an LSTM unit, the input vector x at the current time t is... t The hidden state h can be obtained through the following series of steps. t .

[0045] f t =σ(W f ·[h t-1 ,x t ]+b f )

[0046] i t =σ(W i ·[h t-1 ,x t ]+b i )

[0047]

[0048]

[0049] o t =σ(W o ·[h t-1 ,x t ]+b o )

[0050] h t =o t *tanh(c t )

[0051] In the formula, f t 、i t , c t and o t These represent the forget gate, memory gate, temporary cell state, current cell state, and output gate value, respectively; b f 、b i 、b c 、b o Both are bias vectors; W f 、W i 、W c 、W oThese are all connection weight matrices. The specific steps for establishing the BiLSTM-CRF model in this embodiment are as follows:

[0052] Step 1: Input the segmented characters from the message sample. The characters enter the Look-up layer, where the word2ver method is used to map the i-th character to a character embedding vector x. i (m-dimensional), before proceeding to the next step, the character embedding vector x i Perform dropout to prevent overfitting.

[0053] Step 2: Embed the characters into vector x i The forward hidden state serves as the input to the BiLSTM layer at each time step t. and backward hidden state Directly concatenating the hidden states at that moment yields the complete hidden state. (m-dimensional) and dropout processing is applied. A linear layer is used to store the hidden state h. t The predicted probability p mapped from m dimensions to k dimensions i (k is the number of labeled tags), where p ij W represents the probability that the i-th character is the j-th label. h 、b h These are the connection weight matrix and the bias vector, respectively;

[0054] p i =W h ·h t +b h

[0055] Step 3: Utilize the predicted probability p output by the BiLSTM layer i The optimal labeling is obtained by solving f(x,y) using the Viterbi algorithm, based on the state transition probability matrix A of the CRF layer (with additional initial and final labels). Given the sequence w = {w1, w2, ... w...} n The predicted label sequence is y = {y1, y2, ... y}. n}, Indicates label y i To tag y i+1 Given the transition probabilities, the optimal solution output by the model is obtained by the following formula:

[0056]

[0057] In the formula, y * This represents the optimal labeling output by the model. Indicates label y i To tag y i+1 The transition probability, This indicates that the i-th character is predicted to be the label y.i The probability of.

[0058] For the CRF layer, this invention adds some constraints to ensure that the final prediction result is valid. These constraints can be automatically learned by the CRF layer during training: (1) The label of the original input data of the model or the starting character of the keyword cannot be "I-"; (2) "B-label1 I-label2 I-label3…", in this mode, label1, label2, and label3 must be the same keyword label; (3) The message keywords must appear in "key-value" pairs, and the "_KEY" label of the keyword must appear before the "_VALUE" label.

[0059] 3) Output of model test results

[0060] Using valid message samples from the test set generated by data preprocessing as the object, the established BiLSTM-CRF model is substituted to obtain the model's prediction results. These results are then compared with the keywords that can be extracted from the test set, and the evaluation results of the model test are output.

[0061] The key technologies of the HTTP message keyword extraction method based on BiLSTM-CRF in this invention are as follows:

[0062] 1) During data preprocessing, the original HTTP message is divided into several message samples such as request line, request header, request content, response header, and response content according to certain rules, and keyword annotation is performed on each sample for effective and valuable keywords, while message samples without keywords are filtered out.

[0063] 2) During data preprocessing, when segmenting message samples into characters, while ensuring the keyword order dependency, it is necessary to truncate excessively long message samples and divide them into multiple samples according to certain rules.

[0064] 3) Train the model using valid message samples and their extractable keywords to establish a BiLSTM-CRF model for keyword extraction and mining in HTTP messages.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for extracting keywords from HTTP messages based on BiLSTM-CRF, characterized in that... Includes the following steps: Step S1, establish a keyword feature library: use existing messages to establish a keyword feature library, store keyword tags, keyword extraction rules and original message full-text address information in the database, so that the BiLSTM-CRF model can extract data from the database as the source of model training samples and test samples; the keywords in the message appear in key-value pairs, and the keyword must have both a key value and a value value at the same time; Step S2, Data Preprocessing: The data of the keyword feature library established in Step S1 is preprocessed sequentially, including: filtering dirty data in the database, segmenting the original HTTP message according to the "carriage return + newline" rule, converting special characters of keywords containing special characters, labeling keywords according to the retrieved keyword tags and extraction rules, and using part of the preprocessed data as training samples and the remaining part as test samples. Step S3: Establish the BiLSTM-CRF model (1) The characters after the message sample is segmented enter the look-up layer, and the i-th character is transformed using the word2ver method. i Mapped to an m-dimensional character embedding vector x i The character embedding vector x i The dimensions are preset; (2) Embed the character into the vector x i The forward hidden state serves as the input to the BiLSTM layer at time t. and backward hidden state Directly concatenating the hidden states yields the complete m-dimensional hidden state at that moment. The hidden state h is obtained by using the linear layer formula (1). t The predicted probability p mapped from m dimensions to k dimensions i The k-dimensional dimension is the total number of pre-set keyword tags; p i =W h ·h t +b h (1) In the formula, W h To connect the weight matrix, b h It is the bias vector; (3) For the character sequence w = {w1, w2, ..., w...} i ,…,w n The label sequence y = {y1, y2, ..., y} is obtained by labeling. i ,…,y n Then, based on the predicted probability p output by the BiLSTM layer... i The optimal label y is obtained by solving the state transition probability matrix A of the CRF layer using the Viterbi algorithm (2). * ; In the formula, y * This represents the optimal labeling output by the model. Indicates label y i To tag y i+1 The transition probability, Represents the i-th character w i Predicted label y i The probability; n represents the number of characters in the character sequence w; Step S4: Using the valid message samples in the test set generated by the data preprocessing in step S2 as the object, substitute them into the BiLSTM-CRF model established in step S3 to obtain the model prediction results. Compare the prediction results with the labeled keywords in the test samples obtained in step S2, and output the evaluation results of the model test.

2. The HTTP message keyword extraction method based on BiLSTM-CRF according to claim 1, characterized in that: The specific process of data preprocessing in step S2 is as follows: (1) Filter dirty data in the database, which includes keywords that do not conform to the definition and requirements; (2) Based on the characteristics of the original HTTP message, the original HTTP message is segmented according to the rule of "carriage return + newline character" to form multiple messages, namely request line, request content, response header and response content; (3) For keywords containing special characters, the special characters are converted into ordinary characters through escape processing. The special characters include \n and \r. (4) Based on the keyword tags and extraction rules extracted from the database, the keywords in the message are labeled with keywords, and the keywords in the message appear in key-value pairs; (5) When converting the message sample into a training or test sample suitable for model input, while ensuring the keyword order dependency, the message sample obtained in step (4) that exceeds a certain length is truncated and divided into multiple new samples. The segmentation conditions are: the length reaches a certain threshold, and the label of the keyword matched for the first time in reverse search at the truncated point cannot be the "_KEY" label, and if there are characters after the truncated point, the next character label must be a label without specific meaning.

3. The HTTP message keyword extraction method based on BiLSTM-CRF according to claim 2, characterized in that: In step (4), the keywords are labeled with keywords. The keyword key value is labeled as "label value_KEY" and the keyword value is labeled as "label value_VALUE". At the same time, the BIO labeling system is used for each character in the segmented sample. B indicates that the character is the beginning of the keyword key value or value value, that is, the character is labeled as "B-label value_KEY" or "B-label value_VALUE". I indicates that the character is the middle or end of the keyword key value or value value, that is, the character is labeled as "I-label value_KEY" or "I-label value_VALUE". O indicates that the character has no specific meaning, that is, the character is labeled as "O".

4. The HTTP message keyword extraction method based on BiLSTM-CRF according to claim 1, characterized in that: In step S3, the m-dimensional character embedding vector x obtained in step (1) i Dropout processing is also required to prevent overfitting; step (2) concatenates the complete m-dimensional hidden states h at that moment. t Dropout processing is also required.

5. The HTTP message keyword extraction method based on BiLSTM-CRF according to claim 4, characterized in that: In the process of solving for the optimal annotation in step (2) of step S3, the following constraints are introduced: (1) The label of the first character of the original input data or keywords of the model cannot be "I-"; (2) "B-label1I-label2I-label3…", in this mode, label1, label2, and label3 must be the same keyword tag; (3) Message keywords must appear in "key-value" pairs, and the "_KEY" tag of the keyword must appear before the "_VALUE" tag.

Images