Open source intelligence semantic credibility analysis and Bi-LSTM-Atten model self-optimization label extraction method

Abstract

The invention relates to the field of natural language processing, and discloses an open source intelligence semantic credibility analysis and Bi-LSTM-Atten model self-optimization label extraction method, which comprises the following steps: collecting and preprocessing open source intelligence, and establishing a data source value evaluation model; extracting open source intelligence features, performing word segmentation processing on the crawled text, calculating the weight of each segmented word to obtain a candidate tag set, and performing word embedding on selected candidate tags by using BERT; performing semantic credibility analysis on open source intelligence, constructing a credible intelligence classifier, and screening out a credible intelligence set; extracting a credible open source information label, constructing a label evaluation model, constructing a label scoring training set, and training a label quality scoring device based on a Bi-LSTM-Atten model; obtaining the score of each label by using a label quality scoring device; and scoring all the labels in the candidate label set according to the scoring result of the scoring device, and extracting the first k labels defined by the user. According to the invention, the accuracy and credibility of open source information label extraction can be improved.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a self-optimizing label extraction method of an open source intelligence semantic credibility analysis and a Bi-LSTM-Atten model. Background Art

[0002] With the explosive growth of information, it is becoming increasingly important to effectively extract useful information from massive data. Open source intelligence is a method of collecting, analyzing, and utilizing information from publicly available information sources. In this era of information explosion, open source intelligence label extraction has become an important tool for automatically identifying and extracting key information from text data. By using natural language processing and machine learning techniques, open source intelligence label extraction can help users quickly and accurately understand text content, thereby accelerating the decision-making process, discovering new insights, and discovering potential threats and opportunities.

[0003] The goal of open source intelligence tag extraction is to extract key information from text and convert it into structured tags for further analysis and utilization. These tags can cover a wide range of information, including entity names, geographic locations, dates, events, topics, and more. By converting text data into tags, users can more easily search, filter, and analyze the data, thereby discovering the value and meaning hidden behind the massive amount of information.

[0004] Open source intelligence extraction includes the use of supervised, semi-supervised, and unsupervised techniques. Supervised keyword extraction methods mainly include machine learning and neural network models. Machine learning focuses on increasing the number of text features, while neural network models focus more on the application of complex features in keyword extraction. There are many unsupervised extraction methods, which are broadly divided into simple statistics and graph-based extraction methods, and each has its own different extraction methods. Supervised and unsupervised keyword extraction based on semi-supervised keyword extraction methods reduces the difficulty of manual labeling and provides a learning reference for keyword extraction for machine learning. Summary of the invention

[0005] The purpose of the present invention is to provide a self-optimizing label extraction method of open source intelligence semantic trust analysis and Bi-LSTM-Atten model, which can automatically collect open source intelligence, screen out a trustworthy intelligence set based on semantic analysis, and use the Bi-LSTM-Atten model to select labels.

[0006] To achieve the above object, the present invention comprises the following steps:

[0007] Step 1: Open source intelligence collection and preprocessing, including:

[0008] Step 1.1: Identify the source of open source intelligence data.

[0009] Step 1.2: Establish a data source value assessment model to assess the value of each item in each category of open source intelligence data source, and select items with higher value according to needs and put them into the list of URLs to be crawled;

[0010] Step 1.3: Build a distributed crawler cluster in master-slave mode, send crawler tasks to the server according to the data source category, and execute data crawling tasks;

[0011] Step 1.4: Preprocess the crawled data, including data cleaning and text extraction;

[0012] Step 2: Extract open source intelligence features, including:

[0013] Step 2.1: Segment the crawled text;

[0014] Step 2.2: Use the TF-IDF algorithm and the TextRank algorithm to select the candidate tag set for each piece of intelligence;

[0015] Step 2.3: Use BERT technology to embed candidate label words and obtain the word embedding vector of each candidate label;

[0016] Step 3: Open source intelligence semantic trust analysis, including:

[0017] Step 3.1: Use a deep belief network to embed semantic features for each piece of intelligence. Group the word embedding vectors of the candidate tags extracted from each piece of intelligence according to content, time, and domain knowledge and concatenate them into vectors. Pre-train the deep belief network through an unsupervised greedy layer-by-layer method to obtain m feature embedding values.

[0018] Step 3.2: Build an open source intelligence trust classifier based on random forest technology, use the trained random forest model to filter the trustworthy intelligence in the crawled intelligence, and form a trustworthy intelligence set;

[0019] Step 4: Extract trusted open source intelligence labels, including:

[0020] Step 4.1: Build a label evaluation model, calculate the value of candidate labels, and obtain a label scoring training set;

[0021] Step 4.2: Use the label scoring training set to train the label quality scorer based on the Bi-LSTM-Atten model, input the previously obtained word embedding vector into the label quality scorer, and obtain the score of each label;

[0022] Step 4.3, select the top k tags defined by the user according to the scoring results of the tag quality scorer;

[0023] Step 4.4: According to user needs, the label quality scorer adjusts the predicted scores, and then uses the label and the adjusted scoring results to iteratively train the label quality scorer.

[0024] Furthermore, in step 1.2, the data source value assessment model includes relevance evaluation, accuracy evaluation, comprehensiveness evaluation and timeliness evaluation, among which the relevance evaluation includes the fit of strategic intelligence articles and the fit of tactical intelligence articles; the accuracy evaluation includes information detail and citation accuracy; the comprehensive evaluation includes botnet information, vulnerability information, network weapon information, attack techniques and tactics information and virus information; the timeliness evaluation includes virus information, sensitive message update frequency and emergency release time.

[0025] Furthermore, step 1.3 specifically includes: in the master-slave mode, there is a master server to maintain the queue of URLs to be crawled, which is responsible for distributing the URLs to be crawled to different slave servers responsible for crawling each data source, monitoring and balancing the load of the slave servers, and the slave servers are responsible for web page crawling and parsing; in the initialization stage, the URLs contained in the screened data source are first put into the queue of URLs to be crawled as seed URLs. After the crawler starts, the URLs to be crawled are taken from the head of the queue of URLs to be crawled, the URLs are accessed and the texts are parsed and stored in the database, and then these URLs are put into the queue of crawled URLs, and the other URLs contained in them are put into the tail of the queue of URLs to be crawled, and the next cycle is entered until the queue of URLs to be crawled is empty.

[0026] Furthermore, step 2.2 specifically includes: using the TF-IDF algorithm and the TextRank algorithm to calculate the weight of each word of each piece of information, and normalizing the weights of each word calculated by the two algorithms, calculating the mean of the normalized weight of each word, arranging them in descending order according to the mean, selecting the first 50 words in turn, and obtaining the final set of candidate tags.

[0027] Furthermore, step 2.3 specifically includes: converting the intelligence text to which each candidate tag belongs into a tag ID in the BERT vocabulary, converting the input sequence after tokenization and special tag addition into the input format of the BERT model, inputting the input sequence into a pre-trained BERT model for inference, and extracting the embedded representation of each candidate tag in the output of the BERT model.

[0028] Furthermore, in step 3.1, the deep belief network is composed of multiple restricted Boltzmann machines (RBMs) stacked together, and an open source intelligence training set is constructed to perform network pre-training in an unsupervised greedy layer-by-layer method, wherein the RBM is trained layer by layer using the CD algorithm, and finally m feature embedding values ​​are obtained.

[0029] Furthermore, in step 3.2, the credibility of each piece of intelligence is manually determined by manual labeling, and the intelligence is divided into three categories: credible intelligence, uncredible intelligence, and undecidable intelligence. The input features are the values ​​obtained by embedding the features of each piece of intelligence. A training data set is formed based on the labeling results. The random forest model is trained using the open source intelligence training set. The performance of the random forest model is evaluated using the test data set, including accuracy, precision, recall, and F1 score. After the expected performance is achieved, the trained random forest model is used to screen credible intelligence from the crawled intelligence.

[0030] Furthermore, step 4.1 specifically includes: establishing a label evaluation model based on weighted calculations of frequency, timeliness, closeness, information and completeness, and scoring each candidate label in each piece of intelligence; recording the keyword set and each label score of each candidate intelligence as a training data, and finally forming the entire label scoring training set.

[0031] Furthermore, in step 4.3, the first k tags defined by the user are selected by sorting them from high to low according to the scores.

[0032] Furthermore, in step 4.4, the label to be scored is input into the label quality scorer based on Bi-LSTM-Atten to obtain a predicted score, and the predicted score is adjusted according to user needs. The adjusted score data is used to train the label quality scorer online, and the optimization is iterated in a loop.

[0033] Compared with the existing intelligence label extraction method, the present invention has the following advantages:

[0034] 1. The present invention uses an open source intelligence data source evaluation model to select data sources, and at the same time uses a method for allocating distributed crawler tasks according to the data source type, thereby balancing the task load of crawling open source intelligence from different servers.

[0035] 2. The present invention uses a deep belief network based on semantic analysis to represent the features of open source intelligence text, and uses random forest to perform credible analysis and classification of open source intelligence.

[0036] 3. The present invention uses a label quality scorer based on the Bi-LSTM-Atten model, and constructs a training data set based on a label value evaluation mechanism. At the same time, the label quality scorer uses a self-optimization method to optimize parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a flow chart of the present invention;

[0038] Figure 2 A specific step diagram provided for a specific embodiment of the present invention;

[0039] Figure 3 A block diagram of a data source evaluation method provided by a specific embodiment of the present invention;

[0040] Figure 4 An open source intelligence collection diagram provided for a specific embodiment of the present invention;

[0041] Figure 5 A diagram showing the internal structure of a deep belief network provided by a specific embodiment of the present invention;

[0042] Figure 6 A flowchart of trustworthy analysis based on random forests is provided for a specific embodiment of the present invention;

[0043] Figure 7 A label evaluation model diagram provided for a specific embodiment of the present invention;

[0044] Figure 8 A label quality scorer diagram based on a Bi-LSTM-Atten structure provided in a specific embodiment of the present invention;

[0045] Fig. 9 A self-optimization flow chart provided for a specific embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the features and advantages of the present invention more obvious and understandable, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] Figure 1 Specific steps of a specific embodiment of the present invention are as follows: Figure 2 As shown, the present invention proposes a self-optimizing label extraction method of open source intelligence semantic trust analysis and Bi-LSTM-Atten model, including:

[0048] Step S101, open source intelligence collection and preprocessing, includes the following implementation steps:

[0049] First, determine the data sources of open source intelligence, including various news media websites, commercial websites, and social media accounts; second, establish a data source value assessment model, such as Figure 3 As shown, the data source value evaluation model includes relevance evaluation, accuracy evaluation, comprehensiveness evaluation and timeliness evaluation.

[0050] The relevance evaluation includes the fit of strategic intelligence articles and tactical intelligence articles; the accuracy evaluation includes information detail and citation accuracy; the comprehensiveness evaluation includes botnet information, vulnerability information, network weapon information, attack techniques and tactics information, and virus information; the timeliness evaluation includes the frequency of sensitive information updates and the time of emergency releases.

[0051] Evaluate the value of each item in each category of open source intelligence data source, select the items with higher value according to the needs, and put them into the list of URLs to be crawled; then build a distributed open source intelligence crawler cluster in master-slave mode, such as Figure 4 As shown in the figure, the data sources are classified according to their categories and respectively assigned to the slave servers for crawling. In the master-slave mode, there is a dedicated master server to maintain the queue of URLs to be crawled, which is responsible for distributing the URLs to be crawled to different slave servers responsible for crawling each data source, monitoring and balancing the load of the slave servers. The slave servers are responsible for the actual web page crawling and parsing work; in the initialization stage, the uniform resource locator URL contained in the screened data source is first put into the queue of URLs to be crawled as the seed URL. After the crawler starts, the URL to be crawled is taken from the head of the queue of URLs to be crawled, the URL is accessed and the text is parsed and stored in the database, and then these URLs are put into the queue of crawled URLs, and the other URLs contained in them are put into the tail of the queue of URLs to be crawled, and the next cycle is entered until the queue of URLs to be crawled is empty.

[0052] Finally, the crawled data is preprocessed, including data cleaning and text extraction.

[0053] Step S102, extracting open source intelligence features, includes the following implementation steps:

[0054] First, the crawled text is segmented, and then the TF-IDF algorithm and TextRank algorithm are used to calculate the weight of each segmentation of each intelligence, and the weights of each segmentation obtained by the two algorithms are normalized respectively. The average of the normalized weights of each segmentation obtained by the two algorithms is calculated, and the averages are arranged in descending order. The first 50 segmentations are selected in turn to obtain the final candidate tag set. The above operation is performed on each open source intelligence until the candidate tag set of each open source intelligence is obtained. After that, the candidate tag set of each intelligence is obtained, and the BERT technology is used to embed the candidate tag words to obtain the word embedding vector of each candidate tag;

[0055] Step S103, open source intelligence semantic trust analysis, includes the following implementation steps:

[0056] Use a deep belief network to embed semantic features into each piece of intelligence. The components of the deep belief network are as follows: Figure 5 As shown in the figure, the word embedding vectors of the candidate tags extracted from each piece of intelligence are grouped according to content, time, and domain knowledge and concatenated into vectors. The vectors of the candidate tags in each piece of intelligence are used to form a vector. The deep belief network is pre-trained through an unsupervised greedy layer-by-layer method to obtain m feature embedding values.

[0057] The deep belief network is usually composed of multiple stacked restricted Boltzmann machines (RBMs). The training set is pre-trained using an unsupervised greedy layer-by-layer method, where the RBM is trained layer by layer using the CD algorithm, and finally m feature embeddings are obtained.

[0058] Build an open source intelligence trustworthy classifier based on random forest technology. The process is as follows Figure 6 As shown, the trained random forest model is used to filter the credible intelligence in the crawled intelligence and form a credible intelligence set;

[0059] Construct an open source intelligence trust classifier based on random forest technology, use manual labeling methods to manually judge the credibility of each piece of intelligence, and divide it into three categories: trustworthy intelligence, untrustworthy intelligence, and undecidable intelligence. The input features are the values ​​obtained by embedding the corresponding features of each piece of intelligence, and form a training data set based on the labeling results; use the data set to train the random forest model, and use the test data set to evaluate the performance of the random forest model, including accuracy, precision, recall, and F1 score; after the evaluation reaches the expected performance, use the trained random forest model to screen trustworthy intelligence in the crawled intelligence. The F1 score is an indicator used in statistics to measure the accuracy of a binary classification (or multi-task binary classification) model.

[0060] Step S104, extracting trusted open source intelligence labels, includes the following implementation steps:

[0061] First, a label evaluation model is constructed. Labels are manually annotated and the value of each label is calculated to obtain a label scoring training set. A label evaluation model is established based on the weighted calculation of frequency, timeliness, closeness, information and completeness. Figure 7 As shown, the value of each candidate tag in each intelligence is evaluated; the frequency needs to meet the defined tag to meet a certain frequency threshold in the intelligence, the timeliness is determined according to the intelligence release time, and the closeness is mainly used to evaluate the co-occurrence strength of continuous elements in new phrases, including T distribution test, Pearson chi-square test, point-by-point mutual information, likelihood ratio, and the information degree needs to be based on the inverse document frequency, part-of-speech distribution, and stop word distribution of the phrase in the intelligence expectation. The completeness is evaluated based on the closeness of the subset words and superset words of the label phrase; the keyword set and each label score of each candidate intelligence are recorded as a training data, which eventually constitutes the entire training data set;

[0062] Use the label training set to train the label quality scorer based on the Bi-LSTM-Atten model, such as Figure 8 As shown, the word embedding vector obtained previously is input into the label quality scorer to obtain the score of each label;

[0063] Use the Bi-LSTM-Atten model scorer as the score for each tag, sort by score from high to low, and select the first k tags defined by the user, such as Fig. 9 shown.

[0064] Self-optimization of the label quality scorer: the label to be scored is input into the label quality scorer model based on Bi-LSTM-Atten to obtain the predicted score. The predicted score is adjusted according to user needs, and the label quality scorer is trained online using the adjusted score data. This cycle is repeated to optimize the label quality scorer model.

[0065] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. Open source intelligence semantic trust analysis and self-optimizing label extraction method of Bi-LSTM-Atten model, characterized by: The following steps are involved: Step 1: Open source intelligence collection and preprocessing, including: Step 1.1: Identify the source of open source intelligence data. Step 1.2: Establish a data source value assessment model to assess the value of each item in each category of open source intelligence data source, and select items with higher value according to needs and put them into the list of URLs to be crawled; Step 1.3: Build a distributed crawler cluster in master-slave mode, send crawler tasks to the server according to the data source category, and execute data crawling tasks; Step 1.4: Preprocess the crawled data, including data cleaning and text extraction; Step 2: Extract open source intelligence features, including: Step 2.1: Segment the crawled text; Step 2.2: Use the TF-IDF algorithm and the TextRank algorithm to select the candidate tag set for each piece of intelligence; Step 2.3: Use BERT technology to embed candidate label words and obtain the word embedding vector of each candidate label; Step 3: Open source intelligence semantic trust analysis, including: Step 3.1: Use a deep belief network to embed semantic features for each piece of intelligence. Group the word embedding vectors of the candidate tags extracted from each piece of intelligence according to content, time, and domain knowledge and concatenate them into vectors. Pre-train the deep belief network through an unsupervised greedy layer-by-layer method to obtain m feature embedding values. Step 3.2: Build an open source intelligence trust classifier based on random forest technology, use the trained random forest model to filter the trustworthy intelligence in the crawled intelligence, and form a trustworthy intelligence set; Step 4: Extract trusted open source intelligence labels, including: Step 4.1: Build a label evaluation model, calculate the value of candidate labels, and obtain a label scoring training set; Step 4.2: Use the label scoring training set to train the label quality scorer based on the Bi-LSTM-Atten model, input the previously obtained word embedding vector into the label quality scorer, and obtain the score of each label; Step 4.3: Select the top k tags defined by the user according to the scoring results of the tag quality scorer; Step 4.4: According to user needs, the label quality scorer adjusts the predicted scores, and then uses the label and the adjusted scoring results to iteratively train the label quality scorer.

2. The method for self-optimizing label extraction of open source intelligence semantic trust analysis and Bi-LSTM-Atten model according to claim 1, characterized in that: In step 1.2, the data source value assessment model includes relevance evaluation, accuracy evaluation, comprehensiveness evaluation and timeliness evaluation. The relevance evaluation includes the fit of strategic intelligence articles and tactical intelligence articles; the accuracy evaluation includes information detail and citation accuracy; the comprehensiveness evaluation includes botnet information, vulnerability information, network weapon information, attack techniques and tactics information, and virus information; the timeliness evaluation includes virus information, sensitive message update frequency, and emergency release time.

3. The method for self-optimizing label extraction of open source intelligence semantic trust analysis and Bi-LSTM-Atten model according to claim 2 is characterized in that: Step 1.3 specifically includes: in the master-slave mode, there is a master server to maintain the queue of URLs to be crawled, which is responsible for distributing the URLs to be crawled to different slave servers responsible for crawling each data source, monitoring and balancing the load of the slave servers, and the slave servers are responsible for web page crawling and parsing; in the initialization stage, the URLs contained in the filtered data source are first put into the queue of URLs to be crawled as seed URLs. After the crawler starts, the URLs to be crawled are taken from the head of the queue of URLs to be crawled, the URLs are accessed and the texts are parsed and stored in the database, and then these URLs are put into the queue of crawled URLs, and the other URLs contained in them are put into the tail of the queue of URLs to be crawled, and the next cycle is entered until the queue of URLs to be crawled is empty.

4. The method for self-optimizing label extraction of open source intelligence semantic trust analysis and Bi-LSTM-Atten model according to claim 3 is characterized in that: Step 2.2 specifically includes: using the TF-IDF algorithm and the TextRank algorithm to calculate the weight of each word of each piece of information, and normalizing the weights of each word calculated by the two algorithms, calculating the mean of the normalized weight of each word, arranging them in descending order according to the mean, and selecting the first 50 words in turn to obtain the final set of candidate tags.

5. The method for self-optimizing label extraction of open source intelligence semantic trust analysis and Bi-LSTM-Atten model according to claim 4 is characterized in that: Step 2.3 specifically includes: converting the intelligence text to which each candidate tag belongs into a tag ID in the BERT vocabulary, converting the input sequence after tokenization and special tags added into the input format of the BERT model, inputting the input sequence into the pre-trained BERT model for inference, and extracting the embedded representation of each candidate tag in the output of the BERT model.

6. The method for self-optimizing label extraction of open source intelligence semantic trust analysis and Bi-LSTM-Atten model according to claim 5, characterized in that: In step 3.1, the deep belief network is composed of multiple restricted Boltzmann machines (RBMs) stacked together, and an open source intelligence training set is constructed to perform network pre-training in an unsupervised greedy layer-by-layer method. RBMs are trained layer by layer using the CD algorithm, and finally m feature embedding values ​​are obtained.

7. The method for self-optimizing label extraction of open source intelligence semantic trust analysis and Bi-LSTM-Atten model according to claim 6, characterized in that: In step 3.2, the credibility of each piece of intelligence is manually determined by manual labeling, and divided into three categories: credible intelligence, untrustworthy intelligence, and undecidable intelligence. The input features are the values ​​obtained by embedding the corresponding features of each piece of intelligence. A training data set is formed based on the labeling results. The random forest model is trained using the open source intelligence training set, and the performance of the random forest model is evaluated using the test data set, including accuracy, precision, recall, and F1 score. After the evaluation reaches the expected performance, the trained random forest model is used to screen the credible intelligence in the crawled intelligence.

8. The method for self-optimizing label extraction of open source intelligence semantic trust analysis and Bi-LSTM-Atten model according to claim 7, characterized in that: Step 4.1 specifically includes: establishing a label evaluation model based on weighted calculations of frequency, timeliness, closeness, information and completeness, and scoring each candidate label in each piece of intelligence; recording the keyword set and label scores of each candidate intelligence as a training data, ultimately forming the entire label scoring training set.

9. The method for self-optimizing label extraction of open source intelligence semantic trust analysis and Bi-LSTM-Atten model according to claim 8, characterized in that: In step 4.3, the first k tags defined by the user are selected by sorting them from high to low scores.

10. The method for self-optimizing label extraction of open source intelligence semantic trust analysis and Bi-LSTM-Atten model according to claim 9, characterized in that: In step 4.4, the label to be scored is input into the label quality scorer based on Bi-LSTM-Atten to obtain the predicted score. The predicted score is adjusted according to user needs, and the label quality scorer is trained online using the adjusted score data, and the optimization is iterated in a loop.

Images