Open source intelligence semantic credible analysis and bi-lstm-atten model self-optimizing label extraction method

Abstract

This invention relates to the field of natural language processing and discloses a self-optimizing label extraction method for open-source intelligence semantic credibility analysis and a Bi-LSTM-Atten model. The method includes: open-source intelligence collection and preprocessing, establishing a data source value assessment model; open-source intelligence feature extraction, performing word segmentation on the crawled text, calculating the weight of each segment, obtaining a candidate label set, and using BERT to embed the selected candidate labels; open-source intelligence semantic credibility analysis, constructing a credible intelligence classifier, and filtering out the credible intelligence set; credible open-source intelligence label extraction, constructing a label evaluation model, building a label scoring training set, and training a label quality scorer based on the Bi-LSTM-Atten model; obtaining scores for each label using the label quality scorer; scoring all labels in the candidate label set according to the scorer's results, and extracting the user-defined top k labels. This invention can improve the accuracy and credibility of open-source intelligence label extraction.

Description

Technical Field

[0001] This invention relates to the field of natural language processing, specifically to an open-source intelligence semantic credibility analysis and a self-optimizing label extraction method for the Bi-LSTM-Atten model. Background Technology

[0002] With the explosive growth of information, effectively extracting useful information from massive amounts of data has become increasingly important. Open source intelligence is a method for collecting, analyzing, and utilizing information from publicly available sources. In this era of information overload, open source intelligence tag extraction has become an important tool for automatically identifying and extracting key information from text data. By using natural language processing and machine learning techniques, open source intelligence tag extraction can help users quickly and accurately understand text content, thereby accelerating the decision-making process, uncovering new insights, and identifying potential threats and opportunities.

[0003] The goal of open-source intelligence tagging is to extract key information from text and transform it into structured tags for further analysis and utilization. These tags can cover a wide variety of information, including entity names, geographical locations, dates, events, topics, and more. By converting text data into tag form, users can more easily search, filter, and analyze the data, thereby uncovering the value and meaning hidden behind massive amounts of information.

[0004] Open-source keyword extraction encompasses supervised, semi-supervised, and unsupervised techniques. Supervised keyword extraction methods primarily include machine learning and neural network models. Machine learning focuses on increasing the number of text features, while neural network models concentrate on applying complex features in keyword extraction. Numerous unsupervised extraction methods can be broadly categorized into those based on simple statistics and graphs, each with its own sub-methods. Semi-supervised keyword extraction methods, combining supervised and unsupervised approaches, reduce the difficulty of manual annotation while providing learning references for machine learning in keyword extraction. Summary of the Invention

[0005] The purpose of this invention is to provide a self-optimizing label extraction method for open-source intelligence semantic credibility analysis and Bi-LSTM-Atten model. This method can automatically collect open-source intelligence, filter out a set of credible intelligence based on semantic analysis, and select labels using a Bi-LSTM-Atten model.

[0006] To achieve the above objectives, the present invention includes the following steps:

[0007] Step 1: Open source intelligence gathering and preprocessing, specifically including:

[0008] Step 1.1: Determine the data source for open-source intelligence;

[0009] Step 1.2: Establish a data source value assessment model to evaluate the value of each item in each category of open-source intelligence data sources, and select items with higher value according to requirements and add them to the list of URLs to be crawled.

[0010] Step 1.3: Build a distributed crawler cluster in master-slave mode, and distribute crawler tasks to the server according to the data source category to execute the data crawling tasks;

[0011] Step 1.4: Preprocess the crawled data, including data cleaning and text extraction;

[0012] Step 2, open-source intelligence feature extraction, specifically including:

[0013] Step 2.1: Perform word segmentation on the crawled text;

[0014] Step 2.2: Use the TF-IDF algorithm and TextRank algorithm to select the candidate tag set for each piece of information;

[0015] Step 2.3: Use BERT technology to embed candidate tags into words, and obtain the word embedding vector for each candidate tag;

[0016] Step 3: Open source intelligence semantic credibility analysis, specifically including:

[0017] Step 3.1: Use a deep belief network to embed semantic features for each piece of intelligence. Group the word embedding vectors of the candidate tags extracted from each piece of intelligence according to content, time, and domain knowledge, and concatenate them into a vector. Pre-train the deep belief network using an unsupervised greedy layer-by-layer method to obtain m feature embedding values.

[0018] Step 3.2: Construct an open-source intelligence trust classifier based on random forest technology, use the trained random forest model to filter trustworthy intelligence from the crawled intelligence, and form a trustworthy intelligence set;

[0019] Step 4, extraction of trusted open-source intelligence tags, specifically including:

[0020] Step 4.1: Construct a tag evaluation model, calculate the value of candidate tags, and obtain a tag score training set;

[0021] Step 4.2: Train a label quality scorer based on the Bi-LSTM-Atten model using the label score training set. Input the previously obtained word embedding vectors into the label quality scorer to obtain the score for each label.

[0022] Step 4.3: Select the top k user-defined tags based on the tag quality scorer's rating results;

[0023] Step 4.4: Based on the user's needs, the label quality scorer adjusts the predicted scores, and then uses the label and the adjusted score results to iteratively train the label quality scorer.

[0024] Furthermore, in step 1.2, the data source value assessment model includes relevance evaluation, accuracy evaluation, comprehensiveness evaluation, and timeliness evaluation. Among them, relevance evaluation includes the relevance of strategic intelligence articles and the relevance of tactical intelligence articles; accuracy evaluation includes the detail of information and the accuracy of citations; comprehensiveness evaluation includes botnet information, vulnerability information, cyber weapon information, attack tactics information, and virus information; and timeliness evaluation includes virus information, the frequency of sensitive message updates, and the release time of emergencies.

[0025] Furthermore, step 1.3 specifically includes: In master-slave mode, there is one master server to maintain the queue of URLs to be crawled, responsible for distributing the URLs to be crawled to different slave servers responsible for crawling each data source, monitoring and balancing the load of the slave servers, and the slave servers are responsible for web page crawling and parsing; In the initialization phase, the URLs contained in the data source are first selected as seed URLs and put into the queue of URLs to be crawled. After the crawler starts, the URLs to be crawled are taken from the head of the queue of URLs to be crawled, the URLs are accessed and the text is parsed and stored in the database, and then these URLs are put into the queue of already crawled URLs, and other URLs contained therein are put into the tail of the queue of URLs to be crawled, and the next loop is entered until the queue of URLs to be crawled is empty.

[0026] Furthermore, step 2.2 specifically includes: calculating the weight of each word segment for each piece of information using the TF-IDF algorithm and the TextRank algorithm respectively, normalizing the weights of each word segment calculated by the two algorithms respectively, calculating the mean of the normalized weights of each word segment, arranging them in descending order of mean, and selecting the first 50 words in sequence to obtain the final candidate tag set.

[0027] Furthermore, step 2.3 specifically includes: converting the intelligence text to which each candidate label belongs into a tag ID in the BERT vocabulary, converting the input sequence after tokenization and special tag addition into the input format of the BERT model, inputting the input sequence into the pre-trained BERT model for inference, and extracting the embedding representation of each candidate label from the output of the BERT model.

[0028] Furthermore, in step 3.1, the deep belief network is composed of multiple Restricted Boltzmann Machines (RBMs) stacked together. An open-source intelligence training set is constructed to pre-train the network using an unsupervised greedy layer-by-layer method. The RBMs are trained layer by layer using the CD algorithm, ultimately obtaining m feature embedding values.

[0029] Furthermore, in step 3.2, the credibility of each piece of intelligence is manually determined using a manual annotation method, and it is divided into three categories: credible intelligence, untrustworthy intelligence, and indeterminate intelligence. The input features are the values ​​obtained by embedding the features of each piece of intelligence. A training dataset is formed based on the annotation results. A random forest model is trained using an open-source intelligence training set. The performance of the random forest model is evaluated using a test dataset, including accuracy, precision, recall, and F1 score. After the evaluation reaches the expected performance, the trained random forest model is used to filter credible intelligence from the crawled intelligence.

[0030] Furthermore, step 4.1 specifically includes: establishing a tag evaluation model based on a weighted calculation of frequency, timeliness, density, information content, and completeness, and scoring each candidate tag in each piece of intelligence; recording the keyword set and tag scores of each candidate piece of intelligence as a training data set, which ultimately constitutes the entire tag scoring training set.

[0031] Furthermore, in step 4.3, the top k tags, defined by the user, are selected according to the scores from highest to lowest.

[0032] Furthermore, in step 4.4, the labels to be scored are input into a label quality scorer based on Bi-LSTM-Atten to obtain predicted scores. The predicted scores are adjusted according to user needs, and the label quality scorer is trained online using the adjusted scores, and then iteratively optimized.

[0033] Compared with existing intelligence tag extraction methods, the present invention has the following advantages:

[0034] 1. This invention uses an open-source intelligence data source evaluation model to select data sources, and at the same time, it uses a method of distributed crawler task allocation according to data source type to balance the task load of crawling open-source intelligence from different servers.

[0035] 2. This invention uses a deep belief network based on semantic analysis to represent the features of open-source intelligence text, and uses random forest to perform credibility analysis and classification of open-source intelligence.

[0036] 3. This invention uses a label quality scorer based on the Bi-LSTM-Atten model and constructs a training dataset based on a label value evaluation mechanism. At the same time, the label quality scorer uses a self-optimizing method to optimize parameters. Attached Figure Description

[0037] Figure 1 This is a flowchart of the present invention;

[0038] Figure 2 Specific step diagrams provided for a specific embodiment of the present invention;

[0039] Figure 3 This is a block diagram of a data source evaluation method provided in a specific embodiment of the present invention;

[0040] Figure 4 An open-source intelligence gathering diagram provided for a specific embodiment of the present invention;

[0041] Figure 5 This is a diagram illustrating the internal structure of a deep belief network according to a specific embodiment of the present invention.

[0042] Figure 6 A flowchart of a trust analysis based on random forest provided in a specific embodiment of the present invention;

[0043] Figure 7 This is a diagram of a label evaluation model provided in a specific embodiment of the present invention;

[0044] Figure 8 A diagram of a tag quality scorer based on a Bi-LSTM-Atten structure is provided in a specific embodiment of the present invention;

[0045] Figure 9 This is a self-optimization flowchart provided for a specific embodiment of the present invention. Detailed Implementation

[0046] To make the features and advantages of the present invention more apparent and understandable, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] Figure 1 This is a flowchart of the present invention. Specific steps of a particular embodiment of the present invention are as follows: Figure 2 As shown, this invention proposes an open-source intelligence semantic credibility analysis and a self-optimizing label extraction method for the Bi-LSTM-Atten model, including:

[0048] Step S101, open-source intelligence gathering and preprocessing, includes the following implementation steps:

[0049] First, identify the data sources for open-source intelligence, including various news media websites, commercial websites, and social media accounts; second, establish a data source value assessment model, such as... Figure 3 As shown, the data source value evaluation model includes relevance evaluation, accuracy evaluation, comprehensiveness evaluation, and timeliness evaluation.

[0050] Relevance evaluation includes the relevance of strategic intelligence articles and tactical intelligence articles; accuracy evaluation includes the detail and accuracy of information citations; comprehensiveness evaluation includes information on botnets, vulnerabilities, cyber weapons, attack tactics, and viruses; and timeliness evaluation includes the frequency of sensitive information updates and the release time of breaking news.

[0051] Evaluate the value of each item in each category of open-source intelligence data sources, and select the more valuable items according to requirements, adding them to the list of URLs to be crawled; then build a master-slave distributed open-source intelligence crawler cluster, such as... Figure 4 As shown, data sources are categorized and assigned to slave servers for crawling. In master-slave mode, a dedicated master server maintains the queue of URLs to be crawled, distributing them to different slave servers responsible for crawling each data source, monitoring and balancing the load on the slave servers, and handling the actual webpage crawling and parsing. During the initialization phase, URLs containing Uniform Resource Locators (URLs) from the selected data sources are placed into the queue of URLs to be crawled as seed URLs. Once the crawler starts, URLs to be crawled are retrieved from the head of the queue, accessed, and their text parsed and stored in the database. These URLs are then added to the queue of already crawled URLs, while other URLs contained within them are added to the tail of the queue of URLs to be crawled. The process continues until the queue of URLs to be crawled is empty.

[0052] Finally, the crawled data is preprocessed, including data cleaning and text extraction.

[0053] Step S102, Open Source Intelligence Feature Extraction, includes the following implementation steps:

[0054] First, the crawled text is segmented into words. Then, the weights of each word segment for each piece of intelligence are calculated using the TF-IDF and TextRank algorithms respectively. The weights of each word segment obtained by the two algorithms are normalized. The mean of the normalized weights of each word segment obtained by the two algorithms is calculated. The words are arranged in descending order of mean, and the first 50 words are selected to obtain the final candidate tag set. The above operation is performed on each piece of open source intelligence until the candidate tag set for each piece of open source intelligence is obtained. Then, BERT technology is used to embed the candidate tag words to obtain the word embedding vector of each candidate tag.

[0055] Step S103, Open Source Intelligence Semantic Trust Analysis, includes the following implementation steps:

[0056] Semantic features are embedded into each piece of intelligence using a deep belief network. The components of a deep belief network are as follows: Figure 5 As shown, the word embedding vectors of the candidate tags extracted from each intelligence are grouped and concatenated into vectors according to content, time, and domain knowledge. Based on the vectors of each candidate tag in each intelligence, a vector is formed. A deep belief network is pre-trained using an unsupervised greedy layer-by-layer method to obtain m feature embedding values.

[0057] Deep belief networks are typically composed of multiple Restricted Boltzmann Machines (RBMs) stacked together. The training set is pre-trained using an unsupervised greedy layer-by-layer method, where the RBMs are trained layer by layer using the CD algorithm, ultimately yielding m feature embeddings.

[0058] The process of building an open-source intelligence trust classifier based on random forest technology is as follows: Figure 6 As shown, a trained random forest model is used to filter credible intelligence from the crawled intelligence and form a credible intelligence set.

[0059] An open-source intelligence credibility classifier based on random forest technology is constructed. The credibility of each intelligence report is manually determined using annotation, categorizing it into three classes: credible, untrustworthy, and undeterminable. The input features are the values ​​obtained from feature embedding for each intelligence report. A training dataset is formed based on the annotation results. The random forest model is trained using this dataset, and its performance is evaluated using a test dataset, including accuracy, precision, recall, and F1 score. After achieving the expected performance, the trained random forest model is used to filter credible intelligence from the crawled reports. The F1 score is a statistical metric used to measure the accuracy of binary (or multi-task binary) classification models.

[0060] Step S104, extraction of trusted open-source intelligence tags, includes the following implementation steps:

[0061] First, a tag evaluation model is constructed. Tags are manually labeled, and the value of each tag is calculated to obtain a tag score training set. The tag evaluation model is then established by weighting the scores based on frequency, timeliness, density, informativeness, and completeness. Figure 7 As shown, the value of each candidate tag in each intelligence is evaluated; frequency needs to meet a certain frequency threshold defined in the intelligence; timeliness is determined according to the intelligence release time; density is mainly used to evaluate the co-occurrence strength of continuous elements in new phrases, including T-distribution test, Pearson chi-square test, pointwise mutual information, and likelihood ratio; informationality needs to be based on the inverse document frequency, part-of-speech distribution, and stop word distribution of the phrase in the intelligence; completeness is evaluated based on the density of the subset and superset words of the tag phrase; the keyword set and tag score of each candidate intelligence are recorded as training data, which finally constitute the entire training dataset;

[0062] A label quality scorer based on the Bi-LSTM-Atten model is trained using the labeled training set, such as... Figure 8 As shown, the previously obtained word embedding vectors are input into the label quality scorer to obtain the score for each label;

[0063] A Bi-LSTM-Atten model-based scorer is used as the score for each label. The top k labels, representing a user-defined number, are selected based on their scores from highest to lowest. Figure 9 As shown.

[0064] The label quality scorer is self-optimizing. The labels to be scored are input into a Bi-LSTM-Atten label quality scorer model to obtain predicted scores. The predicted scores are adjusted according to user needs. The adjusted scores are then used to train the label quality scorer online. This process is repeated iteratively to optimize the label quality scorer model.

[0065] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An open source intelligence semantic trust analysis and self-optimizing label extraction method of a Bi-LSTM-Atten model, characterized in that, Comprising the following steps: Step 1, open source intelligence collection and preprocessing, specifically including: Step 1.1, determine the data source of open source intelligence; Step 1.2, establish a data source value evaluation model to evaluate the value of each item in each category of open source intelligence data source, and select the items with higher value according to the demand and put them into the list of URLs to be crawled; Step 1.3, build a distributed crawler cluster in master-slave mode, distribute crawler tasks to servers according to data source categories, and execute data crawling tasks; Step 1.4, preprocessing the crawled data, including data cleaning and text extraction; Step 2, open source intelligence feature extraction, specifically including: Step 2.1, segment the crawled text; Step 2.2, use TF-IDF algorithm and TextRank algorithm to select the candidate label set of each piece of intelligence; Step 2.3, use BERT technology for candidate label word embedding to get the word embedding vector of each candidate label; Step 3, open source intelligence semantic credibility analysis, specifically including: Step 3.1, use deep belief network to perform semantic feature embedding on each piece of intelligence, group and concatenate the word embedding vectors of the candidate labels extracted from each piece of intelligence into vectors according to content, time and domain knowledge, pretrain the deep belief network through unsupervised greedy layer-by-layer method, and get m feature embedding values; Step 3.2, build an open source intelligence credibility classifier based on random forest technology, use the trained random forest model to filter credible intelligence from the crawled intelligence, and form a credible intelligence set; Step 4, credible open source intelligence label extraction, specifically including: Step 4.1, build a label evaluation model to calculate the value of candidate labels to get a label score training set; Step 4.2, use the label score training set to train a label quality scorer based on Bi-LSTM-Atten model, input the word embedding vector obtained before into the label quality scorer to get the score of each label; Step 4.3, select the top k labels defined by the user according to the score of the label quality scorer; Step 4.4, according to the user's demand, the label quality scorer adjusts the predicted scores, and then iteratively trains the label quality scorer using the label and the adjusted score.

2. The open source intelligence semantic credibility analysis and Bi-LSTM-Atten model self-optimizing label extraction method of claim 1, wherein in step 1.2, the data source value evaluation model includes relevance evaluation, accuracy evaluation, comprehensiveness evaluation and timeliness evaluation, wherein the relevance evaluation includes strategic intelligence article fit degree and tactical intelligence article fit degree; the accuracy evaluation includes information detail and reference accuracy; the comprehensiveness evaluation includes botnet information, vulnerability information, network weapon information, attack technique information and virus information; and the timeliness evaluation includes virus information, sensitive message update frequency and emergency event release time.

3. The open source intelligence semantic credibility analysis and Bi-LSTM-Atten model self-optimizing label extraction method of claim 2, wherein ​ Step 1.3 specifically includes: in the master-slave mode, there is one master server to maintain the URL queue to be crawled, responsible for distributing the URL to be crawled to different slave servers responsible for crawling each data source, monitoring and balancing the load of the slave servers, and the slave servers are responsible for web page crawling and parsing; in the initialization stage, the URLs contained in the screened data sources are put into the seed URL queue as seed URLs, and after the crawler starts, the URL to be crawled is taken out from the head of the URL queue to be crawled, the URL is accessed and the text is parsed and stored in the database, and then the URL is put into the crawled URL queue, and the other URLs contained therein are put into the tail of the URL queue to be crawled, and the next loop is entered until the URL queue to be crawled is empty.

4. The open source intelligence semantic credible analysis and self-optimizing label extraction method of the Bi-LSTM-Atten model according to claim 3, characterized in that, Step 2.2 specifically includes: using the TF-IDF algorithm and the TextRank algorithm respectively to calculate the weight of each word segmentation of each piece of intelligence, and normalizing the weight of each word segmentation calculated by the two algorithms respectively, calculating the mean of the normalized weight of each word segmentation, arranging the mean in descending order, and selecting the first 50 word segmentations in turn to obtain the final candidate label set.

5. The open source intelligence semantic credible analysis and self-optimizing label extraction method of the Bi-LSTM-Atten model according to claim 4, characterized in that, Step 2.3 specifically includes: converting the intelligence text to which each candidate label belongs into token ID in the BERT vocabulary table, converting the input sequence added with tokenization and special tokens into the input format of the BERT model, inputting the input sequence into the pre-trained BERT model for inference, and extracting the embedding representation of each candidate label in the output of the BERT model.

6. The open source intelligence semantic credible analysis and self-optimizing label extraction method of the Bi-LSTM-Atten model according to claim 5, characterized in that, In step 3.1, the deep belief network is stacked by a plurality of restricted Boltzmann machines (RBM), and the open source intelligence training set is constructed to pre-train the network by an unsupervised greedy layer-by-layer method, wherein the RBM is trained layer by layer by using the CD algorithm, and finally m feature embedding values are obtained.

7. The open source intelligence semantic credible analysis and self-optimizing label extraction method of the Bi-LSTM-Atten model according to claim 6, characterized in that, In step 3.2, the credibility of each piece of intelligence is artificially judged by using the artificial labeling method, and is divided into credible intelligence, non-credible intelligence and undetermined intelligence, the input feature is the value obtained by the feature embedding of each piece of intelligence, the training data set is formed according to the labeling result, the random forest model is trained using the open source intelligence training set, the performance of the random forest model is evaluated using the test data set, including accuracy, precision, recall and F1 score, and after the evaluation reaches the expected performance, the credible intelligence in the crawled intelligence is screened using the trained random forest model.

8. The open source intelligence semantic trust analysis and self-optimizing label extraction method of the Bi-LSTM-Atten model according to claim 7, characterized in that, Step 4.1 specifically includes: according to the frequency, timeliness, closeness, information degree and integrity weighting calculation to establish a label evaluation model, score each candidate label in each piece of intelligence; record the keyword set of each candidate intelligence and the label score as a piece of training data, and finally form the entire label score training set.

9. The open source intelligence semantic trust analysis and self-optimizing label extraction method of the Bi-LSTM-Atten model according to claim 8, characterized in that, In step 4.3, the top k labels of the user-defined number are selected in order from high to low according to the score.

10. The open source intelligence semantic trust analysis and self-optimizing label extraction method of the Bi-LSTM-Atten model according to claim 9, characterized in that, In step 4.4, the label to be scored is input into the Bi-LSTM-Atten-based label quality scorer to obtain a predicted score, the predicted score is adjusted according to user requirements, the data after the adjustment is used to train the label quality scorer online, and the optimization is iterated cyclically.

Images