Network asset risk assessment method and system based on dark web threat intelligence analysis

Abstract

The application discloses a network asset risk assessment method and system based on dark web threat intelligence analysis, and relates to the technical field of network asset risk early warning, and comprises the following steps: obtaining dark web threat intelligence data used for assessing network asset risks to obtain a threat intelligence data set, performing Chinese text segmentation on the threat intelligence data set by using a WCA-WSA algorithm, performing NLP analysis, and obtaining internal features of the threat intelligence data; obtaining the timeliness of the threat intelligence data by using an ITAA-SE algorithm; and respectively calculating a dark web threat intelligence related asset value A and a dark web threat intelligence threat occurrence probability T by using a NARA-DW algorithm, taking the timeliness of the threat intelligence data as a dark web threat intelligence related vulnerability severity V, calculating a risk value of the threat intelligence related asset, and performing network asset risk assessment on an asset to be evaluated. The application realizes the determination of the asset risk from the perspective of an attacker by using dark web threat data.

Description

Technical Field

[0001] This invention relates to the field of network asset risk early warning technology, specifically to a network asset risk assessment method and system based on dark web threat intelligence analysis. Background Technology

[0002] With the popularization of global informatization and the continuous development of network technology, cyberspace has become the "fifth space" after land, sea, air, and space, making the construction of cyberspace security an urgent matter. However, as cyberattack methods become increasingly sophisticated, traditional passive network security defense technologies are struggling to cope with the ever-evolving forms of attacks. Active defense measures, such as network asset risk assessment and early warning, have become a research hotspot.

[0003] Network asset risk assessment and early warning utilizes acquired threat intelligence data to analyze the assets to be assessed. By considering asset vulnerabilities, related threats, and asset value, it identifies security risks and completes risk assessment and early warning. Risk assessment and early warning can determine the degree and priority of risk, enabling proactive asset defense with limited resources before an attack occurs. As a powerful means of network asset security management, its risk assessment standards have been extensively studied in related fields. The risk calculation function is generally a function of assets, threats, and vulnerabilities, expressed as:

[0004] However, current research on cyber asset risk assessment methods mainly focuses on innovation in the assessment process and risk determination, without fully considering the rationality of assessment intelligence data. Traditional risk assessment intelligence data suffers from problems such as data fragmentation and difficulty in unification, scattered sources, low timeliness and information value, difficulty in predicting actual hacker attack tendencies, and subjectivity in indicator assessment. Summary of the Invention

[0005] This invention proposes a network asset risk assessment method and system based on dark web threat intelligence analysis. Based on timely dark web threat intelligence analysis, it proposes a network asset risk assessment and early warning method from the attacker's perspective.

[0006] This invention provides a method for assessing network asset risks based on dark web threat intelligence analysis, comprising the following steps: Obtain dark web threat intelligence data for assessing network asset risks, and obtain a threat intelligence dataset; Obtain a specialized terminology segmentation dictionary and load the jieba segmentation algorithm. Dic Input the text from the threat intelligence dataset that needs to be segmented. text ,Finish text Primary word segmentation, obtaining primary word segmented text W pri ; Build and train a word vector model w2v Using word vector models w2v Using words as elements, a secondary adaptive word segmentation method is used to traverse the primary word segmentation text. The segmented text of the threat intelligence data obtained W And perform NLP analysis on it to obtain the internal characteristics of threat intelligence data; Initial intelligence clone text bilocation And obtain the segmented text of threat intelligence data. W substring set substring ; When intelligence If the string is in Chinese, then determine the substring set. substring Does the current string exist in the string? If substring set substring With the current string If the match is successful, then the current string will be... Add to clone text in sequence bilocation ; When intelligence If it is in English, then determine the substring set. substring Does the text contain segmented threat intelligence data? W The current text word If substring set substring With the current text word If a match is successful, the current text word will be... Add to clone text in sequence bilocation ; The similarity between the cloned text of threat intelligence data and its segmented text is calculated. If the similarity exceeds a specified threshold, the webpage is determined to be related to event-level intelligence, and the number of related webpages is obtained. Based on the obtained number of webpages, the network propagation degree of the threat intelligence data is calculated using the P@n algorithm, and then substituted into a custom timeliness function to obtain the timeliness of the threat intelligence data. The custom timeliness function is: in, spread For online dissemination, The function contraction coefficient; Based on the threat intelligence dataset and the internal characteristics of the threat intelligence data, the value of dark web threat intelligence-related assets is calculated respectively. A Dark web threat intelligence: probability of threat occurrence T The timeliness of threat intelligence data is used as a measure of the severity of vulnerability associated with dark web threat intelligence. V Then calculate the risk value of the assets associated with the threat intelligence: Based on the risk values ​​of assets associated with threat intelligence, conduct a cyber asset risk assessment of the assets to be evaluated; The timeliness of obtaining the threat intelligence data includes the following steps: Input the segmented text of the threat intelligence data into a search engine to obtain the result domain set. ,in l To obtain the number of web pages, initialize the relevant web page counters. num =0; Traversing the result domain set P This includes using the intelligence clone technique to obtain intelligence. clone text Through text similarity algorithms Word segmentation text for obtaining threat intelligence data W With clone text Text similarity When text similarity Greater than or equal to the threshold t ,but num = num+ 1; The network propagation degree of threat intelligence data obtained is calculated using the P@n algorithm. spread The calculation formula is as follows: The network propagation degree of threat intelligence data is substituted into a custom timeliness function to obtain the timeliness of the threat intelligence data.

[0007] Furthermore, the acquisition of dark web threat intelligence data for assessing network asset risks, resulting in a threat intelligence dataset, includes: Based on the data characteristics of the crawled websites, a custom crawler was designed using the requests, lxml, and scrapy libraries in Python to crawl and parse data from dark web websites and store dark web threat intelligence data. The crawled information includes product information such as post titles, prices, content, publishers, and purchase quantities.

[0008] Furthermore, the method of obtaining a specialized terminology segmentation dictionary includes: Threat intelligence dataset in Chinese text words Request a search engine to obtain the result domain. ; Match all of the results domain words Substring, get the set of substrings substring .

[0009] Statistical substring set substring The number of occurrences of each substring in the text is obtained by reversing the order of occurrence counts to obtain the co-occurrence ratio. nThe substring of is stored in a professional term segmentation dictionary. Dic ; The secondary adaptive word segmentation method includes: Through word vector model w2v Matching primary word segmentation text The word vectors corresponding to the words in the text are adaptively selected to determine whether the word vectors exist. If the match fails and the length of the matched word exceeds the threshold... t v Then the word will be added to the professional term segmentation dictionary. Dic The jieba word segmentation algorithm is used to perform secondary word segmentation on the words.

[0010] Furthermore, NLP analysis is performed on the segmented text of the threat intelligence data to obtain its internal features, including: The internal characteristics of threat intelligence data are obtained by clustering, classifying, fuzzy matching, and keyword extraction of the segmented threat intelligence text.

[0011] Furthermore, the value of assets associated with dark web threat intelligence is calculated separately. A Dark web threat intelligence: probability of threat occurrence T ,include: Calculate the buyer's confidence level The calculation formula is as follows: in, As a sample publisher id Number of products published; The number of products that have been published and have a purchase history; Extracting word count features Calculate intelligence confidence level The calculation formula is as follows: Extracting intelligence on product purchase volume Calculate the probability of dark web threat intelligence occurrence. T The calculation formula is as follows: in , , As the indicator weight, satisfying ; The risk function shrinkage coefficient for content word count characteristics. The risk function contraction coefficient for the purchase volume of intelligence goods; This is used to minimize the risk of control indicators; Extracting intelligence price characteristics price Calculate the value of assets associated with dark web threat intelligence. A The calculation formula is as follows: in, The risk function contraction coefficient is the characteristic of intelligence pricing.

[0012] Furthermore, the network asset information in the dark web threat intelligence database includes hardware and software information, unique identifier information, and geographic information; The aforementioned hardware and software information includes software name, operating system description, and port number; The unique identification information includes the device IP address and asset communication number; The geographical information includes geographical location and coordinates; The information of the asset to be assessed is associated with the network asset information in the dark web threat intelligence database. When the asset to be assessed has an information leakage threat and the information leakage subject of the asset to be assessed is matched in the dark web threat intelligence database, the asset to be assessed is at risk of being attacked due to the information leakage threat, and the information leakage threat of the asset to be assessed is regarded as its associated asset threat.

[0013] Furthermore, the formula for calculating the risk value of the threat intelligence-related assets is as follows: in The first of multiple related intelligence reports for the asset to be assessed i The asset risk value of the related intelligence; m The number of related intelligence; in, in For the first i The value of related assets associated with dark web threat intelligence; For the first i The probability of a threat occurring from related intelligence on the dark web; For the first i The severity of vulnerability associated with related intelligence on the dark web.

[0014] This invention also provides a network asset risk assessment system based on dark web threat intelligence analysis, including: The intelligence acquisition module is used to acquire dark web threat intelligence data for assessing network asset risks, and obtain a threat intelligence dataset; The intelligence preprocessing module obtains a specialized terminology segmentation dictionary and loads the jieba segmentation algorithm, thus loading the specialized terminology segmentation dictionary. Dic Input the text from the threat intelligence dataset that needs to be segmented. text ,Finish text Primary word segmentation, obtaining primary word segmented text W pri ; Construct and train a word vector model w2v Using word vector models w2v Using words as elements, a secondary adaptive word segmentation method is used to traverse the primary word segmentation text. The segmented text of the threat intelligence data obtained W And perform NLP analysis on it to obtain the internal characteristics of threat intelligence data; The first intelligence analysis module is used to initialize intelligence. clone text bilocation And obtain the segmented text of threat intelligence data. W substring set substring When intelligence If the string is in Chinese, then determine the substring set. substring Does the current string exist in the string? If substring set substring With the current string If the match is successful, then the current string will be... Add to clone text in sequence bilocation When intelligence If it is in English, then determine the substring set. substring Does the text contain segmented threat intelligence data? W The current text word If substring set substring With the current text word If a match is successful, the current text word will be... Add to clone text in sequence bilocation Calculate the similarity between the cloned text of the threat intelligence data and its segmented text. If the similarity exceeds a specified threshold, the webpage is determined to be related to event-level intelligence, and the number of related webpages is obtained. Based on the obtained number of webpages, the network propagation degree of the threat intelligence data is calculated using the P@n algorithm, and then substituted into a custom timeliness function to obtain the timeliness of the threat intelligence data. The custom timeliness function is: in, spread For online dissemination, The function contraction coefficient; The timeliness of obtaining the threat intelligence data includes the following steps: Input the segmented text of the threat intelligence data into a search engine to obtain the result domain set. ,in l To obtain the number of web pages, initialize the relevant web page counters. num =0; Traversing the result domain set P This includes using the intelligence clone technique to obtain intelligence. clone text Through text similarity algorithms Word segmentation text for obtaining threat intelligence data W With clone text Text similarity When text similarity Greater than or equal to the threshold t ,but num = num+ 1; The network propagation degree of threat intelligence data obtained is calculated using the P@n algorithm. spread The calculation formula is as follows: Substitute the network propagation degree of threat intelligence data into a custom timeliness function to obtain the timeliness of threat intelligence data; The second intelligence analysis module is used to calculate the value of dark web threat intelligence-related assets based on the threat intelligence dataset and the internal characteristics of the threat intelligence data. A Dark web threat intelligence: probability of threat occurrence T The timeliness of threat intelligence data is used as a measure of the severity of vulnerability associated with dark web threat intelligence. V Then calculate the risk value of the threat intelligence-related assets; The intelligence utilization module is used to conduct network asset risk assessments of assets to be evaluated based on the risk values ​​of assets associated with threat intelligence.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention addresses the problem of Chinese word segmentation in specialized fields by designing a web-based adaptive Chinese word segmentation algorithm, WCA-WSA. This algorithm utilizes the co-occurrence phenomenon of word strings and automatically obtains a dictionary of specialized terms through the co-occurrence of web substrings. It then constructs a secondary adaptive word segmentation algorithm to perform fine-grained adjustments to words, thereby improving the accuracy of Chinese threat intelligence word segmentation.

[0016] This invention addresses the problem of automatically acquiring timely intelligence by designing the ITAA-SE algorithm, which is based on a search engine. The algorithm constructs intelligence copies using search engine results, determines webpage relevance by combining text similarity, and calculates intelligence timeliness through network propagation, thus achieving automatic acquisition of threat intelligence timeliness.

[0017] This invention addresses the shortcomings of traditional network asset risk assessment methods by designing the NARA-DW algorithm, which is based on dark web threat intelligence analysis. This algorithm uses dark web characteristics as assessment indicators to calculate the asset value (A) associated with dark web threat intelligence and the probability of threat occurrence (T). Combined with the timeliness of intelligence obtained by the ITAA-SE algorithm, it constructs multi-dimensional and multi-granular dark web assessment indicators from the attacker's perspective, enabling comprehensive network asset risk assessment and early warning. By fully utilizing dark web threat data, it achieves asset risk determination from the attacker's perspective. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0019] Figure 1 This is a schematic diagram of the framework structure of the network asset risk assessment method based on dark web threat intelligence analysis provided by the present invention; Figure 2 This is a schematic diagram comparing the NARA-DW algorithm in this invention with the traditional risk assessment process; Figure 3 This is an example of the word segmentation result of the WCA-WSA algorithm of the present invention; Figure 4 This is a visualization diagram of the threat intelligence k-means clustering of the present invention; Figure 5 The function contraction coefficient of this invention A diagram showing the comparison of the effects of different values; Figure 6 This is a schematic diagram illustrating the accuracy of single-sample correlation determination in this invention. Figure 7 Confidence level of the purchaser of this invention Value distribution diagram; Figure 8 Information price features of the present invention price Buyer confidence Purchase volume of intelligence products Data distribution diagram; Figure 9 Confidence level of the purchaser of this invention Content word count characteristics Purchase volume of intelligence products A diagram illustrating the data distribution; Figure 10 Purchase volume of information products for this invention Timeliness timeliness Intelligence pricing characteristics price Data distribution diagram; Figure 11 The risk function contraction coefficient of this invention A diagram illustrating the changes in the risk function for different values; Figure 12 The risk function contraction coefficient of this invention A diagram illustrating the changes in the risk function for different values; Figure 13 The risk function contraction coefficient of this invention A diagram illustrating the changes in the risk function for different values; Figure 14 Information risk value of the present invention risk A schematic diagram of the distribution. Detailed Implementation

[0020] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. However, it should be understood that the scope of protection of the present invention is not limited to the specific implementation.

[0021] Traditional methods for assessing and warning about network asset risks suffer from fragmented data sources, low timeliness and information value, difficulty in predicting actual hacker attack tendencies, and subjective assessment indicators. Analysis has revealed that the characteristics of threat intelligence data from the dark web address these issues to some extent. However, current research in this area is still in its early stages, lacking methods that combine these two domains. Furthermore, in dark web threat intelligence analysis, there are problems with specialized Chinese word segmentation and the automatic acquisition of timely intelligence. Dark web threat intelligence primarily exists in dark web marketplaces and forums, with dark web marketplaces offering more comprehensive and valuable threat intelligence information. Therefore, this invention uses dark web marketplaces as a source of threat intelligence for research and analysis.

[0022] This invention uses dark web threat intelligence as risk assessment data for in-depth analysis and research. Through intelligence text analysis, timely acquisition of intelligence, and risk assessment of intelligence-related assets, a dark web threat intelligence database is obtained from multiple granularities and dimensions. Finally, based on the threat intelligence database, an asset-intelligence association algorithm is designed to complete the risk assessment and early warning of the assets to be assessed. This method is divided into intelligence acquisition and preprocessing, intelligence analysis, and intelligence utilization, such as... Figure 1 As shown. The innovative points of this invention are as follows:

[0023] (1) To address the problems in word segmentation of Chinese text in professional fields, a web-based Chinese adaptive word segmentation algorithm (WCA-WSA) is proposed. Compared to English text which uses spaces for automatic word segmentation, Chinese text requires a Chinese word segmentation algorithm. Due to the limitations of existing Chinese word segmentation algorithms, their accuracy is not high in professional texts with complex semantics and niche meanings. This is because professional words often express deep semantics related to the profession, which are difficult to understand through simple literal meanings, and some invention terms are niche and belong to out-of-vocabulary words. Therefore, traditional word segmentation algorithms are not effective in identifying word segmentation ambiguity and out-of-vocabulary words. To solve this problem, the model is based on web big data and combines the co-occurrence phenomenon of word strings to design the WCA-WSA algorithm.

[0024] (2) To address the problems in the automatic acquisition of intelligence timeliness, an Information Timeliness Acquisition Algorithm based on SearchEngine (ITAA-SE) is proposed. Threat intelligence on the dark web consists of semi-structured and unstructured text, making it difficult to guarantee the acquisition of indicators such as timestamps. Furthermore, in the field of cybersecurity, the time interval between intelligence releases does not account for a large proportion of its timeliness, and its timeliness cannot be reflected by conventional indicators. Therefore, a new perspective is needed to automatically determine and acquire intelligence timeliness. For an intelligence event, in the information age internet, inputting the short text of the event as a search term into a search engine and returning whether the web pages are relevant and how many are involved indirectly reflects the degree to which the event is known to the public, i.e., its online dissemination. Therefore, the ITAA-SE algorithm is constructed based on search engine result analysis to automatically acquire intelligence timeliness.

[0025] (3) To address the problems of traditional network asset risk assessment and early warning methods, a Network Asset Risk Assessment algorithm based on Dark Web threat intelligence analysis (NARA-DW) is proposed. Dark Web threat information appears earlier than surface web information, and most security incidents that occur on the surface web actually occurred on the dark web in their early stages. There are patterns in the emergence of surface and dark web threats. From the perspective of the defender, it is necessary to consider and deploy defenses against all threats and vulnerabilities of assets, which is not targeted and leads to a waste of resources. Dark Web threat data is a product of the attacker, and its attack conversion rate is high and highly targeted. Therefore, the model changes the traditional risk assessment mode of internal control. Based on the characteristics of dark web market commodity intelligence such as purchase volume, seller, and price and the threat significance they contain, combined with the timeliness of intelligence obtained by the ITAA-SE algorithm, from the attacker's perspective, a multi-dimensional and multi-granular dark web assessment index is constructed, and the NARA-DW algorithm is proposed to complete network asset risk assessment and early warning from the outside.

[0026] Example 1 The present invention provides a network asset risk assessment method based on dark web threat intelligence analysis, comprising the following steps: Step S1: Obtain dark web threat intelligence data for assessing network asset risks, and obtain a threat intelligence dataset; Step S2: Use the web-based Chinese adaptive word segmentation WCA-WSA algorithm to segment the Chinese text of the threat intelligence dataset. This includes automatically obtaining a professional terminology dictionary based on the web, loading the obtained dictionary into the jieba algorithm, and eliminating the fine-grained words in the threat intelligence dataset through secondary adaptive word segmentation to obtain the segmented text of the threat intelligence data. Perform NLP analysis on the segmented text of the threat intelligence data to obtain the internal features of the threat intelligence data. Step S3: Based on the segmented text of the threat intelligence data, the ITAA-SE algorithm, which is based on search engines, is used to automatically obtain the timeliness of the threat intelligence data. This includes extracting the text clones of the threat intelligence data from the two domains of the webpage based on the segmented text of the threat intelligence data, calculating the similarity between the text clones of the threat intelligence data and their segmented text, and determining that the webpage is related to the event-level intelligence if the similarity exceeds a specified threshold, thus obtaining the number of related webpages; based on the number of webpages obtained, the P@n algorithm is used to calculate the network propagation degree of the threat intelligence data, and this degree is then substituted into a custom timeliness function to obtain the timeliness of the threat intelligence data. Step S4: Based on the threat intelligence dataset and the internal characteristics of the threat intelligence data, and using the NARA-DW algorithm for network asset risk assessment based on dark web threat intelligence analysis, calculate the value of assets associated with dark web threat intelligence. A Dark web threat intelligence: probability of threat occurrence T The timeliness of threat intelligence data is used as a measure of the severity of vulnerability associated with dark web threat intelligence. V Then calculate the risk value of the assets associated with the threat intelligence: Step S5: Based on the risk values ​​of the assets associated with the threat intelligence, conduct a cyber asset risk assessment of the assets to be evaluated.

[0027] In step S1, dark web threat intelligence data for assessing network asset risks is obtained, resulting in a threat intelligence dataset, including: designing a custom crawler using Python's requests, lxml, and scrapy libraries based on the data characteristics of the crawled websites, completing the crawling and parsing of dark web website data, and storing the dark web threat intelligence data; the crawled information includes product intelligence release titles, prices, content, publishers, and purchase quantities.

[0028] Regarding the web-based Chinese adaptive word segmentation WCA-WSA algorithm in step S2: After a search query is entered, the search engine returns relevant web pages. These pages contain a title field and a text field. The top ten pages are selected to form a set, called the result field. For the search query, a set of substrings is matched within the result field. The number of times each element in the result field appears is called the co-occurrence of that substring. When a word exists and carries a specific semantic meaning, its appearance in relevant text indicates that the strings it contains co-occur in a specific order. This objective phenomenon does not change with different text content or context. Examples of word string co-occurrence are shown in Table 1.

[0029] Table 1 Examples of word string co-occurrence phenomena A business website 10 10 An insurance company 10 10 In subsequent NLP tasks such as clustering of intelligence Chinese texts, word segmentation is required. However, due to the characteristics of texts in fields such as cybersecurity, traditional word segmentation algorithms have low accuracy. Examples of bottlenecks are shown in Table 2.

[0030] Table 2 Examples of bottlenecks in Chinese word segmentation in professional fields Fine-grained problems Shopping website | Shopping cart | Data Shopping website | Shopping cart | Data Unregistered words recognition A business website, an insurance company A business website, an insurance company Many professional-related web pages on the internet rely heavily on text descriptions, and the co-occurrence of word strings objectively exists across the text of different web pages in both domains. When a short text containing a specific professional term is entered into a search engine, its search mechanism prioritizes displaying web pages containing the core term. Furthermore, despite differences in the content focus and relevance of related web pages, the professional term consistently matches as a substring of the input text. This co-occurrence of word strings reflects that the word's co-occurrence is among the highest in the substring set. Therefore, substrings with high co-occurrence in the result domain can be considered to represent a specific word entity in the text.

[0031] Based on the above phenomena, the WCA-WSA algorithm automatically obtains a professional term segmentation dictionary from the web. It loads the dictionary based on the jieba algorithm and eliminates the fine-grained word problem through secondary adaptive segmentation, thus completing the Chinese text segmentation.

[0032] Step S2 includes: Step S2.1: Convert the threat intelligence dataset into Chinese text. words Request a search engine to obtain the result domain. ; Match all in the result domain words Substring, get the set of substrings substring .

[0033] Step S2.2: Count the substring set substring The number of occurrences of each substring in the text is obtained by reversing the order of occurrence counts to obtain the co-occurrence ratio. n The substring of is stored in the professional term segmentation dictionary. Dic ; Step S2.3: Repeat steps S2.1 and S2.2 to complete the traversal search of the threat intelligence dataset and obtain a web-based professional term segmentation dictionary. Dic Proceed to the word segmentation step; Step S2.4: Load the jieba word segmentation algorithm and load the professional word segmentation dictionary. Dic Input the text from the threat intelligence dataset that needs to be segmented. text ,Finish text Primary word segmentation, obtaining primary word segmented text W pri ; Step S2.5: Construct and train the word vector model w2v Using word vector models w2v Using words as elements, a secondary adaptive word segmentation method is used to traverse the primary word segmentation text. Obtain Chinese threat intelligence segmented text W ; The secondary adaptive word segmentation method includes the following steps: Through word vector model w2v Matching primary word segmentation text The word vectors corresponding to the words in the text are adaptively selected to determine whether the word vectors exist. If the match fails and the length of the matched word exceeds the threshold... t v Then the word will be added to the professional term segmentation dictionary. Dic The jieba word segmentation algorithm is used to perform secondary word segmentation on the words.

[0034] In the quadratic adaptive word segmentation method, when element word vector matching fails, two cases are discussed: (1) If the particle size is too fine, it can be decomposed again.

[0035] When the WCA-WSA algorithm executes steps S2.1 and S2.2, due to the Chinese text in the threat intelligence dataset... words The relevant content in this context is widely discussed on the web, with numerous web pages describing the events it contains, leading to a proliferation of substring sets. substring Sort and store in a professional terminology segmentation dictionary Dic Word segmentation and Chinese text words The matching is too high, and its granularity is too high, so it can be further decomposed.

[0036] This situation manifests as the word length exceeding a threshold. t v At this point, the granularity is reduced through secondary word segmentation to obtain the final word segmentation result. An example of this is shown in Table 3.

[0037] Table 3 Examples of words with excessively fine-grained segmentation. Detailed census data of a certain country Population census of a certain country Population of a certain country Email data Email account password Email account password account (2) Due to the limitations of the training corpus for the word vector model, the model lacks word vectors for specialized terms.

[0038] This issue stems from a problem with the model training set, resulting in missing word vectors for specialized terms, which is related to the specialized term segmentation dictionary. Dic Irrelevant. This type of situation occurs when the word length does not exceed the threshold. t v Secondary word segmentation is not performed.

[0039] Co-occurrence degree n Parameters in the substring n The selection of the threshold needs to consider how to reduce the computational waste and impact on word segmentation results caused by noisy substrings. t v The value of needs to be considered in order to better preserve words that are accurately segmented but whose word vectors fail to match.

[0040] In step 2, NLP analysis is performed on the segmented Chinese threat intelligence text to obtain the internal features of the threat intelligence data. This includes clustering, classifying, fuzzy matching, and keyword extraction of the segmented threat intelligence text to obtain the internal features of the threat intelligence data, providing multi-granular threat intelligence information for subsequent network asset risk warning.

[0041] In step S3, regarding the ITAA-SE algorithm for automatically acquiring information timeliness based on search engines: For an intelligence event, in the interconnected internet, using the intelligence text as input to request a search engine, the relevance and quantity of the returned results indirectly reflect the degree of public awareness of the event, i.e., the intelligence network's dissemination. Traditional intelligence timeliness assessment indicators are not applicable to the field of cybersecurity, requiring new perspectives and indicators to determine intelligence timeliness. This invention, based on a big data search engine, constructs the ITAA-SE algorithm to calculate intelligence timeliness from the perspective of intelligence network dissemination. Intelligence network dissemination depends on the relevance of search engine request results; however, objectively determining whether the returned webpage content is relevant to the intelligence presents the following challenges:

[0042] Challenge 1: Returned web pages contain both title and text fields. For keyword intelligence relevance assessment, it's sufficient to determine whether the keyword appears in both fields. However, for event-level intelligence relevance assessment, whether a web page is relevant is difficult to determine through simple keyword matching.

[0043] Challenge 2: Even if the returned webpage content is relevant to the search intelligence event, there may be situations where both domains of the webpage contain excessive content related to the event. For example, if a webpage expresses opinions, discussions, or in-depth analysis of the event, this type of text can introduce a lot of noise into the computer's ability to determine whether the webpage is relevant.

[0044] To address the aforementioned research challenges, the ITAA-SE algorithm proposes an intelligence cloning method. This method extracts text clones of search intelligence from two domains of a webpage. A text similarity algorithm is used to calculate the matching degree between the clone and the original text. If the match exceeds a predetermined threshold, the webpage is determined to be related to event-level intelligence. Based on this, the P@n algorithm is used to calculate the intelligence network propagation degree. Substituting this into a custom timeliness function, the final timeliness of the intelligence is automatically obtained.

[0045] Step S3 includes the following steps: Step S3.1: Input the segmented Chinese threat intelligence text into the search engine and obtain the result domain set. ,in l To obtain the number of web pages, initialize the relevant web page counters. num =0; Step S3.2: Traverse the result domain set P This includes using the intelligence clone technique to obtain intelligence. clone text Through text similarity algorithms Obtain Chinese threat intelligence segmented text W With clone text Text similarity When text similarity Greater than or equal to the threshold t ,but num = num+ 1; The network propagation degree of threat intelligence data obtained is calculated using the P@n algorithm. spread The calculation formula is as follows: (1) Step S3.3: Substitute the network propagation degree of the threat intelligence data into the custom timeliness function to obtain the timeliness of the threat intelligence data, as follows: (2) in is the function contraction coefficient.

[0046] The intelligence cloning method in step S3.2 includes: Initial intelligence clone text bilocation ; And obtain the segmented text of threat intelligence data from both domains of the webpage. W substring set substring ; When intelligence If the string is in Chinese, then determine the substring set. substring Does the current string exist in the string? If substring set substring With the current string If the match is successful, then the current string will be... Add to clone text in sequence bilocation ; When intelligence If it is in English, then determine the substring set. substring Does the text contain segmented threat intelligence data? W The current text word If substring set substring With the current text word If a match is successful, the current text word will be... Add to clone text in sequence bilocation .

[0047] The intelligence cloning method obtains intelligence text clones based on string sequence matching, preserving relevant intelligence text content in both domains of the webpage while filtering out text noise in both domains. Combined with a text similarity algorithm, it objectively and effectively determines whether the webpage is related to the intelligence.

[0048] In the ITAA-SE algorithm, there is a text similarity algorithm. Timeliness calculation function Parameter threshold t With the number of web pages l The construction and selection of [the data] were analyzed.

[0049] (1) Number of web pages l The selection.

[0050] l The selection of data needs to consider whether it can fully include relevant web content that is currently available for intelligence events, and at the same time, it should help in the expression and calculation of network dissemination.

[0051] (2) Text similarity algorithm The selection.

[0052] Since the intelligence clone is based on the sequential matching of intelligence strings, the difference between the intelligence and clone texts lies in the difference of missing elements. For this type of text, a text similarity algorithm based on string distance is suitable for similarity calculation.

[0053] (3) Threshold t The choice.

[0054] Different text similarity algorithms will result in different effective judgment ranges. After determining the similarity algorithm, effective judgment values ​​can be selected through test text experiments. t The value of .

[0055] (4) Timeliness calculation function analyze.

[0056] Intelligence network dissemination spread This indirectly reflects its current timeliness, with a value between 0 and 1. spread The higher the value, the lower the timeliness, and vice versa. Furthermore, considering practical analysis... spread The impact on intelligence timeliness should gradually decrease as its value increases; when spread When it approaches 0, the acceleration of the timeliness increase is less than -1. spread As the value approaches 1, the rate of decrease in timeliness approaches 0. When... spread When =1, the information still retains its timeliness. Therefore, it is defined that the relevant webpage does not exist in the network, i.e. spread When =0, timeliness =1, timeliness With function Decreasing.

[0057] From the above constraints, the shrinkage coefficient can be obtained. The value should be greater than 1. The proof is as follows: (3) in, ,but ,but .

[0058] In step S4, regarding the NARA-DW algorithm for network asset risk assessment based on dark web threat intelligence analysis: Among them, the probability of threat occurrence from dark web threat intelligence. T Dark web threat intelligence indicates the likelihood of a threat occurring, reflecting the potential for related assets to be attacked by that threat; the value of assets associated with dark web threat intelligence. A The value of the assets involved in the intelligence indicates the importance of the assets threatened by that intelligence; the severity of the vulnerability associated with dark web threat intelligence is also indicated. V This indicates the severity of a vulnerability in an asset that the intelligence relates to, reflecting the extent of the impact of an attack on the asset based on that intelligence threat.

[0059] Traditional asset risk assessment methods rely on fragmented data sources, hindering the standardization of subsequent assessment criteria. Furthermore, the acquisition of assessment indicators from the defender's perspective introduces a degree of subjectivity. For example, the defender might internally value an asset at... ， This value incorporates the defender's subjective judgment; from the attacker's perspective, the asset's value is assessed as follows: This value represents the actual worth of an asset relative to the attacker, thus possessing objectivity. A piece of dark web threat intelligence contains the target of the threat, the value of the attack, and the severity of the associated vulnerability reflected in its timeliness. Through the intelligence text, the price of intelligence products, purchase volume, and other related threat information, indicators such as the value of the assets associated with the intelligence, the severity of the associated vulnerability, and the probability of the threat occurring can be indirectly assessed.

[0060] This invention analyzes intelligence features and designs multi-dimensional, multi-granular dark web evaluation indicators from the attacker's perspective, thus creating the NARA-DW algorithm from an external perspective.

[0061] In dark web marketplace forums, threat intelligence information includes features such as product titles, content, prices, purchase volume, and publishers. Textual features can be linked to threatened assets, while features like purchase volume can indicate the probability of a threat occurring on the dark web. T Dark web threat intelligence related asset value A This can be reflected through characteristics such as price.

[0062] The timeliness of threat intelligence data can reflect the severity of vulnerabilities associated with dark web threat intelligence. V Generally speaking, the lower the timeliness of intelligence, the higher the degree of public awareness of the threat on the internet, the lower the severity of the vulnerability, and vice versa.

[0063] Step S4 includes: Calculate the buyer's confidence level The calculation formula is as follows: (4) in, As a sample publisher id Number of products published; The number of products that have been published and have a purchase history; Extracting word count features Calculate intelligence confidence level The calculation formula is as follows: (5) Extracting intelligence on product purchase volume Calculate the probability of dark web threat intelligence occurrence. T The calculation formula is as follows: (6) in , , As the indicator weight, satisfying ; The risk function shrinkage coefficient for content word count characteristics. The risk function contraction coefficient for the purchase volume of intelligence goods; This is used to minimize the risk of control indicators; Extracting intelligence price characteristics price Calculate the value of assets associated with dark web threat intelligence. A The calculation formula is as follows: (7) in, The risk function contraction coefficient is the characteristic of intelligence pricing.

[0064] Analysis of the NARA-DW algorithm (1) Dark Web Threat Intelligence Threat Occurrence Probability T Analysis of the calculation formula.

[0065] Buyer confidence The design is for the number of products that have been published and have a purchase history. With the sample publisher id Number of products published The ratio. When a certain intelligence product has a purchase record, the threat feasibility and transaction value of that intelligence are reflected, and the seller's credibility increases; conversely, the seller's credibility decreases. At the same time, the buyer's confidence level... The higher the value, the less the seller's reputation will be affected when listing questionable products.

[0066] Intelligence confidence The value and the buyer's confidence level Content word count characteristics The confidence level of a dark web threat intelligence report can be assessed by the confidence level of its buyers. With content word count characteristics Obtained. Buyer's confidence level. This reflects the credibility of the intelligence source and the characteristics of the number of words in the content. This reflects the credibility of the intelligence product's content, as well as the characteristics of its word count. The larger the value, the more detailed and abundant the threat information, and the more credible the associated threats.

[0067] The value of the probability T of a threat occurring in dark web threat intelligence and the confidence level of the intelligence. Purchase volume of intelligence products Relevant. Intelligence confidence level. This reflects the authenticity of the intelligence threat and the volume of intelligence product purchases. This reflects the extent to which threats are spread among hackers and their willingness to attack. (Intelligence product purchase volume) The higher the value, the more likely hackers are to target this threat.

[0068] Weighting of indicators , , The selection can be set through prior analysis to satisfy... .

[0069] From formulas (3), (6), and (7), it can be seen that... A , T , V The values ​​are all between 0 and 1.

[0070] Risk function contraction coefficient , , This determines the convergence speed of the corresponding indicator value. The risk posed by such indicators increases with their value, and convergence begins at a certain threshold. For example, define when... Values ​​greater than 1000 indicate a high-risk impact. The contraction coefficient can be determined based on the data distribution of relevant characteristics in actual dark web threat intelligence datasets.

[0071] The NARA-DW algorithm in this invention is compared with traditional risk assessment methods. Figure 2 As shown.

[0072] In step S1: The network asset information in the dark web threat intelligence database includes software and hardware information, unique identifier information, and geographic information; wherein the software and hardware information includes software name, operating system description, and port number; the unique identifier information includes device IP address and asset communication number; the geographic information includes geographical location and coordinates; the information of the asset to be assessed is associated with the network asset information in the dark web threat intelligence database; when the asset to be assessed has an information leakage threat, and the information leakage subject of the asset to be assessed is matched in the dark web threat intelligence database, then the asset to be assessed has the risk of being attacked due to the information leakage threat, and the information leakage threat of the asset to be assessed is regarded as its associated asset threat.

[0073] When there are multiple related intelligence reports about an asset to be assessed, the formula for calculating the risk value of the related assets is as follows: (8) in The first of multiple related intelligence reports for the asset to be assessed i Risk values ​​of related assets; m The number of related intelligence; in, (9) in For the first i The value of related assets associated with dark web threat intelligence; For the first i The probability of a threat occurring from related intelligence on the dark web; For the first i The severity of vulnerability associated with related intelligence on the dark web.

[0074] Example 2 This invention also provides a network asset risk assessment system based on dark web threat intelligence analysis, comprising: The intelligence acquisition module is used to acquire dark web threat intelligence data for assessing network asset risks, and obtain a threat intelligence dataset; The intelligence preprocessing module obtains a specialized terminology dictionary and adaptively adjusts and eliminates the fine-grained words in the threat intelligence dataset to obtain the segmented text of the threat intelligence data. Then, it performs NLP analysis on the segmented text to obtain the internal features of the threat intelligence data. The first intelligence analysis module is used to extract the text clones of the threat intelligence data from the two domains of the webpage based on the segmented text of the threat intelligence data, calculate the similarity between the text clones of the threat intelligence data and their segmented text, and determine that the webpage is related to the event-level intelligence if the similarity exceeds a specified threshold, thereby obtaining the number of related webpages; based on the number of webpages obtained, the network propagation degree of the threat intelligence data is calculated using the P@n algorithm, and then substituted into a custom timeliness function to obtain the timeliness of the threat intelligence data; The second intelligence analysis module is used to calculate the value of dark web threat intelligence-related assets based on the threat intelligence dataset and the internal characteristics of the threat intelligence data. A Dark web threat intelligence: probability of threat occurrence T The timeliness of threat intelligence data is used as a measure of the severity of vulnerability associated with dark web threat intelligence. V Then calculate the risk value of the threat intelligence-related assets; The intelligence utilization module is used to conduct network asset risk assessments of assets to be evaluated based on the risk values ​​of assets associated with threat intelligence.

[0075] The present invention will now be described in detail with reference to specific embodiments.

[0076] This embodiment accesses real dark web websites and obtains a dark web threat intelligence database through the intelligence acquisition and analysis modules. In the intelligence utilization module, an assessment asset dataset is loaded to complete the asset risk assessment and early warning. Through result analysis of each stage and comparative experiments, the model's effectiveness and value are demonstrated.

[0077] 1. Dark Web Threat Intelligence Acquisition To facilitate the verification of the WCA-WSA algorithm's effectiveness, an active Chinese dark web trading forum was selected as the threat data source. This forum has facilitated several transactions that subsequently led to significant cybersecurity incidents, making it of significant research value. The experiment constructed a dark web access environment using the model's dark web threat intelligence acquisition and preprocessing module. A custom crawler was built based on the website's content to obtain threat intelligence data. The data was preprocessed using libraries such as pandas to obtain a dark web threat intelligence dataset. The dataset contains 8225 samples, covering threat product intelligence published on the website from October 2018 to October 2021. Examples of some sample features are shown in Table 4.

[0078] Table 4. Examples of partial features from a threat intelligence dataset sample Price 79 Content A certain bank in a certain region has released data on 150,000 new customers who purchased wealth management products this year, representing its high-quality wealth management clients… Publisher frequency 586118 Quantity 10 2. Text segmentation based on WCA-WSA algorithm The WCA-WSA algorithm was used to segment the product titles and content text of the intelligence dataset, providing a foundation for subsequent intelligence NLP analysis. The experiment used product title features from the threat intelligence dataset as the WCA-WSA corpus to automatically obtain a dictionary of specialized terms. Dic It uses Tencent AI Lab's open-source word vector model as the w2v model. Considering practical factors, the co-occurrence rate is among the top... n parameters of the substring n The value is 2, the threshold. t v The value is 5. Example of the algorithm's word segmentation results. Figure 3 As shown.

[0079] To verify the effectiveness of the WCA-WSA algorithm, jieba, jieba+Dic, and WCA-WSA-no were used respectively. The algorithm performs word segmentation on the same text set. Table 5 shows the differences in performance when the text is segmented using the four methods described above.

[0080] Table 5 Examples of differences in word segmentation effects As shown in Table (5), adding the automatically acquired professional terminology segmentation dictionary... Dic Subsequently, the latter three algorithms could identify important entities such as "blockchain," while the jieba algorithm failed to do so, which will significantly impact subsequent NLP tasks. Meanwhile, Algorithm 2 suffers from overly granular word segmentation, producing terms like "a company's email account." The latter three algorithms, based on actual word vector models, scientifically reduced the granularity of the words. For specialized entities like "a recruitment website," due to limitations in the word vector training corpus, the word vector model lacks word vectors for this term, and Algorithm 3 did not set a threshold. t v The decision to perform secondary word segmentation resulted in a decrease in fine-grainedness. The WCA-WSA algorithm considers all of the above situations and correctly completes the word segmentation.

[0081] In summary, the WCA-WSA algorithm effectively addresses some of the problems existing in current professional fields of Chinese word segmentation, demonstrating its effectiveness and value.

[0082] 3. Threat intelligence database acquisition 3.1 Intelligence Clustering Based on WCA-WSA This invention categorizes dark web threat intelligence into four types: information leakage, hacking techniques, malware, and other threats. Due to the limited availability of dark web datasets and the difficulty in collecting relevant data, supervised algorithm analysis is not suitable. Therefore, the experiment uses the intelligence title text, obtains word vectors through Tencent's open-source word vector model, takes the average of the sentence word vectors as the text representation, and uses the k-means algorithm for clustering to obtain intelligence label classifications. Four clusters are selected, and the TSNE algorithm is used to reduce the text representation vector to 2 dimensions, enabling cluster visualization. The visualization is shown below. Figure 4 As shown.

[0083] By manually analyzing the cluster labels and mapping them to the actual intelligence classification categories, threat category features of the threat intelligence dataset are obtained.

[0084] 3.1. Information Timeliness Acquisition Based on ITAA-SE Algorithm The ITAA-SE algorithm was used to traverse the intelligence dataset and obtain the timeliness of all intelligence. The short text titles of dark web intelligence products possess high intelligence threat generalization; therefore, these titles were used as input for the ITAA-SE algorithm's web search.

[0085] A certain company is currently the most comprehensive search engine covering Chinese web pages, and it is used as the search source for the ITAA-SE algorithm. The search engine retrieval algorithm returns web pages in reverse order based on their comprehensive score for the search query. As the ranking decreases, the web pages gradually lose value. Considering the operation of formula (2), the number of web pages is taken. l The value is 10.

[0086] Since the similarity matching object is Chinese text, based on the intelligence doppelganger characteristic of the ITAA-SE algorithm, the ratio algorithm is selected as the most suitable algorithm. The ratio algorithm is commonly used for text similarity measurement and is one of the most frequently used text similarity algorithms, focusing on the differences between texts. Therefore, The formula is:

[0087] (10) Among them, the denominator H The sum of the lengths of the two strings before deduplication; numerator M The length of the overlapping strings.

[0088] Based on the test text comparison experiment, the threshold t The value is 0.82.

[0089] For the function contraction coefficient The value of needs to be considered spread When =1, its timeliness is determined by the minimum value of timeliness. Different values ​​have the following effects: Figure 5As shown, based on experience, the minimum timeliness value is set at around 0.15. Therefore, The value is 2.

[0090] During the intelligence network dissemination acquisition phase, 10 representative intelligence samples (100 web pages) were selected, and the relevance of the web pages was determined using both the text matching method and the ITAA-SE intelligence doppelganger method. The text matching method's determination rule is to iterate through the web page set and calculate the similarity between it and the input text. If any result exceeds a threshold, the web page is considered relevant. The similarity algorithm and threshold selection are consistent with ITAA-SE.

[0091] In this experiment, result=1 indicates that the webpage is considered relevant, and result=0 indicates that it is irrelevant. Relevance determination is essentially a binary classification, with accuracy as the determining factor. Precision Recall rate Recall , F 1 -score As an evaluation index of algorithm results, the index calculation formula is shown in equations (11)-(13).

[0092] (11) (12) (13) in, P , R , F 1 Accuracy Precision Recall rate Recall , F 1 -score Evaluation indicators; a Indicates that it is predicted to be a Class and actually belong to a The number of classes; b Indicates that it is predicted to be a The class is actually non a The number of classes; c Indicates that it is predicted to be non- a The class is actually a The number of classes.

[0093] The algorithm performance comparison is shown in Table 6.

[0094] Table 6 Comparison of Webpage Relevance Judgment Algorithms As shown in Table 6: Intelligence clone technique F 1 -score, Recall It exhibits significant advantages in terms of metrics and demonstrates excellent judgment performance, with its accuracy-average value far exceeding that of the text matching method. The text matching method, however, achieves a significantly higher accuracy rate when result=0. Precision While the algorithm outperforms the ITAA-SE algorithm, its precision for result=1 is only 0.39, indicating that the algorithm's results are not valid. Analysis of web pages with incorrect relevance determinations revealed that the main reasons for the errors were: 1) low-quality input information text containing noise from modifiers and quantifiers.

[0095] The function is based on string matching and does not consider keyword weight, resulting in high text matching scores even when the content is irrelevant, thus exceeding the threshold.

[0096] The intelligence contained multiple threats, such as content involving both a cheat program and certain data, which interfered with search results and text matching accuracy. Figure 6 The accuracy of the algorithm for each sample (10 returned web pages) is displayed. Figure 6 In the diagram, line L1 represents the accuracy of determining webpage relevance using the intelligence cloning method on the selected samples, and line L2 represents the accuracy of determining webpage relevance using the text matching method on the selected samples. Figure 6 It can be seen that, for determining the relevance of similar web pages returned by a certain intelligence, the intelligence clone method is generally more effective than the text matching method. Figure 6 In the analysis, samples 3 and 10 showed anomalies, which were attributed to the randomness of the relevant web pages. Specifically, when acquiring intelligence clones, the intelligence string had fewer elements than the string with more elements, resulting in high text matching but irrelevant web page content, leading to incorrect relevance assessments. This problem worsened with the increasing number of web pages. l As the value increases, the impact of this problem gradually decreases. Comparative analysis proves that the intelligence clone method is effective.

[0097] After obtaining the intelligence network's propagation extent, it was constructed through ITAA-SE. time ( spread The timeliness of the data is obtained using the function, and the results are shown in Table 7.

[0098] Table 7 Examples of Timely Intelligence Acquisition As shown in Table 7, the closer the keywords are to the information disseminated on the Table Network, the lower its timeliness. Even with complex and detailed text, information events can still be acquired in a timely manner. Combined with manual analysis, Samples 1 and 2 in the table pertain to daily life and have been widely discussed on the Table Network, with numerous related web pages. Sample 3 pertains to information leakage in a professional field; the Table Network has reported on related leakage incidents, indicating some dissemination and a certain degree of timeliness. Samples 4 and 5 pertain to hacking techniques, involving relatively small and sensitive areas, with low circulation on the Table Network, but a small number of related web pages exist, indicating relatively high timeliness. Sample 6 pertains to malicious software; the content is not circulated on the Table Network. If no cybersecurity incidents caused by this software occur temporarily, the intelligence value is high and its timeliness is strong. The experiment demonstrates that the timeliness of intelligence automatically acquired through the ITAA-SE algorithm is consistent with reality and is effective.

[0099] 4. Risk assessment of intelligence-related assets based on NARA-DW After obtaining the timeliness of the intelligence dataset, the indicators required for NARA-DW risk assessment are analyzed and acquired. The NARA-DW algorithm is then used to complete the risk assessment of assets associated with the intelligence dataset. Sample publishers corresponding to the threat intelligence dataset are then extracted. id Number of products published The number of products that have been published and have purchase records Content word count characteristics Purchase volume of intelligence products Intelligence pricing characteristics price With timeliness timeliness The characteristics are calculated using formulas (5)-(8) to obtain the values ​​of each indicator, and the distribution of the characteristics and indicator data is analyzed.

[0100] Figure 7 This demonstrates the buyer's confidence level. The value distribution shows that values ​​above 0.8 indicate high-threat hackers. The presence of hackers on this website who have posted more than 20 threat intelligence reports and possess high credibility suggests the website is worthy of research. Furthermore, this type of metric can be used in the field of identifying key hackers.

[0101] Figure 8 This demonstrates the intelligence pricing characteristics of intelligence indicators. price Buyer confidence Sample publisher id Number of products published Data distribution relationship, with darker colored points representing... A value of 0 indicates a lighter color. Not equal to 0. (By) Figure 8 It can be seen that the buyer's confidence level The higher the value, the more likely it is to be published by the sample publisher. id Number of products published Overall, the trend is upward; the more valuable the intelligence product, the higher the likelihood of a threat occurring. Simultaneously, the buyer's confidence level... Features of intelligence pricing price High intelligence, and buyer confidence. Low, intelligence price characteristics price The difference in purchase success rate is significant with higher intelligence; the former still has purchase records, indicating higher confidence among buyers. The released intelligence is highly credible and comprehensively threatening. Based on the above analysis, the confidence level of buyers of algorithmic metrics... The effectiveness of this has been proven, and the characteristics of intelligence pricing have been demonstrated. price It can be used for the valuation of related assets.

[0102] Figure 9 Demonstrates the characteristics of intelligence pricing price Content word count characteristics Purchase volume of intelligence products Data distribution. (By...) Figure 9 It can be seen that when the confidence level of intelligence buyers... After reaching a certain level, the content word count characteristic The value range gradually increases, and the success rate of product purchases is high. This indicates that the higher the credibility of the intelligence, the higher the word count characteristic of the content. A wider value range allows for better analysis of the probability of threats occurring on the dark web. T The assessment.

[0103] Figure 10 Displays the volume of goods purchased based on key indicators. Timeliness timeliness Intelligence pricing characteristics price Data distribution. (By...) Figure 10 It can be seen that the characteristics of intelligence pricing price Overall, with timeliness timeliness It increases as it rises. At the same time, timeliness... timeliness The higher the level, the greater the purchase volume of intelligence products. As the value range widens, the intelligence becomes more valuable for trading. This aligns with actual patterns and demonstrates the importance of timely acquisition. timeliness The effectiveness of the NARA-DW algorithm is thus proven.

[0104] Based on prior knowledge, the weight vector The values ​​[0.6, 0.3, 0.1] are assigned to minimize the risk of the control indicator. The value is set to 0.2. The risk function contraction coefficient is determined based on the actual characteristic data distribution of the dark web. , , Values.

[0105] Combination Figure 8 , 9 Related feature distribution, including content word count features The high-risk threshold is set at 4000; the purchase volume of intelligence products will be... The high-risk threshold is set at 4000; the intelligence price characteristic is... price The high-risk threshold is set at 100,000. Figure 11 , Figure 12 and Figure 13 The risk function contraction coefficients are shown respectively. , , The different values ​​correspond to changes in the risk function. Ultimately, The value is 1000. The value is 1000. The value is 25000.

[0106] Substituting the parameter values, the risk value of intelligence-related assets is calculated using formula (1). risk The value is between 0 and 1, and the experimental threat intelligence database acquisition is complete. The result distribution is as follows: Figure 14 As shown, the darker color is risk Threat intelligence with a value >0.064. Examples of intelligence risk indicator assessments are shown in Table 8.

[0107] Table 8 Examples of Intelligence Risk Assessment Hacker training toolkit 0.078 0.181 0.432 1.000 Selling antivirus evasion viruses 0.025 0.183 0.206 0.670 Selling user account passwords for a certain country / region 0.004 0.182 0.149 0.165 Table 8 shows an example of threat intelligence assessment in the cyber asset domain. Regarding the asset value (A) indicator for dark web threat intelligence, threats 1, 2, and 3 have essentially the same value in the dark web, belonging to the same level of intelligence. Regarding the probability of occurrence (T) indicator for dark web threat intelligence, considering multiple intelligence indicators, hacker techniques have a higher probability of occurrence compared to single-asset viruses and important assets on the surface web of equivalent value. Regarding the vulnerability severity (V) indicator for dark web threat intelligence, hacker techniques are not circulated on the surface web, asset viruses have already occurred in related network incidents and have a certain degree of network propagation, and the asset data involved in threat 3 has already received attention on the surface web. In summary, the risk values ​​of threats 1, 2, and 3 decrease sequentially, consistent with actual threat situations. Therefore, the effectiveness of the NARA-DW algorithm is proven. Furthermore, this algorithm can be used for risk assessment of threat intelligence in any dark web market, providing technical support for dark web-related research and possessing high value.

[0108] 5. Network asset risk assessment and early warning The collected and organized dataset of assets to be assessed will be loaded into the overall risk assessment value. The calculation formula is shown in Table 9, and the sample asset dataset is also shown. Based on the matching intelligence information returned by the algorithm's rel_list, the overall risk assessment value is obtained through formula (8). Complete risk assessment and early warning.

[0109] Table 9 Sample Examples of Asset Datasets A certain communication software Table 10 shows examples of asset risk assessment and early warning results. Detailed information on related intelligence can be viewed through the threat intelligence database index in rel_list; Table 11 shows examples of threat intelligence database samples.

[0110] Table 10 Asset Risk Assessment Early Warning Results Table 11 Sample examples of threat database 13 Prior to 2019, a certain communication software's 16GB of data was deduplicated, removing most duplicates. This batch of data belongs to old data from a certain communication software, mostly from 2018 and 2019. The credential stuffing efficiency is generally low; it's collected from credential stuffing over the past two years. I basically... 0.07103 Data breach After manual comparison and verification, the assessed assets were found to be well matched with relevant threat intelligence, and the overall quantitative risk and early warning were automatically obtained.

[0111] Based on the proposed model, this experiment studies an active Chinese darknet market. First, an intelligence gathering and analysis module obtains a darknet threat intelligence database. Then, in the intelligence utilization module, an assessment asset dataset is loaded, ultimately achieving risk assessment and early warning for assets. Experimental results and analysis demonstrate that the proposed model can comprehensively analyze darknet threat intelligence and complete network asset risk assessment and early warning based on the darknet threat intelligence database.

[0112] Finally, it should be noted that the above-disclosed embodiment is only one specific embodiment of the present invention. However, the embodiments of the present invention are not limited thereto, and any variations that can be conceived by those skilled in the art should fall within the protection scope of the present invention.

Claims

1. A method for network asset risk assessment based on dark web threat intelligence analysis, characterized in that, Includes the following steps: Obtain dark web threat intelligence data for assessing network asset risks, and obtain a threat intelligence dataset; Obtain a professional word segmentation dictionary, and load the jieba word segmentation algorithm, load the professional word segmentation dictionary Dic , input the text in the threat intelligence data set that needs to be segmented text , complete text primary segmentation, obtain the primary segmented text W pri ; Build and train a word vector model w2v Using word vector models w2v Using words as elements, a secondary adaptive word segmentation method is used to traverse the primary word segmentation text. The segmented text of the threat intelligence data obtained W And perform NLP analysis on it to obtain the internal characteristics of threat intelligence data; Initialization information of the avatar text bilocation ; And obtain the threat intelligence data of the segmented text W The substring set substring ; When the information is Chinese, it is determined whether the current substring set substring contains the current substring ; If the substring set substring matches the current string , the current string is sequentially added to the clone text bilocation ; When intelligence If it is in English, then determine the substring set. substring Does the text contain segmented threat intelligence data? W The current text word If the substring set substring With the current text word If a match is successful, the current text word will be... Add to clone text in sequence bilocation ; The similarity between the cloned text of threat intelligence data and its segmented text is calculated. If the similarity exceeds a specified threshold, the webpage is determined to be related to event-level intelligence, and the number of related webpages is obtained. Based on the obtained number of webpages, the network propagation degree of the threat intelligence data is calculated using the P@n algorithm, and then substituted into a custom timeliness function to obtain the timeliness of the threat intelligence data. The custom timeliness function is: in, spread For online dissemination, The function contraction coefficient; Based on the threat intelligence dataset and the internal characteristics of the threat intelligence data, the value of dark web threat intelligence-related assets is calculated respectively. A Dark web threat intelligence: probability of threat occurrence T The timeliness of threat intelligence data is used as a measure of the severity of vulnerability associated with dark web threat intelligence. V Then calculate the risk value of the assets associated with the threat intelligence: Based on the risk values ​​of assets associated with threat intelligence, conduct a cyber asset risk assessment of the assets to be evaluated; The timeliness of obtaining the threat intelligence data includes the following steps: Input the segmented text of the threat intelligence data into a search engine to obtain the result domain set. ,in l To obtain the number of web pages, initialize the relevant web page counters. num =0; Traversing the result domain set P This includes using the intelligence clone technique to obtain intelligence. clone text Through text similarity algorithms Word segmentation text for obtaining threat intelligence data W With clone text Text similarity When text similarity Greater than or equal to the threshold t ,but num = num+ 1; The network propagation degree of threat intelligence data obtained is calculated using the P@n algorithm. spread The calculation formula is as follows: The network propagation degree of threat intelligence data is substituted into a custom timeliness function to obtain the timeliness of the threat intelligence data.

2. The network asset risk assessment method based on dark web threat intelligence analysis according to claim 1, characterized in that: The acquisition of dark web threat intelligence data for assessing network asset risks results in a threat intelligence dataset, including: Based on the data characteristics of the crawled websites, a custom crawler was designed using the requests, lxml, and scrapy libraries in Python to crawl and parse data from dark web websites and store dark web threat intelligence data. The crawled information includes product information such as post titles, prices, content, publishers, and purchase quantities.

3. The network asset risk assessment method based on dark web threat intelligence analysis according to claim 1, characterized in that: The terminology dictionary obtained includes: Threat intelligence dataset in Chinese text words Request a search engine to obtain the result domain. ; Match all of the results domain words Substring, get the set of substrings substring ; Statistical substring set substring The number of occurrences of each substring in the text is obtained by reversing the order of occurrence counts to obtain the co-occurrence ratio. n The substring of is stored in a professional term segmentation dictionary. Dic ; The secondary adaptive word segmentation method includes: Through word vector model w2v Matching primary word segmentation text The word vectors corresponding to the words in the text are adaptively selected to determine whether the word vectors exist. If the match fails and the length of the matched word exceeds the threshold... t v Then the word will be added to the professional term segmentation dictionary. Dic The jieba word segmentation algorithm is used to perform secondary word segmentation on the words.

4. The network asset risk assessment method based on dark web threat intelligence analysis according to claim 3, characterized in that: The segmented text of the threat intelligence data is subjected to NLP analysis to obtain the internal features of the threat intelligence data, including: Clustering, classification, fuzzy matching, and keyword extraction are performed on the segmented threat intelligence text to obtain the internal characteristics of the threat intelligence data.

5. The network asset risk assessment method based on dark web threat intelligence analysis according to claim 1, characterized in that: The value of assets associated with dark web threat intelligence is calculated separately. A Dark web threat intelligence: probability of threat occurrence T ,include: Calculate the buyer's confidence level The calculation formula is as follows: in, As a sample publisher id Number of products published; The number of products that have been published and have a purchase history; Extracting word count features Calculate intelligence confidence level The calculation formula is as follows: Extracting intelligence on product purchase volume Calculate the probability of threat occurrence from dark web threat intelligence. T The calculation formula is as follows: in , , As the indicator weight, satisfying ; The risk function shrinkage coefficient for content word count characteristics. The risk function contraction coefficient for the purchase volume of intelligence goods; This is used to minimize the risk of control indicators; Extracting intelligence price characteristics price Calculate the value of assets associated with dark web threat intelligence. A The calculation formula is as follows: in, The risk function contraction coefficient is the characteristic of intelligence pricing.

6. The network asset risk assessment method based on dark web threat intelligence analysis according to claim 2, characterized in that: The network asset information in the dark web threat intelligence database includes hardware and software information, unique identifier information, and geographic information; The aforementioned hardware and software information includes software name, operating system description, and port number; The unique identification information includes the device IP address and asset communication number; The geographical information includes geographical location and coordinates; The information of the assets to be evaluated is associated with the network asset information in the dark web threat intelligence database; When the asset to be assessed is threatened with information leakage, and the entity responsible for the information leakage of the asset to be assessed is matched in the dark web threat intelligence database, then the asset to be assessed is at risk of being attacked due to the information leakage threat, and the information leakage threat of the asset to be assessed is considered as a threat to its associated assets.

7. The network asset risk assessment method based on dark web threat intelligence analysis according to claim 6, characterized in that: The formula for calculating the risk value of the threat intelligence-related assets is as follows: in The first of multiple related intelligence reports for the asset to be assessed i The asset risk value of the related intelligence; m The number of related intelligence; in, in For the first i The value of related assets associated with dark web threat intelligence; For the first i The probability of a threat occurring from related intelligence on the dark web; For the first i The severity of vulnerability associated with related intelligence on the dark web.

8. A network asset risk assessment system based on dark web threat intelligence analysis, characterized in that: include: The intelligence acquisition module is used to acquire dark web threat intelligence data for assessing network asset risks, and obtain a threat intelligence dataset; The intelligence preprocessing module obtains a specialized terminology segmentation dictionary and loads the jieba segmentation algorithm, thus loading the specialized terminology segmentation dictionary. Dic Input the text from the threat intelligence dataset that needs to be segmented. text ,Finish text Primary word segmentation, obtaining primary word segmented text W pri ; Construct and train a word vector model w2v Using word vector models w2v Using words as elements, a secondary adaptive word segmentation method is used to traverse the primary word segmentation text. The segmented text of the threat intelligence data obtained W And perform NLP analysis on it to obtain the internal characteristics of threat intelligence data; The first intelligence analysis module is used to initialize intelligence. clone text bilocation ; And obtain the segmented text of threat intelligence data. W substring set substring When intelligence If the string is in Chinese, then determine the substring set. substring Does the current string exist in the string? ; If substring set substring With the current string If the match is successful, then the current string will be... Add to clone text in sequence bilocation When intelligence If it is in English, then determine the substring set. substring Does the text contain segmented threat intelligence data? W The current text word If the substring set substring With the current text word If a match is successful, the current text word will be... Add to clone text in sequence bilocation ; The similarity between the cloned text of threat intelligence data and its segmented text is calculated. If the similarity exceeds a specified threshold, the webpage is determined to be related to event-level intelligence, and the number of related webpages is obtained. Based on the obtained number of webpages, the network propagation degree of the threat intelligence data is calculated using the P@n algorithm, and then substituted into a custom timeliness function to obtain the timeliness of the threat intelligence data. The custom timeliness function is: in, spread For online dissemination, The function contraction coefficient; The timeliness of obtaining the threat intelligence data includes the following steps: Input the segmented text of the threat intelligence data into a search engine to obtain the result domain set. ,in l To obtain the number of web pages, initialize the relevant web page counters. num =0; Traversing the result domain set P This includes using the intelligence clone technique to obtain intelligence. clone text Through text similarity algorithms Word segmentation text for obtaining threat intelligence data W With clone text Text similarity When text similarity Greater than or equal to the threshold t ,but num = num+ 1; The network propagation degree of threat intelligence data obtained is calculated using the P@n algorithm. spread The calculation formula is as follows: Substitute the network propagation degree of threat intelligence data into a custom timeliness function to obtain the timeliness of threat intelligence data; The second intelligence analysis module is used to calculate the value of dark web threat intelligence-related assets based on the threat intelligence dataset and the internal characteristics of the threat intelligence data. A Dark web threat intelligence: probability of threat occurrence T The timeliness of threat intelligence data is used as a measure of the severity of vulnerability associated with dark web threat intelligence. V Then calculate the risk value of the threat intelligence-related assets; The intelligence utilization module is used to conduct network asset risk assessments of assets to be evaluated based on the risk values ​​of assets associated with threat intelligence.

Images