Dark web detection system

Abstract

The invention provides a dark network detection system comprising an information crawling module used for accessing a Tor anonymous network and crawling dynamic content; the information analysis module is used for carrying out preprocessing, entity identification, theme classification, association analysis and multi-modal fusion analysis on the dynamic content and generating structured data; the data storage module is used for constructing a database and storing the structured data into the database; the result display module is used for displaying the analysis result in a visual mode; the information crawling module can accurately access a dark web and crawl related dynamic content, then the information analysis module can perform multi-level analysis processing based on the dynamic content, the problem of strong confusion of dark web information is effectively solved, the obtained structured data can be stored, and visual display and retrieval are provided, so that the information retrieval efficiency is improved. Automatic and deep crawling and high-precision and multi-mode intelligent analysis of dark web information are realized, and powerful technical support is provided for dark web information management.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of dark web detection, and in particular to a dark web detection system. BACKGROUND

[0002] With the deep development of the Internet, users have a basic demand for anonymous communication, aiming to effectively hide identity information in network activities. The anonymous communication technology represented by Tor (The Onion Router) provides users with high anonymity and privacy protection through protocol camouflage, traffic confusion and multi-layer encryption, effectively resisting traffic analysis and identity tracking. However, it is this difficult to trace feature that makes the dark web full of data leakage, illegal trading and other information. These illegal information quickly spread due to their low-cost transmission characteristics, posing a persistent threat to social governance and security.

[0003] Currently, the monitoring means for the dark web faces many challenges, making information governance difficult. First, it is difficult to access the dark web network environment. Second, the links of the dark web are time-varying, often invalid or updated, making it difficult to continuously track. In addition, many dark web pages use dynamic rendering technology (such as JavaScript) to generate content, and traditional static crawlers cannot effectively extract complete data. In addition, the challenge of the dark web to the information analysis level is particularly prominent: strong content confusion: dark web information often has strong confusion, which is difficult to accurately judge from the text content itself, and needs to be analyzed in combination with user, context and other multi-dimensional information. Low recognition accuracy: traditional monitoring methods, such as manual screening or single-feature machine learning methods, can identify some information, but have problems such as high time and labor costs, low recognition accuracy, etc. In addition, dark web content is presented in multiple modalities such as text and pictures, and mixed with multiple languages such as Chinese, English and Russian, which puts higher requirements on analysis technology. The spread structure of illegal information in the dark web has particularity, spreads rapidly, and the negative impact will rapidly expand in a short time, so there is an urgent need for early detection capability of monitoring technology. In summary, the existing technology has obvious shortcomings in dealing with the anonymity, dynamics, content complexity, etc. of the dark web, and there is an urgent need for a dark web monitoring technology that can realize automatic, efficient crawling and intelligent, high-precision analysis of dark web information, to effectively support the content governance work in cyberspace. SUMMARY

[0004] In view of this, the present application proposes a dark web detection system, which can realize automatic, deep crawling and high-precision, multi-modal intelligent analysis of dark web information, thereby providing strong technical support for dark web information governance.

[0005] The technical solution of the present application is as follows: A dark web detection system, comprising: an information crawling module configured to access a Tor anonymous network and crawl dynamic content; an information analysis module configured to preprocess, entity recognition, topic classification, correlation analysis and multi-modal fusion analysis of the dynamic content, and generate structured data; a data storage module configured to build a database and store the structured data into the database; a result display module configured to display the analysis result in a visualized manner; The information crawling module, the information analysis module, the data storage module and the result display module are sequentially connected.

[0006] Preferably, the specific steps of accessing the Tor anonymous network by the information crawling module are as follows: Integrating a Tor client into the information crawling module, configuring a mixed proxy and a dynamic path selection strategy to access the Tor anonymous network, and simultaneously deploying a crawler environment by using a container isolation technology.

[0007] Preferably, the specific steps of crawling the dynamic content by the information crawling module are as follows: Accessing a dark web link directory website, searching and collecting.onion addresses of target dark web communities or markets as starting points for crawling; Integrating a headless browser into the information crawling module, the headless browser simulates user behavior, executes JavaScript scripts in pages from the starting points for crawling, and parses and obtains complete page content generated by dynamic rendering technology; Using a depth-first crawling strategy to recursively crawl sites of the target dark web based on the complete page content, and obtain the dynamic content.

[0008] Preferably, the specific steps of preprocessing the dynamic content by the information analysis module are as follows: Extracting text content from the dynamic content and performing cleaning and structured analysis; Extracting image content from the dynamic content, and using super-resolution reconstruction, denoising convolution network and contrast limited adaptive histogram equalization technology to preprocess the image content.

[0009] Preferably, the specific steps of entity recognition of the dynamic content by the information analysis module are as follows: Inputting the text content into a predefined entity type and an initialized keyword dictionary for parallel processing; Performing rule matching on the text content by using a regular expression, recognizing predefined entities with a specific format, using a jieba word segmentation tool to segment the text content, matching specific patterns in the keyword dictionary in the segmentation result, and obtaining entity information; Context information around the extracted entity information is extracted and deduplication processing is performed.

[0010] Preferably, the specific steps of the information analysis module for subject classification of dynamic content are: Based on the pre-defined keyword library and weight rules, the text content is subject classified.

[0011] Preferably, the specific steps of the information analysis module for correlation analysis are: TF-IDF is used to vectorize the text content, and the cosine similarity of the vectorized text content is calculated, and the calculation formula is: ; Where A and B are two vectorized text contents, is the cosine similarity, and the closer the cosine similarity is to 1, the higher the similarity of the two text contents.

[0012] Preferably, the specific steps of the information analysis module for multi-modal fusion analysis of dynamic content are: The OCR tool is used to extract the text information contained in the image content, and the text information is incorporated into the text content; A multi-modal pre-training model is used to map the text content and image content to the same semantic space.

[0013] Preferably, the data storage module is used to construct a relational database or a No SQL database, and the structured data is formatted and stored in the relational database or the No SQL database.

[0014] Preferably, the result display module provides a human-computer interaction interface, and the analysis results and the retrieved dark web related information are displayed through the human-computer interaction interface.

[0015] Compared with the prior art, the beneficial effects of the present application are: 1. The success rate and depth of dark web information acquisition are improved: by integrating Tor client, container isolation and headless browser technology, the problems of anonymity, dynamics and anti-crawler mechanism of the dark web are effectively solved, and stable crawling of dynamically rendered pages and deep crawling of structured data are realized.

[0016] 2. The identification accuracy of illegal information is improved: the multi-modal semantic enhancement method is combined with text, context and picture information for comprehensive analysis, effectively solving the problem of strong misleading of dark web information.

[0017] 3. The automation and efficiency of monitoring are realized: covering the collection, analysis, storage and display modules, replacing the inefficient and high-cost manual screening method, providing an efficient technical solution for dark web content governance. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only the preferred embodiments of the present application, and other drawings can also be obtained by those skilled in the art without any creative effort based on these drawings.

[0019] Fig. 1 A schematic diagram of a dark web detection system according to the present application; Fig. 2 A flowchart of entity recognition by an information analysis module of a dark web detection system according to the present application; In the figure, 1, information crawling module; 2, information analysis module; 3, data storage module; 4, result display module. DETAILED DESCRIPTION

[0020] In order to better understand the technical content of the present application, a specific embodiment will be provided below, and the present application will be further described in combination with the drawings.

[0021] Referring to Figs. 1-2 The present application provides a dark web detection system, which comprises: An information crawling module 1 is used to access the Tor anonymous network and crawl dynamic content; An information analysis module 2 is used to preprocess, identify entities, classify topics, perform correlation analysis and multi-modal fusion analysis on the dynamic content, and generate structured data; A data storage module 3 is used to build a database and store the structured data in the database; A result display module 4 is used to display the analysis results in a visual manner; The information crawling module 1, the information analysis module 2, the data storage module 3 and the result display module 4 are sequentially connected in data.

[0022] The main research goal of the present application is to overcome the deficiencies of the existing dark web monitoring methods, and to solve the problems of difficult information acquisition due to strong anonymity and dynamic changes of content of the dark web, and low analysis and recognition accuracy due to strong information confusion and multi-modal confusion. A dark web detection system is provided, which is deployed on one or more servers and comprises an information crawling module 1, an information analysis module 2, a data storage module 3 and a result display module 4. The information crawling module 1 anonymously accesses the dark web and crawls data. Then, the crawled data is sent to the information analysis module 2 for processing, identification and classification. The structured data after analysis is stored in the data storage module 3. Finally, the result display module 4 is used by the analysis personnel in a visual manner.

[0023] Since most of the dark web will be set with anti-crawling mechanism, the access function of Tor anonymous network is provided in the information crawling module 1, which solves the problem of difficult access to the dark web, and can evade the anti-crawling mechanism and ensure the accurate crawling of dynamic content. After crawling the dynamic content of the webpage, the information analysis module 2 processes the dynamic content in multiple levels. The preprocessing can remove irrelevant information in the dynamic content and retain useful information. Entity recognition can analyze the entity types contained in the webpage and obtain exact entity information, such as poster nickname, price and contact information, etc. Topic classification can classify the posts on the webpage by topic, and the topic types include virtual resources, etc. Association analysis can evaluate the association between different webpages, realize the detection and classification of related events and similar illegal information. Finally, through multi-modal fusion analysis, the strong confusion of dark web information can be dealt with, and the robustness of detection can be effectively improved. After the multi-level analysis and processing of the information analysis module 2, structured information containing post URL, entity information, topic classification, association relationship and picture path, etc. can be obtained. The structured information is stored in the data storage module 3 for long-term storage, fast query and deep mining. The result display module 4 can read the structured information from the data storage module 3 for analysis, helping the analyst to master the dark web dynamics and trends, and providing decision support for information governance and crime fighting.

[0024] Preferably, the specific steps of the information crawling module 1 accessing the Tor anonymous network are as follows: Integrate Tor client in the information crawling module 1, configure mixed proxy and dynamic path selection strategy to access Tor anonymous network, and use container isolation technology to deploy crawler environment.

[0025] In order to ensure effective access to Tor anonymous network, Tor client is integrated in the information crawling module 1, and mixed proxy and dynamic path selection strategy are configured to access Tor anonymous network. The crawler framework (such as Python's requests library) is configured to send all network requests through the local Tor proxy (usually listening on 127.0.0.1:9050 port) to access the onion routing network. To enhance anonymity, the Tor control protocol (Control Port) can be used to dynamically switch the exit node and realize IP rotation.

[0026] In order to ensure security and evade anti-crawler mechanism, container isolation technology is used to deploy crawler environment. The crawler program is deployed in Docker container for isolation to prevent real identity information from being leaked.

[0027] Preferably, the specific steps of the information crawling module 1 crawling dynamic content are as follows: Access the dark web link directory website, search and collect the.onion address of the target dark web community or market as the starting point of crawling; A headless browser is integrated into the information crawling module 1. The headless browser simulates user behavior, executes the JavaScript scripts in the page from the starting point of the crawl, and parses and obtains the complete page content generated by dynamic rendering technology. A depth-first crawling strategy is used to recursively crawl the target dark web site based on the complete page content to obtain dynamic content.

[0028] Information crawling module 1 employs a comprehensive extraction method to obtain initial connections. This involves accessing well-known dark web link directories (such as Ahmia.fi) to search for and collect .onion addresses of target dark web communities or markets as the starting point for crawling. Addressing the common use of JavaScript to dynamically render pages on the dark web, information crawling module 1 integrates a headless browser (Selenium-Wire). When a page is accessed, the headless browser simulates real user actions, executing JavaScript scripts within the page to parse and obtain the complete page content generated by dynamic rendering technology. This effectively solves the problem that traditional static crawlers cannot capture dynamic content. Furthermore, for Chinese dark web trading markets, a depth-first crawling strategy is adopted, recursively crawling along links within the complete page content to obtain as many posts and transaction information as possible from a site, thereby obtaining dynamic content.

[0029] Preferably, the specific steps for the information analysis module 2 to preprocess dynamic content are as follows: Extract text content from dynamic content and perform cleaning and structured analysis; Image content is extracted from dynamic content, and preprocessed using super-resolution reconstruction, denoising convolutional networks, and contrast-limited adaptive histogram equalization techniques.

[0030] Information Analysis Module 2 uses an HTML parsing tool to extract text content from crawled web pages, and performs cleaning and structured analysis to remove irrelevant data and retain important data. The HTML parsing tool is the BeautifulSoup library in Python, which is highly fault-tolerant and can effectively handle irregular dark web page code, and extract text content such as title, body, poster, and time.

[0031] Meanwhile, the information analysis module 2 will also extract image content from dynamic content. Then, to address the common problems of low resolution and noise interference in dark web images, image enhancement algorithms such as super-resolution reconstruction (e.g., SRGAN), denoising networks (e.g., DnCNN), and contrast-limited adaptive histogram equalization (CLAHE) can be used to enhance the image quality and improve the accuracy of subsequent analysis.

[0032] Preferably, the specific steps for entity recognition of dynamic content by the information analysis module 2 are as follows: The text content is input into a predefined entity type and an initialized keyword dictionary for parallel processing; Regular expressions are used to match text content according to rules, predefined entities with specific formats are identified, and the jieba word segmentation tool is used to segment the text content. Specific patterns in the keyword dictionary are matched in the segmentation results, and entity information is obtained. Extract the contextual information surrounding the entity information and perform deduplication.

[0033] For the text content, regular expressions and natural language processing technology (jieba segmentation) are used in combination. By using predefined entity types (such as poster nickname, price, contact information, etc., a total of 9 categories) and a keyword dictionary, the system accurately matches and extracts entities with key information. The system first loads the predefined entity types and keyword dictionary. Then, the system processes the input text data in parallel in two ways: on the one hand, it uses regular expressions based on rule matching to match predefined entities with specific formats, such as price patterns; on the other hand, it uses the jieba segmentation tool to segment the text based on word segmentation matching, and then matches specific patterns in the keyword dictionary in the word segmentation results. After matching entity information through the above two methods, the system extracts the context information around the entity information. Finally, all extracted entity information is deduplicated to ensure the uniqueness of the results.

[0034] Preferably, the specific steps for the information analysis module 2 to classify dynamic content by topic are as follows: Based on a predefined keyword library and weighting rules, text content is classified into 10 categories (such as virtual resources) to achieve efficient identification of illegal content on the dark web.

[0035] Preferably, the specific steps for the association analysis performed by the information analysis module 2 are as follows: The text content is vectorized using TF-IDF, and the cosine similarity of the vectorized text content is calculated using the following formula: ; Where A and B are the vectorized text contents, and cosine similarity is the similarity between the two text contents. The closer the cosine similarity is to 1, the higher the similarity between the two text contents.

[0036] Information analysis module 2 can also determine the similarity of text content. Currently, commonly used methods for calculating text similarity are roughly as follows: string matching-based methods, statistical feature-based methods, semantic similarity calculation methods based on word vectors, and semantic modeling based on deep learning. This invention selects the statistical feature-based method, and the representative model of this method is the bag-of-words model. In this model, the text content is first represented by TF-IDF vectors, which is the word frequency-inverse document frequency. The second step is to calculate the cosine similarity between two vectors. The result of the cosine similarity calculation is between 0 and 1. The closer it is to 1, the higher the similarity between the two text contents, that is, the stronger the correlation. Through correlation analysis, the detection and classification of related events and similar illegal information can be achieved.

[0037] Preferably, the specific steps of the information analysis module 2 in performing multimodal fusion analysis on dynamic content are as follows: Use OCR tools to extract text information contained in the image content and incorporate the text information into the text content; A multimodal pre-trained model is used to map text content and image content to the same semantic space.

[0038] To address the highly deceptive nature of dark web information, information analysis module 2 also needs to perform multimodal fusion analysis, which includes image text extraction and multimodal semantic alignment. Image text extraction involves using OCR tools to extract text information and then incorporating the text information into the text content for unified analysis, in order to counter the highly deceptive nature of dark web information.

[0039] Multimodal semantic alignment uses CLIP (Contrastive Language-Image Pre-training, a multimodal pre-trained model) to map text content and image content to the same semantic space, thereby enabling the retrieval of relevant image content using text content descriptions, or the reverse utilization of image content features to verify and enhance the analysis results of text content, effectively improving the robustness of detection.

[0040] Preferably, the data storage module 3 is used to construct a relational database or a No SQL database, and to store structured data in the relational database or No SQL database after formatting.

[0041] All structured data extracted in Information Analysis Module 2, including post URLs, entity information, topic categories, relationships, and image paths, are formatted and stored in a relational database or No SQL database for long-term storage, fast querying, and in-depth analysis.

[0042] Preferably, the result display module 4 provides a human-computer interaction interface to display analysis results and retrieve dark web related information.

[0043] Results Display Module 4 provides analysts with a human-computer interaction interface. Analysts can use this interface to search for dark web information with specific keywords or topics, view entity relationship networks, and intuitively grasp the dynamics and trends of the dark web through visual charts (such as transaction volume changes, topic distribution pie charts, etc.), providing decision support for information governance and crime fighting.

[0044] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A dark web detection system, characterized in that, include: The information crawling module is used to access the Tor anonymous network and crawl dynamic content; The information analysis module is used to preprocess dynamic content, identify entities, classify topics, perform correlation analysis and multimodal fusion analysis, and generate structured data. The data storage module is used to build the database and store structured data in the database; The results display module is used to present the analysis results in a visual manner; The information crawling module, information analysis module, data storage module, and result display module are sequentially connected.

2. The dark web detection system according to claim 1, characterized in that, The specific steps for the information crawling module to access the Tor anonymous network are as follows: The Tor client is integrated into the information crawling module, and a hybrid proxy and dynamic path selection strategy is configured to access the Tor anonymous network. At the same time, container isolation technology is used to deploy the crawling environment.

3. The dark web detection system according to claim 1, characterized in that, The specific steps for the information crawling module to crawl dynamic content are as follows: Visit dark web link directories, search for and collect .onion addresses of target dark web communities or marketplaces as the starting point for crawling; A headless browser is integrated into the information crawling module. The headless browser simulates user behavior, executes the JavaScript scripts in the page from the starting point of the crawl, and parses and obtains the complete page content generated by dynamic rendering technology. A depth-first crawling strategy is used to recursively crawl the target dark web site based on the complete page content to obtain dynamic content.

4. The dark web detection system according to claim 1, characterized in that, The specific steps for the information analysis module to preprocess dynamic content are as follows: Extract text content from dynamic content and perform cleaning and structured analysis; Image content is extracted from dynamic content, and preprocessed using super-resolution reconstruction, denoising convolutional networks, and contrast-limited adaptive histogram equalization techniques.

5. The dark web detection system according to claim 4, characterized in that, The specific steps for entity recognition of dynamic content by the information analysis module are as follows: The text content is input into a predefined entity type and an initialized keyword dictionary for parallel processing; Regular expressions are used to match text content according to rules, predefined entities with specific formats are identified, and the jieba word segmentation tool is used to segment the text content. Specific patterns in the keyword dictionary are matched in the segmentation results, and entity information is obtained. Extract the contextual information surrounding the entity information and perform deduplication.

6. The dark web detection system according to claim 4, characterized in that, The specific steps for the information analysis module to classify dynamic content by topic are as follows: Based on a predefined keyword library and weighting rules, the text content is classified into topics.

7. The dark web detection system according to claim 4, characterized in that, The specific steps for the information analysis module to perform correlation analysis are as follows: The text content is vectorized using TF-IDF, and the cosine similarity of the vectorized text content is calculated using the following formula: ； Where A and B are the two vectorized text contents, respectively. The cosine similarity is the highest value for a given cosine similarity score. The closer the cosine similarity score is to 1, the higher the similarity between the two texts.

8. The dark web detection system according to claim 4, characterized in that, The specific steps of the information analysis module in performing multimodal fusion analysis on dynamic content are as follows: Use OCR tools to extract text information contained in the image content and incorporate the text information into the text content; A multimodal pre-trained model is used to map text content and image content to the same semantic space.

9. A dark web detection system according to claim 1, characterized in that, The data storage module is used to build a relational database or a No SQL database, and to store structured data in a formatted relational database or a No SQL database.

10. A dark web detection system according to claim 1, characterized in that, The results display module provides a human-computer interaction interface, through which analysis results are displayed and dark web-related information is retrieved.

Images