Dark web monitoring and analysis system

Abstract

The present disclosure relates to the technical field of Internet, and provides a dark web monitoring and analyzing system, comprising: a dark web access subsystem, configured to establish a communication link with the dark web based on a dark web access agent, to ensure the realization of dark web access; a collection agent subsystem, configured to collect target site data in the dark web according to a collection task; a data processing subsystem, configured to process the target site data, to obtain standardized data and add data labels to the standardized data; a data storage subsystem, configured to store and manage the target site data and the standardized data in a distributed storage manner; a data analysis subsystem, configured to analyze the standardized data with the added data labels, to generate data analysis results; and a dark web data situation display subsystem, configured to search the target site data from different levels based on search terms, and to display the target site data in a multi-dimensional graphical manner. The present disclosure can realize real-time monitoring of the dark web, hidden service discovery, data collection and analysis, and safe and stable storage and management of data.

Description

Technical Field

[0001] This disclosure relates to the field of Internet technology, and in particular to a dark web monitoring and analysis system. Background Technology

[0002] Dark web services refer to hidden service networks built within anonymous networks. Currently, the largest dark web is the Tor (Onion Router) network. Besides this, there are also the I2P (Invisible Internet Project) dark web and the Zeronet dark web, but their scale is relatively small. Dark web services are based on anonymous communication technologies, providing an untraceable network system. On the dark web, both service providers and visitors are completely anonymous and cannot be traced.

[0003] The dark web employs a distributed network structure, unlike traditional centralized networks. Therefore, it's impossible to search the dark web using general web crawling techniques. Consequently, discovering hidden services on the dark web has become a pressing technical problem to be solved. Summary of the Invention

[0004] This disclosure aims to address at least one of the problems existing in the prior art by providing a dark web monitoring and analysis system, including:

[0005] The dark web access subsystem is used to establish a communication link with the dark web based on the dark web access proxy, and to replace the communication link in a timely manner according to the status of the communication link to ensure dark web access. The dark web includes at least one of Tor dark web, I2P dark web, and ZeroNet dark web.

[0006] The data collection agent subsystem is used to collect data from target sites on the dark web using a web crawler program according to the data collection task.

[0007] The data storage subsystem is used to store and manage target site data using a distributed storage approach.

[0008] The data processing subsystem is used to process the target site data according to the pre-configured data processing rules, obtain the normalized data corresponding to the target site data, and add data tags to the normalized data;

[0009] The data analysis subsystem is used to analyze and process the standardized data after adding data tags using statistical analysis methods, and generate data analysis results corresponding to the target site data.

[0010] The Dark Web Data Situation Display Subsystem is used to perform different levels of retrieval of target site data based on search terms, and to display the target site data in a graphical way through multiple dimensions.

[0011] The data storage subsystem is also used to store and manage standardized data using a distributed storage approach.

[0012] Optionally, the dark web access subsystem includes:

[0013] The Tor access module is used to establish a communication link with the Tor dark network based on the Tor dark network access proxy, and to replace the communication link in a timely manner according to the status of the communication link to ensure Tor dark network access.

[0014] The I2P access module is used to establish a communication link with the I2P dark web based on the I2P dark web access proxy. It adds multiple proxy services to the I2P dark web access proxy. When one proxy service is unavailable, the other proxy services are used to establish a communication link with the I2P dark web to ensure I2P dark web access.

[0015] The ZeroNet access module is used to establish a communication link with the ZeroNet dark network based on the ZeroNet dark network access proxy, and to replace the communication link in a timely manner according to the status of the communication link to ensure the realization of ZeroNet dark network access.

[0016] Optionally, the data collection agent subsystem includes:

[0017] The data acquisition task scheduling module is used to configure data acquisition tasks and acquisition strategies;

[0018] The crawler module is used to divide the collection task into multiple collection sub-tasks. Using the collection strategy, a distributed web crawler is used to crawl the target sites in the dark web according to the collection sub-tasks in order to obtain the target site data.

[0019] The content parsing module is used to parse the target site data to obtain the data content of the target site data;

[0020] The feature extraction module is used to extract features and feature values ​​from the data content.

[0021] Optionally, the data storage subsystem includes a data distribution module, a statement parsing module, a parallel processing module, and a summary processing module, wherein:

[0022] The data distribution module is used to receive data query statements and distribute them to the statement parsing module using a round-robin scheduling strategy.

[0023] The statement parsing module is used to perform syntax and semantic checks on data query statements, parse the data query statements, generate a syntax parse tree and a data query execution plan, and send the syntax parse tree and data query execution plan to the data distribution module;

[0024] The data distribution module is also used to perform data segmentation on the syntax parse tree and data query execution plan according to the preset data segmentation strategy, and distribute the segmented syntax parse tree and data query execution plan to the corresponding areas of the parallel processing module;

[0025] The parallel processing module is used to determine whether the data-segmented parse tree has a preset execution order relationship based on the data query execution plan after data segmentation. If the data-segmented parse tree has a preset execution order relationship, it is processed according to the preset execution order relationship. If the data-segmented parse tree does not have a preset execution order relationship, it is processed according to the preset parallel processing method.

[0026] The summary processing module is used to summarize the processing results of the parallel processing module to obtain the data query results corresponding to the data query statement.

[0027] Optionally, the data processing subsystem includes:

[0028] The data cleaning module is used to extract target data from the target site data according to the pre-configured data extraction rules; remove useless symbols and stop words from the target data and perform deduplication according to the pre-configured data cleaning rules to clean the target data; and map the cleaned target data to the pre-configured structured data fields according to the pre-configured field mapping rules to obtain the normalized data corresponding to the target data.

[0029] The data tag management module is used to perform word segmentation on standardized data using at least one of the following methods: string matching-based word segmentation, statistical word segmentation, and understanding-based word segmentation, to obtain the word segmentation results of the standardized data, and to set data tags for the standardized data based on the word segmentation results.

[0030] Optionally, the data analysis subsystem includes:

[0031] The semantic recognition module is used to perform semantic recognition on the normalized data after adding data labels using at least one of the Skip-gram model-based method or LSA algorithm to obtain the semantic recognition result.

[0032] The named entity recognition module is used to perform named entity recognition on normalized data after adding data labels using a named entity recognition model based on long short-term memory network, and obtain the named entity recognition result. The named entity recognition model based on long short-term memory network includes a lookup layer, a bidirectional LSTM (Long Short-Term Memory) layer and a CRF (Conditional Random Field) layer.

[0033] The key sentence recognition module is used to segment the standardized data after adding data labels, represent the words and sentences obtained from the segmentation process into vectors through word embedding, and use machine learning methods to rank the vector-represented words and sentences. The words and / or sentences ranked before the preset value are identified as key sentences to obtain the key sentence recognition results.

[0034] The classification analysis module is used to classify the normalized data after adding data labels based on the text features of the normalized data after adding data labels, based on the trained classifier model, and obtain the classification analysis results. The trained classifier model is obtained by training a preset text classification algorithm in advance based on the text vector space of the sample text features.

[0035] The clustering analysis module is used to extract features from the normalized data after adding data labels using TF-IDF, and to perform clustering using the density-based DBSCAN algorithm to obtain the clustering analysis results.

[0036] The association analysis module is used to filter relevant data from the data storage subsystem based on preset data characteristics, perform association analysis using preset association analysis algorithms, and obtain association analysis results.

[0037] Optional, the dark web data situational awareness subsystem includes:

[0038] The data search module is used to retrieve data from target sites based on search terms, employing relational databases and distributed full-text search technology. It optimizes and automatically aggregates the search results. An improved inverted incremental indexing technique is used to store and automatically update the search results. Query expansion technology in information retrieval is used to establish a query expansion retrieval model. Through local and global analysis methods, search terms are expanded, the retrieval semantics are clarified, and data related to the search terms is obtained from the target site data to arrive at the search results.

[0039] The data display module is used to convert target site data into graphical information through visualization technology. Based on the visualization display and analysis engine tool, it displays the graphical information in the form of charts through preset layout and display methods, and realizes human-computer interaction.

[0040] Optionally, the dark web monitoring and analysis system may also include:

[0041] The source tracing subsystem is used to trace the origins of dark web nodes and network attacks, perform source tracing analysis based on the tracing results, and establish a dark web source tracing database.

[0042] Optionally, the traceability subsystem includes:

[0043] The dark web node tracing module is used to trace dark web nodes, obtain the descriptive information of the traced dark web nodes, perform dark web node tracing analysis based on the descriptive information and clear web information, and establish a dark web tracing database based on the descriptive information.

[0044] The network attack tracing module is used to detect network attack behavior. After detecting network attack behavior, it tracks and locates the attack data stream to identify the network device or host that sent the attack data. It analyzes the relevant information of the network device or host to determine whether the network device or host is controlled by a third party. If the network device or host is controlled by a third party, it sequentially identifies the control nodes at all levels of the network device or host down to the source control node. It then identifies and analyzes the source control node to determine its attribute information.

[0045] Optionally, the dark web monitoring and analysis system may also include:

[0046] The IP address pool management subsystem is used to maintain and manage the IP addresses used by dark web access proxies based on the IP address pool, so that there are enough available IP addresses in the IP address pool for dark web access proxies to use.

[0047] Compared with the prior art, this disclosure sets up a dark web access subsystem, a data collection agent subsystem, a data storage subsystem, a data processing subsystem, a data analysis subsystem, and a dark web data status display subsystem in the dark web monitoring and analysis system. It realizes real-time monitoring of the dark web, discovery of hidden services, data collection and analysis, and can also maximize the secure and stable storage and management of data. Attached Figure Description

[0048] One or more embodiments are illustrated by way of example with the corresponding pictures in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0049] Figure 1 A schematic diagram of the structure of a dark web monitoring and analysis system provided in one embodiment of this disclosure;

[0050] Figure 2A schematic diagram of the structure of a dark web access subsystem provided for another embodiment of this disclosure;

[0051] Figure 3 A schematic diagram of the structure of a data acquisition agent subsystem provided for another embodiment of this disclosure;

[0052] Figure 4 A schematic diagram of the structure of a data storage subsystem provided for another embodiment of this disclosure;

[0053] Figure 5 A schematic diagram of the structure of a data processing subsystem provided for another embodiment of this disclosure;

[0054] Figure 6 A schematic diagram of the structure of a data analysis subsystem provided for another embodiment of this disclosure;

[0055] Figure 7 A schematic diagram of the structure of a dark web data situation display subsystem provided for another embodiment of this disclosure;

[0056] Figure 8 A schematic diagram of the structure of a dark web monitoring and analysis system provided for another embodiment of this disclosure;

[0057] Figure 9 A schematic diagram of the structure of a traceability subsystem provided for another embodiment of this disclosure;

[0058] Figure 10 This is a schematic diagram of the structure of a dark web monitoring and analysis system provided for another embodiment of this disclosure. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the various embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been presented in the various embodiments of this disclosure to facilitate a better understanding of this application. However, the technical solutions claimed in this application can be implemented even without these technical details and with various variations and modifications based on the following embodiments. The division of the various embodiments below is for ease of description and should not constitute any limitation on the specific implementation of this disclosure. The various embodiments can be combined with and referenced by each other without contradiction.

[0060] One embodiment of this disclosure relates to a dark web monitoring and analysis system, such as Figure 1 As shown, it includes a dark web access subsystem 110, a data collection agent subsystem 120, a data storage subsystem 130, a data processing subsystem 140, a data analysis subsystem 150, and a dark web data situation display subsystem 160.

[0061] The dark web access subsystem 110 is used to establish a communication link with the dark web based on the dark web access proxy, and to replace the communication link in a timely manner according to the status of the communication link to ensure dark web access. The dark web includes at least one of Tor dark web, I2P dark web, and ZeroNet dark web.

[0062] The Dark Web Access Subsystem leverages existing dark web resources and the technical characteristics of anonymous networks. By deploying controlled nodes such as traffic detection nodes, anonymous network proxy nodes, and data communication transmission nodes within the dark web, it deeply probes and perceives dark web traffic. This enables the subsequent collection of access behaviors of dark web nodes and clients, thereby achieving resource discovery capabilities for mainstream dark web networks such as Tor, I2P, and ZeroNet.

[0063] The dark web access subsystem can discover dark web sites in two ways. The first method is through site discovery crawlers. This method uses a list of dark web sites as seed addresses. The crawler periodically scans the entire list, determining whether a site is accessible and updating its accessibility status. If a site is inaccessible, it is skipped, and the crawler continues scanning the next site in the list. If a site is accessible, its page is downloaded and parsed to extract the URL (Uniform Resource Locator). The extracted URL is then checked against a new domain. If it is a new domain, it is added to the dark web site list, and information such as the domain name, discovery time, and accessibility status is recorded. The second method involves discovering dark web sites simultaneously with data collection. Since most dark web sites can only obtain more information after logging in, and dark web sites cannot be accessed through site discovery crawlers, dark web sites can be discovered in the data during data collection. The discovered dark web sites are then compared with the dark web site list. If the discovered dark web site does not exist in the dark web site list, it is added to the dark web site list as a new dark web site.

[0064] The data collection agent subsystem 120 is used to collect target site data in the dark web using a crawler program according to the data collection task.

[0065] The data storage subsystem 130 is used to store and manage target site data using a distributed storage method.

[0066] The collected target site data is typically rich in content, diverse in type, and varied in format. Direct storage of this data would result in large volumes, numerous data types, disorganized content, complex formats, and difficulty in estimating its value. Furthermore, the original data files are unsuitable for effective business applications. The data storage subsystem, however, can uniformly store and manage the collected target site data using distributed storage, maximizing the secure and stable storage and retrieval of data, thus providing data storage support for the entire dark web monitoring and analysis system. Simultaneously, the data storage subsystem serves as the data foundation for the entire dark web monitoring and analysis system, providing data sources for the data analysis subsystem.

[0067] The data processing subsystem 140 is used to process the target site data according to the pre-configured data processing rules, obtain the normalized data corresponding to the target site data, and add data tags to the normalized data.

[0068] The data analysis subsystem 150 is used to analyze and process the standardized data after adding data tags using statistical analysis methods, and generate data analysis results corresponding to the target site data.

[0069] The Dark Web Data Situation Display Subsystem 160 is used to perform different levels of retrieval of target site data based on search terms, and to display the target site data in a graphical way through multiple dimensions.

[0070] The data storage subsystem 130 is also used to store and manage standardized data using a distributed storage approach.

[0071] Compared with the prior art, the embodiments disclosed herein set up a dark web access subsystem, a data collection agent subsystem, a data storage subsystem, a data processing subsystem, a data analysis subsystem, and a dark web data situation display subsystem in the dark web monitoring and analysis system. This enables real-time monitoring of the dark web, discovery of hidden services, data collection and analysis, and also maximizes the secure and stable storage and management of data.

[0072] For example, such as Figure 2 As shown, the dark web access subsystem 110 may include a Tor access module 111, an I2P access module 112, and a ZeroNet access module 113.

[0073] The Tor access module 111 is used to establish a communication link with the Tor dark web based on the Tor dark web access proxy, and to replace the communication link in a timely manner according to its status to ensure Tor dark web access. The Tor access module provides Tor dark web access services for the dark web monitoring and analysis system, enabling the data collection proxy subsystem to crawl relevant content from the Tor dark web.

[0074] The I2P access module 112 is used to establish a communication link with the I2P dark network based on the I2P dark network access proxy, and to add multiple proxy services to the I2P dark network access proxy. When one of the proxy services is unavailable, the other proxy services are used to establish a communication link with the I2P dark network to ensure I2P dark network access.

[0075] The I2P access module establishes a Socks5 proxy service via I2P, setting the I2P egress address and port to the Socks5 proxy server's address and port, respectively. When accessing websites using the built-in HTTP (Hypertext Transfer Protocol) and HTTPS (Hypertext Transfer Protocol over Secure Socket Layer) proxy services of I2P, the displayed IP address (Internet Protocol Address) is essentially fixed. However, since the built-in HTTP and HTTPS proxy services are provided by the I2P user, when the I2P user goes offline, these services also go offline, rendering them temporarily unavailable. To address this, the I2P access module modifies the built-in HTTP and HTTPS proxy services, adding multiple proxy services to the I2P dark web access proxy. When one proxy service becomes unavailable, it automatically switches to another available one, thus ensuring I2P dark web access.

[0076] The I2P access module can provide various services for I2P egress routing. Besides the built-in HTTP and HTTPS proxy services, it also includes a wide range of services provided by I2P users, such as Socks5 proxy services, web services, and Tor network access services. The I2P access module can also add I2P clients to use services not built into I2P.

[0077] ZeroNet access module 113 is used to establish a communication link with ZeroNet based on the ZeroNet dark network access proxy, and to replace the communication link in a timely manner according to the status of the communication link to ensure ZeroNet dark network access.

[0078] ZeroNet is a distributed, internet-like network built upon peer-to-peer (P2P) connections. Its most prominent feature is decentralization; each visitor acts as a node, and every connected device functions as both a server and a client, responsible for storing network content and providing bandwidth. The ZeroNet access module provides access to the ZeroNet darknet service for darknet monitoring and analysis systems, enabling the data collection agent subsystem to crawl relevant content from the ZeroNet darknet.

[0079] For example, the data collection agent subsystem 120 can be used to manage dark web data collection agents, including basic configuration of the collection agent, data collection strategies, distribution and execution of data collection tasks, anti-crawler mechanisms, etc. Figure 3 As shown, the data acquisition agent subsystem 120 may include a data acquisition task scheduling module 121, a crawler program module 122, a content parsing module 123, and an element extraction module 124.

[0080] The data acquisition task scheduling module 121 is used to configure data acquisition tasks and acquisition strategies.

[0081] The crawler module 122 is used to divide the collection task into multiple collection sub-tasks. Using the collection strategy, a distributed web crawler is used to crawl the target sites in the dark web according to the collection sub-tasks in order to obtain the target site data.

[0082] The content parsing module 123 is used to parse the target site data to obtain the data content of the target site data.

[0083] The feature extraction module 124 is used to extract features and feature values ​​from the data content.

[0084] The physical architecture of a distributed web crawler can be divided into two parts: a distributed data storage cluster and a crawling service cluster. The distributed data storage cluster and the crawling service cluster can be deployed together, meaning both components are deployed on the same machine. However, to avoid putting pressure on the distributed data storage cluster running the crawler, the two can be deployed separately. In this case, the crawler program can run on the distributed data storage cluster, utilizing its distributed architecture to provide distributed storage and distributed computing capabilities. The crawling task is divided into multiple subtasks that run in parallel on multiple collaborating nodes. The crawled page data is stored in a distributed manner, and relevant metric data generated during the crawling and content parsing phases is written to the message queue of a message queue service in real time. The crawling service then calculates the metric results in real time.

[0085] Distributed web crawlers have the following characteristics: they can be deployed on general-purpose servers, thus saving costs; their distributed architecture is scalable, and overall performance can be improved by enhancing hardware capabilities or increasing the number of nodes; they provide distributed storage capabilities, which can evenly distribute crawled page data across various nodes; and they have high fault tolerance, allowing the number of data replicas to be set, and tasks that fail can be automatically reassigned to other nodes for execution.

[0086] The logical architecture of a distributed web crawler can be divided into two parts: a batch processing part and a real-time computing part. The batch processing part is implemented based on a distributed data storage cluster and is used to execute the data collection tasks, i.e., the crawling tasks, and to distribute and store the crawled data. The real-time computing part is implemented based on a crawling service cluster and is used to calculate the relevant indicator data generated during the crawler's operation in real time, and to store the calculated indicator results in the distributed data storage cluster.

[0087] The workflow of the data acquisition agent subsystem is as follows: The data acquisition function is implemented through a distributed web crawler. After accessing the dark web through the dark web access subsystem, the crawler module collects source data from target sites on the dark web as target site data according to the acquisition tasks and strategies configured by the acquisition task scheduling module. The content parsing module performs content parsing on the target site data to obtain the data content of the target site data. The feature extraction module extracts features and feature values ​​from the data content to classify and store the data content and provide it to other subsystems for use.

[0088] The data collection agent subsystem has a wide data collection scope and rich content, which can provide comprehensive data support for subsequent analysis, statistics and mining of dark web data, thereby gaining a deeper understanding of the dark web situation and trends.

[0089] For example, the data storage subsystem includes a physical infrastructure component and a software implementation component. The physical infrastructure component includes a database server, a web server, and a resource allocation management server, which respectively implement data access, data analysis and processing, and resource workload balancing. The database server, web server, and resource allocation management server within the physical infrastructure component work together to build an efficient and collaborative backend service management system, thereby enabling the data storage subsystem to operate efficiently and with balanced workloads. The software implementation component is the core of the data storage subsystem, used to implement the allocation of data query statements, resource workload scheduling, database server parallel execution strategies, fault tolerance services, and disaster recovery. Based on this, such as... Figure 4 As shown, the data storage subsystem 130 may include a data distribution module 131, a statement parsing module 132, a parallel processing module 133, and a summary processing module 134, wherein:

[0090] The data distribution module 131 is used to receive data query statements and distribute the data query statements to the statement parsing module using a round-robin scheduling strategy.

[0091] The statement parsing module 132 is used to perform syntax and semantic checks on the data query statement, parse the data query statement, generate a syntax parse tree and a data query execution plan, and send the syntax parse tree and data query execution plan to the data distribution module.

[0092] The statement parsing module checks the data query statement against pre-stored standardized statements in the data dictionary to ensure its syntax is correct, the existence of the query objects, and the necessary permissions of the user initiating the query. If the query statement is syntactically correct, the query objects exist, and the user has the appropriate permissions, a syntax parse tree is generated. Otherwise, an exception is thrown and analyzed. After generating the syntax parse tree, the module recursively traverses it to extract the specific location of the data query statement within the data storage subsystem, the specific query functions used, the specific query conditions, and the specific form of result set processing. It then generates a data query execution plan and sends the syntax parse tree and execution plan to the data distribution module.

[0093] The statement parsing module primarily serves three purposes: first, it checks the validity of attributes in data query statements and ensures that the data query statements conform to grammatical rules; second, it generates a new data structure, namely a syntax parse tree; and third, it provides a basis for data segmentation in the data distribution module. By using the statement parsing module to parse data query statements and generate syntax parse trees, the accuracy of statement parsing results is higher, the data hierarchy is clearer, and the data structure is more complete.

[0094] The data distribution module 131 is also used to perform data segmentation on the syntax parse tree and data query execution plan according to a preset data segmentation strategy, and distribute the segmented syntax parse tree and data query execution plan to the corresponding areas of the parallel processing module.

[0095] The data distribution module, combining preset data partitioning strategies and business logic, can employ various methods for data partitioning. It also provides a reserved data partitioning interface, allowing users to flexibly customize and implement their own data partitioning, further improving system availability. Here, data partitioning refers to dividing a data query task, which should originally be executed as a single task, into multiple data query subtasks based on its task type and according to a preset data partitioning strategy. These subtasks are then forwarded to different areas of the parallel processing module for processing, thereby alleviating the performance pressure issues associated with centralized processing.

[0096] The data distribution module can also manage connection pools, thread pools, heartbeat test functions, and system configuration data.

[0097] Connection pool management mainly includes connection creation, distribution, pooling, and recycling. A connection pool can store idle connections without immediate release; once a new connection requirement arises, a connection in the pool can be used directly without establishing a new connection.

[0098] Thread pool management mainly includes monitoring and managing the lifecycle of each thread under concurrent querying, including thread creation, readiness, running, blocking, and death.

[0099] The heartbeat test function management mainly targets the data storage subsystem. It uses heartbeat tests to check whether the database nodes in the data storage subsystem are alive. If an anomaly is found, it is dealt with in a timely manner, thereby avoiding the system resource consumption caused by sending data to unavailable database nodes.

[0100] System configuration data management mainly includes data partitioning strategy configuration, database configuration, connection pool configuration, thread pool configuration, and heartbeat test configuration. When the data distribution module starts, it first initializes the configuration files, including data source configuration, data partitioning strategy configuration, and thread pool configuration. Secondly, the data distribution module implements the data distribution function. After capturing the syntax parse tree and data query execution plan generated by the statement parsing module, the data distribution module divides it into multiple data query subtasks according to the corresponding data partitioning strategy, and forwards these subtasks to the parallel processing module, thereby realizing a parallel query mechanism.

[0101] The parallel processing module 133 is used to determine whether the data-segmented syntax parse tree has a preset execution order relationship based on the data query execution plan after data segmentation. If the data-segmented syntax parse tree has a preset execution order relationship, it is processed according to the preset execution order relationship. If the data-segmented syntax parse tree does not have a preset execution order relationship, it is processed according to the preset parallel processing method.

[0102] The parallel processing module enables multi-process concurrent execution of data query statements, improving the execution efficiency of data query statements and greatly shortening the response time of data query statements.

[0103] The summary processing module 134 is used to summarize the processing results of the parallel processing module to obtain the data query results corresponding to the data query statement.

[0104] When the data query statement is an SQL (Structured Query Language) statement, the workflow of the data storage subsystem is as follows:

[0105] First, the data distribution node of the data distribution module receives SQL statements. Based on factors such as the current workload of each data parsing node, the estimated completion time for parsing, the estimated time required for this SQL statement, and historical response time requirements, it uses a round-robin scheduling strategy to distribute the SQL statements to the various data parsing nodes of the statement parsing module, enabling each data parsing node to parse the SQL statements. When multiple data query statements pass through the same data distribution node, high concurrency issues will inevitably arise. Therefore, to eliminate the performance bottleneck of a single data distribution node, the data distribution module can include multiple data distribution nodes, each of which can distribute data query statements to different data parsing nodes. By using a round-robin scheduling strategy to distribute data query statements sequentially to each data parsing node, the workload of each data parsing node can be kept balanced.

[0106] Secondly, the data parsing node parses the SQL statement, generates a syntax parse tree and a data query execution plan, and sends these to the data distribution module. Then, the data distribution module segments the syntax parse tree and data query execution plan according to a preset data partitioning strategy, and distributes the segmented syntax parse tree and data query execution plan to different areas of the parallel processing module.

[0107] Finally, the parallel processing module determines whether the parse tree after data partitioning has a preset execution order based on the data query execution plan. If it does, the parse tree is processed according to that order. If it does not, it is processed using a preset parallel processing method. Afterward, the aggregation processing module summarizes the results from the parallel processing module to obtain the final data query result.

[0108] The data distribution module in the data storage subsystem can monitor the status of multiple data nodes in the parallel processing module and maintain data information tables for each data node. It can also store the location and replica information of the table blocks corresponding to each data information table. To ensure fault tolerance and prevent the entire data storage subsystem from crashing due to the failure of one data node, a replica data node can be configured for each data node, with a specified address. The replica data node stores the same data as the original data node, achieving read-write separation. Data location information can be stored in the data distribution module. When querying data, the data distribution module first locates the data's location, and then retrieves the data from the corresponding original data node or replica data node based on that location.

[0109] The data distribution nodes in the data distribution module and the data nodes in the parallel processing module can operate in a 1:n ratio, meaning one data distribution node corresponds to n data nodes. Each time a data distribution node receives a data query statement, it continuously monitors the distribution, execution, and result return of that query statement. The execution result of the data query statement is returned to the data distribution node that distributed it. For example, if M data query statements are distributed by data distribution node A to K data nodes, then data distribution node A will monitor the execution of the M data query statements by these K data nodes. Simultaneously, the M execution results corresponding to the M data query statements are integrated and summarized by the aggregation processing module and returned to data distribution node A.

[0110] By partitioning and splitting data into separate databases and tables, storing them on different data nodes, and ensuring that each data node operates independently, performing retrieval, update, or index creation operations independently, parallel queries can be performed, improving data query efficiency. The distributed storage approach adopted by the data storage subsystem also features high availability, high scalability, and high concurrency, solving the problem of massive data storage that traditional databases cannot address.

[0111] For example, such as Figure 5 As shown, the data processing subsystem 140 may include a data cleaning module 141 and a data tag management module 142.

[0112] The data cleaning module 141 is used to extract target data from the target site data according to the pre-configured data extraction rules; remove useless symbols and stop words from the target data according to the pre-configured data cleaning rules, and perform deduplication to clean the target data; and map the cleaned target data to the pre-configured structured data fields according to the pre-configured field mapping rules to obtain the normalized data corresponding to the target data.

[0113] The data cleaning module controls the data cleaning engine through data cleaning tasks. It defines source data using URL matching, allowing the engine to match data sources based on pre-configured data extraction rules. Data from target sites collected by the acquisition agent subsystem serves as the data source and can be reused by multiple data cleaning tasks. Each task requires specific data extraction, cleaning, and field mapping rules. The data cleaning engine extracts and cleans the target site data according to these pre-defined rules, maps it to specific structured data fields, and then sends the standardized data to the data storage subsystem for storage and management.

[0114] The data tag management module 142 is used to perform word segmentation on the standardized data using at least one of the following methods: string matching-based word segmentation, statistical word segmentation, and understanding-based word segmentation, to obtain the word segmentation results of the standardized data, and to set data tags for the standardized data based on the word segmentation results.

[0115] The word segmentation process here refers to the process of recombining consecutive words or phrases into a word sequence according to certain rules.

[0116] String matching-based word segmentation methods use a word segmentation dictionary as the foundation for string segmentation. When segmenting normalized data, the methods first compare the entries in the word segmentation dictionary with the strings in the normalized data. If the comparison result is true, it means the string being compared can be segmented as a single word or phrase from other content in the normalized data. If the comparison result is false, it means the string being compared cannot be segmented as a single word or phrase and cannot be segmented from other content in the normalized data. String matching-based word segmentation methods have low time complexity and are easy to understand.

[0117] Statistical word segmentation methods require building a word segmentation model based on manually labeled parts of speech and statistical features. The word segmentation model is used to calculate the probability of each word or phrase, and the word or phrase with the highest probability is taken as the final word segmentation result. This overcomes the defect of string matching-based word segmentation methods that cannot perform word segmentation when the corresponding word or phrase does not exist in the word segmentation dictionary.

[0118] Understanding-based word segmentation methods simultaneously perform syntactic parsing of sentences and semantic parsing of words when segmenting standardized data, thus eliminating word ambiguity.

[0119] The data label management module can also establish a user word dictionary, add words or phrases outside the word segmentation dictionary to the user word dictionary, and perform word segmentation on the normalized data based on the user word dictionary. To achieve better clustering results, the data label management module can also add unnecessary words as stop words to the stop word dictionary, and remove the stop words in the word segmentation results based on the stop word dictionary after performing word segmentation on the normalized data. To unify synonyms, the data label management module can also use a disambiguation dictionary to convert multiple synonyms into the same word. To remove meaningless single characters such as "yes" and "will", the data label management module can also set a single character dictionary based on the single characters to be retained, and retain the single characters in the single character dictionary in the word segmentation results and remove meaningless single characters such as "yes" and "will" after performing word segmentation on the normalized data.

[0120] The data label management module can set dark web data labels as needed. On the one hand, it can be used as the collection keywords of the collection agent subsystem to guide the collection agent subsystem to collect dark web data targeted. On the other hand, it can provide support for data processing and data analysis.

[0121] The data processing subsystem performs necessary data processing work on multi-source dark web data: one is to clean the data, including removing meaningless data, data deduplication, screening the same or similar data in multiple dark web platforms, etc.; the other is to perform labeling processing on the data, extract the data features of the dark web data, and set data labels for each dark web data. The data processing subsystem realizes the rapid and effective sorting and utilization of data, forms standard business data, and provides a solid data foundation for business applications.

[0122] Exemplarily, as Figure 6 shown, the data analysis subsystem 150 can include a semantic recognition module 151, a named entity recognition module 152, a key sentence recognition module 153, a classification analysis module 154, a clustering analysis module 155, and an association analysis module 156.

[0123] The semantic recognition module 151 is used to perform semantic recognition on the normalized data after adding data labels by using at least one of the methods based on the Skip-gram model or the LSA (Latent Semantic Analysis) algorithm, and obtain the semantic recognition result.

[0124] In natural language processing, the selection of corpora needs to meet two important requirements: First, the corpus must be sufficient. On the one hand, the dictionary must have a large enough vocabulary, meaning it must include a sufficient number of words; on the other hand, it must contain as many sentences as possible that reflect the relationships between words to ensure that the model can learn the semantic and grammatical relationships in sentences. Second, the corpus must be accurate, meaning the selected corpus must correctly reflect the semantic and grammatical relationships. During semantic recognition, the corpus processing method may affect the accuracy of semantic recognition. For example, in an n-gram model, the window size limitation means that the relationship between words outside the window and the current word cannot be correctly reflected in the model, while simply increasing the window size will increase the complexity of model training. Skip-gram models can effectively solve these problems. Skip-gram models allow certain words to be skipped, thus correctly reflecting the semantic information of sentences.

[0125] The LSA algorithm maps documents to vectors in a low-dimensional semantic space through singular value decomposition in linear algebra. However, each dimension in the space has no explicit physical meaning. Topic modeling attempts to represent text from the perspective of probabilistic generative models. Each dimension is a "topic", which is usually a cluster of words. Therefore, the semantics represented by each dimension can be roughly guessed through the topic, which has a certain degree of interpretability.

[0126] The named entity recognition module 152 is used to perform named entity recognition on normalized data after adding data labels using a named entity recognition model based on long short-term memory network, and obtain named entity recognition results. The named entity recognition model based on long short-term memory network includes a lookup layer, a bidirectional LSTM layer and a CRF layer.

[0127] Named entities refer to entities in text that have specific meaning or strong referentiality, typically including time, location, and proper nouns. Named entity recognition models extract these entities from unstructured text and can identify even more categories of entities according to business needs.

[0128] The named entity recognition model employs a Long Short-Term Memory (LSTM) network for named entity recognition. The LSM network selectively forgets some historical information and adds some current input information through three gate structures: an input gate, a forget gate, and an output gate, ultimately integrating these into the current state and generating the output state.

[0129] Taking sentences as units, each character in the dictionary has a unique identifier, thus yielding a one-hot vector for each character, with the dimension being the size of the dictionary. The first layer of the named entity recognition model based on Long Short-Term Memory (LSTM) networks is a lookup layer. This layer uses a pre-trained or randomly initialized word embedding matrix to map each character in the sentence from a one-hot vector to a low-dimensional, dense character vector. Dropout is applied before inputting to the next layer to mitigate overfitting.

[0130] The second layer of the named entity recognition model based on Long Short-Term Memory (LSTM) networks is a bidirectional LSTM layer. This layer automatically extracts sentence features, using the character-embedding sequence of each word in a sentence as input to each time step of the bidirectional LSTM. The hidden state sequence output by the forward LSTM is then concatenated with the hidden states output by the backward LSTM at each position to obtain the complete hidden state sequence. After setting dropout, a linear layer is applied to obtain the automatically extracted sentence features.

[0131] The third layer of the named entity recognition model based on long short-term memory networks is the CRF layer, which performs sentence-level sequence labeling.

[0132] The key statement recognition module 153 is used to segment the standardized data after adding data labels, represent the words and sentences obtained from the segmentation process into vectors through word embedding, and sort the words and sentences after vector representation using machine learning methods. The words and / or sentences ranked before the preset value are regarded as key statements to obtain the key statement recognition results.

[0133] The key statement identification module uses an "extraction-based summarization" approach to identify key statements. Its workflow is as follows: First, the standardized data after adding data labels is segmented into sentences. During sentence segmentation, words and sentences can be separated by commas. After obtaining the segmented words and sentences, they are represented as vectors using word embedding. After obtaining the vector representations of words and sentences, machine learning methods are used to rank them. A feature-based ranking method can be selected, scoring each word and sentence based on different features, and then ranking them according to their scores. Features used in feature-based ranking methods can include word / sentence length, word / sentence position, whether title words are included, and sentence keywords. After ranking, words and / or sentences ranking higher than a preset value are identified as key statements.

[0134] To avoid situations where top-ranked words and / or sentences express similar meanings, a penalty factor can be introduced. Starting from the second-ranked word / sentence after feature-based sorting, each word / sentence is re-scored based on the penalty factor. The words / sentences are then sorted according to their re-scored scores, and the sorted words / sentences are output in the order they were arranged before the sentence segmentation process, thereby improving the coherence of key sentence recognition.

[0135] The classification analysis module 154 is used to classify the normalized data after adding data labels based on the text features of the normalized data after adding data labels, based on the trained classifier model, and obtain the classification analysis results. The trained classifier model is obtained by training a preset text classification algorithm in advance based on the text vector space of the sample text features.

[0136] The clustering analysis module 155 is used to extract features from the normalized data after adding data labels using TF-IDF, and to perform clustering using the density-based DBSCAN algorithm to obtain the clustering analysis results.

[0137] The density-based DBSCAN algorithm can discover clusters of arbitrary shapes and filter outliers, avoiding classifying outliers into a particular cluster and thus preventing clustering bias.

[0138] The association analysis module 156 is used to filter relevant data from the data storage subsystem according to preset data characteristics, perform association analysis using preset association analysis algorithms, and obtain association analysis results.

[0139] The data analysis subsystem analyzes data through various data governance methods and data analysis models, forming effective business data support.

[0140] For example, such as Figure 7 As shown, the dark web data situation display subsystem 160 may include a data search module 161 and a data display module 162.

[0141] The data search module 161 is used to retrieve target site data based on search terms, using relational databases and distributed full-text search technology, and to optimize and automatically aggregate the search results; it uses an improved inverted incremental indexing technology to store and automatically progressively update the search results; it uses query expansion technology in information retrieval to establish a query expansion retrieval model, expands search terms and clarifies search semantics through local and global analysis methods, and obtains data related to the search terms from the target site data to obtain search results.

[0142] The data display module 162 is used to convert target site data into graphical information through visualization technology. Based on the visualization display analysis engine tool, it displays the graphical information in the form of charts through a preset layout and display method, and realizes human-computer interaction.

[0143] The Dark Web Data Situation Display Subsystem improves the system's retrieval performance by enabling different levels of searching based on file type, data tags, and data characteristics through the data search module. The data display module can graphically present the overall situation of the collected dark web data, supporting various display methods such as histograms or pie charts based on dark web type, data type, language type, and statistical characteristics, making the data more vivid, intuitive, and easy to understand.

[0144] For example, such as Figure 8 As shown, the dark web monitoring and analysis system may also include a source tracing subsystem 170.

[0145] The tracing subsystem 170 is used to trace the origins of dark web nodes and network attacks, perform tracing analysis based on the tracing results, and establish a dark web tracing database.

[0146] For example, such as Figure 9 As shown, the tracing subsystem 170 may include a dark web node tracing module 171 and a network attack tracing module 172.

[0147] The dark web node tracing module 171 is used to trace dark web nodes, obtain the description information of the traced dark web nodes, perform dark web node tracing analysis based on the description information and the clear web information, and establish a dark web tracing database based on the description information.

[0148] The dark web node tracing module can trace the origins of dark web nodes, including relay nodes and exit nodes, obtaining descriptive information such as domain names, addresses, open ports, and SSH (Secure Shell) fingerprints. This descriptive information is then compared with surface-level information to perform dark web node tracing analysis. Furthermore, the module can establish a dark web tracing database based on the descriptive information of dark web nodes, storing this information and providing a basis for further analysis.

[0149] The network attack tracing module 172 is used to detect network attack behavior. After detecting a network attack, it tracks and locates the attack data stream to identify the network device or host that sent the attack data. It analyzes the relevant information of the network device or host to determine whether the network device or host is controlled by a third party. If the network device or host is controlled by a third party, it sequentially identifies the control nodes at each level of the network device or host up to the source control node. The source control node is identified and analyzed to determine its attribute information. The attribute information of the source node can include its identity information, location information, etc. The identity information of the source node can be its name, account, or other information that reflects its identity. The location information of the source node can be its physical location information, such as its region and latitude / longitude. The location information of the source node can also be its virtual location information, such as its IP address and MAC address.

[0150] The network attack tracing module can reconstruct the network attack process and perform in-depth tracing of attackers.

[0151] For example, such as Figure 10 As shown, the dark web monitoring and analysis system may also include an IP address pool management subsystem 180. The IP address pool management subsystem 180 is used to maintain and manage the IP addresses used by dark web access proxies based on the IP address pool, so that there are a sufficient number of available IP addresses in the IP address pool for dark web access proxies to use.

[0152] The IP address pool management subsystem can be configured with detection intervals to ensure that the number of available IP addresses in the pool meets the needs of dark web access proxies. The information that the IP address pool management subsystem needs to maintain includes IP addresses, their regions, and communication status.

[0153] Those skilled in the art will understand that the above embodiments are specific implementations of this disclosure, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this disclosure.

Claims

1. A dark web monitoring and analysis system, characterized in that, The dark web monitoring and analysis system includes: The dark web access subsystem is used to establish a communication link with the dark web based on the dark web access proxy, and to replace the communication link in a timely manner according to the status of the communication link to ensure dark web access. The dark web includes at least one of Tor dark web, I2P dark web, and ZeroNet dark web. The data collection agent subsystem is used to collect data from target sites on the dark web using a web crawler program according to the data collection task. A data storage subsystem is used to store and manage the target site data using a distributed storage method; The data processing subsystem is used to process the target site data according to the pre-configured data processing rules, obtain the normalized data corresponding to the target site data, and add data tags to the normalized data; The data analysis subsystem is used to analyze and process the standardized data after adding data tags using statistical analysis methods, and generate data analysis results corresponding to the target site data. The Dark Web Data Situation Display Subsystem is used to perform different levels of retrieval of the target site data based on search terms, and to display the target site data in a graphical manner through multiple dimensions. The data storage subsystem is also used to store and manage standardized data using a distributed storage method; The dark web access subsystem discovers dark web sites in two ways: The first method uses a list of dark web sites as seed addresses. A crawler periodically scans the entire list to determine if a site is accessible and updates its accessibility status. If a site is inaccessible, it is skipped, and the crawler continues scanning the next site in the list. If a site is accessible, its page is downloaded, parsed, and the URL extracted. The extracted URL is then checked to see if it represents a new domain. If it is, it is added to the dark web site list. The second method discovers dark web sites in the data during data collection and compares them with the dark web site list. If a discovered site does not exist in the list, it is added as a new site. The data analysis subsystem includes: The key sentence recognition module is used to segment the standardized data after adding data labels. The words and sentences obtained from the segmentation are represented as vectors through word embedding. Machine learning methods are used to rank the vector-represented words and sentences. The words and / or sentences ranked before the preset value are identified as key sentences to obtain the key sentence recognition results. A penalty factor is introduced. Starting from the second-ranked word / sentence after feature-based ranking, each word / sentence is re-scored based on the penalty factor. The words / sentences are ranked according to the re-scored scores, and the ranked words / sentences are output in the order before the segmentation.

2. The dark web monitoring and analysis system according to claim 1, characterized in that, The dark web access subsystem includes: The Tor access module is used to establish a communication link with the Tor dark network based on the Tor dark network access proxy, and to replace the communication link in a timely manner according to the status of the communication link to ensure Tor dark network access. The I2P access module is used to establish a communication link with the I2P dark web based on the I2P dark web access proxy. It adds multiple proxy services to the I2P dark web access proxy. When one proxy service is unavailable, the other proxy services are used to establish a communication link with the I2P dark web to ensure I2P dark web access. The ZeroNet access module is used to establish a communication link with the ZeroNet dark network based on the ZeroNet dark network access proxy, and to replace the communication link in a timely manner according to the status of the communication link to ensure the realization of ZeroNet dark network access.

3. The dark web monitoring and analysis system according to claim 1, characterized in that, The data acquisition agent subsystem includes: The data acquisition task scheduling module is used to configure data acquisition tasks and acquisition strategies; The crawler module is used to divide the collection task into multiple collection sub-tasks, and use the collection strategy to use a distributed web crawler to crawl target sites in the dark web according to the collection sub-tasks in order to obtain the target site data. The content parsing module is used to parse the target site data to obtain the data content of the target site data; The feature extraction module is used to extract the features and feature values ​​of the data content.

4. The dark web monitoring and analysis system according to claim 1, characterized in that, The data storage subsystem includes a data distribution module, a statement parsing module, a parallel processing module, and a summary processing module, wherein: The data distribution module is used to receive data query statements and distribute the data query statements to the statement parsing module using a round-robin scheduling strategy. The statement parsing module is used to perform syntax and semantic checks on the data query statement, parse the data query statement, generate a syntax parse tree and a data query execution plan, and send the syntax parse tree and the data query execution plan to the data distribution module; The data distribution module is further configured to perform data segmentation on the syntax parse tree and the data query execution plan according to a preset data segmentation strategy, and distribute the segmented syntax parse tree and the data query execution plan to the corresponding areas of the parallel processing module; The parallel processing module is used to determine whether the data-segmented syntax parse tree has a preset execution order relationship based on the data query execution plan after data segmentation. If the data-segmented syntax parse tree has a preset execution order relationship, the data-segmented syntax parse tree is processed according to the preset execution order relationship. If the data-segmented syntax parse tree does not have a preset execution order relationship, the data-segmented syntax parse tree is processed according to a preset parallel processing method. The summary processing module is used to summarize the processing results of the parallel processing module to obtain the data query result corresponding to the data query statement.

5. The dark web monitoring and analysis system according to claim 1, characterized in that, The data processing subsystem includes: The data cleaning module is used to extract target data from the target site data according to pre-configured data extraction rules; remove useless symbols and stop words from the target data and perform deduplication processing according to pre-configured data cleaning rules to clean the target data; and map the cleaned target data to preset structured data fields according to preset field mapping rules to obtain the normalized data corresponding to the target data. The data tag management module is used to perform word segmentation on standardized data using at least one of the following methods: string matching-based word segmentation, statistical word segmentation, and understanding-based word segmentation, to obtain the word segmentation results of the standardized data, and to set data tags for the standardized data based on the word segmentation results.

6. The dark web monitoring and analysis system according to claim 1, characterized in that, The data analysis subsystem also includes: The semantic recognition module is used to perform semantic recognition on the normalized data after adding data labels using at least one of the Skip-gram model-based method or LSA algorithm to obtain the semantic recognition result. The named entity recognition module is used to perform named entity recognition on normalized data after adding data tags using a named entity recognition model based on long short-term memory network, and obtain named entity recognition results. The named entity recognition model based on long short-term memory network includes a lookup layer, a bidirectional LSTM layer and a CRF layer. The classification analysis module is used to classify the normalized data after adding data labels based on the text features of the normalized data after adding data labels, based on the trained classifier model, and obtain the classification analysis results. The trained classifier model is obtained by training a preset text classification algorithm in advance based on the text vector space of the sample text features. The clustering analysis module is used to extract features from the normalized data after adding data labels using TF-IDF, and to perform clustering using the density-based DBSCAN algorithm to obtain the clustering analysis results. The association analysis module is used to filter relevant data from the data storage subsystem according to preset data characteristics, perform association analysis using preset association analysis algorithms, and obtain association analysis results.

7. The dark web monitoring and analysis system according to claim 1, characterized in that, The dark web data situation display subsystem includes: The data search module is used to retrieve data from the target site based on search terms using relational databases and distributed full-text search technology, and to optimize and automatically aggregate the search results; it uses an improved inverted incremental indexing technology to store and automatically progressively update the search results; it utilizes query expansion technology in information retrieval to establish a query expansion retrieval model, expands search terms and clarifies search semantics through local and global analysis methods, and obtains data related to the search terms from the target site data to obtain search results; The data display module is used to convert the target site data into graphical information through visualization technology. Based on the visualization display and analysis engine tool, the graphical information is displayed in the form of charts through a preset layout and display method, and human-computer interaction is realized.

8. The dark web monitoring and analysis system according to any one of claims 1 to 7, characterized in that, The dark web monitoring and analysis system also includes: The source tracing subsystem is used to trace the origins of dark web nodes and network attacks, perform source tracing analysis based on the tracing results, and establish a dark web source tracing database.

9. The dark web monitoring and analysis system according to claim 8, characterized in that, The source tracing subsystem includes: The dark web node tracing module is used to trace dark web nodes, obtain description information of the traced dark web nodes, perform dark web node tracing analysis based on the description information and clear web information, and establish the dark web tracing database based on the description information. The network attack tracing module is used to detect network attack behavior. After detecting network attack behavior, it tracks and locates the attack data stream to identify the network device or host that sent the attack data. It analyzes the relevant information of the network device or host to determine whether the network device or host is controlled by a third party. If the network device or host is controlled by a third party, it sequentially identifies the control nodes at each level of the network device or host up to the source control node. It identifies and analyzes the source control node to determine the attribute information of the source control node.

10. The dark web monitoring and analysis system according to any one of claims 1 to 7, characterized in that, The dark web monitoring and analysis system also includes: The IP address pool management subsystem is used to maintain and manage the IP addresses used by the dark web access proxy based on the IP address pool, so that there are a sufficient number of available IP addresses in the IP address pool for the dark web access proxy to use.

Images