Dark network data collection and extraction system and method

Abstract

The present invention relates to a dark network data collection and extraction system and method. The system comprises a dark network point discovery module, a dark network data crawling module, a page parsing and content extraction module, and a data fusion and storage module that are connected in sequence; the dark network point discovery module obtains dark web urls in multi-source data; the dark network data crawling module is configured with the Tor service, modifies Nutch configuration to make the Nutch interact with a socks protocol, and crawls data; the page parsing and content extraction module carries out page parsing, page vectoring, signature generation, similarity calculation, template set updating, and page content extraction; and the data fusion and storage module reorganizes the extracted content by using a data alignment strategy, and fuses similarly-closed data records into a database. According to the system and method disclosed by the present invention, a complete system design flow from page saving to content extraction is realized, and data support is provided for the discovery of illegal transaction activities in the dark network and the establishment of thedark network knowledge map.

Description

Technical field [0001] The invention relates to a dark network data collection and extraction system and method, and belongs to the technical field of internet network information. Background technique [0002] The Tor (The Onion Router) network provides users with anonymized services. While ensuring privacy, it also encourages criminal behavior. Many websites clearly mark the sale of drugs, guns and ammunition and other prohibited items. The Dark Web is different from the Surface Web and the Deep Web. The former is a page that can be directly crawled by search engines, and the latter is a page that must be accessed through dynamic requests. Tor achieves complete anonymity through the three-hop mechanism of routing. When using the Tor browser to access the dark web, there will be three transfer nodes between the entry node and the final destination server. The entry node knows the user's IP address, and the exit node knows the destination server's IP address and the transmitted d...

AI Technical Summary

Problems solved by technology

However, this patent is mainly aimed at the dynamic data acquisition of the Deep Web mentioned above, that is, the Deep Web. The pages in the Deep Web have no definite links and can only be accessed by constructing a dynamic query request, but after constructing a dynamic query request Conventional crawlers can crawl directly

Images