Dark web data collection and analysis method and system

Through dark web data collection and analysis methods and systems, multi-threading and multi-task technology are used to bypass the protection mechanism and perform data collection and analysis, solving the limitations of dark web information mining in the existing technology, and realizing the dark web data Accurate collection and visual display have enhanced the scope of intelligence data acquisition and promoted social security.

CN118939891AInactive Publication Date: 2024-11-12ZHEJIANG POLICE COLLEGE
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411008199.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-12
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

It is difficult for the existing technology to effectively and systematically mine and supervise illegal information in the dark web, limiting the scope of intelligence data acquisition and affecting social security.

Method used

Provide a dark web data collection and analysis method and system, obtain the URL address of Chinese dark web transaction websites through the URL acquisition module, use multi-threading and multi-task technology to bypass the website protection mechanism, convert the data format, and use Jieba library for word segmentation and filtering. , use burpsuite software to grab login information, generate word cloud diagrams and charts for visual display.

Benefits of technology

It realizes accurate collection and analysis of dark web data, breaks through the limitations of surface networks, provides more information mining directions, enhances information supervision capabilities, and ensures the accuracy of data collection and intuitive visual display.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939891A_ABST
    Figure CN118939891A_ABST
Patent Text Reader

Abstract

The invention discloses a dark web data collection and analysis method and system, the system comprises a URL acquisition module, an analysis module and an analysis storage module, and the method comprises the following steps: S1, obtaining a URL address where key information of a Chinese dark web transaction website is located through the URL acquisition module, and carrying out classified storage on the obtained URL address; s2, the obtained URL addresses are traversed, and key page information in each URL address is obtained; and S3, converting the format of the obtained key page information and then storing the key page information. According to the scheme, technologies such as multi-task, multi-thread and multi-page acquisition, variable-frequency acquisition and bypassing a website protection mechanism are used, automatic data acquisition can be realized, then data analysis is performed, then visual display is performed, data on a dark web can be collected, and targeted analysis is performed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of web crawlers and data analysis, and in particular to a dark web data collection and analysis method and system. Background Art

[0002] The dark web is an encrypted network system based on the Internet and built using anonymous communication technology. It requires special software and special methods to access.

[0003] Through practical observations, it is found that illegal activities on the dark web are highly correlated with the dark web market.

[0004] Based on the limitations of current intelligence mining work on the dark web, it is extremely important to conduct systematic mining research on illegal information on the dark web. Through technical means, systematic mining research on dark web intelligence can effectively carry out information supervision, broaden the scope of intelligence data acquisition, and thus effectively maintain social security. Summary of the invention

[0005] The present invention mainly aims at the above-mentioned problems and provides a dark web data collection and analysis method and system, which can collect data on the dark web and conduct targeted analysis, providing more directions for intelligence mining work.

[0006] The purpose of the present invention is mainly achieved through the following scheme: a dark web data collection and analysis method, comprising the following steps: S1. Obtain the URL address of the key information of the Chinese dark web transaction website through the URL acquisition module, and classify and store the obtained URL addresses; S2. Traverse the acquired URL addresses to obtain key page information in each URL address; S3. Convert the acquired key page information into a new format and store it.

[0007] The socks5 protocol is mostly used to transmit data in the dark web, and the request library is mainly for the http protocol. Therefore, you need to use the cow software to convert the socks5 protocol into the http protocol and then obtain the traffic package of the dark web. You can successfully obtain the pictures, HTML and other content in the dark web pages.

[0008] Preferably, step S3 is specifically as follows: S301, calling the pandas library to convert the crawled data into a two-dimensional array to meet the needs of subsequent keyword selection and geographic information analysis, and then converting the transaction release date data from string format to datetime format through the to_datetime function; S302, call the jieba library, use the precise mode to segment the transaction content, split the crawled transaction content into several phrases, set a filtering function, and filter out single-word phrases; S303. Create a key-value pair: key is a word, value is the number of times it appears, traverse all words, increase the corresponding value by 1 each time it appears, count the number of times each word appears, and save the final result in the dictionary counts.

[0009] Preferably, in step S1, burpsuite software is used to capture packets to obtain login cookies and request header information. The legitimacy and security of crawler operations can be ensured by using burpsuite software to capture the cookies and request header information required for login.

[0010] Preferably, in step S2, the key page information includes the merchant ID and the product description.

[0011] Preferably, in step S2, a threading library is used to create multi-threaded tasks and execute crawler tasks in parallel. Since the access speed of the dark web is slow and the cookies of the website have time limits, it is necessary to enable multi-threading for parallel crawling so that all data can be crawled and output at one time. The threading library can be used to create multi-threaded tasks and execute crawler tasks, tkinter tasks, etc. in parallel.

[0012] This solution uses multi-threading and multi-tasking technology to traverse the URL addresses of websites that store key information, bypass the website protection mechanism, and ensure the accuracy of data collection.

[0013] As a preferred method, the dark web data collection and analysis method also includes a data display step, specifically: first, the phrases after the jieba database segmentation are spliced ​​with " / " using the join function, and then the phrases are input into the WordCloud function to generate a word cloud diagram. The processed data finally needs to be presented in a visual form to make the data more intuitive.

[0014] A dark web data collection and analysis system, which runs the dark web data collection and analysis method as described above, comprises: URL acquisition module, which obtains the URL address of the key information of the Chinese dark web transaction website and classifies and stores the obtained URL addresses; The parsing module traverses the acquired URL addresses and obtains the key page information in each URL address; this is the core part of the entire solution, and the next step of data analysis can be carried out through these key data; The analysis and storage module converts the acquired key page information into a format for storage.

[0015] Preferably, the dark web data collection and analysis system also includes a display module, which analyzes geographic information, traverses the place name list to obtain the frequency of each place name in the data transaction content, and generates relevant charts to display the data transaction situation in different regions; uses the pyecharts library to generate word cloud maps and charts to display the frequency of keyword occurrence and the proportion of each category of data.

[0016] This solution uses technologies such as multi-tasking, multi-threading, multi-page collection, variable frequency collection, and bypassing website protection mechanisms to automatically collect data, perform data analysis, and then perform visual display.

[0017] The URL acquisition module is mainly responsible for obtaining the URL address of the key information of the Chinese dark web trading website and storing them together and classifying them. Because the Chinese dark web website has a clear classification of all the goods on sale, when crawling data, you can directly crawl according to the category, which can be divided into categories such as data and tools. Since the Chinese dark web transaction has a login verification mechanism, this solution uses cookies to bypass the login mechanism, configures the burpsuite software for packet capture, and can obtain login cookies, request headers and other information.

[0018] After obtaining key information, the data needs to be converted into a form that is convenient for data analysis and stored locally. The data obtained through HTML parsing needs to be stored. Common storage methods include: txt, csv, excel, json, etc. This solution uses csv to store data. This method is more convenient and suitable for data with a small storage volume. It can also be directly imported into the database. Save as a relational database, such as mysql, oracle, etc., save as a non-relational database, such as Redis, etc.

[0019] The amount of data extracted from each data selection is counted, and the amount of data for each category of each topic is integrated. If you want to emphasize the proportion of each category, you can use a pie chart to display it. The final statistical data can provide data sources and inspiration for the next step of visualization.

[0020] The process of data analysis is also a modeling process, that is, the process of selecting and extracting massive and disorderly data to make them into meaningful and neat charts.

[0021] First, after completing the module classification, the data is selected by keywords. For example, if you want to extract the data transaction volume related to student occupations, select the table using the keyword "student", and then you can get the data transaction volume related to student occupations. During the selection process, you may find that the data volume related to a certain category is so scarce that it can be ignored, so this category is discarded to ensure the stability of extracting various types of data.

[0022] The transaction content of data transactions often includes the origin of the personal information being sold. Therefore, by performing word segmentation analysis on the transaction content, the personal information leakage situation of each province in the country can be obtained. The names of provinces and cities and their subordinate counties and cities are saved as txt, and the frequency of each place name appearing in the data transaction content is obtained by traversing the place names in place.txt. The frequency of information leakage involving each province and city is the sum of the frequencies of its and its subordinate counties and cities appearing in the data transaction content.

[0023] This plan combines the actual needs of information construction and provides a new direction for maintaining public safety through the mining and research of dark web information. Compared with traditional illegal information mining, this plan will break through the limitations of the surface network and go deep into the dark web, providing more directions for intelligence mining. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flow chart of a dark web data collection and analysis method of the present invention; Figure 2 It is a dark web data packet transmission protocol conversion process diagram of the present invention; Figure 3 This is an overall framework diagram of a dark web data collection and analysis system of the present invention; Figure 4 It is a flow chart of data analysis ideas of the present invention. DETAILED DESCRIPTION

[0025] The technical solution of this solution is further specifically described below through embodiments and in conjunction with the accompanying drawings.

[0026] A dark web data collection and analysis method of this embodiment, such as Figure 1 As shown, the following steps are included: S1. Obtain the URL address of the key information of the Chinese dark web transaction website through the URL acquisition module, and classify and store the obtained URL addresses; S2. Traverse the acquired URL addresses to obtain key page information in each URL address; S3. Convert the acquired key page information into a new format and store it.

[0027] The whole system consists of three modules: URL acquisition module, parsing module and analysis storage module. First, according to the characteristics of the dark web, configure the dark web browser proxy and bypass it (such as Figure 2 As shown in the figure), secondly, the burpsuite software is used to capture the cookies and request header information required for login to ensure the legitimacy of subsequent crawler operations. Subsequently, multi-threading is used to traverse the Chinese dark web trading website to obtain the URL, while bypassing the website protection mechanism to ensure the accuracy of data collection. The parsing module uses multi-threading to achieve parallel crawling, uses cookies to bypass login verification, and extracts key page information for each URL address, including core information such as merchant ID and product description, such as Figure 3 For ease of use, you can use the tkinter module to design an exe interface. Through this program, you can crawl the data in the dark web and save it locally by entering simple information. You can also see the real-time crawling progress during the crawling process, and the list on the right will also display the crawled content in real time.

[0028] In terms of data storage, the csv format is selected as the data storage method to facilitate subsequent analysis and import into the database. During the storage process, the key information obtained is converted into a format to ensure that it is suitable for subsequent data analysis. Using the HTML parsing library, the data is saved locally in the form of a csv file and can also be imported into the database, providing a convenient data source for further intelligence mining.

[0029] In the process of data analysis, we first analyze the crawled data and sort out the general data analysis ideas, such as Figure 4 , and then make full use of the functions of the pandas library and the jieba library. First, use the pandas library to convert the acquired data into a two-dimensional array to facilitate the subsequent keyword selection and geographic information analysis. Then, use the jieba library to segment the transaction content and filter out single-word words to ensure the stability of extracting various types of data. The preparation work at this stage provides strong support for the subsequent visualization display.

[0030] Finally, a visual display is performed. Through geographic information analysis, the place name list is traversed to obtain the frequency of each place name in the data transaction content, and relevant charts are generated to show the data transaction situation in different regions. The pyecharts library is used to generate word cloud maps and charts to show the frequency of keywords and the proportion of each category of data.

[0031] It should be understood that this embodiment is only used to illustrate the present invention and is not used to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope limited by the appended claims of the application.

Claims

1. A dark web data collection and analysis method, characterized in that: The following steps are involved: S1. Obtain the URL address of the key information of the Chinese dark web transaction website through the URL acquisition module, and classify and store the obtained URL addresses; S2. Traverse the acquired URL addresses to obtain key page information in each URL address; S3. Convert the acquired key page information into a new format and store it.

2. A dark web data collection and analysis method according to claim 1, characterized in that: Step S3 is specifically as follows: S301. Call the pandas library to convert the crawled data into a two-dimensional array, and convert the transaction release date data from string format to datetime format through the to_datetime function; S302, call the jieba library, use the precise mode to segment the transaction content, split the crawled transaction content into several phrases, set a filtering function, and filter out single-word phrases; S303. Create a key-value pair: key is a word, value is the number of times it appears, traverse all words, increase the corresponding value by 1 each time it appears, count the number of times each word appears, and save the final result in the dictionary counts.

3. A dark web data collection and analysis method according to claim 1, characterized in that: In step S1, burpsuite software is used to capture packets to obtain login cookies and request header information.

4. A dark web data collection and analysis method according to claim 1, characterized in that: In step S2, the key page information includes the merchant ID and product description.

5. A dark web data collection and analysis method according to claim 1 or 4, characterized in that: In step S2, a threading library is used to create multi-threaded tasks to execute crawler tasks in parallel.

6. A dark web data collection and analysis method according to claim 1, characterized in that: It also includes data display steps, specifically: first use the join function to concatenate the phrases after word segmentation in the jieba library with " / ", and then input the phrases into the WordCloud function to generate a word cloud diagram.

7. A dark web data collection and analysis system, running the dark web data collection and analysis method as claimed in claim 1, characterized in that: include: URL acquisition module, which obtains the URL address of the key information of the Chinese dark web transaction website and classifies and stores the obtained URL addresses; The parsing module traverses the acquired URL addresses and obtains the key page information in each URL address; The analysis and storage module converts the acquired key page information into a format for storage.

8. A dark web data collection and analysis system according to claim 7, characterized in that: It also includes a display module, which analyzes geographic information, traverses the place name list to obtain the frequency of each place name in the data transaction content, and generates relevant charts to display the data transaction situation in different regions; uses the pyecharts library to generate word cloud maps and charts to display the frequency of keyword occurrence and the proportion of each category of data.