A method of monitoring data dark web leaks and apparatus therefor

Abstract

The present application belongs to the field of network security, and in particular to a method and device for monitoring data dark web leakage. The device for monitoring data dark web leakage comprises a dark web resource scheduling and collection module, a data preprocessing and analysis module, a data analysis and correlation matching module, an alarm and report generation module. The dark web resource scheduling and collection module is used to realize the management of dark web resources, anonymous access and adaptive crawling, including: a dark web resource list maintenance unit: maintaining an updatable dark web resource list containing.onion site URLs, covering the entrances of online markets, forums and telegram groups for illegal data transactions. Through anonymous adaptive dark web collection, multi-modal data analysis, enterprise asset fingerprint multi-modal matching and machine learning risk assessment, the present application realizes the full-process automation of dark web data leakage monitoring, significantly improves the monitoring efficiency and accuracy, can quickly trace the leakage event and push real-time alarms, and helps the security team to prevent data misuse risks in advance.

Description

Technical Field

[0001] This invention relates to the field of network security technology, and in particular to a method and apparatus for monitoring data leaks on the dark web. Background Technology

[0003] Currently, the monitoring of data leaks from the dark web mainly faces the following technical challenges and problems:

[0004] Manual monitoring is inefficient: Existing methods mostly rely on security researchers manually browsing dark web sites, subscribing to threat intelligence sources, or joining specific channels. This method has a narrow coverage, is extremely inefficient, and is highly dependent on personal experience, making it unable to cope with the rapidly changing nature of dark web content.

[0005] Insufficient automation: Some basic automation tools can only achieve simple keyword matching or RSS subscription. They lack intelligent data collection, cleaning and correlation analysis capabilities, have a high false alarm rate, and have difficulty extracting effective information from encrypted conversations, images or obfuscated text.

[0006] Lack of precise correlation and tracing: Existing technologies struggle to quickly and accurately correlate fragmented data found on the dark web with protected corporate assets such as domain names, employee email formats, and internal code snippets. This makes it impossible to effectively determine the exact source and authenticity of leaked data, leading to delayed or ineffective early warnings.

[0007] Poor scalability and stability: Dark web access is slow and unstable. Simple crawler programs are difficult to achieve large-scale and sustainable data collection in the complex and dynamic dark web environment and are easily blocked by anti-crawler mechanisms. Therefore, we propose a method and device for monitoring data leakage on the dark web to solve the above problems. Summary of the Invention

[0008] The purpose of this invention is to address the shortcomings of existing technologies by proposing a method and apparatus for monitoring data leakage on the dark web.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] A device for monitoring data leakage on the dark web includes a dark web resource scheduling and acquisition module, a data preprocessing and parsing module, a data analysis and correlation matching module, and an alarm and report generation module;

[0011] The dark web resource scheduling and collection module includes a dark web resource list maintenance unit, an anonymization proxy unit, and an adaptive crawler unit, which are used to realize dark web resource management, anonymized access, and adaptive data crawling.

[0012] The data preprocessing and parsing module includes a data cleaning unit, a natural language processing unit, and a multimedia content extraction unit, which are used to perform data cleaning, text parsing, and multimedia information extraction.

[0013] The data analysis and correlation matching module includes an enterprise asset fingerprint database unit, a multi-mode matching unit, and an intelligent risk assessment unit, which are used to realize asset fingerprint management, data correlation matching, and risk level assessment.

[0014] The alarm and report generation module includes a real-time alarm unit, a visualization dashboard unit, and a report generation unit, which are used to realize real-time early warning, visualization display, and structured report output;

[0015] The four modules are connected sequentially through data interfaces to achieve full automation of the dark web data leakage monitoring process.

[0016] Preferably, the anonymization proxy unit integrates Tor client and other anonymous network protocols, has IP rotation function, and can filter collectable IPs through point-to-point judgment logic; the adaptive crawler unit is based on the Scrapy framework and supports JavaScript rendering, simulated login, bypassing simple CAPTCHAs, and complies with the target robots.txt rules.

[0017] Preferably, the multimedia content extraction unit integrates an OCR module and a hash value calculation function to extract text information from images and screenshots and generate hash values ​​for file samples; the NLP processing unit supports word segmentation, part-of-speech tagging, and named entity recognition, and can identify key entities such as personal names, organization names, currencies, and cryptocurrency wallet addresses.

[0018] Preferably, the enterprise asset fingerprint database unit supports pre-configured or user-configured enterprise digital asset fingerprints, including but not limited to company domain names, employee email naming rules, internal project codes, database pattern fragments, GitHub code fragments, and certificate serial numbers; the multi-pattern matching unit adopts a combination of regular expression precise matching and cosine similarity fuzzy matching to achieve the association and comparison of key information with asset fingerprints.

[0019] Preferably, the intelligent risk assessment unit is based on a classification model trained by SVM or neural network to determine whether dark web content involves data transactions and to assess the risk level of leaked data, which is divided into low, medium, high, and critical. The real-time alarm unit supports pushing alarms through integrated channels such as Email, Slack, SMS, and SIEM. The alarm information includes the leaked data sample, source link, and discovery time.

[0020] A method for monitoring data leaks on the dark web includes the following steps:

[0021] S1: Dark Web Resource Scheduling and Collection: Maintain a dark web resource list, access the dark web through anonymized proxies, and call adaptive crawlers to collect dark web data;

[0022] S2: Data Preprocessing and Parsing: Clean the raw data, extract key entities by parsing the text using NLP, and extract text from multimedia content and calculate file hash values ​​using OCR;

[0023] S3: Data Analysis and Association Matching: Configure an enterprise asset fingerprint database, use multi-mode matching to compare key information with asset fingerprints, and assess risk level through classification models;

[0024] S4: Alarm and Report Generation: If the risk level reaches the threshold, a real-time alarm will be pushed, the visual dashboard will be updated, and a structured security report will be generated.

[0025] Preferably, in step S1, the dark web resource list includes illegal data transaction entry points such as .onion sites, dark web markets, forums, and Telegram groups; the adaptive crawler calls the corresponding crawling strategy according to the target resource type, which includes static web pages, dynamic JS, and chat streams.

[0026] Preferably, in step S1, resource management of dark web sites is performed: an updatable dark web resource list is maintained through a device, including online marketplaces, forums, and Telegram group entrances suspected of being dark web sites used for illegal data transactions.

[0027] Preferably, in step S3, multi-pattern matching includes regular expression exact matching, including domain name, email rules and cosine similarity fuzzy matching for matching fixed-format asset fingerprints, including internal project code names and code snippets for matching non-fixed-format asset fingerprints; risk level assessment determines the "data transaction relevance" of dark web content through a classification model and outputs the risk level.

[0028] Preferably, in step S4, the trigger condition for real-time alarms is a risk level ≥ "high"; the visual dashboard displays a monitoring overview, historical alarms, risk trend charts, and source distribution charts; structured reports support daily, weekly, and on-demand generation, including event details, risk levels, and handling suggestions.

[0029] The beneficial effects of this invention are:

[0030] 1. Overall System Architecture: The integration and collaboration of four major modules—collection, preprocessing, analysis and matching, and alarm—achieves full-process automation of dark web data leakage monitoring, solving the problem of fragmented monitoring in existing technologies;

[0031] 2. Anonymous resource scheduling and collection: Integrating the Tor client and adaptive crawler engine, it enables stable and covert collection of dynamic dark web resources, solving the problems of slow access speed and easy blocking of dark web, and improving the scale and sustainability of collection.

[0032] 3. Multimodal data parsing: Based on NLP and OCR technologies, it extracts key entities from unstructured text and images, solving the problem that existing tools cannot parse multimedia content and obfuscated text, and improving the effective information extraction rate;

[0033] 4. Enterprise Digital Asset Fingerprint Database and Multi-pattern Matching: Build a customized asset fingerprint database and combine precise and fuzzy matching algorithms to achieve accurate association between dark web data and enterprise assets, solve the problem of fragmented data being untraceable, and reduce false alarm rate.

[0034] 5. Machine learning-driven risk assessment: By using SVM and neural network models to classify data transaction content and assess risks, the inefficiency of manual judgment is solved, and the risk level is automatically classified.

[0035] 6. Fully automated process: It connects the collection, parsing, analysis, and alarm processes in a series, eliminating the need for manual intervention, solving the problem of low efficiency in manual monitoring, and enabling real-time response to leakage events. Attached Figure Description

[0036] Figure 1 This is a block diagram of a device for monitoring data leakage on the dark web, as proposed in this invention. Detailed Implementation

[0037] The following is in conjunction with the appendix Figure 1 This application will be described in further detail.

[0038] This application discloses a method and apparatus for monitoring data leakage on the dark web.

[0039] Example 1

[0040] Reference Figure 1 A device for monitoring data leaks on the dark web includes a dark web resource scheduling and acquisition module, a data preprocessing and parsing module, a data analysis and correlation matching module, and an alarm and report generation module;

[0041] Dark Web Resource Scheduling and Collection Module: Used to manage dark web resources, anonymize access, and perform adaptive crawling, including:

[0042] Dark Web Resource List Maintenance Unit: Maintains an updatable dark web resource list including .onion site URLs, covering online marketplaces, forums, and Telegram groups for illicit data transactions.

[0043] Anonymization Proxy Unit: Integrates Tor client or other anonymous network protocols, providing anonymization and IP rotation functions. It filters collectable IPs through point-to-point judgment logic to ensure the concealment and stability of collection.

[0044] Adaptive crawler unit: Based on the Scrapy framework, it calls the corresponding crawler strategy for different types of resources such as static web pages, dynamic JS loading content, and chat streams, handles JavaScript rendering, simulates login, bypasses simple CAPTCHAs, and complies with the target robots.txt rules.

[0045] Data preprocessing and parsing module: Used to clean, parse, and extract key information from the collected raw data, including:

[0046] Data cleaning unit: Based on preset preprocessing strategies, it removes noise such as HTML tags, advertisements, and irrelevant characters from the original data.

[0047] NLP processing unit: performs word segmentation, part-of-speech tagging, and named entity recognition (NER) on unstructured text, identifying key entities such as people's names, locations, organization names, currencies, and cryptocurrency wallet addresses.

[0048] Multimedia content extraction unit: integrates an OCR optical character recognition module to extract text information from images and screenshots, and calculates the file sample hash value for subsequent matching.

[0049] Data analysis and correlation matching module: used to perform correlation analysis and risk assessment between dark web data and enterprise assets, including:

[0050] Enterprise Asset Fingerprint Unit: Supports pre-configured or user-configured "digital asset fingerprints" of monitoring targets, such as company domain names, employee email naming rules, internal project codes, database schema fragments, GitHub code fragments, and certificate serial numbers.

[0051] Multi-pattern matching unit: It adopts a combination of regular expression precise matching and cosine similarity fuzzy matching to compare the extracted key information with the asset fingerprint database.

[0052] Intelligent Risk Assessment Unit: Based on classification models trained using SVM, neural networks, etc., it determines whether the content involves data transactions and assesses the potential risk level of leaked data.

[0053] Alarm and report generation module: used to realize real-time alerts, visualization, and report output, including:

[0054] Real-time alert unit: When a high-confidence matching result is detected, an alert is pushed through preset channels such as Email, Slack, SMS, and SIEM integration, including the context of the leaked data sample, source link, and discovery time.

[0055] Visual dashboard unit: Provides a web management interface to display monitoring overview, historical alarms, risk trend charts, and source distribution charts.

[0056] Report generation unit: Supports the generation of structured security reports on a regular daily, weekly, and on-demand basis for archiving and upward reporting.

[0057] A method for monitoring data leaks on the dark web, characterized by comprising the following steps:

[0058] S1: Dark Web Resource Scheduling and Collection: Maintain a dark web resource list, access the dark web through anonymized proxies, and call adaptive crawlers to collect dark web data;

[0059] The steps involved in dark web resource scheduling and collection are as follows:

[0060] 1.1 Maintaining a Dark Web Resource List: The device's dark web resource list maintenance unit updates and stores a list of dark web URLs containing illegal data transaction entry points (such as .onion sites, forums, and Telegram groups).

[0061] 1.2 Anonymization Access Configuration: Start the anonymization proxy unit, integrate Tor client or other anonymization protocols, enable IP rotation function, and filter collectable IPs to ensure the concealment of collection;

[0062] 1.3 Adaptive Data Crawling: The adaptive crawler unit is invoked to select the appropriate strategy based on the target resource type (static webpage, dynamic JS, chat stream), handle JavaScript rendering, simulate login, bypass simple CAPTCHA, crawl dark web content and store the raw data;

[0063] S2: Data Preprocessing and Parsing: Clean the raw data, extract key entities by parsing the text using NLP, and extract text from multimedia content and calculate file hash values ​​using OCR;

[0064] The data preprocessing and parsing steps include:

[0065] 2.1 Data Cleaning: The data cleaning unit removes noise such as HTML tags, advertisements, and irrelevant characters from the original data to obtain preliminarily cleaned data.

[0066] 2.2 NLP Analysis: The NLP processing unit is activated to perform word segmentation, part-of-speech tagging, and NER on the cleaned text to extract key entities such as person names, organization names, and cryptocurrency addresses.

[0067] 2.3 Multimedia Parsing: The multimedia content extraction unit is invoked to extract text from images and screenshots using OCR, calculate the file sample hash value, and integrate the text with the multimedia parsing results.

[0068] S3: Data Analysis and Association Matching: Configure an enterprise asset fingerprint database, use multi-mode matching to compare key information with asset fingerprints, and assess risk level through classification models;

[0069] The data analysis and association matching steps include:

[0070] 3.1 Configure asset fingerprints: In the enterprise asset fingerprint database unit, pre-set or user-entered enterprise digital asset fingerprints to be monitored, domain names, email rules, and project codes;

[0071] 3.2 Multi-pattern matching: Through the multi-pattern matching unit, the key information obtained by parsing is compared with the asset fingerprint database using the regular expression of precise matching and the cosine similarity of fuzzy matching to screen suspected leaked data;

[0072] 3.3 Risk Assessment: Activate the intelligent risk assessment unit to determine whether suspected data involves data transactions through a classification model, and assess the risk level, which is divided into low, medium, high, and critical.

[0073] S4: Alarm and Report Generation: If the risk level reaches the threshold, a real-time alarm will be pushed, the visual dashboard will be updated, and a structured security report will be generated.

[0074] The steps involved in generating alarms and reports are as follows:

[0075] 4.1 Real-time alarm: If the risk level reaches the preset threshold, the real-time alarm unit will push alarm information through preset channels, including key context;

[0076] 4.2 Visualization: The monitoring overview, risk trend chart, and source distribution are updated in the visualization dashboard unit for the security team to view in real time;

[0077] 4.3 Report Generation: The report generation unit generates structured security reports according to preset cycles or user needs, archives them, and supports reporting.

[0078] Example 2

[0079] Taking "monitoring the dark web leakage of XYZCorp company's financial documents" as an example, the implementation process of this invention is explained in detail:

[0080] Step 1: Configure the enterprise asset fingerprint database

[0081] In the enterprise asset fingerprint database unit of the device, XYZCorp's customized asset fingerprint is entered: company name "XYZCorp", name of finance department head "JohnDoe", and financial document identifier "Confidential".

[0082] Step 2: Dark Web Data Collection

[0083] Dark web resource scheduling and collection module started:

[0084] The maintained dark web resource list includes the URL of a dark web forum (.onion site);

[0085] Anonymization proxy unit integrates Tor client, allowing access to the forum after IP rotation;

[0086] The adaptive crawler unit identifies the "posts with attachments" type in the forum and crawls the post titled "Q1EarningsLeak" along with an attached image.

[0087] Step 3: Data Preprocessing and Parsing

[0088] Data preprocessing and parsing module work:

[0089] The data cleaning unit removes advertisements and irrelevant comments from posts, while retaining titles and image attachments;

[0090] The multimedia content extraction unit calls the OCR interface to recognize text from images: "XYZCorp", "Confidential", "JohnDoe", "Q1 Revenue: XXX million yuan", "Net Profit: XXX million yuan";

[0091] The NLP processing unit performs NER on the OCR results to extract key entities: organization name "XYZCorp", person name "JohnDoe", and financial data "Q1 revenue XXX million yuan".

[0092] Step 4: Data Analysis and Association Matching

[0093] Data analysis and correlation matching module operation:

[0094] The multi-mode matching unit accurately matches the “XYZCorp” and “JohnDoe” extracted by OCR with the asset fingerprint database to confirm the target asset.

[0095] The intelligent risk assessment unit uses a neural network model to determine that if a post's title contains the word "Leak" and its content contains financial data, it falls under the category of "data transaction-related content" and is classified as "critical."

[0096] Step 5: Alarm and Report Generation

[0097] Alarm and report generation module response:

[0098] The real-time alert unit pushes alerts to the XYZCorp security team via Email and Slack, including image attachments, forum links, and the discovery time "202X Year Month Day XX Hour";

[0099] The visualization dashboard unit marks this event in the "Risk Trend Chart" and labels it with the risk level "Critical";

[0100] The report generation unit adds a new entry for "XYZCorp Financial Data Breach Incident" to the structured report of the day, recording the incident details, risk level, and handling recommendations.

[0101] Through the above implementation, the XYZCorp security team received an alert within 10 minutes of the financial documents being leaked on the dark web, and promptly took measures such as removing the dark web content and investigating the source of the internal data leak to prevent further misuse of the data.

[0102] This invention enables automated, large-scale, and continuous data collection and updating of the dark web, including marketplaces, forums, chat rooms, and blogs. Through intelligent semantic analysis, natural language processing, and pattern recognition technologies, it performs deep cleaning, extraction, and classification of the collected unstructured data, significantly reducing the false positive rate. It constructs a precise matching model based on enterprise digital asset fingerprints, which can efficiently associate dark web data with specific protected entities, enabling rapid tracing and verification of leaks, providing real-time and actionable security alerts, and generating detailed analysis reports to help security teams take countermeasures before data is misused.

[0103] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A device for monitoring data leakage on the dark web, characterized in that, It includes a dark web resource scheduling and collection module, a data preprocessing and parsing module, a data analysis and correlation matching module, and an alarm and report generation module; The dark web resource scheduling and collection module includes a dark web resource list maintenance unit, an anonymization proxy unit, and an adaptive crawler unit, which are used to realize dark web resource management, anonymized access, and adaptive data crawling. The data preprocessing and parsing module includes a data cleaning unit, a natural language processing unit, and a multimedia content extraction unit, which are used to perform data cleaning, text parsing, and multimedia information extraction. The data analysis and correlation matching module includes an enterprise asset fingerprint database unit, a multi-mode matching unit, and an intelligent risk assessment unit, which are used to realize asset fingerprint management, data correlation matching, and risk level assessment. The alarm and report generation module includes a real-time alarm unit, a visualization dashboard unit, and a report generation unit, which are used to realize real-time early warning, visualization display, and structured report output; The four modules are connected sequentially through data interfaces to achieve full automation of the dark web data leakage monitoring process.

2. The device for monitoring data leakage on the dark web according to claim 1, characterized in that, The anonymization proxy unit integrates Tor clients and other anonymous network protocols, has IP rotation capabilities, and can filter and collect IPs through point-to-point judgment logic. The adaptive crawler unit is customized based on the Scrapy framework, supports JavaScript rendering, simulated login, bypassing simple CAPTCHAs, and adheres to the target robots.txt rules.

3. The device for monitoring data leakage on the dark web according to claim 1, characterized in that, The multimedia content extraction unit integrates an OCR module and a hash value calculation function to extract text information from images and screenshots and generate hash values ​​for file samples. The NLP processing unit supports word segmentation, part-of-speech tagging, and named entity recognition, and can identify key entities such as personal names, organization names, currencies, and cryptocurrency wallet addresses.

4. The device for monitoring data leakage on the dark web according to claim 1, characterized in that, The enterprise asset fingerprint database unit supports pre-configured or user-configured enterprise digital asset fingerprints, including but not limited to company domain names, employee email naming rules, internal project codes, database schema fragments, GitHub code fragments, and certificate serial numbers; The multi-mode matching unit uses a combination of regular expression precise matching and cosine similarity fuzzy matching to achieve the association and comparison of key information with asset fingerprints.

5. The method and apparatus for monitoring data leakage on the dark web according to claim 1, characterized in that, The intelligent risk assessment unit is based on a classification model trained by SVM or neural network to determine whether dark web content involves data transactions and to assess the risk level of leaked data. The risk level is divided into low, medium, high and critical. The real-time alarm unit supports pushing alarms through integrated channels such as Email, Slack, SMS, and SIEM. The alarm information includes the leaked data sample, the source link, and the discovery time.

6. A method for monitoring data leaks on the dark web, characterized in that, Includes the following steps: S1: Dark Web Resource Scheduling and Collection: Maintain a dark web resource list, access the dark web through anonymized proxies, and call adaptive crawlers to collect dark web data; S2: Data Preprocessing and Parsing: Clean the raw data, extract key entities by parsing the text using NLP, extract text from multimedia content using OCR and calculate file hash values; S3: Data Analysis and Association Matching: Configure an enterprise asset fingerprint database, use multi-mode matching to compare key information with asset fingerprints, and assess risk levels through classification models; S4: Alarm and Report Generation: If the risk level reaches the threshold, a real-time alarm will be pushed, the visual dashboard will be updated, and a structured security report will be generated.

7. The method for monitoring data leakage on the dark web according to claim 6, characterized in that, In step S1, the dark web resource list includes illegal data transaction entry points such as .onion sites, dark web markets, forums, and Telegram groups; the adaptive crawler calls the corresponding crawling strategy according to the target resource type, which includes static web pages, dynamic JS, and chat streams.

8. The method for monitoring data leakage on the dark web according to claim 1, characterized in that, Step S1 involves resource management of dark web sites: an updatable list of dark web resources is maintained through a device, including online marketplaces, forums, and Telegram group entrances suspected of being dark web sites used for illegal data transactions.

9. A method for monitoring data leakage on the dark web according to claim 6, characterized in that, In step S3, multi-pattern matching includes regular expression exact matching, including domain name, email rules and cosine similarity fuzzy matching for matching fixed-format asset fingerprints, and internal project code names and code snippets for matching non-fixed-format asset fingerprints; The risk level assessment uses a classification model to determine the relevance of data transactions in dark web content and outputs a risk level.

10. A method for monitoring data leakage on the dark web according to claim 6, characterized in that, In step S4, the trigger condition for real-time alarms is a risk level ≥ high; the visual dashboard displays a monitoring overview, historical alarms, risk trend charts, and source distribution charts; structured reports support daily, weekly, and on-demand generation, including event details, risk level, and handling suggestions.

Images