Black production application collection system based on propagation channel

Abstract

The present application belongs to the technical field of network security and network crime governance, and specifically relates to a black and grey industry application collection system. The system comprises a black production portal website screening system based on multi-dimensional features and a black production application dynamic collection system. The former is used to extract features from a seed website to be tested and identify black production portal websites that can provide a large number of application collections. The latter is used to automatically and massively download applications from the black production portal websites. The black production portal website screening system comprises a multi-dimensional feature extraction module and a learning and identification module. The black production application dynamic collection system comprises a high-adversarial dynamic content capture scheme and a collection logic scheme based on semantic priority. The present application can efficiently collect wild black production applications and effectively solve the lag in black production application discovery. It can also provide help or inspiration for other work in the field of black production plugging.

Description

Technical Field

[0001] This invention belongs to the field of network security and cybercrime governance technology, specifically involving a data collection system for black and gray industries (black market). Background Technology

[0002] The black and gray industries are currently developing rapidly and migrating to mobile devices. With rapid economic and social development, the methods and scope of these industries are also expanding rapidly. These black market applications seriously impact people's lives and livelihoods; telecommunications and online fraud perpetrated through fraudulent apps has become a major form of such crime. As people become increasingly reliant on mobile applications, traditional forms of crime are further migrating to mobile platforms, necessitating various measures to control these black market applications.

[0003] Traditional black market application data collection methods based on reports have shortcomings. Currently, monitoring departments and related agencies mainly rely on reports from victims or keyword matching, which are highly reactive. This approach has the following disadvantages:

[0004] 1. Delay: It can only respond to black market applications that have already been reported, by which time the black market applications may have already caused significant financial losses to people;

[0005] 2. Long cycle: By the time a report is received, the criminal gang may have already fled, and by the time the monitoring department intercepts it, the criminal application may have already gone offline and cannot collect data.

[0006] Large-scale data collection by cybercriminal applications faces challenges. Cybercriminal and gray-market applications often exhibit high levels of adversarial activity, which presents numerous challenges to large-scale data collection efforts:

[0007] 1. Black market apps often choose to spread covertly to evade regulation, and therefore will not be listed and promoted in regular app stores;

[0008] 2. Criminal gangs promote illicit applications on their own websites, but these websites are often deployed in batches and are highly mobile, with their numbers far exceeding the number of illicit applications they promote. Therefore, the efficiency of data collection methods through these websites is low.

[0009] 3. Black market websites employ anti-crawler techniques such as CAPTCHA recognition, webpage anti-debugging, and asynchronous dynamic loading, rendering traditional crawling methods gradually ineffective.

[0010] Discovering distribution channels provides an effective and sufficient data source for application collection. Black market application collection solutions based on distribution channels utilize black market portals as the source of application data, effectively solving the problems of traditional report-based solutions. Black market portals often provide numerous download links for black market applications, driving traffic to these applications. The distribution channel-based solution has the following advantages: 1. Large collection volume and high efficiency; a single black market portal can often collect a large number of applications. 2. Fresh data source, helping to identify black market applications still in the distribution stage, making them more valuable for analysis and tracking. Summary of the Invention

[0011] The purpose of this invention is to provide a black market application data collection system based on dissemination channels that can collect large amounts of data, is highly efficient, and has analytical and tracking value.

[0012] The present invention provides a black market application collection system based on dissemination channels, which includes identifying black market portal websites as dissemination channels and automating the collection of black market applications. Specifically, it identifies black market portal websites as dissemination channels based on their multi-dimensional characteristics; and employs highly adversarial dynamic collection tools to achieve automated, large-scale dynamic collection of applications associated with black market portals.

[0013] To this end, this invention constructs two black market portal website identification systems based on multi-dimensional features and a black market application dynamic acquisition system. The multi-dimensional feature-based black market portal website identification system extracts features from seed websites to be tested and identifies black market portal websites that can provide a large number of application acquisitions; the black market application dynamic acquisition system is used to automatically and in large quantities download applications from black market portal websites. This invention also designs two auxiliary modules to assist the operation of the core modules: a data storage module and a task scheduling module. The overall system architecture is as follows: Figure 1 As shown.

[0014] (I) A black market portal website identification system based on multi-dimensional features, the structure of which is as follows: Figure 2 As shown.

[0015] Black market portals are websites that integrate download links for various types of illicit applications, using eye-catching colors to entice users to download them. They are one of the primary channels for victims to download illicit applications. The applications they direct traffic to are frequently updated, making this channel a popular route to access a large number of illicit applications. However, black market portals differ from typical websites that spread illegal content. Their primary purpose is traffic generation, and the semantics of their content lack strong illicit characteristics. Traditional identification methods, such as keyword matching, are ineffective against black market portals, necessitating the search for new key characteristics.

[0016] A black market portal website identification system based on multi-dimensional features needs to accurately identify black market portal websites from seed websites. To address these challenges, this invention proposes a feature point selection scheme based on portal traffic generation functionality and a precise identification scheme based on ensemble learning. The specific schemes are described below.

[0017] (1) Feature point selection scheme based on portal traffic generation function (multi-dimensional feature extraction module)

[0018] This invention starts with the traffic-driving function of black market portal websites and deeply analyzes the webpage structure and image features of these websites.

[0019] Specifically, webpage structural features include: black market portals often contain more backlinks and channels than ordinary websites. Backlinks refer to external links outside the website; the more backlinks, the more obvious the traffic-driving characteristic. Channels are a special type of backlink used to calculate website traffic; the click-through rate of these backlinks is often used as the basis for settling promotion fees for portals, and is the main profit method for black market portals. Therefore, these are all significant characteristics that can distinguish black market portals. Structural features are extracted from webpages by combining information such as webpage layout, traffic, and keywords.

[0020] Specifically, this invention extracts five different structural features, including: ① Number of external links to the webpage: the number of links to external websites on this website; ② Number of external link channels: the number of external links with channel codes; ③ Number of lists: the number of table and list elements in the webpage layout; ④ Number of visits: the maximum number of visits to the website within 30 days; ⑤ Text theme: the number of times pornographic and gambling-related keywords are hit in the webpage text.

[0021] Regarding image features, black market portals need to attract user clicks to increase traffic, and this need is reflected in webpage images as features such as high saturation and high contrast. Specifically, image features are extracted from aspects such as brightness, saturation, and contrast of webpage images for analysis of webpage screenshots;

[0022] Specifically, this invention extracts five different image features, including: ① Image brightness: average RGB values ​​of pixels; ② Image luminance: weighted average RGB values ​​of pixels (luminosity method); ③ Image saturation: ratio of pixel RGB range to maximum value; ④ Blank contrast: pixel range of grayscale image; ⑤ Contrast: variance of adjacent pixel values ​​in grayscale image.

[0023] (2) Accurate recognition scheme based on ensemble learning (learning recognition module)

[0024] In the learning module, this invention utilizes an ensemble learning model to accurately identify the input seed website based on the features extracted by the feature extraction module, and outputs the identification results. In the previous approach, this invention extracted the aforementioned ten categories of features, but different black market portals have distinct prominent features, making them impossible to distinguish through simple linear differentiation. This invention designs a nonlinear ensemble learning model based on random forests, which allows for different feature selections, facilitating the handling of imbalanced data and balancing classification errors. In this learning model, 200 independent binary classification decision trees are first constructed, forming a forest through randomization. In each training iteration, the Bagging algorithm is used to calculate and output accurate identification results from the aggregated results of each classification tree.

[0025] Due to the simplicity of decision trees and the randomness of forest construction, this algorithm can run effectively on large datasets compared to other algorithms. It can also determine the importance and mutual influence of different features, exhibiting strong generalization ability. This method boasts high accuracy and stable classification results.

[0026] (II) The structure of the black market application dynamic data collection system is as follows: Figure 3 As shown.

[0027] The black market application dynamic data collection system takes as input the verified black market portal websites output by a multi-dimensional feature-based black market portal website identification system. Its aim is to comprehensively, accurately, and quickly download black market applications promoted within each website. However, due to the increasing deployment of various anti-scraping mechanisms on websites, especially black market websites which are becoming more secretive to avoid exposing their criminal activities and evading regulatory oversight, the system faces significant challenges in automated data collection. Besides website-specific access restrictions, the application download logic varies across websites, and webpage operation methods such as pop-ups and CAPTCHAs are becoming increasingly frequent.

[0028] Based on the above, a dynamic data collection system for black market applications must achieve the following objectives:

[0029] (a) Bypassing various anti-crawling measures set up by black market portals;

[0030] (b) Application data collection can be completed from different websites with high applicability.

[0031] The black market application dynamic collection system achieves the above objectives through the following scheme: it automatically and dynamically collects black market applications from input black market portal websites.

[0032] (1) Highly adversarial dynamic content capture scheme

[0033] Anti-crawler measures use technical means to prevent others from obtaining information from a website in bulk. Most black market portals or their cloud service providers (such as Alibaba Cloud and Cloudflare) filter fields such as UserAgent and Referer in the headers, blocking crawlers with abnormal headers or frequency. Some websites also use dynamic page rendering, and pages dynamically generated by JavaScript cannot have their source code viewed statically. This renders many traditional static crawling frameworks ineffective. Therefore, more advanced automated content capture tools are needed.

[0034] This invention proposes a dynamic content capture scheme that simulates mobile browser users. It captures the real-time rendered content and structure of a website by mimicking the browser's rendering process, and bypasses the anti-crawling mechanisms of malicious portals by disguising browser parameters and data packet headers, making it highly resistant to attack. Specifically, Puppeteer is used to monitor and manipulate browser pages via the Chrome DevTools protocol. By disguising parameters and data packet headers, it simulates a real mobile browser accessing a malicious portal website, thus bypassing the portal's anti-crawling mechanisms.

[0035] This solution is suitable for large-scale automated extraction processes, adapts to different rendering methods on various websites, and has strong versatility. Simultaneously, the disguised mobile browser effectively counters the filtering of potential users by black market portals. Based on the dynamic content capture scheme, this invention can capture rich webpage structure and semantic information rendered in real-time by black market portals, facilitating further analysis of website content.

[0036] (2) Semantic-first acquisition logic scheme

[0037] Different black market portals have vastly different layouts and logics, and they generally contain page elements that require user interaction, such as advertisements and pop-ups. Many legitimate download links are often hidden behind pop-ups, which interferes with automated data scraping tools. Therefore, in addition to capturing the dynamically rendered content of a website, a rigorously designed scraping logic is needed to comprehensively and accurately extract black market applications from numerous backlinks.

[0038] Based on a semantic-first data collection logic, this approach builds upon breadth-first search by prioritizing backlinks captured using semantic evaluation tools related to downloads. Backlinks with semantically more relevant to downloads are given higher priority, optimizing collection efficiency. Considering the multi-layered click logic of black market portals, this solution continues to monitor new links pointed to by backlinks that failed to be collected after dynamic triggering, ensuring comprehensive data collection. Specifically, a priority task queue based on download semantics is constructed. For each black market portal, all captured page backlink entries are retrieved, and priority is assigned based on their semantic characteristics before adding them to the download task queue. While executing queued tasks, not only are black market applications downloaded from URLs pointing to APKs, but new tasks are also created and added to the queue from failed tasks pointing to new links. Repeating this process allows all associated APKs of that black market portal to be downloaded.

[0039] The core of this invention is a black market portal website identification system based on multi-dimensional features and a black market application dynamic collection system. An auxiliary module assists the two core systems in completing the entire black market application collection based on dissemination channels. Specific details of these two modules will be described in the detailed implementation section.

[0040] The beneficial effects of this invention are:

[0041] (1) This invention addresses the pain points of traditional discovery schemes that rely on whistleblowing and investigation, and designs and implements a black market application collection technology based on dissemination channels, which effectively solves the problem of the lag in the discovery of black market applications.

[0042] (2) The distribution channel selected by this invention based on the characteristics of black and gray industry dissemination behavior, namely the black industry portal, can provide help or inspiration for other work in the field of black industry blocking.

[0043] (3) The framework for black market application data collection constructed by this invention can also provide assistance or inspiration for other work in the field of website content acquisition. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the data collection system architecture for black market applications based on dissemination channels, as described in this invention.

[0045] Figure 2 A flowchart illustrating a black market portal website identification system based on multi-dimensional features.

[0046] Figure 3 A flowchart illustrating the dynamic data collection system for black market applications. Detailed Implementation

[0047] This invention designs and implements the aforementioned data collection technology for black market applications based on dissemination channels. This section provides a detailed description of the specific implementation details of this invention.

[0048] (I) Black Market Portal Identification System Based on Multi-Dimensional Features

[0049] The data source for this invention is the input seed website. For a specific input website, the black market portal website identification system based on multi-dimensional features will first extract multi-dimensional features of the website, and then further utilize a random forest model to complete the seed website identification. In this invention, the black market portal website identification system based on multi-dimensional features includes a feature extraction module and a learning and recognition module.

[0050] The feature extraction module first crawls the input website and saves its page content and webpage screenshots. Combining the webpage structure and image features of the black market portal, it evaluates different dimensions such as backlinks, channels, traffic, and image color, and selects a total of 10 different dimensions to extract website features.

[0051] In the learning and recognition module, this invention utilizes an ensemble learning model to accurately identify the input seed website based on the features extracted by the feature extraction module, and outputs the recognition result. This invention chooses a dimensionless random forest classification model (reference: Ho, Tin Kam. "Random decision forests." Proceedings of the 3rd international conference on document analysis and recognition. Vol. 1. IEEE, 1995.) to avoid incompatibility issues related to data distribution, order of magnitude, and units across different dimensions. In the model, results are predicted using independent binary classification trees, and then the results of each classification tree are summarized by a randomly constructed forest. Finally, the Bagging strategy algorithm is used to calculate and output the accurate discrimination result. For a random forest of m trees, the majority voting principle applies if and only if there are... The random forest model will only make a wrong classification if all the decision trees make incorrect judgments; therefore, the overall classification error rate is [missing value].

[0052]

[0053] Based on random forest, this invention conducted gradient experiments on different numbers of decision trees, minimum leaf node sample number, and minimum split leaf node sample number. Through grid search, it was determined that the minimum leaf node sample number in the final random forest model is 2, the minimum split leaf node sample number is 2, and 200 decision trees are used.

[0054] (II) Dynamic Data Collection System for Black Market Applications

[0055] The black market application dynamic collection system takes black market portals identified by a multi-dimensional feature-based black market portal website identification system as input, and downloads applications associated with the input black market portals through large-scale, automated parallel crawling, thus completing a comprehensive and efficient collection of black market applications.

[0056] In this invention, the black market application dynamic acquisition system employs a highly adversarial dynamic content capture framework. Specifically, it uses Puppeteer to monitor and manipulate browser pages via the Chrome DevTools protocol. By employing methods such as spoofing parameters and data packet headers, it simulates access to black market portal websites from a real mobile browser, bypassing the portal's anti-scraping mechanisms. Furthermore, this invention utilizes a semantic-priority-based acquisition logic scheme, constructing a priority task queue based on download semantics. For each black market portal, the system retrieves all captured external link entries and assigns priority to them based on their semantic characteristics, adding them to the download task queue. While executing queued tasks, the system not only downloads the black market application from URLs pointing to APKs but also creates new tasks from failed tasks pointing to new links and adds them to the queue. By repeating the above operations, all associated APKs of the black market portal can be downloaded.

[0057] This invention also employs targeted anti-interference measures against dozens of common interference factors during the download process. When these interference factors occur, the IntersectingViewport API is used to monitor the rendering visibility of elements on the page. When advertisements or pop-ups are detected, their positions are located via XPath, and they are automatically blocked or closed, greatly improving the download success rate and the robustness of this invention.

[0058] Regarding the testing of the black market portal website identification system based on multi-dimensional features, this invention used 300 black market portals and 300 non-black market portals, both manually verified, as samples. The system achieved an accuracy of 94.3% and a recall of 94% on the validation set. Regarding the testing of the black market application dynamic acquisition system, this invention ran for 49 minutes and 22 seconds under light load on a 32-core, 212GB Linux server, successfully downloading applications from 159 of the 202 test portals, achieving a download coverage of 78.8%.

[0059] (III) Auxiliary Module

[0060] The auxiliary modules are the data storage module and the task scheduling module. These modules will assist the two core modules in completing the data collection process for black market applications.

[0061] The data storage module is used to store the crawled data and analysis results from the feature extraction module, learning and recognition module, and the black market application dynamic collection system in this invention. It establishes a unified data storage, retrieval, and sharing interface between different modules, facilitating the acquisition of information extracted by the previous module. This module also saves intermediate results from the process of this invention, facilitating verification testing and data analysis.

[0062] The task scheduling module is an optimization measure for tasks that acquire large amounts of data. This module receives task information from other main modules, maintains the task queue with the Redis database as the core, and ensures that the overall system can run smoothly.

Claims

1. A black market application data collection system based on dissemination channels, characterized in that, This includes a multi-dimensional feature-based black market portal website identification system and a black market application dynamic acquisition system. The multi-dimensional feature-based black market portal website identification system extracts features from seed websites to be tested and identifies black market portal websites that can provide a large number of application acquisitions. The black market application dynamic acquisition system is used to automatically and in large quantities download applications from black market portal websites. Among them: The black market portal website identification system based on multi-dimensional features includes a multi-dimensional feature extraction module and a learning and recognition module; wherein: The multi-dimensional feature extraction module starts from the traffic-driving function of black market portal websites and deeply analyzes to obtain the webpage structure features and image features of the traffic-driving websites. Regarding webpage structural characteristics, black market portals often contain more backlinks and channels than ordinary websites. Backlinks refer to external links outside the website; the more backlinks, the more obvious the traffic-driving characteristics. Channels are a special type of backlink used to calculate website traffic; the click volume of these backlinks is often used as the basis for settling portal website promotion fees and is the main profit method for black market portals. These are used as significant characteristics to distinguish black market portals. Structural features are extracted from webpages by comprehensively considering information such as webpage layout, traffic, and keywords. Regarding image features, black market portals need to attract user clicks to increase traffic. This need is reflected in web page images by features such as high saturation and high contrast. Specifically, image features are extracted from aspects such as brightness, saturation, and contrast of web page images to analyze web page screenshots. The learning and recognition module is designed with a nonlinear ensemble learning model based on random forest. In this learning model, the results are predicted by independent binary classification trees, and then the results of each classification tree are summarized by a forest built in a random manner. Finally, the Bagging strategy algorithm is used to calculate and output accurate discrimination results. The aforementioned black market application dynamic collection system takes the black market portal websites identified as genuine by the multi-dimensional feature-based black market portal website identification system as input, and comprehensively, accurately, and quickly downloads the black market applications promoted within each website. Specifically, it achieves the above objectives through the following two schemes, namely, automating the dynamic collection of black market applications from the input black market portal websites: (1) Highly adversarial dynamic content capture scheme Specifically, this method simulates dynamic content capture from mobile browser users. It captures the real-time rendered content and structure of a website by mimicking the browser's rendering, and bypasses the anti-crawling mechanisms of malicious portals by disguising browser parameters and data packet headers. This solution is suitable for large-scale automated extraction processes, adapts to different rendering methods on various websites, and has strong versatility. Simultaneously, disguising the mobile browser effectively counters the filtering of potential users by malicious portals. Based on this dynamic content capture solution, it can capture rich webpage structure and semantic information rendered in real-time by malicious portals, facilitating further analysis of website content. (2) Semantic-first acquisition logic scheme Based on capturing dynamically rendered content from websites, a rigorously designed collection logic is implemented to comprehensively and accurately extract black market applications from numerous backlinks. This logic prioritizes semantics, prioritizing backlinks captured by semantic evaluation tools related to downloads, and assigning higher priority to backlinks whose semantics better match download characteristics, thus optimizing collection efficiency. Considering the multi-layered click logic set up by black market portals, this solution continues to monitor new links pointed to by backlinks that fail to be collected after dynamic triggering, ensuring the comprehensiveness of the collection.

2. The black market application data collection system based on dissemination channels according to claim 1, characterized in that: The extracted webpage structural features are divided into 5 categories, specifically: ① Number of backlinks: the number of links from this website to external websites; ② Number of backlink channels: the number of backlinks with channel codes; ③ Number of lists: the number of table and list elements in the webpage layout; ④ Number of visits: the maximum number of visits to the website in 30 days; ⑤ Text theme: the number of times pornographic and gambling-related keywords are hit in the webpage text. The extracted image features are divided into 5 categories, specifically: ① Image brightness: average RGB value of pixels; ② Image luminance: weighted average RGB value of pixels; ③ Image saturation: ratio of pixel RGB range to maximum value; ④ Blank contrast: pixel range of grayscale image; ⑤ Contrast: variance of adjacent pixel values ​​in grayscale image.

3. The black market application data collection system based on dissemination channels according to claim 1, characterized in that: The highly adversarial dynamic content capture scheme uses Puppeteer to monitor and manipulate browser pages via the Chrome DevTools protocol. It employs methods such as spoofing parameters and data packet headers to simulate access to black market portal websites by a real mobile browser, thereby bypassing the anti-crawler mechanisms of black market portals. The semantic-first acquisition logic scheme constructs a priority task queue based on download semantics. For each black market portal, all captured external link entries are retrieved, and priority is assigned based on their semantic characteristics and added to the download task queue. While executing queue tasks, not only are black market applications downloaded from URLs pointing to APKs, but new tasks are also created from failed tasks pointing to new links and added to the queue. The above operation is repeated to download all associated APKs of the black market portal.

4. The black market application data collection system based on dissemination channels according to claim 1, characterized in that, It also includes a data storage module and a task scheduling module. The data storage module is used to store the crawled data and analysis results from the feature extraction module, the learning and recognition module, and the black market application dynamic collection system. It establishes a unified data storage, acquisition, and sharing interface between different modules, making it easier for different modules to obtain the information extracted by the previous module. The task scheduling module is an optimization processing measure for tasks involving the acquisition of large amounts of data. It receives task information from other main modules and maintains the task queue with the Redis database as the core, ensuring that the overall system can run smoothly.

Images