Mobile webpage camouflage detection method and system

Abstract

The invention provides a mobile webpage camouflage detection method and system. The method comprises the following steps: determining search keywords and search results of mobile webpage camouflage; performing webpage access on a preset simulation desktop end and a preset simulation mobile end according to the search result to obtain webpage access data; the webpage access data comprises training data and test data; performing feature extraction on the test data to obtain classification features; the classification features are used for distinguishing normal webpage camouflage and mobile webpage camouflage; the types of feature extraction comprise text features, visual features, network features, element features and URL features; inputting the classification features to a pre-trained classification model, and obtaining a mobile webpage camouflage detection result output by the classification model; wherein the classification model is obtained by training based on training data. According to the method, the normal webpage camouflage and the mobile webpage camouflage can be effectively identified and classified, and efficient and accurate mobile webpage camouflage detection is realized.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a mobile web page camouflage detection method and system. Background Art

[0002] Web camouflage technology refers to using different technical means to make web pages display different content in different situations. Web camouflage technology can optimize user experience when used in web page optimization. Therefore, effective camouflage detection technology is of great significance to ensuring network security and maintaining the health of the Internet ecosystem.

[0003] With the widespread popularity of smartphones, mobile devices have become the main tool for people to obtain information and use the Internet. Since smartphones are currently used by the vast majority of Internet users around the world, in order to adapt to this trend, search engines have introduced a mobile-first indexing mechanism, which gives priority to the mobile version of website content for indexing and ranking. This makes mobile SEO crucial.

[0004] Existing web cloaking detection methods focus on web cloaking behaviors that provide completely different content to search engines and desktop users, but lack attention to mobile web cloaking. In addition, the benign behavior of web mobile optimization may also lead to large differences between the desktop and mobile content of web pages, further increasing the difficulty of detection. Therefore, it is necessary to design a method that can effectively detect mobile web cloaking. Summary of the invention

[0005] Aiming at the problem of mobile web page disguise detection, the present invention provides a mobile web page disguise detection method and system from the perspective of automated discovery and detection, which can effectively distinguish between mobile web page optimization and mobile web page disguise, and realize efficient and accurate mobile web page disguise detection.

[0006] The present invention provides a mobile web page disguise detection method, comprising: determining a search keyword and search result of mobile web page disguise; the search result is a result of searching the search keyword on a search engine; accessing a web page on a preset simulated desktop terminal and a preset simulated mobile terminal according to the search result, to obtain web page access data; the web page access data includes training data and test data; extracting features from the test data to obtain classification features; the classification features are used to distinguish between normal web pages and mobile web page disguises; the types of feature extraction include text features, visual features, network features, element features and URL features; inputting the classification features into a pre-trained classification model to obtain a mobile web page disguise detection result output by the classification model; wherein the classification model is trained based on the training data.

[0007] According to a mobile web page disguise detection method provided by the present invention, the determining of the search keywords of the mobile web page disguise includes: extracting seed keywords from a preset illegal website; based on the seed keywords, performing keyword-related search in a preset search engine to obtain extended keywords; the extended keywords are keywords ranked within a preset range in the search results of the search engine; based on the seed keywords, denoising the extended keywords to obtain the search keywords of the mobile web page disguise.

[0008] According to a mobile web page disguise detection method provided by the present invention, the web page is accessed on a preset simulated desktop terminal and a preset simulated mobile terminal respectively according to the search results to obtain web page access data, including: forming the preset simulated desktop terminal and the preset simulated mobile terminal by setting a Chrome browser on a Mac OSX and an Android device; determining the URL of a web page according to the search results; using Playwright to start the Chrome browser to access the web page through the URL of the web page, and recording the page HTML content, page screenshots, redirection chains and HAR files of the preset simulated desktop terminal and the preset simulated mobile terminal respectively as the web page access data.

[0009] According to a mobile web page disguise detection method provided by the present invention, the text features include HTML text similarity features, core content similarity features and HTML structure similarity features; the feature extraction of the test data to obtain classification features includes: determining the full HTML text according to the HTML content of the page, so as to extract the HTML text similarity features based on a preset text feature extraction algorithm; determining the core content according to the HTML content of the page, so as to extract the core content similarity features based on a preset text feature extraction algorithm; the core content is the content after the tag information is removed from the HTML content of the page; determining the HTML structure according to the HTML content of the page, so as to extract the HTML structure similarity features based on a preset text feature extraction algorithm; the HTML structure is the content after the tag information and the full HTML text are removed from the HTML content of the page; the preset text feature extraction algorithm includes blank character removal processing, content segmentation processing and Hamming distance calculation of simhash value.

[0010] According to a mobile web page camouflage detection method provided by the present invention, the visual features include Phash distance features, ORB features and histogram distance features; the feature extraction of the test data to obtain classification features includes: calculating the Hamming distance of the pHash value according to the page screenshot to obtain the Phash distance feature; according to the page screenshot, using the ORB algorithm and a matcher to extract the ORB feature; performing HSV color space conversion and image histogram Bhattacharyya distance similarity calculation on the page screenshot to extract the histogram distance feature.

[0011] According to a mobile web page disguise detection method provided by the present invention, the network features include network behavior similarity features, request quantity ratio features, image quantity ratio features, Js file quantity ratio features, back-end redirection features, cross-domain redirection features and false 404 features; the feature extraction of the test data to obtain classification features includes: calculating the Jaccard similarity coefficient according to the HAR file to obtain the network behavior similarity features; calculating the ratio of the total number of mobile network requests to the total number of desktop network requests according to the HAR file to obtain the request quantity ratio feature; calculating the ratio of mobile picture type network requests to the total number of desktop picture network requests according to the HAR file to obtain the picture quantity ratio feature; calculating the ratio of mobile Js type network requests to the total number of desktop Js network requests according to the HAR file to obtain the Js file quantity ratio feature; extracting the back-end redirection feature, the cross-domain redirection feature and the false 404 feature according to the redirection chain.

[0012] According to a mobile web page camouflage detection method provided by the present invention, the element features include desktop Iframe features, mobile Iframe features and image similarity features; the feature extraction of the test data to obtain classification features includes: extracting the desktop Iframe features and the mobile Iframe features according to the HTML content of the page; creating a picture file list according to the HTML content of the page, and calculating the Jaccard similarity of the picture file list to obtain the image similarity feature.

[0013] According to a mobile web page disguise detection method provided by the present invention, the URL features include URL similarity features, KL divergence features and URL entropy features; the feature extraction of the test data to obtain classification features includes: creating a URL list according to the redirection chain, and calculating the Jaccard similarity of the URL list to obtain the URL similarity feature; extracting the KL divergence feature and the URL entropy feature according to the redirection chain.

[0014] According to a mobile web page camouflage detection method provided by the present invention, the classification model is an XGBoost model.

[0015] The present invention also provides a mobile web page disguise detection system, comprising: a determination module, used to determine the search keywords and search results of mobile web page disguise; the search results are the results of searching the search keywords on the search engine; an access module, used to access the web page on a preset simulated desktop terminal and a preset simulated mobile terminal according to the search results, and obtain web page access data; the web page access data includes training data and test data; a feature extraction module, used to extract features from the test data to obtain classification features; the classification features are used to distinguish normal web pages from mobile web page disguises; the types of feature extraction include text features, visual features, network features, element features and URL features; a detection module, used to input the classification features into a pre-trained classification model to obtain the mobile web page disguise detection results output by the classification model; wherein the classification model is trained based on the training data.

[0016] The present invention provides a method and system for detecting mobile web page disguise, which includes: determining search keywords and search results of mobile web page disguise; accessing web pages on a preset simulated desktop terminal and a preset simulated mobile terminal according to the search results to obtain web page access data; the web page access data includes training data and test data; extracting features from the test data to obtain classification features; the classification features are used to distinguish normal web pages from mobile web page disguises; the types of feature extraction include text features, visual features, network features, element features and URL features; inputting the classification features into a pre-trained classification model to obtain the mobile web page disguise detection results output by the classification model; wherein the classification model is obtained by training based on the training data. The present invention can effectively identify and classify normal web pages and mobile web page disguises, and realizes efficient and accurate mobile web page disguise detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of a mobile web page disguise detection method provided by the present invention.

[0019] Figure 2 It is a specific flow chart of a mobile web page disguise detection method provided by the present invention.

[0020] Figure 3 This is a classification feature importance ratio diagram provided by the present invention.

[0021] Figure 4 It is a structural schematic diagram of a mobile web page camouflage detection system provided by the present invention.

[0022] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] Web camouflage technology refers to using different technical means to make web pages display different content in different situations. Common web camouflage technologies include user agent detection, IP (Internet Protocol) address detection, redirection, and JavaScript detection. This technology can be used to optimize user experience, but it is also often used in illegal activities, such as illegal search engine optimization and phishing websites.

[0025] User agent detection technology checks the visitor's user agent string to determine the device and browser type used by the visitor, thereby displaying different web page content. For example, a website can determine whether the visitor is using a mobile device or a desktop device based on the user agent string, and thus display different content. This technology is widely used in mobile optimization web pages, displaying specific advertisements, and hiding content that you do not want to display. In addition, by detecting the visitor's IP address, the website can determine its geographic location and display different content based on the location. For example, a website can display localized content based on the visitor's country or region, or block users in certain areas. This technology has been widely used in localized content services, regional restrictions, and content review. Another common web page camouflage technology is redirection. Redirection refers to automatically transferring users from one URL (Uniform Resource Locator) to another URL. Through redirection, visitors can be directed to a specific version of a web page based on their conditions (such as device type, geographic location). Redirection technology is often used in device optimization, A / B testing (controlled experiments), and content distribution network optimization. Through JavaScript detection, visitors' browser environment, screen resolution, plug-ins and other information can be detected to dynamically adjust web page content. JavaScript detection is usually performed on the client side, and the DOM (Document Object Model) structure can be modified based on the detection results. This technology is widely used in responsive design, dynamic content loading, and personalized recommendations.

[0026] In malicious scenarios, web camouflage technology is used in various fraud and attack activities, posing a serious threat to Internet security and user privacy. It improves the search ranking of malicious websites by displaying different content to search engine crawlers and ordinary users, misleading users to visit low-quality or malicious websites, resulting in theft of personal information, financial losses, and identity theft. In addition, camouflage technology is also used to hide download links of malware, spread illegal or harmful content, bypass content filtering and censorship, and guide users to perform uninformed operations by disguising web page elements, such as clicking on malicious ads and links, resulting in computer damage, privacy leakage, and illegal advertising revenue. These behaviors not only seriously damage the normal browsing experience of users, but also undermine the security and credibility of the Internet. Therefore, effective camouflage detection technology is of great significance to ensuring network security and maintaining the health of the Internet ecosystem.

[0027] With the widespread popularity of smartphones, mobile devices have become the main tool for people to obtain information and use the Internet. Globally, most Internet users access the Internet through mobile devices. In order to adapt to this trend, search engines have launched a mobile-first indexing mechanism, which gives priority to the mobile version of website content for indexing and ranking. This makes mobile search engine optimization (SEO, Search Engine Optimization) crucial.

[0028] The technology of promoting illegal industry content only on mobile terminals and displaying normal content on desktop terminals is defined as mobile web page cloaking. Differentiated from benign web page mobile optimization behaviors, mobile web page cloaking is essentially a web page that uses user agent detection technology to promote illegal industry content, providing completely different content on the desktop and mobile terminals of the web page (web pages using this technology only display the illegal industry content to be promoted to mobile users, and display normal content to desktop users), or redirecting mobile users to the website of the illegal industry. However, in order to achieve the purpose of circumventing censorship and evading detection, mobile web page cloaking is more about using JavaScript code on the client to implement the functions of identifying visitor types and promoting illegal industry content. Existing web page cloaking detection methods focus on web page cloaking behaviors that provide completely different content to search engines and desktop users, and lack attention to mobile web page cloaking. In addition, the benign behavior of web page mobile optimization may also lead to large differences between the desktop and mobile terminal content of the web page, further increasing the difficulty of detection. Therefore, it is necessary to design a method that can effectively detect mobile web page cloaking.

[0029] Since it is a legal and proactive optimization practice aimed at improving the experience of mobile users, it follows several basic principles:

[0030] (1) Content consistency: Major search engines such as Google emphasize that in order to ensure a good user experience, websites should provide consistent content on different devices. Even if the content and layout are optimized, the main information of the page should remain consistent.

[0031] (2) URL consistency: Although you can use different subdomains, such as m.example.com, the URL structure should be consistent to facilitate understanding and indexing by users and search engines.

[0032] (3) User experience optimization: Since the performance of mobile devices is usually not as good as that of desktop devices, mobile optimization reduces page loading time by optimizing images and resources, thereby improving user experience.

[0033] (4) Technical compliance: Mobile SEO uses HTTP (Hyper Text Transfer Protocol) status code 301 or 302 to correctly redirect, ensuring that search engines and users understand that this is a legal mobile optimization behavior.

[0034] Please refer to Figure 1 , Figure 1 A schematic diagram of a flow chart of a mobile web page camouflage detection method provided by the present invention.

[0035] A method for detecting mobile web page camouflage, comprising:

[0036] 101: Determine the search keyword and search results disguised as a mobile web page; the search results are the results of searching the search keyword on a search engine;

[0037] 102: Accessing web pages on a preset simulated desktop terminal and a preset simulated mobile terminal respectively according to the search results to obtain web page access data; the web page access data includes training data and test data;

[0038] 103: Extract features from the test data to obtain classification features; the classification features are used to distinguish between normal web pages and mobile web page disguises; the types of feature extraction include text features, visual features, network features, element features and URL features;

[0039] 104: Input classification features to a pre-trained classification model to obtain a mobile web page camouflage detection result output by the classification model; wherein the classification model is trained based on training data.

[0040] There are some similarities between the performance of mobile optimization of web pages and mobile web page camouflage. Both may cause large differences in the text content and visual effects of web pages on desktop and mobile terminals, resulting in false positives caused by mobile optimization of web pages. In order to solve the technical problems existing in the prior art, the present invention starts from the technical characteristics of mobile web page camouflage. How to accurately distinguish between benign mobile optimization and mobile web page camouflage that promotes illegal industry content is the focus of this detection method. The present invention constructs a data set (web page access data) of web pages and normal web pages that include mobile web page camouflage technology, extracts features that help distinguish between mobile optimization and mobile web page camouflage, and selects a suitable machine learning model to train a supervised classifier (classification model), thereby achieving accurate classification of web pages.

[0041] Specifically, the present invention first obtains the search keywords disguised as mobile web pages. For example, only a batch of seed keywords related to illegal industries need to be manually collected, and then the "related search" function provided by the search engine can be used to automatically collect sufficient related keywords of illegal industries, which can effectively increase the number and coverage of keywords.

[0042] Then, the preset simulated desktop and mobile terminals are used to obtain the web page access data corresponding to the search results for subsequent feature extraction. For example, two browser configurations are used in mainstream search engines such as Google, Baidu, and Bing to obtain the desktop and mobile terminal data of the web pages corresponding to the search results of the search keywords. In order to trigger the mobile web page camouflage technology implemented using JavaScript, JavaScript code is executed when obtaining web page access data.

[0043] Then, the classification features are designed by using the difference in behavior performance between normal web pages and mobile web page disguises. By classifying and extracting the test data, the features that help distinguish between mobile optimization and mobile web page disguises are used to achieve accurate classification of web pages. Finally, based on the classification features, the task of detecting mobile web page disguises is completed through a pre-trained classification model. The present invention can effectively identify and classify normal web pages and mobile web page disguises, and realizes efficient and accurate mobile web page disguise detection.

[0044] It should be noted that a normal web page refers to a web page that does not use camouflage technology or uses camouflage technology for benign purposes, such as mobile web page optimization.

[0045] Please refer to Figure 2 , Figure 2 A schematic diagram of a specific process of a mobile web page camouflage detection method provided by the present invention.

[0046] As a preferred embodiment, determining the search keywords of the mobile web page disguise includes: extracting seed keywords from a preset illegal website; based on the seed keywords, performing a keyword-related search in a preset search engine to obtain extended keywords; the extended keywords are keywords that are ranked within a preset range in the search results of the search engine; based on the seed keywords, denoising the extended keywords to obtain the search keywords of the mobile web page disguise.

[0047] Promoters use mobile web page camouflage technology to enhance their ability to evade detection, but their main goal is still to obtain high rankings in search results through relevant keywords, thereby increasing the visibility of their promotional content to users. Based on this observation, this embodiment searches for relevant keywords of illegal industries through search engines to find potential mobile web page camouflage. Specifically, a series of keywords of illegal industries are searched as seed keywords on the three major search engines of Google, Baidu and Bing, and the first 50 search results of each keyword are collected as detection targets. In specific implementation, the search result crawler is implemented by the SERP Scraper API provided by Python and oxylabs.io, and the results are stored in a database for subsequent processing. Of course, other third-party APIs (such as serpapi.com) or official APIs provided by search engines can also be used to implement it.

[0048] At the same time, the "related search" function provided by the search engine is used to expand the existing keywords (seed keywords) of illegal industries, and the keywords ranked in the preset range in the search engine's search results are used as extended keywords. Finally, the irrelevant keywords are manually reviewed and deleted (denoising) from the extended keywords to obtain the search keywords of mobile web page disguise, thereby expanding the number and coverage of related keywords of illegal industries, being able to adapt to the ever-changing mobile web page disguise technology, and ensuring the long-term effectiveness of the detection method. For subsequent research, different search engines will be used to collect keyword search results.

[0049] As a preferred embodiment, web page access is performed on a preset simulated desktop and a preset simulated mobile terminal respectively according to the search results to obtain web page access data, including: forming a preset simulated desktop and a preset simulated mobile terminal by setting a Chrome browser on a Mac OSX and an Android device; determining the URL of the web page according to the search results; using Playwright to start the Chrome browser to access the web page through the URL of the web page, and recording the page HTML (Hyper Text Markup Language) content, page screenshots, redirection chains and HAR files of the preset simulated desktop and the preset simulated mobile terminal respectively as web page access data.

[0050] Mobile web page disguise displays benign content to desktop users while promoting illegal content to mobile users. This embodiment detects mobile web page disguise by comparing the differences between the desktop version and the mobile version of the web page. To this end, in the implementation of the web crawler, two types of browser environments are configured to simulate the scenes of desktop and mobile users accessing web pages and collect data. Specifically, the user agent is set to the latest version of the Chrome browser on Mac OSX (preset simulated desktop end) and Android device (preset simulated mobile end), and the Referer field is set to the URL of the corresponding web page, for example, set to https: / / www.google.com to simulate the user behavior of accessing the website through the Google search engine. Then use the Playwright framework to start the Chrome browser instance to load the web page. Of course, other frameworks for loading browsers to access web pages can also be used instead, such as Selenium. Each time a web page is accessed, the page HTML content, page screenshots, redirection chains and HAR files (HTTP Archive) are recorded. For each web page, if the initial access fails, up to three retries will be performed to ensure the integrity and reliability of the data.

[0051] These web page access data constitute the complete research dataset. Then a manual annotation method was used to obtain a training set (training data) for a binary supervised classifier (classification model). For each search engine's search results, 15,000 samples were randomly selected, and the web page screenshots and marked labels were manually compared. Pages that promote illegal content on mobile devices but display normal content on desktops are marked as "positive samples". At the same time, pages that do not meet the above characteristics are marked as "negative samples". Through this process, 5,358 positive samples and 39,642 negative samples were finally obtained. In binary classification problems, excessive imbalance of sample classes will cause the model to be biased towards the majority class and reduce its ability to identify minority class samples. To address this challenge, the negative samples were appropriately undersampled, and the ratio of positive samples to negative samples was adjusted to 2:8 to achieve a balanced dataset. After adjustment, the training set consisted of 26,790 samples from three different search engines, including 5,358 positive samples and 21,432 negative samples.

[0052] Please refer to Figure 3 , Figure 3 This is a graph showing the importance ratio of classification features provided by the present invention.

[0053] Mobile web page camouflage may cause significant differences in text, visual presentation, and network behavior between the desktop and mobile versions of the web page. This embodiment detects mobile web page camouflage by evaluating whether these differences are significant. Mobile search engine optimization is an active optimization practice that also causes significant differences between desktop and mobile versions of web pages. Therefore, the key to improving the accuracy of mobile web page camouflage is to select features that can distinguish mobile search engine optimization. Mobile optimization follows some basic principles so that it can be distinguished from mobile web page camouflage that promotes content of illegal industries. This embodiment designs 19 classification features to distinguish normal web pages (such as web pages that have achieved mobile optimization) from mobile web page camouflage. These features include text features, visual features, network features, element features, and URL features. Under the premise of limited resources, some relatively unimportant features can be deleted to improve efficiency. Some features proposed in this embodiment can also be used for other types of web page camouflage detection.

[0054] As a preferred embodiment, text features include HTML text similarity features, core content similarity features and HTML structure similarity features; feature extraction is performed on test data to obtain classification features, including: determining the full HTML text according to the HTML content of the page, so as to extract the HTML text similarity features based on a preset text feature extraction algorithm; determining the core content according to the HTML content of the page, so as to extract the core content similarity features based on a preset text feature extraction algorithm; the core content is the content after removing tag information from the HTML content of the page; determining the HTML structure according to the HTML content of the page, so as to extract the HTML structure similarity features based on a preset text feature extraction algorithm; the HTML structure is the content after removing tag information and the full HTML text from the HTML content of the page; the preset text feature extraction algorithm includes removing blank character processing, content segmentation processing and Hamming distance calculation of simhash value.

[0055] Text features include HTML text similarity features, core content similarity features, and HTML structure similarity features. Although mobile optimization may also cause changes in text content, according to the principle of content consistency, these changes are mainly attributed to non-core content such as page layout, external links, and user comments. In addition, if the page uses CSS (Cascading Style Sheets) to achieve differences in content layout, the desktop and mobile terminals may have similar HTML structures, but different CSS files are loaded. For mobile web camouflage, the content extracted by the three methods may be quite different.

[0056] Use the full amount of HTML text in the HTML content of the page to calculate text similarity and extract HTML text similarity features.

[0057] Use the core content text of the page HTML content to calculate text similarity and extract the core content similarity features. The method of extracting the core content of the page is: first remove the page <header> 、 <footer> 、 、 <iframe>Tags and all tags with the words "header" and "footer" in their class attributes, and then extract the remaining content text of the page as the core content of the page.

[0058] The text similarity is calculated using the HTML structure of the HTML content of the page, and the HTML structure similarity feature is extracted. The HTML structure of the page is extracted, and the attributes (tag information) of all tags and the full amount of HTML text are removed, and only the sequence of tags is retained to represent the page structure. For the three types of text to be compared , a unified post-processing step (preset text feature extraction algorithm) is performed: all blank characters in the document are removed (blank character removal processing), the content is segmented using n-gram technology (n=4) to generate a token list (content segmentation processing), the 64-bit simhash value of each token is calculated as the digital representation of the document, and finally the text difference is quantified by calculating the Hamming distance of the simhash values ​​of two documents as the text feature.

[0060] As a preferred embodiment, the visual features include Phash distance features, ORB features and histogram distance features; feature extraction is performed on the test data to obtain classification features, including: calculating the Hamming distance of the pHash value according to the page screenshot to obtain the Phash distance feature; extracting the ORB feature according to the page screenshot using the ORB algorithm and the matcher; performing HSV color space conversion on the page screenshot and calculating the image histogram Bhattacharyya distance similarity to extract the histogram distance feature.

[0061] In this embodiment, the visual features include Phash distance features, ORB features and histogram distance features.

[0062] Based on the page screenshots, the global features of the image are extracted by the pHash algorithm. pHash can capture the overall visual information of the image, so the similarity of the images can be comprehensively compared. Specifically, the overall visual similarity of the page can be quantified by calculating the Hamming distance between the pHash values ​​of two screenshots to obtain the Phash distance feature.

[0063] Since mobile optimization may introduce factors such as layout structure changes, content adjustment and image compression, these changes may lead to larger pHash distances. Therefore, the ORB (Oriented FAST and Rotated BRIEF) algorithm is introduced to describe the layout structure and local features of the image. Specifically, according to the page screenshot, the key points of the image are extracted and the feature descriptors of the key points are generated, and then reliable matching pairs are screened by the brute force matcher.The ratio of the number of reliable matching pairs to the total number of key points is used as the ORB feature.

[0064] Finally, a color histogram is used to describe the color distribution characteristics of an image. Unlike mobile web page camouflage, mobile optimization may cause a large change in the page layout structure, but it usually does not cause a dramatic change in color distribution. According to the page screenshot, the image is converted into an HSV (hue, saturation, brightness) color space, which can more intuitively and effectively describe color information. The histogram of each image is calculated, and the similarity is quantified by comparing the Bhattacharyya distance between the histograms. The smaller the distance value, the higher the image similarity, thereby extracting the histogram distance feature.

[0065] As a preferred embodiment, the network features include network behavior similarity features, request quantity ratio features, picture quantity ratio features, Js file quantity ratio features, back-end redirection features, cross-domain redirection features and false 404 features; feature extraction is performed on the test data to obtain classification features, including: calculating the Jaccard similarity coefficient according to the HAR file to obtain the network behavior similarity features; calculating the ratio of the total number of mobile network requests to the total number of desktop network requests according to the HAR file to obtain the request quantity ratio feature; calculating the ratio of the mobile picture type network requests to the total number of desktop picture network requests according to the HAR file to obtain the picture quantity ratio feature; calculating the ratio of the mobile Js type network requests to the total number of desktop Js network requests according to the HAR file to obtain the Js file quantity ratio feature; extracting the back-end redirection feature, cross-domain redirection feature and false 404 feature according to the redirection chain.

[0066] This embodiment uses the collected redirection chain and HAR file to extract network features. First, the network behaviors of the desktop and mobile terminals are compared for similarity. A network event is defined as a tuple of (Method, URL, Response Code), and a HAR file is regarded as a sequence of network events.

[0067] By calculating the Jaccard similarity coefficient of the network event sequences on the desktop and mobile terminals, the similarity of network behaviors is quantified. The higher the similarity coefficient, the greater the similarity of network behaviors, and the network behavior similarity characteristics are obtained.

[0068] Mobile optimization usually performs image compression, reduces the number of HTTP requests, and uses lightweight JavaScript code to optimize the loading speed of pages on mobile terminals. This results in the number and complexity of requests processed by mobile devices when accessing these pages being usually lower than that of desktop devices. In contrast, mobile web pages are disguised to promote illegal industry content to mobile users, and mobile devices generate more network requests when accessing them.Based on this observation, this embodiment heuristically extracts the ratio of the total number of network requests, the ratio of the number of image requests, and the ratio of the number of JavaScript requests when the desktop and mobile terminals access the web page as features. According to the HAR file, the ratio of the total number of network requests on the mobile terminal to the total number of network requests on the desktop terminal is calculated to obtain the feature of the proportion of the number of requests; according to the HAR file, the ratio of the network requests of the mobile terminal image type to the total number of network requests of the desktop terminal image is calculated to obtain the feature of the proportion of the number of images; according to the HAR file, the ratio of the network requests of the mobile terminal Js type to the total number of network requests of the desktop terminal Js is calculated to obtain the feature of the proportion of the number of Js files.

[0069] In addition, mobile optimization is more inclined to redirect via HTTP status code 301 or 302, rather than using JavaScript redirection. Therefore, this embodiment also takes whether the mobile end page has back-end redirection and cross-domain redirection as two features.

[0070] Some mobile web pages pretend to display a 404 error on the desktop, but return a status code of 200. Given that a normal website should ensure that the page content is consistent with the status code, this contradictory phenomenon is used as a detection feature (false 404 feature).

[0071] As a preferred embodiment, the element features include desktop Iframe features, mobile Iframe features and image similarity features; feature extraction is performed on the test data to obtain classification features, including: extracting desktop Iframe features and mobile Iframe features based on the HTML content of the page; creating a picture file list based on the HTML content of the page, and calculating the Jaccard similarity of the picture file list to obtain the image similarity feature.

[0072] Considering that some illegal pages use full-screen iframe or frame tags to display and promote illegal industry content. Some mobile web page disguises also use this technology to promote illegal content to mobile users. In order to distinguish mobile web page disguises from other illegal pages, this embodiment uses whether there are full-screen iframe or frame tags on the desktop and mobile terminals as two features. The desktop terminal Iframe feature and the mobile terminal Iframe feature are extracted based on the page HTML content.

[0073] In addition, this embodiment also evaluates the similarity of web page embedded images on desktop and mobile terminals. Specifically, this embodiment extracts all tags in HTML, collects the file names of URLs corresponding to these tags, and creates a list of picture files. Then, the similarity of embedded images is quantified by calculating the Jaccard similarity of the two lists to obtain image similarity features.

[0074] As a preferred embodiment, URL features include URL similarity features, KL divergence features and URL entropy features; feature extraction is performed on test data to obtain classification features, including: creating a URL list based on the redirection chain, and calculating the Jaccard similarity of the URL list to obtain the URL similarity feature; extracting KL divergence features and URL entropy features based on the redirection chain.

[0075] Considering that some pages using mobile web page camouflage have very random features in the path and file name of the URL, which are obviously different from the URL pattern of normal web pages, this randomness is similar to the URL of a phishing website. In this embodiment, the KL (Kullback-Leibler) divergence feature and the URL entropy feature are two effective features in the method of detecting phishing websites based on URLs, and this embodiment also extracts these two features for classification. As a preferred embodiment, the classification model is an XGBoost model. In this embodiment, the extracted classification features are used to train a supervised binary classifier. A series of classic machine learning algorithms are extensively experimented, including Naive Bayes, K nearest neighbor, logistic regression, decision tree, random forest, support vector machine and XGBoost model. The purpose of the experiment is to determine the classification model with the best performance. The performance of the classifier based on the training data is evaluated using the standard 5-fold cross validation method. In the evaluation process, the performance indicators used in the binary classification include recall (R), precision (P) and F1 score. Among the seven classification algorithms, the XGBoost model achieves the best classification performance, which is superior to other classifiers in all three classification indicators. Finally, this embodiment uses the XGBoost classifier to detect mobile camouflage on the classification features and identifies 95,588 search results related to mobile web page camouflage.

[0078] In the importance weights of various classification features in the XGBoost model, the higher the weight, the greater the contribution to the model decision. The results show that the color histogram distance is the most decisive feature, and its weight is significantly higher than other classification features, exceeding 0.14. This finding may be related to the trend of hiding pages on mobile devices to promote illegal content. The distribution of pages disguised by mobile web pages is significantly different from the distribution of original pages, while normal pages usually use more consistent color styles.In addition, the weight ratios of nine features, including the network behavior similarity feature and the request quantity ratio feature, exceed 0.05, confirming the importance of multi-dimensional feature extraction. It is worth noting that although the cross-domain redirection feature and the full-screen iframe feature are considered suspicious behaviors, their weights in the model are not significant. This may be because the features extracted from the text, visual, and network behavior dimensions already include the influence of these features.

[0079] The mobile web page camouflage detection framework proposed in the present invention can be directly deployed in a real network environment, and the detection results can provide relevant threat intelligence of illegal industries. The beneficial effects of the present invention are as follows: (1 ) Efficient and accurate mobile web page camouflage detection

[0082] The present invention achieves efficient and accurate detection by constructing a data set of web pages containing mobile web page camouflage technology and normal web pages, extracting features that effectively distinguish the two, and selecting a suitable machine learning model to train a supervised classifier. Compared with traditional detection methods, the present invention can identify malicious web pages using mobile web page camouflage technology with a lower false alarm rate. ( 2 ) Can instantly identify and classify mobile web page camouflage

[0084] Existing mobile web page disguise detection methods usually require long-term behavior analysis, resulting in a certain lag in detection time. The present invention realizes instant recognition and classification of mobile web page disguise by extracting the difference characteristics of mobile optimization and mobile web page disguise in behavior performance and combining the data automatically collected by the search engine. ( 3 ) Good scalability and coverage

[0086] The present invention utilizes the "related search" function of the search engine to expand the number and coverage of related keywords of illegal industries and designs a set of scalable detection frameworks. By expanding the keyword and feature set, it is possible to adapt to the ever-changing mobile web page camouflage technology and ensure the long-term effectiveness of the detection method.

[0087] The mobile web page disguise detection system provided by the present invention is described below. The mobile web page disguise detection system described below and the mobile web page disguise detection method described above can be referred to each other.

[0088] Please refer to Figure 4. Figure 4 is a schematic structural diagram of a mobile web page camouflage detection system provided by the present invention.

[0089] The present invention also provides a mobile web page camouflage detection system, including: a determination module 1, configured to obtain search keywords and search results of mobile web page camouflage; the search results are the results of searching for the search keywords on a search engine; an access module 2, configured to perform web page access on a preset simulated desktop end and a preset simulated mobile end respectively according to the search results to obtain web page access data; the web page access data includes training data and test data; a feature extraction module 3, configured to extract features from the test data to obtain classification features; the classification features are used to distinguish normal web pages and mobile web page camouflage; the types of feature extraction include text features, visual features, network features, element features, and URL features; a detection module 4, configured to input the classification features into a pre-trained classification model to obtain a mobile web page camouflage detection result output by the classification model; wherein, the classification model is trained based on the training data.

[0090] Figure 5 illustrates a schematic structural diagram of an electronic device. As shown in Figure 5, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540. Among them, the processor 510, the communication interface 520, and the memory 530 complete mutual communication through the communication bus 540.The processor 510 can call the logic instructions in the memory 530 to execute the mobile web page disguise detection method, which includes: determining the search keywords and search results of the mobile web page disguise; the search results are the results of searching the search keywords on the search engine; according to the search results, web pages are accessed on a preset simulated desktop terminal and a preset simulated mobile terminal respectively to obtain web page access data; the web page access data includes training data and test data; feature extraction is performed on the test data to obtain classification features; the classification features are used to distinguish between normal web pages and mobile web page disguises; the types of feature extraction include text features, visual features, network features, element features and URL features; the classification features are input into a pre-trained classification model to obtain the mobile web page disguise detection results output by the classification model; wherein the classification model is trained based on the training data. In addition, the logic instruction in the above- mentioned memory 530 can be implemented in the form of software functional unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including some instructions to make a computer device (can be a personal computer, server, or network equipment, etc.) perform all or part of the steps of the method described in each embodiment of the present invention. And the aforementioned storage medium includes: various media that can store program code such as USB flash drive, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk.

[0092] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the mobile web page disguise detection method provided by the above methods, the method including: determining the search keywords and search results of the mobile web page disguise; the search results are the results of searching the search keywords on the search engine; according to the search results, web pages are accessed on a preset simulated desktop terminal and a preset simulated mobile terminal respectively to obtain web page access data; the web page access data includes training data and test data; feature extraction is performed on the test data to obtain classification features; the classification features are used to distinguish between normal web pages and mobile web page disguises; the types of feature extraction include text features, visual features, network features, element features and URL features; the classification features are input into a pre-trained classification model to obtain the mobile web page disguise detection results output by the classification model; wherein the classification model is trained based on the training data.

[0093] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the mobile web page disguise detection method provided by the above-mentioned methods, the method comprising: determining search keywords and search results of mobile web page disguise; the search results are the results of searching the search keywords on a search engine; according to the search results, web pages are accessed on a preset simulated desktop terminal and a preset simulated mobile terminal respectively to obtain web page access data; the web page access data includes training data and test data; feature extraction is performed on the test data to obtain classification features; the classification features are used to distinguish between normal web pages and mobile web page disguises; the types of feature extraction include text features, visual features, network features, element features and URL features; the classification features are input into a pre-trained classification model to obtain the mobile web page disguise detection result output by the classification model; wherein the classification model is trained based on the training data.

[0094] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement them without paying creative labor. Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software plus the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above-mentioned technical scheme is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., including some instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) perform the method described in some parts of each embodiment or an embodiment.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.< / iframe> < / footer> < / header>

Claims

1. A method for detecting mobile web page camouflage, characterized in that: include: Determine the search keywords and search results disguised by mobile web pages; The search results are the results of searching the search keywords on the search engine; Accessing web pages on a preset simulated desktop terminal and a preset simulated mobile terminal respectively according to the search results to obtain web page access data; the web page access data includes training data and test data; Extracting features from the test data to obtain classification features; the classification features are used to distinguish between normal web pages and mobile web page disguises; the types of feature extraction include text features, visual features, network features, element features and URL features; Inputting the classification features into a pre-trained classification model to obtain a mobile web page camouflage detection result output by the classification model; Wherein, the classification model is obtained through training based on the training data.

2. The method for detecting mobile web page camouflage according to claim 1, characterized in that: The step of determining the search keyword disguised as a mobile webpage includes: Extract seed keywords from preset illegal websites; Based on the seed keyword, a keyword-related search is performed in a preset search engine to obtain an extended keyword; the extended keyword is a keyword that is ranked within a preset range in the search results of the search engine; Based on the seed keyword, the extended keyword is denoised to obtain the disguised search keyword of the mobile web page.

3. The method for detecting mobile web page camouflage according to claim 1, characterized in that: The step of accessing a web page on a preset simulated desktop terminal and a preset simulated mobile terminal respectively according to the search results to obtain web page access data includes: By setting the Chrome browser on Mac OSX and Android device, the preset simulated desktop terminal and the preset simulated mobile terminal are formed; Determining the URL of a webpage according to the search results; Through the URL of the web page, Playwright is used to start the Chrome browser to access the web page, and the page HTML content, page screenshots, redirection chains and HAR files of the preset simulated desktop and the preset simulated mobile terminals are recorded respectively as the web page access data.

4. The method for detecting mobile web page camouflage according to claim 1, characterized in that: The text features include HTML text similarity features, core content similarity features and HTML structure similarity features; The extracting features of the test data to obtain classification features includes: Determine the full amount of HTML text according to the HTML content of the page, and extract the HTML text similarity feature based on a preset text feature extraction algorithm; Determine the core content according to the HTML content of the page, and extract the core content similarity feature based on a preset text feature extraction algorithm; the core content is the content after removing tag information from the HTML content of the page; Determine the HTML structure according to the HTML content of the page, and extract the HTML structure similarity feature based on a preset text feature extraction algorithm; the HTML structure is the content after removing the tag information and the full HTML text from the HTML content of the page; The preset text feature extraction algorithm includes blank character removal processing, content word segmentation processing and Hamming distance calculation of simhash value.

5. The method for detecting mobile web page camouflage according to claim 1, characterized in that: The visual features include Phash distance features, ORB features and histogram distance features; The extracting features of the test data to obtain classification features includes: Calculate the Hamming distance of the pHash value according to the page screenshot to obtain the Phash distance feature; According to the page screenshot, the ORB feature is extracted using an ORB algorithm and a matcher; The page screenshot is converted into HSV color space and the image histogram Bhattacharyya distance similarity is calculated to extract the histogram distance feature.

6. The method for detecting mobile web page camouflage according to claim 1, characterized in that: The network features include network behavior similarity features, request quantity ratio features, image quantity ratio features, Js file quantity ratio features, backend redirection features, cross-domain redirection features and false 404 features; The extracting features of the test data to obtain classification features includes: Calculate the Jaccard similarity coefficient according to the HAR file to obtain the network behavior similarity feature; Calculate the ratio of the total number of mobile terminal network requests to the total number of desktop terminal network requests according to the HAR file to obtain the request quantity ratio feature; Calculate the ratio of the total number of network requests for mobile-end image types and desktop-end image network requests according to the HAR file to obtain the image quantity ratio feature; Calculate the ratio of the total number of Js type network requests on the mobile terminal and the total number of Js network requests on the desktop terminal according to the HAR file, and obtain the Js file quantity ratio feature; The backend redirection feature, the cross-domain redirection feature and the false 404 feature are extracted according to the redirection chain.

7. The method for detecting mobile web page camouflage according to claim 1, characterized in that: The element features include desktop Iframe features, mobile Iframe features and image similarity features; The extracting features of the test data to obtain classification features includes: Extracting the desktop Iframe features and the mobile Iframe features according to the HTML content of the page; A picture file list is created according to the HTML content of the page, and the Jaccard similarity of the picture file list is calculated to obtain the image similarity feature.

8. The method for detecting mobile web page camouflage according to claim 1, characterized in that: The URL features include URL similarity features, KL divergence features and URL entropy features; The extracting features of the test data to obtain classification features includes: Creating a URL list according to the redirection chain, and calculating the Jaccard similarity of the URL list to obtain the URL similarity feature; The KL divergence feature and the URL entropy feature are extracted according to the redirection chain.

9. The method for detecting mobile web page camouflage according to any one of claims 1 to 8, characterized in that: The classification model is an XGBoost model.

10. A mobile web page camouflage detection system, characterized in that: include: A determination module, used to determine the search keywords and search results disguised as mobile web pages; The search results are the results of searching the search keywords on the search engine; An access module, used to access web pages on a preset simulated desktop terminal and a preset simulated mobile terminal respectively according to the search results to obtain web page access data; the web page access data includes training data and test data; A feature extraction module is used to extract features from the test data to obtain classification features; the classification features are used to distinguish between normal web pages and mobile web page disguises; the types of feature extraction include text features, visual features, network features, element features and URL features; A detection module, used for inputting the classification features into a pre-trained classification model to obtain a mobile web page camouflage detection result output by the classification model; Wherein, the classification model is obtained through training based on the training data.

Images