A webpage dark link suspiciousness evaluation method, system, device and storage medium

Abstract

The application discloses a webpage suspicious dark link evaluation method, system, device and storage medium, the method is scanning and identifying whether the resources in the webpage exist hidden links and scoring; according to whether js suspicious behavior is contained, secondary rendering detection scoring is carried out; according to the comparison results before and after, the suspicious degree of the dark link contained in the page is quantitatively evaluated by combining a quantitative algorithm. The application optimizes the page source dark link discovery function under the scene of js implicit modification of html source code, and improves the accuracy of dark link rule regular matching; according to the source characteristics, it is judged whether secondary calculation of page suspected dark link characteristics is needed, and the dark link matching score is adjusted according to the sensitive word matching result in the calculation process, the accuracy is improved; for the page source code without js implicit modification, single matching calculation is adopted, unnecessary resource consumption is reduced; finally, the reliable quantitative evaluation result is obtained by comprehensively calculating the score sequence.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network security, more particularly, it relates to a webpage suspicious link suspiciousness evaluation method, system, device and storage medium. BACKGROUND

[0002] Dark link, also known as black link, is a hidden link maliciously implanted on a webpage by a hacker, and is one of the cheating methods of black hat SEO. The purpose is to use high-weight website external links to improve the search engine search ranking of the website. There are several types of existing detection schemes:

[0003] a. Direct matching method. As one of the existing dark link detection general technical solutions, it is to obtain the webpage source code first, and then use dark link keywords or regular expressions to traverse and match the source code to find the dark link in the webpage.

[0004] b. Direct secondary comparison method based on before and after rendering. For example: Patent No. 2016102803259 (a stack-based dark link detection method), which is a method of exposing hidden links by calling headless browser rendering to obtain html source code

[0005] c. Secondary comparison method based on historical data. For example: Patent No. 2020106803894 (a website abnormal dark link detection method, device and storage medium), which uses historical pages as a reference to compare and evaluate the pages obtained at the time of detection to find newly added hidden dark links.

[0006] d. Dark link target evaluation method. For example: Patent No. 2021111246058 (a dark link detection method and system based on link content), which obtains hidden html links according to the regular method, then accesses the hidden link page and evaluates the page, and according to the evaluation result, it is determined whether it is a dark link.

[0007] e. Machine learning or semantic analysis method. For example: Patent No. 2014104522212 (an internet dark link detection method based on statistical machine learning), which establishes a prediction model through language analysis to directly predict whether there is a dark link.

[0008] For the direct matching method, since the webpage source code supports the javascript (js) programming language, theoretically there are countless ways to write and encrypt dark links, and their complexity far exceeds the matching range of the keyword library and regular library in the existing dark link detection method, resulting in more omissions. In addition, the webpage written in js programming language will not have any webpage writing features that are exclusive to hackers, and general developers will also use the same writing method. The dark link detection using the direct matching method will produce a large number of false positives.

[0009] Similarly, the rendering before and after the secondary comparison method, historical data secondary comparison method, machine learning or semantic analysis method, etc. The starting point is to affirm the idea that the dark chain "must be" written by hackers, ignoring that website developers will also write a large number of hidden elements for interactive effects, display effects, etc. The links will be marked as dark links by the above methods and will produce false positives.

[0010] For the dark chain pointing to the target evaluation method, it is foreseeable that more system resources will be consumed, the efficiency will be relatively low, and there will be problems of being unable to access due to the effectiveness of the domain name of the dark chain. SUMMARY

[0011] The technical problem to be solved by the present application is to solve the above-mentioned deficiencies of the prior art. The first object of the present application is to provide a webpage suspicious degree evaluation method.

[0012] The second object of the present application is to provide a webpage suspicious degree evaluation system.

[0013] The third object of the present application is to provide a computer device.

[0014] The fourth object of the present application is to provide a computer storage medium.

[0015] In order to achieve the above-mentioned first object, the present application provides a webpage suspicious degree evaluation method, which scans and identifies whether each resource in the webpage contains hidden links and scores; according to whether it contains js suspicious behavior, it performs secondary rendering detection and scoring; according to the comparison results before and after, the suspicious degree of the page containing the dark chain is quantitatively evaluated by combining the quantification algorithm.

[0016] As a further improvement, the following steps are included:

[0017] Step S1. Initialize the configuration, obtain the source code of the webpage to be detected;

[0018] Step S2. Prepare a regular feature library to detect resources, including a dark link regular library, a js illegal operation regular library, and a sensitive word library;

[0019] Step S3. Source code preprocessing, first load the sensitive word library from the regular feature library, obtain the sensitive word list, and insert the content of the sensitive word library as a new proper noun into the word segmentation library to improve the accuracy, and then use the word segmentation library to segment the source code;

[0020] Step S4. Use multi-threading technology to process the matching and positioning of each detection rule and the source code in parallel;

[0021] For each detection rule, the matching position, the contained url and the corresponding matching score are obtained according to the matching result;

[0022] According to the word segmentation result of the source code in step S3, sensitive words existing in the source code are recognized in combination with a sensitive word library; when the sensitive words are contained, a sensitive word score sequence is matched out, the original matching score is weighted by a weighting algorithm according to the sensitive word score sequence, and the highest score does not exceed N points, and finally the matching result and the matching score are returned;

[0023] Step S5. Obtain js illegal operation detection rules from the js illegal operation regular library, use the js illegal operation detection rules to detect whether the source code contains encryption; if yes, call the renderer to render the source code, intercept the rendered html source code, and repeat step 4 for the rendered page information; if no, go to step S7;

[0024] Step S6. Obtain dark chain detection rules from the dark chain regular library, traverse and compare the matching results before and after rendering according to the dark chain detection rules, and weight or de-weight the rendering result score sequence by comparing the differences between the files before and after rendering; if there is no before rendering and new after rendering, it is determined that there is a hidden link, and the matching result score is weighted to the highest; if there are both before and after rendering, it is determined that the page content has no hidden item, and the matching result score is de-weighted to reduce the possible false positive impact; if there is before rendering and no after rendering, no change is made; go to step S7;

[0025] Step S7. Page comprehensive score calculation, the score sequence of the matching item detection result of the whole page is calculated by the total score algorithm, and the page overall evaluation score is obtained after rounding, and the higher the score is, the more likely the page contains a dark chain.

[0026] Further, the N = 9, in step S6, the matching result score is weighted to the highest of 9 points; the matching result score is de-weighted to 70%; in step S7, the page overall evaluation score is 0-9 points.

[0027] Further, each detection rule in the dark chain regular library is a regular expression for detecting a specific html element hidden mode. According to the frequency of sampling statistics of known real dark chains, an integer score of 1-9 points is given after linear transformation formula calculation, and the higher the frequency is, the higher the score is, which means that this hidden mode is more likely to be used by hackers. The linear transformation formula is:

[0028]

[0029] Wherein, S represents the original frequency; Smax represents the highest frequency; Smin represents the lowest frequency; F represents the transformed score.

[0030] Further, the js illegal operation regular library is used to check whether the js changes the behavior of the html source code tag in the source code; whether the browser rendering needs to be called is judged by checking whether the hidden link or the promotion vocabulary is hidden by using js, so as to improve the detection accuracy and reduce unnecessary system resource consumption.

[0031] Further, in the sensitive word library, the high-frequency vocabulary table is counted according to the follow-up words of the real hidden link sample, and after manual or machine screening, the calculation score of each word after linear conversion is obtained by the same calculation method as the hidden link regular library, as a special sensitive word list.

[0032] Further, the weight increasing algorithm is:

[0033]

[0034] Wherein, Sf represents the final weighted result score; ai represents the score sequence of the sensitive word detection result; St represents the upper limit of the score, which is set to 9 points; b represents the basic score of the hidden link itself;

[0035] When the number of ai is 0:

[0036] S f =b

[0037] The total score algorithm is:

[0038]

[0039] Wherein, S: page total score; pmax: maximum value in the page score sequence; pi: sequence composed of the array except the maximum value in the page score sequence;

[0040] When the number of score sequences is equal to 1:

[0041] S=p max

[0042] When the number of score sequences is equal to 0:

[0043] S=0.

[0044] In order to realize the above-mentioned purpose two, the application provides a web page hidden link suspicious degree evaluation system, comprising:

[0045] The scanning module is used for scanning and identifying whether the hidden link exists in each resource in the webpage;

[0046] The scoring module is used for scoring the hidden link; and the secondary rendering detection score is obtained according to whether the js suspicious behavior is included;

[0047] The evaluation module is configured to quantitatively evaluate the suspicious degree of the dark link contained in the page according to the comparison result and a quantitative algorithm.

[0048] To achieve the above-mentioned purpose three, the application provides a computer device comprising a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned web page dark link suspicious degree evaluation method when executing the computer program.

[0049] To achieve the above-mentioned purpose four, the application provides a computer readable storage medium, which stores a computer program, and the computer program implements the above-mentioned web page dark link suspicious degree evaluation method when executed by a processor.

[0050] Advantages

[0051] Compared with the prior art, the application has the following advantages:

[0052] The application uses a large number of real dark link pages for classification and statistics, combines a linear transformation algorithm, gives a basic score of each type of dark link hiding mode and each dark link promotion word, and obtains a quantitative evaluation score of whether the page contains a real dark link based on the sensitive word weight increasing algorithm and the page total score algorithm. Compared with other dark link detection methods, the method is more scientific and comprehensive, increases the running efficiency, accuracy and reduces the missed detection, and reduces the influence of false positives as much as possible, and is more suitable for integrated into the existing security detection system for large-scale scanning detection or other security engineering applications. BRIEF DESCRIPTION OF DRAWINGS

[0053] Fig. 1 The mainstream diagram of the application;

[0054] Fig. 2 The detection sub-flow diagram of the application. DETAILED DESCRIPTION

[0055] The application will be further described below in combination with specific embodiments in the drawings.

[0056] Reference Figs. 1-2 A web page dark link suspicious degree evaluation method, the method comprising: scanning and identifying whether each resource in the web page contains a hidden link and scoring; performing secondary rendering detection and scoring according to whether the web page contains js suspicious behavior; and quantitatively evaluating the suspicious degree of the dark link contained in the page according to the comparison result and a quantitative algorithm.

[0057] Specifically, the method comprises the following steps:

[0058] Step S1. Initialization configuration, obtaining the source code of the web page to be detected, i.e. the html source code to be detected, through the get or post method;

[0059] Step S2. Prepare regular feature library detection resources, including dark chain regular library, js illegal operation regular library, sensitive word library;

[0060] Step S3. Source code preprocessing, first load the sensitive word library, obtain the sensitive word list, and insert the content of the sensitive word library as new proper nouns into the segmentation library to improve accuracy, and then use the segmentation library to segment the source code;

[0061] Step S4. Use multi-threading technology to process the matching and positioning of each detection rule and the source code in parallel;

[0062] For each detection rule, the matching position, contained url and corresponding matching score are obtained according to the matching result; for example: assuming that among all the detection rules, two detection rules are found to match in the html source code, the first one is a text indentation method dark chain regular, and its basic score is 4, which is evenly divided into the first and second positions (4 points); the second one is a "display:none" direct hidden dark chain regular, and its basic score is 1, which matches 3 positions (3rd, 4th and 5th positions) in the html source code, then the matching score sequence of this page is [2, 2, 1, 1, 1];

[0063] According to the segmentation result of the source code in step S3, the sensitive words existing in the source code are identified in combination with the sensitive word library; when the sensitive words are contained, the sensitive word score sequence is matched, and the original matching score is weighted by the weighting algorithm according to the sensitive word score sequence, and the highest is not more than N points, preferably, N = 9, and finally the matching result and the matching score are returned;

[0064] Step S5. Obtain js illegal operation detection rules from the js illegal operation regular library, use the js illegal operation detection rules to detect whether the source code exists encryption; if it exists, call the renderer to render the source code (html source code), the renderer can be chrome or firefox, and the rendered html source code is intercepted, and step 4 is repeated for the rendered page information; if it does not exist, step S7 is entered;

[0065] Step S6. Obtain dark chain detection rules from the dark chain regular library, traverse and compare the matching results before and after rendering according to the dark chain detection rules, and weight or de-weight the rendering result score sequence by comparing the differences between the files before and after rendering; if there is no before rendering, and new after rendering, it is determined that there is a hidden link, and the matching result score is weighted to the highest, i.e. weighted to 9 points; if there are both before and after rendering, it is determined that the page content has no hidden items, the matching result score is de-weighted to reduce the possible false positive impact, which can be de-weighted to 70%, and rounded, less than 1 point is calculated as 1 point; if there is before rendering and there is no after rendering, no change is made; step S7 is entered;

[0066] Step S7. Page comprehensive score calculation, the score sequence of the matching item detection result of the whole page is calculated by the total score algorithm and then rounded, and the page overall evaluation score is obtained. The page overall evaluation score is 0-9 points, and the higher the score is, the more likely the page contains a dark link.

[0067] Each detection rule in the dark link regular library is a regular expression for detecting a specific html element hiding method. According to the frequency of sampling statistics of known real dark links (sample source is not limited), an integer score of 1-9 is given after linear transformation formula calculation. The higher the frequency is, the higher the score is, which means that this hiding method is more likely to be used by hackers. The linear transformation formula is:

[0068]

[0069] Wherein, S represents the original frequency; Smax represents the highest frequency; Smin represents the lowest frequency; F represents the transformed score.

[0070] For example: there are 1000 real dark link samples from different websites. After classifying and counting the hidden methods, it is found that there are three kinds of dark link hiding methods. The first one is to directly hide in the html tag using "display:none", a total of 11, which is the least used among all dark link samples. The second one is to hide in the source code <script>标签中以特殊方式(以10、16进制字符或分割字符)将包含链接的js写入html标签的方式，共计700条，为所有暗链隐藏方式中最多的方式；第三种，通过文本缩进隐藏暗链，共计289条。则根据公式，得到三种隐藏方式各自的最终评分，第一种：(11-11) / (700-11)*8+1＝1分；第二种：(700-11) / (700-11)*8+1＝9分；第三种：(289-11) / (700-11)*8+1≈4.2分，取整后为4分。作为本发明中后续页面整体评分的基础分数。需要指出的是，本案例仅作为说明统计计算的样例，不代表实际统计结果或者暗链正则库的实际数据。

[0071] js非法操作正则库用于检查源码中是否包含js更改html源码标签的行为的规则；使用js隐藏暗链和推广词汇是黑客常用的技巧，通过检查是否存在使用js隐藏暗链或推广词汇来判断是否需要调用浏览器渲染，，以达到提高检测精准度，同时减少不必要系统资源消耗的目的。

[0072] 在敏感词库中，根据真实暗链样本的跟随词(暗链所在标签中包含的中英文名词、形容词等)统计出来的高频词汇表，经人工或机器(电脑)筛查后，再通过与暗链正则库同样的计算方式得到每个词线性转换后的计算分数，作为专用的敏感词列表。

[0073] 增权算法为：

[0074]

[0075] 其中，Sf代表最终加权结果分数；ai代表敏感词检测结果的分数序列；St代表分数上限，这里设定为9分；b代表暗链本身基础得分；

[0076] 当ai个数为0时：

[0077] Sf＝b

[0078] 例如：假设原始分数序列为[2,2,1,1,1]，即发现了5个疑似包含暗链的标签，其中第2个匹配位置的标签内还匹配到了两个敏感词："办理假毕业证”、"办理假身份证”，其在敏感词库的基础分数分别为5分、6分，则根据增权计算公式，原来2分的匹配位置调整为(52+62) / (5+6)*(9-2) / 9+2≈6.3，取整为6分，其他位置均未发现敏感词，则校准分数序列为：[2,6,1,1,1]。

[0079] 总分算法为：

[0080]

[0081] 其中，S：页面总分；pmax：页面分数序列中的最大值；pi：页面分数序列除了最大值以外的数组成的序列；

[0082] 当分数序列个数等于1时：

[0083] S＝pmax

[0084] 当分数序列个数等于0时：

[0085] S＝0。

[0086] 例如：假设最终分数序列为[2,6,1,1,1,4]，则基础因子为6，加权序列为[2,1,1,1,4]，总分算法，页面总分为(22+12+12+12+42) / (2+1+1+1+4)*(9-6) / 9+6≈6.9，取整为7分。在分值区间0～9处于高位，属于高可疑页面。

[0087] 一种网页暗链可疑度评估系统，包括：

[0088] 扫描模块，用于扫描识别网页内各项资源是否存在隐藏链接；

[0089] 评分模块，用于对隐藏链接进行评分；根据是否包含js可疑行为进行二次渲染检测评分；

[0090] 评估模块，用于根据前后比对结果结合量化算法对页面包含暗链的可疑程度作出量化评估。

[0091] 一种计算机设备，包括存储器和处理器，所述存储器存储有计算机程序，所述处理器执行所述计算机程序时实现上述的一种网页暗链可疑度评估方法。

[0092] 一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现上述的一种网页暗链可疑度评估方法。

[0093] 以上仅是本发明的优选实施方式，应当指出对于本领域的技术人员来说，在不脱离本发明结构的前提下，还可以作出若干变形和改进，这些都不会影响本发明实施的效果和专利的实用性。< / script>

Claims

1. A method for assessing the suspiciousness of hidden links on a webpage, characterized in that, Scan and identify hidden links in various resources within a webpage and score them; perform secondary rendering detection and scoring based on whether suspicious JavaScript behavior is present; and make a quantitative assessment of the suspiciousness of the page containing hidden links based on the comparison results before and after and a quantitative algorithm. Includes the following steps: Step S1. Initialize the configuration and obtain the source code of the webpage to be tested; Step S2. Prepare regular expression feature library detection resources, including a dark link regular expression library, a JavaScript illegal operation regular expression library, and a sensitive word library; Step S3. Source code preprocessing: First, load the sensitive word library, obtain the list of sensitive words, and insert the content of the sensitive word library as new proper nouns into the word segmentation library to improve accuracy. Then, use the word segmentation library to segment the source code. Step S4. Utilize multi-threading technology to process the matching and location of each detection rule with the source code in parallel; For each detection rule, obtain the matching location, the included URL, and the corresponding matching score based on its matching results; Based on the word segmentation results of the source code in step S3, and combined with the sensitive word library, the sensitive words present in the source code are identified; when a sensitive word is contained, the sensitive word score sequence is matched, and the original matching score is weighted by a weighting algorithm based on the sensitive word score sequence, with a maximum of N points, and finally the matching result and matching score are returned; Step S5. Obtain the JavaScript illegal operation detection rules from the JavaScript illegal operation regular expression library, and use the JavaScript illegal operation detection rules to check whether the source code is encrypted; if it is, call the renderer to render the source code, and capture the rendered HTML source code. Repeat step 4 for the rendered page information; if it is not, proceed to step S7. Step S6. Obtain the dark link detection rules from the dark link regular expression library, traverse and compare the matching results before and after rendering according to the dark link detection rules, and perform weighting or weighting processing on the rendering result score sequence by comparing the differences between the files before and after rendering. If a hidden link is not present before rendering but is added after rendering, the matching result score is considered to have a hidden link and is given the highest weight. If a hidden link exists both before and after rendering, the page content is considered to have no hidden items and the matching result score is given a lower weight to reduce the impact of possible false alarms. If it exists before rendering but not after rendering, then no changes are made; proceed to step S7. Step S7. Page overall score calculation: The score sequence of the matching detection results for the entire page is calculated by the total score algorithm and then rounded to obtain the overall page score. The higher the score, the greater the likelihood that the page contains hidden links.

2. The method for assessing the suspiciousness of hidden links on a webpage according to claim 1, characterized in that, When N=9, in step S6, the matching result score is weighted up to the maximum of 9 points; the matching result score is weighted down to 70%; in step S7, the overall page rating score is obtained as 0 to 9 points.

3. The method for assessing the suspiciousness of hidden links on a webpage according to claim 1, characterized in that, Each detection rule in the hidden link regular expression library is a regular expression that detects a specific way of hiding HTML elements. Based on the frequency of sampling statistics of known real hidden links, a linear transformation formula is used to calculate an integer score from 1 to 9. The higher the frequency, the higher the score, indicating that this hiding method is more likely to be exploited by hackers. The linear transformation formula is as follows: Where S represents the original frequency; Smax represents the highest frequency; Smin represents the lowest frequency; and F represents the transformed score.

4. The method for assessing the suspiciousness of hidden links on a webpage according to claim 1, characterized in that, The JavaScript illegal operation regular expression library is used to check whether the source code contains rules for JavaScript to modify HTML source code tags; by checking for the use of hidden links or promotional keywords in JavaScript, it determines whether browser rendering should be called, thereby improving detection accuracy and reducing unnecessary system resource consumption.

5. The method for assessing the suspiciousness of hidden links on a webpage according to claim 1, characterized in that, In the sensitive word database, a high-frequency vocabulary list is compiled based on the follow words of real dark link samples. After being screened manually or by machine, each word is linearly transformed and scored using the same calculation method as the dark link regularization database. This score serves as a dedicated sensitive word list.

6. The method for assessing the suspiciousness of hidden links on a webpage according to claim 1, characterized in that, The weighting algorithm is as follows: Where Sf represents the final weighted result score; ai represents the score sequence of sensitive word detection results; St represents the upper limit of the score, which is set to 9 points here; b represents the base score of the dark link itself; When the number of ai is 0: The total score algorithm is as follows: Where S: total page score; pmax: maximum value in the page score sequence; pi: sequence of numbers in the page score sequence excluding the maximum value; When the number of fraction sequences is equal to 1: When the number of fractions in the sequence is 0: 。 7. A system for assessing the suspiciousness of hidden links on web pages, characterized in that, include: The scanning module is used to scan and identify whether there are hidden links in various resources within a webpage; The rating module is used to rate hidden links; A secondary rendering detection score is performed based on whether suspicious JavaScript behavior is included. The evaluation module is used to quantitatively assess the suspiciousness of a page containing hidden links based on before-and-after comparison results and a quantitative algorithm. The work process is as follows: Step S1. Initialize the configuration and obtain the source code of the webpage to be tested; Step S2. Prepare regular expression feature library detection resources, including a dark link regular expression library, a JavaScript illegal operation regular expression library, and a sensitive word library; Step S3. Source code preprocessing: First, load the sensitive word library, obtain the list of sensitive words, and insert the content of the sensitive word library as new proper nouns into the word segmentation library to improve accuracy. Then, use the word segmentation library to segment the source code. Step S4. Utilize multi-threading technology to process the matching and location of each detection rule with the source code in parallel; For each detection rule, obtain the matching location, the included URL, and the corresponding matching score based on its matching results; Based on the word segmentation results of the source code in step S3, and combined with the sensitive word library, the sensitive words present in the source code are identified; when a sensitive word is contained, the sensitive word score sequence is matched, and the original matching score is weighted by a weighting algorithm based on the sensitive word score sequence, with a maximum of N points, and finally the matching result and matching score are returned; Step S5. Obtain the JavaScript illegal operation detection rules from the JavaScript illegal operation regular expression library, and use the JavaScript illegal operation detection rules to check whether the source code is encrypted; if it is, call the renderer to render the source code, and capture the rendered HTML source code. Repeat step 4 for the rendered page information; if it is not, proceed to step S7. Step S6. Obtain the dark link detection rules from the dark link regular expression library, traverse and compare the matching results before and after rendering according to the dark link detection rules, and perform weighting or weighting processing on the rendering result score sequence by comparing the differences between the files before and after rendering. If a hidden link is not present before rendering but is added after rendering, the matching result score is considered to have a hidden link and is given the highest weight. If a hidden link exists both before and after rendering, the page content is considered to have no hidden items and the matching result score is given a lower weight to reduce the impact of possible false alarms. If it exists before rendering but not after rendering, then no changes are made; proceed to step S7. Step S7. Page overall score calculation: The score sequence of the matching detection results for the entire page is calculated by the total score algorithm and then rounded to obtain the overall page score. The higher the score, the greater the likelihood that the page contains hidden links.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the webpage hidden link suspicion assessment method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the webpage hidden link suspicion assessment method according to any one of claims 1-6.

Images