Malicious website identification method based on PageRank algorithm

Abstract

The invention discloses a malicious website identification method based on a PageRank algorithm, and belongs to the field of network security protection. According to the method, malicious website identification is realized through multi-dimensional feature analysis, PageRank algorithm optimization and an intelligent research and judgment model; according to the method, the problem that a traditional malicious website studying and judging method is insufficient in accuracy and efficiency when facing novel and hidden malicious websites is solved, and the accuracy of studying and judging the malicious websites is improved; the timeliness of research and judgment is enhanced; the comprehensive analysis of multi-dimensional features is realized, and the comprehensiveness and reliability of research and judgment are improved.

Description

Technical Field

[0001] The invention belongs to the field of network security protection, and in particular relates to a malicious website identification method based on PageRank algorithm. Background Art

[0002] With the rapid development of Internet technology, website security issues have become increasingly prominent and have become an important issue that needs to be addressed urgently. In particular, in recent years, traffic fragmentation behavior of bad websites has gradually become a new attack method widely used by bad elements. This attack method cleverly mixes normal traffic with malicious traffic, making it difficult to effectively identify and distinguish normal traffic, which may lead to serious consequences such as significant degradation of website performance and leakage of sensitive data. Traditional security protection measures seem to be powerless in the face of this new type of attack and it is difficult to provide effective protection.

[0003] Conventional malicious behavior traffic detection mainly relies on content restoration or behavior feature detection of traffic response packets (such as HTTP POST), and request content restoration or action detection of traffic request packets (such as HTTP GET). However, in the TCP protocol design, the maximum limit of a single TCP payload packet is 1480 bytes, while the content of HTTP GET packets is usually small, often not exceeding 500 bytes, so traffic fragmentation rarely occurs. This makes the previous detection methods appear stretched when facing fragmentation behavior, and it is impossible to effectively detect and deal with it.

[0004] At the same time, the current traffic slicing mechanism is widely used by criminals in illegal areas to evade detection by existing regulatory measures. These black and gray traffic slicing behaviors are mixed in normal traffic slicing, which brings great challenges to detection and disposal. Traditional research and judgment methods mostly rely on domain name feature analysis, filing information query, blacklist filtering and other means. Although these methods can identify and block malicious URLs to a certain extent, they often seem powerless when facing advanced threats and new attack methods.

[0005] Specifically, the problems existing in the prior art mainly include:

[0006] Insufficient accuracy: Traditional methods are difficult to fully and accurately identify all types of malicious URLs, especially those that evade detection through disguise and covert means.

[0007] Obvious lag: The analysis and judgment methods that rely on blacklists and preset rules have obvious lags and are unable to cope with the rapid emergence of new malicious URLs.

[0008] Lack of multi-dimensional analysis: Traditional methods often only focus on a single or limited number of features, and lack comprehensive analysis of the multi-dimensional features of malicious URLs, resulting in incomplete and inaccurate research and judgment results.

[0009] Poor timeliness: For newly launched web pages, due to the lack of sufficient links and external references, it is difficult for the traditional PageRank algorithm to give them a reasonable ranking, which affects the timeliness of the judgment.

[0010] In order to solve the above problem, the present invention proposes a method for identifying malicious websites based on PageRank algorithm. Summary of the invention

[0011] In response to the problems mentioned in the background technology, the present invention proposes a malicious URL identification method based on the PageRank algorithm to solve the problems of insufficient accuracy and efficiency of traditional malicious URL analysis methods when facing new and hidden malicious URLs, improve the accuracy of malicious URL analysis; enhance the timeliness of analysis; realize comprehensive analysis of multi-dimensional features, and improve the comprehensiveness and reliability of analysis.

[0012] Technical solution: In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0013] A method for identifying malicious URLs based on the PageRank algorithm realizes malicious URL identification through multi-dimensional feature analysis, PageRank algorithm optimization, and intelligent judgment model, and specifically includes the following steps:

[0014] S1: Collection and organization of data to be studied;

[0015] S2: Conduct basic research and judgment capability analysis on the sorted data to obtain the research and judgment results;

[0016] S3: Based on the research results, feature extraction and PageRank algorithm model construction are performed to calculate the PageRank value;

[0017] S4: Comprehensively collect and analyze the multi-dimensional characteristics of malicious URLs, build a comprehensive research and judgment system, and optimize the PageRank algorithm, including:

[0018] S41: Content quality assessment: Use natural language processing technology to conduct in-depth analysis of web page content, build a multi-dimensional content scoring system, and calculate content quality scores;

[0019] S42: User behavior data analysis: collect and analyze multi-dimensional behavior data of users on web pages, build a user behavior score model, and calculate user behavior scores;

[0020] S43: time decay factor calculation: calculate the time decay score according to the time information of the web page;

[0021] S5: Build an intelligent analysis model and visualize the analysis and summary results.

[0022] Preferably, in S2, basic research and judgment capability analysis includes domain name feature analysis, registration information query, domain name collection search, blacklist-based filtering method, feature matching and machine learning-based method, data mining and deep learning-based method, and heuristic analysis.

[0023] Preferably, in S3, the specific process of calculating the PageRank value is:

[0024] The calculation formula of PageRank value PR(A) is as follows:

[0025] ,

[0026] in, is the PageRank value of page A, Is a page pointing to page A The PageRank value, Is the page The number of links pointing to other pages, Is a page pointing to page A The PageRank value, Is the page The number of links pointing to other pages, and d is the damping coefficient.

[0027] Preferably, in S41, content quality assessment: using natural language processing technology to deeply analyze web page content, construct a multi-dimensional content scoring system, and calculate the content quality score in the following specific process:

[0028] The multi-dimensional content scoring system includes: text content analysis, originality scoring, information richness scoring, language fluency scoring and credibility analysis;

[0029] The weighted comprehensive ranking index is introduced to evaluate the comprehensive quality of web pages. The specific calculation formula is:

[0030] Final ranking index = α×PR+(1-α)×content quality score;

[0031] Specifically:

[0032] ,

[0033] Among them, α is the weight coefficient, PR is the PageRank value of the web page, is the PageRank value of page A, and the content quality score is the score calculated based on the NLP analysis results. is the probability of randomly jumping to a page, N is the total number of web pages; d is the damping coefficient; is the sum of the PageRank values ​​passed from all pages pointing to page A, weighted by the quality and quantity of the links; (1-d) is the probability that the user will continue to click on the link instead of randomly jumping; (1+α)×CQ(A) is the direct contribution of the content quality score to the PageRank value, adjusted by the weight coefficient α; InLinks(A) is the sum of the PageRank values ​​passed from all pages pointing to page A; Is a page pointing to page A The PageRank value, Is the page The number of links pointing to other pages.

[0034] Preferably, in S42, user behavior data analysis: collecting and analyzing multi-dimensional behavior data of users on web pages, building a user behavior score model, and calculating the specific process of user behavior score is:

[0035] The multi-dimensional behavior data of users on web pages includes: visits, nighttime behavior, association analysis, number of visits, historical tags and user reviews;

[0036] By setting the weight coefficient, the user behavior score is combined with the PageRank value to generate a ranking index. The calculation formula of the ranking index is as follows:

[0037] CRS(A)=β×PR(A)+(1-β)×UB(A),

[0038] Among them, CRS(A) is the final ranking indicator of page A; β is the weight coefficient; PR(A) is the PageRank value of page A, and UB(A) is the user behavior score of page A, which is calculated based on user behavior data, as follows:

[0039] ,

[0040] Where n is the number of user behavior indicators; is the weight of the i-th user behavior indicator, is the score of page A on the i-th user behavior indicator.

[0041] Preferably, in S43, the time decay factor is calculated: according to the time information of the web page, the specific process of calculating the time decay score is:

[0042] ,

[0043] Among them, PR(A) is the PageRank value of page A; is the time decay factor of page A, which is calculated based on the online time and content update status of page A; It is page T i is the inverse of the time decay factor; d is the damping coefficient; InLinks(A) is the sum of the PageRank values ​​passed from all pages pointing to page A; Is a page pointing to page A The PageRank value, Is the page The number of links pointing to other pages;

[0044] The calculation formula of the time decay factor τ is:

[0045] τ=e-λ×Δt,

[0046] Among them, λ is the decay rate, e is the base of the natural logarithm, and Δt is the difference between the page online time and the current time.

[0047] As a preferred embodiment, in S5, the specific process of building the intelligent analysis model is as follows:

[0048] Combining natural language processing technology, machine learning and time decay mechanism, comprehensively considering the characteristics of multi-dimensional feature data, selecting classifiers, and building intelligent judgment models;

[0049] S51: Data preprocessing: Standardize the extracted feature data; clean the collected URL samples to ensure the accuracy of the data set;

[0050] S52: Model training and validation: Divide the experimental data set into training set and test set; Use K-fold cross validation method to evaluate the model;

[0051] S53: Comprehensive analysis: Based on the optimized PageRank algorithm, calculate the comprehensive PageRank value of each URL.

[0052] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0053] (1) Improving the accuracy of malicious URL identification: The present invention uses multi-dimensional feature analysis and comprehensively utilizes domain name features, filing information, search engine inclusion, content quality, user behavior data, and time information to comprehensively construct a malicious URL analysis and judgment system.

[0054] The present invention realizes PageRank algorithm optimization: on the basis of the traditional PageRank algorithm, three key elements, content quality evaluation, user behavior data and time decay factor, are introduced to improve the accuracy and timeliness of research and judgment; malicious websites that evade detection through disguise and concealment can be better identified.

[0055] (2) Enhanced timeliness: By introducing a time decay factor, the present invention can dynamically adjust the PageRank value of a web page to reflect its timeliness and importance. By real-time monitoring and analysis of user behavior data, emerging malicious URLs can be discovered and processed in a timely manner. The system can monitor network traffic in real time, issue early warnings and intercept malicious URLs, and achieve real-time monitoring and early warning.

[0056] (3) Realize multi-dimensional comprehensive analysis: The present invention uses multi-dimensional features such as domain name features, filing information, search engine inclusion, content quality, user behavior data, and time information to build a comprehensive analysis and judgment system.

[0057] Data visualization and reporting: The system provides data visualization tools to intuitively display the analysis results in the form of charts, which is convenient for users to analyze and make decisions; at the same time, the generated analysis reports also provide users with detailed analysis basis, further enhancing the transparency and credibility of the analysis.

[0058] (4) Improving the level of intelligence: The present invention combines natural language processing technology, machine learning and time decay mechanism to construct an intelligent analysis and judgment model. By continuously optimizing and iterating model parameters, the level of intelligence and real-time response capability of the analysis and judgment are improved. It can identify malicious URLs with high precision and efficiency, and realize comprehensive and in-depth analysis of malicious URLs.

[0059] Adaptive adjustment: The system can automatically adjust the analysis and judgment strategy according to the dynamic changes of the network environment, improving the system's adaptability and flexibility.

[0060] (5) The present invention can improve the accuracy of malicious URL analysis and ensure the effective identification of new and hidden malicious URLs; enhance the timeliness of analysis and timely respond to the threats of emerging malicious URLs; realize comprehensive analysis of multi-dimensional features and improve the comprehensiveness and reliability of analysis; improve the efficiency and accuracy of malicious URL processing through intelligent analysis models, and provide strong support for network security protection. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 is a data processing flow chart of the present invention;

[0062] Figure 2 It is a schematic diagram of the quantitative assumption relationship structure of the present invention;

[0063] Figure 3 It is a schematic diagram of the quality assumption relationship structure of the present invention;

[0064] Figure 4 This is a logic architecture diagram of malicious website analysis and judgment of the present invention;

[0065] Figure 5 It is a schematic diagram of the data experiment processing flow of the present invention;

[0066] Figure 6 The present invention Figure 5 Schematic diagram of the basic research and judgment ability analysis;

[0067] Figure 7 It is a business timing diagram of the present invention;

[0068] Figure 8 The present invention Figure 4 A flowchart of content quality assessment in ;

[0069] Fig. 9 The present invention Figure 4 Flow chart of user behavior data in;

[0070] Fig.10 The present invention Figure 4 Flow chart of the time decay factor in ;

[0071] Fig.11 The present invention Figure 4 A flowchart of a blacklist-based filtering method in FIG.

[0072] Fig.12 The present invention Figure 4 Flowchart of the method based on feature matching and machine learning in ;

[0073] Fig.13 The present invention Figure 4 Flowchart of the data mining and deep learning-based methods in ;

[0074] Fig.14 The present invention Figure 4 Schematic diagram of the heuristic analysis process in . DETAILED DESCRIPTION

[0075] The present invention is further illustrated below in conjunction with specific examples. The examples are implemented based on the technical solutions of the present invention. It should be understood that these examples are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0076] Example 1

[0077] The present invention aims at the problems of insufficient accuracy, obvious lag, lack of multi-dimensional analysis and poor timeliness in the existing malicious website identification technology. This embodiment proposes a malicious website identification method based on PageRank algorithm, which mainly focuses on the identification and interception of malicious websites. The method can be integrated into network security equipment, security software or cloud platform to provide real-time malicious website detection and protection services for network users and enterprises.

[0078] In this embodiment, the abbreviations and key terms are defined as follows:

[0079] Data packet: Packet is the data unit in TCP / IP protocol communication transmission, generally also called "data packet".

[0080] Traffic fragmentation: The Internet Protocol allows IP fragmentation, when a data packet is larger than the link maximum transmission unit, it can be broken into many small enough fragments to be transmitted on it.

[0081] Three-way handshake: In the TCP protocol, the communicating parties will understand the above information through three TCP messages and establish a TCP connection on this basis. The process of exchanging three TCP message segments between the communicating parties is what is commonly known as the three-way handshake process for TCP connection establishment.

[0082] Filtering conditions in traffic mirroring: Contains inbound rules and outbound rules, which are used to filter the network traffic mirrored in the mirroring session.

[0083] IP header length (Header Length): This field is used to describe the length of the IP header, because there is an optional part of variable length in the IP header. This part occupies 4 bits, and the unit is 32 bits (4 bytes), that is, the value of this area = IP header length (in bits) / (8×4), so the maximum length of an IP header is "1111", that is, 15×4=60 bytes. The minimum length of the IP header is 20 bytes.

[0084] Total Length of IP Packet: The length of the IP packet (including header and data) calculated in bytes, so the maximum length of an IP packet is 65535 bytes.

[0085] PageRank: PageRank, also known as web page level, Google left ranking or Page Rank, is a technology that is calculated based on the hyperlinks between web pages. As one of the elements of web page ranking, Google uses it to reflect the relevance and importance of web pages. It is one of the factors often used to evaluate the effectiveness of web page optimization in search engine optimization operations.

[0086] The method provided in this embodiment aims to achieve high-precision and high-efficiency malicious URL identification by comprehensively using multi-dimensional feature analysis, PageRank algorithm optimization, and intelligent judgment model. The specific implementation steps are:

[0087] S1: Collection and organization of data to be studied;

[0088] Data collection includes network traffic data, web page content data, user behavior data and other related data. The specific information is as follows:

[0089] Network traffic data: Capture HTTP / HTTPS request and response packets from network traffic. These packets contain detailed information about the web pages visited by users, such as URLs, request headers, response bodies, etc.

[0090] Use traffic mirroring or traffic capture tools to collect network traffic data in real time or at regular intervals to ensure the timeliness and integrity of the data.

[0091] Web page content data: Obtain the HTML content, JavaScript code, CSS style, etc. of the web page to be analyzed through crawler technology or API interface. Parse and store the web page content to facilitate subsequent natural language processing and feature extraction.

[0092] User behavior data: collects user access behavior data on web pages, such as number of visits, access time, access path, click behavior, etc.

[0093] Use browser plug-ins, log analysis tools or third-party data providers to obtain these behavioral data.

[0094] Other related data: collect other data related to the webpage, such as domain name registration information, record information, search engine inclusion, etc. This data can be obtained from channels such as domain name registrars, record management agencies or search engines.

[0095] Data organization includes data cleaning, data standardization, data storage, data labeling, etc. Specifically:

[0096] Data cleaning: Clean the collected data to remove duplicate, invalid or abnormal data. Process missing values, such as using mean filling, median filling or model-based filling methods.

[0097] Data standardization: Standardize data from different sources and formats to ensure data consistency and comparability. For example, standardize date and time formats, convert numerical data to the same dimension, etc.

[0098] Data storage: Store the organized data in a database or data warehouse for subsequent query and analysis. Choose a suitable storage solution based on the type and characteristics of the data, such as a relational database, NoSQL database, or distributed storage system.

[0099] Data labeling: Label known normal and malicious URLs to provide sample data for subsequent machine learning model training. Labeling can be done manually or using semi-automatic or fully automatic labeling tools.

[0100] S2: Conduct basic research and judgment capability analysis on the sorted data to obtain the research and judgment results;

[0101] Basic research and judgment capability analysis includes domain name feature analysis, registration information query, domain name inclusion search, blacklist-based filtering methods, feature matching and machine learning-based methods, data mining and deep learning-based methods, and heuristic analysis.

[0102] Domain name feature analysis: Extract domain name length, character composition (such as the ratio of letters, numbers, and special characters), whether it contains official names or abbreviations, etc. By analyzing the structure, character combination, and length of the domain name, potential malicious URLs can be preliminarily screened out.

[0103] Filing information query: query the filing information of relevant national competent departments, and obtain basic information of the website, such as the filing unit, registration time, filing status and consistency of the filing subject, etc. Comprehensively consider the consistency between the filing information and the actual content of the website to improve the accuracy of the research and judgment.

[0104] Domain name index search: Search engine indexing status, check whether the website is indexed by mainstream search engines, as well as the number of indexed web pages and the indexing time. By analyzing the search engine's indexing strategy and update cycle, the legitimacy of the website can be determined.

[0105] Blacklist-based filtering method: The blacklist-based filtering method is a simple and effective means of detecting malicious URLs. It relies on a pre-built and maintained blacklist database of malicious URLs. In the process of malicious URL analysis, the blacklist-based filtering method will first match the URL to be analyzed with the URLs in the blacklist database. The matching algorithm usually uses string matching, such as exact matching or fuzzy matching, to ensure that the URLs in the blacklist can be accurately identified. If the URL to be analyzed successfully matches a URL in the blacklist, the system will immediately determine that the URL is a malicious URL and take appropriate disposal measures. The detailed process is as follows. Fig.11 shown.

[0106] Method based on feature matching and machine learning: The method based on feature matching and machine learning is a method of malicious URL analysis that combines traditional feature matching technology and modern machine learning algorithms. This method first uses feature extraction technology to extract multi-dimensional features from the web page to be analyzed, such as domain name features, content features, behavioral features, etc. These features can fully reflect the properties and behaviors of the web page, providing rich data support for subsequent analysis. On the basis of feature extraction, this method uses machine learning algorithms to learn and analyze the extracted features. Machine learning algorithms automatically mine the correlation and patterns between features from large amounts of data, thereby accurately identifying malicious URLs. Commonly used machine learning algorithms include support vector machines (SVM), random forests, deep neural networks (DNN), etc. These algorithms can train efficient classification models based on known normal URLs and malicious URL samples to predict the maliciousness of unknown URLs. The detailed process is as follows. Fig.12 shown.

[0107] Methods based on data mining and deep learning: Methods based on data mining and deep learning are cutting-edge and powerful malicious URL analysis technologies, which combine the data exploration and pattern discovery capabilities of data mining with the automatic feature learning and complex pattern recognition capabilities of deep learning. This method first uses data mining technology to extract multi-dimensional features related to malicious URLs from massive network data. On the basis of feature extraction, deep learning algorithms are introduced to further explore the deep associations and complex patterns between features. Deep learning models, such as convolutional neural networks (CNN), recurrent neural networks (RNN), long short-term memory networks (LSTM), etc., can automatically learn the intrinsic representation of data without the need for manual feature design. These models convert raw features into high-level abstract features through multiple layers of nonlinear transformations, thereby achieving accurate identification of malicious URLs. The detailed process is as follows. Fig.13 shown.

[0108] Heuristic analysis: Heuristic analysis is a reasoning method based on experience and intuition. In the field of malicious URL research, it combines expert knowledge and domain rules to conduct comprehensive analysis and evaluation of web pages by simulating the decision-making process of human analysts. Heuristic analysis does not rely on a large amount of training data and complex mathematical models, but relies on the analyst's understanding of the characteristics of malicious URLs and keen insight into the network environment.

[0109] In heuristic analysis, we first need to summarize a series of heuristic rules related to malicious URLs based on experience. These rules may be based on multiple dimensions such as domain name characteristics, web page content, user behavior, etc. Then, the analyst will apply these rules to the web pages to be judged, and through comparison and analysis, determine whether the web pages meet the characteristics of malicious URLs. Heuristic analysis is flexible and highly interpretable. It can quickly adapt to new network environments and malicious URL characteristics, and improve the accuracy and timeliness of judgment by continuously adjusting and optimizing heuristic rules. At the same time, because the heuristic rules are built based on human-understandable logic and common sense, the judgment results are easy to explain and verify. The detailed process is as follows. Fig.14 shown.

[0110] S3: Based on the research results, feature extraction and PageRank algorithm model construction are performed to calculate the PageRank value;

[0111] The PageRank algorithm is one of the core algorithms used by Google search engine to evaluate the importance of web pages. Its core concept is link analysis. The algorithm believes that the more times a web page is cited by other web pages (that is, the more inbound links there are), the more important it is in the network. In addition, the PageRank algorithm also takes into account the quality of links, that is, not all links contribute equally to the importance of the target web page. Links from high-quality, authoritative web pages have a more significant effect on increasing the importance of the target web page. This detailed distinction in link quality allows the PageRank algorithm to more accurately evaluate the true value of a web page, avoiding possible misjudgments that may result from relying solely on the number of links.

[0112] S31: Calculate PageRank value;

[0113] Suppose there is a group of 4 web pages: A, B, C, and D. If all pages link only to A, then the PR (PageRank) value of A will be the sum of the PageRanks of B, C, and D, that is, PR(A)=PR(B)+PR(C)+PR(D).

[0114] Suppose again that B links to A and C, C links only to A, and D links to all three other pages. A page only has one vote total, so B gives half a vote to each of A and C. By the same logic, only one-third of D's vote counts toward A's PageRank, i.e. .

[0115] In summary, for a page A, the calculation formula for its PageRank value PR(A) can be expressed as follows:

[0116] ,

[0117] in, is the PageRank value of page A, Is a page pointing to page A The PageRank value, Is the page The out-degree of the page The number of links pointing to other pages, Is a page pointing to page A The PageRank value, Is the page The number of links pointing to other pages. d is the damping coefficient, which is usually set to be greater than or equal to 0.7. It represents the probability that a user will continue to click on random links after reaching a page. n represents the nth element in a sequence corresponding to time t, which is an index of a time or sequence here.

[0118] like Figure 4 As shown in the figure, the process of PageRank logic algorithm is schematically illustrated (initial score of the page / number of outbound links = score corresponding to each outbound link), such as 0.9 / 3=0.3; 1 / 2=0.5; the corresponding calculation nodes will accumulate scores as parameters for subsequent calculations, such as 0.5+0.3=0.8 pointed to in the previous step.

[0119] S4: Comprehensively collect and analyze the multi-dimensional characteristics of malicious URLs, build a comprehensive research and judgment system, and optimize the PageRank algorithm, specifically:

[0120] S41: Content quality assessment: Use natural language processing technology to conduct in-depth analysis of web page content, build a multi-dimensional content scoring system, and calculate content quality scores;

[0121] The multi-dimensional content scoring system includes key aspects such as text content analysis, originality scoring, information richness scoring, language fluency scoring and credibility analysis.

[0122] In order to improve the comprehensiveness and accuracy of web page quality assessment in malicious URL analysis, the present invention introduces a content quality assessment mechanism based on the classic PageRank algorithm. The traditional PageRank algorithm mainly relies on the link relationship between web pages to evaluate the importance of web pages, but this method is difficult to fully reflect the true quality of web page content, especially for malicious websites full of spam, malicious advertisements or false content. Its limitations are particularly obvious. Therefore, the present invention combines natural language processing (Natural Language Processing, NLP), conducts an in-depth analysis of web page content, and constructs a multi-dimensional content scoring system, which aims to more comprehensively evaluate web page quality, thereby improving the accuracy of web page rankings and the effectiveness of malicious URL identification. The multi-dimensional content scoring system specifically covers the following key aspects:

[0123] 1. Bad information detection: that is, text content analysis, using NLP technology to deeply analyze web page text, accurately identify and eliminate potential malicious content or spam, such as false advertisements, phishing website tricks, etc. The presence of such bad information will directly lower the quality score of the web page.

[0124] 2. Originality score: The originality of webpage content is strictly checked through advanced text duplication detection technology. If there is a lot of plagiarism or repeated content, a lower originality score will be given, which will affect its overall quality evaluation.

[0125] 3. Information richness score: Comprehensively consider the hierarchical structure, content type and information richness of the webpage. For example, whether the content structure is complete, whether detailed and effective background information is provided, and whether a variety of media resources (such as pictures, videos, charts, etc.) are integrated, all of these factors have an important impact on the quality of the webpage.

[0126] 4. Language Fluency Scoring: The language quality of a web page is directly related to the user experience. The present invention further reflects the content quality of a web page by carefully analyzing the language fluency of the web page text, such as grammatical accuracy, correct spelling, and smooth expression.

[0127] 5. Credibility analysis: In-depth inspection of whether the webpage contains elements that can enhance credibility, such as company background introduction, business registration information, etc. The lack of credibility elements may indicate that the webpage has a risk of fraud, thereby reducing its quality score.

[0128] The introduction of this multi-dimensional content scoring system not only makes up for the shortcomings of the traditional PageRank algorithm in content quality assessment, but also significantly improves the accuracy and effectiveness of malicious URL research and judgment. When combining the content quality analysis results with the PageRank algorithm, a weighted comprehensive ranking indicator is also introduced. This indicator combines the PageRank value and content quality score of the web page, thereby achieving an evaluation of the comprehensive quality of the web page.

[0129] The specific calculation formula is: final ranking index = α×PR+(1-α)×content quality score, where α is the weight coefficient, which can be flexibly adjusted according to actual application needs; PR is the PageRank value of the web page, and the content quality score is the score calculated based on the NLP analysis results.

[0130] The detailed formula is as follows:

[0131]

[0132] in, is the PageRank value of page A, is the probability of randomly jumping to a page, N is the total number of web pages, this part is the basic item in the PageRank algorithm, used to simulate the user's random click behavior; d is the damping factor; It is the sum of the PageRank values ​​passed from all pages pointing to page A, weighted by the quality and quantity of the links; (1-d) is the probability that the user continues to click on the link instead of randomly jumping; (1+α)×CQ(A) is the direct contribution of the content quality score to the PageRank value, adjusted by the weight coefficient α. The influence of the two factors can be flexibly balanced according to actual application requirements. InLinks(A) is the sum of the PageRank values ​​passed from all pages pointing to page A; Is a page pointing to page A The PageRank value, Is the page The number of links pointing to other pages.

[0133] The following is a simplified example showing the PageRank values, content quality scores and final ranking indicators of five web pages; in this example, α is set to 0.7.

[0134] Table 1 Schematic diagram of content quality score calculation

[0135]

[0136] In Table 1, the final ranking index reflects the comprehensive quality of the web page; web page A has a higher PageRank value and a better content quality score, and its final ranking index is 0.72; web page C has a lower content quality score, and although its PageRank value is higher, its final ranking index is lower.

[0137] In order to verify the optimization effect of introducing content quality evaluation, the present invention conducted a series of experiments. The experimental results show that compared with the traditional PageRank algorithm, the comprehensive ranking method combined with content quality scoring can significantly improve the accuracy of web page quality evaluation, especially when identifying low-quality content and junk websites. The specific effect is reflected in: by comprehensively considering the quality of web page content, the ranking of junk websites (such as false advertisements, malware propagation pages, etc.) is significantly reduced, effectively avoiding the problem of these websites using a large number of external links to improve their rankings. Websites with strong originality, rich information, and good user experience have also been appropriately improved in the final ranking, avoiding the problem of high-quality websites ranking low due to few external links or other factors.

[0138] S42: User behavior data analysis: Collect and analyze multi-dimensional behavior data of users on web pages, build a refined user behavior score model, dynamically adjust the ranking weight of web pages, and calculate user behavior scores;

[0139] The multi-dimensional behavior data of users on web pages include: visits, nighttime behavior, association analysis, number of visits, historical tags, and user reviews. User behavior data has become a crucial factor when exploring the optimization path of search engine ranking algorithms. By collecting and analyzing the multi-dimensional behavior data of users on web pages, the popularity and quality of web pages can be more accurately evaluated, further improving the accuracy of search engines. The present invention proposes an optimization strategy that combines user behavior data. By constructing a refined user behavior scoring model, the ranking weight of web pages is dynamically adjusted, thereby achieving a more accurate web page quality assessment. The introduction of user behavior data not only reflects the popularity of web pages, but also deeply reveals the actual satisfaction and engagement of users with web page content. Specifically, the core indicators of user behavior data include:

[0140] 1. Visits analysis: record the number of users who visit a web page in a unit of time. Web pages with large visits usually have content that is more popular with users and has a higher click-through rate.

[0141] 2. Nighttime behavior analysis: By analyzing the user's access behavior at night, we can determine whether the webpage has all-day appeal. If a webpage can still receive a large number of visits at night, it means that its content is of high quality or highly consistent with user needs.

[0142] 3. Association analysis: This is to analyze the access behavior between web pages of the same origin. For example, after a user visits a certain page, does he immediately jump to other pages associated with that page?

[0143] 4. Visit (times) analysis: examine the number of times a user visits the same web page within a unit of time. This data helps determine whether the user has a high degree of stickiness to the web page content.

[0144] 5. History tag (analysis): By analyzing the changes in user visits to a web page over different time periods (such as the past 1 day, the past 3 days, the past 7 days), the long-term popularity of a web page can be identified.

[0145] 6. User evaluation (analysis): Collect and analyze users’ evaluation of the webpage on public platforms, including comment content, ratings, etc.

[0146] In order to effectively integrate user behavior data with the traditional PageRank algorithm, this application proposes a new comprehensive ranking mechanism. This mechanism combines user behavior scores with PageRank values ​​by setting reasonable weight coefficients to generate a new ranking index. Specifically, the ranking index can be calculated using the following formula:

[0147] Final ranking index = β × PageRank value + (1-β) × user behavior score,

[0148] That is, CRS(A)=β×PR(A)+(1-β)×UB(A),

[0149] Among them, CRS(A) is the final ranking indicator of page A; β is the weight coefficient, which is used to balance the impact of PageRank value and user behavior score on the final ranking; usually, the value of β can be adjusted according to the actual application scenario to find the best balance point; PR(A) is the PageRank value of page A, calculated according to the traditional PageRank algorithm; UB(A) is the user behavior score of page A, which can be calculated based on user behavior data. The specific calculation formula is:

[0150] ,

[0151] Where n is the number of user behavior indicators; is the weight of the i-th user behavior indicator, which is used to reflect the importance of this indicator in the comprehensive evaluation; is the score or value of page A on the i-th user behavior indicator.

[0152] In addition, for pages that are frequently marked or reported as bad URLs by users, the PageRank value of the webpage will be adjusted in a timely manner by combining the number of times the bad URLs are marked. For example, if a webpage frequently has negative feedback in user behavior data (such as a high number of bad URL markings), the PageRank value of the webpage will be reduced.

[0153] In order to verify the effectiveness of the proposed optimization method, this application has conducted a large number of experiments and verified it through web page data in the actual network. The experimental results show that the PageRank algorithm combined with user behavior data has achieved a significant improvement in ranking accuracy. In the experiment, after combining the user behavior score, the ranking of web pages with high negative user reviews and frequently marked as bad URLs dropped significantly. Through the detailed analysis of user behavior data, the quality of web page content can be accurately quantified, avoiding the blind spots of traditional ranking algorithms that rely purely on link structure. The experimental results are shown in the following table:

[0154] Table 2 Schematic diagram of user behavior score calculation

[0155]

[0156] The final ranking indicators in Table 2 reflect the ranking of web pages after comprehensively considering the PageRank value and the user behavior score. Web pages A and C are ranked relatively high, and their user behavior scores are high, indicating that these web pages are more secure and reliable, and have higher content quality.

[0157] Combine user behavior data: Combine user behavior data with PageRank value and generate a new ranking indicator by setting a reasonable weight coefficient β.

[0158] S43: time decay factor calculation: calculate the time decay score according to the time information of the web page;

[0159] The traditional PageRank algorithm is too dependent on the link structure between web pages, which makes it difficult for newly launched web pages to obtain high scores in the ranking due to the limited number of links. In order to overcome this limitation, this embodiment introduces a time decay factor to construct a more dynamic and fair scoring mechanism that can reflect the timeliness of web pages. Specifically, the calculation of the time decay factor includes the following aspects:

[0160] 1. Online time: that is, the time since the webpage was launched. Newer webpages have more growth potential.

[0161] 2. Last updated time: reflects the updating frequency of web page content. Frequently updated pages may be more timely and credible.

[0162] 3. Response time: that is, the response efficiency of the web page. A good response speed means a higher user experience.

[0163] 4. User access time distribution: Analyze users’ access behavior in different time periods to determine the timeliness and relevance of web page content.

[0164] 5. Registration time: The registration time of the domain name. Domain names registered earlier may have higher trust.

[0165] 6. TTL (Time To Live): The lifetime of a domain name. A longer TTL usually means a more stable and healthier website.

[0166] The specific process of calculating the time decay factor is:

[0167]

[0168] Among them, PR(A) is the PageRank value of page A; is the time decay factor of page A, which is calculated based on the online time and content update status of page A; It is page T i The inverse of the time decay factor is used to weight new pages when passing PageRank values; d is the damping coefficient; InLinks(A) is the sum of the PageRank values ​​passed from all pages pointing to page A; Is a page pointing to page A The PageRank value, Is the page The number of links pointing to other pages.

[0169] In order to introduce timeliness factors into the PageRank algorithm, the calculation formula of the time decay factor τ is set here:

[0170] τ=e-λ×Δt,

[0171] Wherein, λ is the decay rate, which is set to 0.1 in this embodiment (this value can be adjusted according to actual conditions), Δt is the difference between the online time of the page and the current time (unit: month); e is the base of the natural logarithm, which is approximately equal to 2.71828. According to the online time difference of each page, the corresponding time decay factor τ is calculated.

[0172] By introducing the time decay factor, the PageRank value of a page will be dynamically adjusted according to its online time and content update status; for new pages, their PageRank values ​​are weighted and multiplied by 1 / τ (the inverse of the time decay factor) to improve their scores. In this way, newer pages can make up for the score disadvantage caused by the small number of links. For old pages, their PageRank values ​​are directly multiplied by τ to reflect the possible decline in their value over time. As the page exists for a longer time, its importance will gradually decrease due to the influence of the time decay factor. The test results are as follows:

[0173] Table 3 Schematic diagram of time decay factor calculation

[0174]

[0175] As can be seen from Table 3, as the time of the page online increases, the time decay factor τ gradually decreases, and for new pages, the τ value is larger, which can effectively improve its initial score. For old pages that have existed for a long time, the quality and relevance of the content may change over time. The time decay factor acts as a "filter", gradually reducing the PageRank value of these pages to prevent their scores from being too high and affecting the accuracy of the results.

[0176] By introducing a time decay factor, the PageRank value of a web page is dynamically adjusted to reflect its timeliness and importance. PageRank algorithm optimization: Introduce content quality assessment, user behavior data, and time decay factors to optimize the ranking mechanism of web pages and improve the accuracy of research and judgment. Functional description: Based on the traditional PageRank algorithm, the system introduces content quality assessment, user behavior data, and time decay factors to optimize the ranking mechanism of web pages and improve the accuracy of research and judgment. User interaction: Users can view the comprehensive PageRank value of each URL through the interface to understand its importance and credibility in the network. UI interface example: The interface displays the comprehensive PageRank value of the URL and the contribution of each influencing factor (such as content quality score, user behavior score, etc.) in the form of a bar chart or pie chart.

[0177] The values ​​in this embodiment are all normalized example values, used to illustrate the algorithm optimization effect: Content quality score (0-1): reflects the credibility and quality of web page content. User behavior score (0-1): quantifies the health of user interaction behavior. Time decay factor (dynamic value): balances the timeliness difference between new and old pages. Weight coefficient (α, β): adjusts the impact of different dimensions on the final ranking.

[0178] S5: Build an intelligent analysis model;

[0179] The system combines natural language processing technology, machine learning and time decay mechanism to build an intelligent analysis model to achieve fast and accurate identification of malicious URLs.

[0180] S51: Data preprocessing: Standardize the extracted feature data to eliminate bias; perform strict cleaning on the collected URL samples to ensure the accuracy of the data set.

[0181] According to the specific needs of the experimental setting, multi-dimensional features are extracted to provide support for subsequent model construction.

[0182] S52: Model training and verification: Divide the experimental data set into a training set and a test set, and ensure that the training set contains a sufficient number of normal URLs and malicious URL samples. Use the training set data to train the constructed malicious URL recognition model, and continuously adjust the model parameters to achieve the best performance.

[0183] The K-fold cross-validation method was used to evaluate the stability and generalization ability of the model.

[0184] S53: Comprehensive analysis: Based on the optimized PageRank algorithm, calculate the comprehensive PageRank value of each URL.

[0185] Comprehensively consider the characteristics of multi-dimensional feature data, select high-performance classifiers such as support vector machine (SVM), random forest (RandomForest) or deep neural network (DNN) to build a malicious URL identification model. Take timely and effective disposal measures for confirmed malicious URLs, such as blocking, warning or deletion.

[0186] User interaction: Users can view the analysis results through the interface, including performance indicators such as the recognition rate of malicious URLs, false positive rate, and missed negative rate. UI interface example: The interface displays summary information of the analysis results, such as the list of identified malicious URLs, the accuracy of the analysis, etc., and provides a detailed analysis report for users to review.

[0187] Real-time monitoring and early warning function: The system can monitor network traffic in real time, issue early warnings and intercept malicious websites, and ensure the safety of network users. User interaction: Users can set early warning thresholds through the interface, receive real-time early warning information, and view interception records. UI interface example: The interface provides early warning setting options, and users can set early warning thresholds according to their needs (such as the number of malicious websites, visits, etc.). At the same time, real-time early warning information and interception records are displayed in the form of a timeline or list.

[0188] Data visualization and reporting functions: The system provides data visualization tools to intuitively display the analysis and judgment results in the form of charts, which is convenient for users to analyze and make decisions. At the same time, detailed analysis and judgment reports are generated for users to review and archive. User interaction: Users can select different types of visualization charts (such as line charts, bar charts, pie charts, etc.) through the interface to view the detailed data of the analysis and judgment results. At the same time, the analysis and judgment report can be exported to PDF or Excel format. UI interface example: A data visualization area is provided on the interface to display charts of the analysis and judgment results (such as a trend chart of the number of malicious URLs, a distribution chart of the comprehensive PageRank value, etc.). At the same time, a report export button is provided, and users can download the analysis and judgment report after clicking it.

[0189] In this embodiment, startup and configuration: After the user starts the system, enter the configuration interface and set the system parameters (such as data collection frequency, analysis model parameters, etc.). After the configuration is completed, it starts to automatically collect and analyze the characteristics of malicious URLs in network traffic. Multi-dimensional feature analysis: Automatically extract the multi-dimensional features of the URL and output the analysis results. PageRank algorithm optimization: According to the optimized PageRank algorithm, calculate the comprehensive PageRank value of each URL and display it. Intelligent analysis: The system combines multi-dimensional features and PageRank algorithm to perform intelligent analysis on malicious URLs and output the analysis results.

[0190] Software and hardware environment:

[0191] 1. Hardware environment: Server configuration. The experiment uses three high-performance servers, each equipped with Intel(R) Xeon(R) Gold 6238R CPU @ 2.20GHz, 32-core CPU, 256G memory, 20M bandwidth, Gigabit Ethernet card, and NVIDIA GeForce RTX 3080 Ti*4 graphics card to ensure sufficient computing resources. Network architecture: The server is connected to the Internet and connected to the network traffic monitoring point through a switch or router to collect network traffic data in real time.

[0192] 2. Software environment: Operating system. The server runs the CentOS Linux release 7.4.1708 operating system, which provides a stable operating environment. Development tools and libraries: Use programming languages ​​such as Python and Java, combined with natural language processing libraries (such as NLTK and spaCy), machine learning libraries (such as scikit-learn, TensorFlow / Keras), and big data processing tools (such as Hadoop and Spark) for development and implementation; use databases such as MySQL or MongoDB to store multi-dimensional feature data, research and judgment results, and user behavior data.

[0193] Example 2

[0194] In this embodiment, the multi-dimensional feature analysis is improved as follows:

[0195] 1. Feature extraction method: In addition to the domain name features, filing information, search engine inclusion status and other features mentioned in Example 1, other features can also be introduced, such as IP address geographic location analysis, DNS resolution records, SSL certificate information, etc., to further enrich the basis for analysis and judgment.

[0196] 2. Feature weight allocation: When constructing a multi-dimensional feature scoring system, in addition to the fixed weight allocation method in the first embodiment, a dynamic weight allocation method can also be used to automatically adjust the weight of each feature according to the actual application scenario and data distribution.

[0197] Improvements to PageRank algorithm optimization:

[0198] Optimization methods: In addition to introducing content quality evaluation, user behavior data and time decay factors, other optimization methods can also be considered, such as topic models (such as LDA) to classify web page content by topic, and further refine the ranking basis of web pages. Topic models can help identify the theme of web pages. For malicious URL analysis, malicious URLs with specific themes such as malware downloads and phishing websites can be distinguished.

[0199] PageRank algorithm: Although the PageRank algorithm performs well in web page ranking, other link analysis algorithms, such as the HITS (Hyperlink-Induced Topic Search) algorithm, can also be considered as an alternative. The HITS algorithm evaluates the importance of a web page by calculating the authority and hub of the web page, which may be more advantageous in certain specific scenarios.

[0200] Application scenario expansion: This embodiment can be applied to the following scenarios:

[0201] 1) E-commerce security: In the field of e-commerce, the present invention can be used to identify and block malicious websites to protect users' personal information and property security. For example, in the payment process, the transaction website is analyzed and judged to prevent fraudulent activities of malicious websites such as phishing websites.

[0202] 2) Search engine optimization: In the field of search engine optimization (SEO), the present invention can be used to evaluate the true value and credibility of web pages and help search engines rank web pages more accurately. For example, by introducing content quality evaluation and user behavior data, the search engine's ranking algorithm can be optimized to improve the accuracy and relevance of search results.

[0203] 3) Online advertisement filtering: In the field of online advertisement, the present invention can be used to identify and filter malicious advertisements to improve the user's browsing experience.

[0204] 4) Enterprise information security: In the field of enterprise information security, the present invention can be used to monitor and prevent the spread of malicious URLs in internal networks, and protect the sensitive information and business security of enterprises. For example, the system of the present invention is deployed in the enterprise intranet to monitor and judge the access behavior of internal employees in real time, and prevent the spread and infection of malicious URLs.

[0205] 5) Internet of Things Security: In the field of Internet of Things, the present invention can be used to identify and prevent attacks on the Internet of Things system by malicious devices or malicious websites. For example, by analyzing the access patterns and communication data of Internet of Things devices, potential malicious devices or websites can be discovered and isolated in a timely manner.

[0206] 6) Social media security: In the field of social media, the present invention can be used to identify and filter malicious links or malicious content to protect the privacy and security of users. For example, the shared links on social media can be analyzed to prevent the spread and harm of malicious links.

[0207] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for identifying malicious websites based on PageRank algorithm, characterized by: Malicious URLs can be identified through multi-dimensional feature analysis, PageRank algorithm optimization, and intelligent judgment models. The specific steps include: S1: Collection and organization of data to be studied; S2: Conduct basic research and judgment capability analysis on the sorted data to obtain the research and judgment results; S3: Based on the research results, feature extraction and PageRank algorithm model construction are performed to calculate the PageRank value; S4: Comprehensively collect and analyze the multi-dimensional characteristics of malicious URLs, build a comprehensive research and judgment system, and optimize the PageRank algorithm, including: S41: Content quality assessment: Use natural language processing technology to conduct in-depth analysis of web page content, build a multi-dimensional content scoring system, and calculate content quality scores; S42: User behavior data analysis: collect and analyze multi-dimensional behavior data of users on web pages, build a user behavior score model, and calculate user behavior scores; S43: time decay factor calculation: calculate the time decay score according to the time information of the web page; S5: Build an intelligent analysis model and visualize the analysis and summary results.

2. The method for identifying malicious websites based on PageRank algorithm according to claim 1, characterized in that: In S2, basic research and judgment capability analysis includes domain name feature analysis, registration information query, domain name collection search, blacklist-based filtering methods, feature matching and machine learning-based methods, data mining and deep learning-based methods, and heuristic analysis.

3. The method for identifying malicious websites based on PageRank algorithm according to claim 1, characterized in that: In S3, the specific process of calculating the PageRank value is: The calculation formula of PageRank value PR(A) is as follows: ； in, is the PageRank value of page A, Is a page pointing to page A The PageRank value, It is a page The number of links pointing to other pages, Is a page pointing to page A The PageRank value, It is a page The number of links pointing to other pages, and d is the damping coefficient.

4. The method for identifying malicious websites based on PageRank algorithm according to claim 1, characterized in that: In S41, content quality assessment: using natural language processing technology to conduct in-depth analysis of web page content, building a multi-dimensional content scoring system, and calculating the content quality score in the following specific process: The multi-dimensional content scoring system includes: text content analysis, originality scoring, information richness scoring, language fluency scoring and credibility analysis; The weighted comprehensive ranking index is introduced to evaluate the comprehensive quality of web pages. The specific calculation formula is: Final ranking index = α × PR + (1-α) × content quality score; Specifically: ； Among them, α is the weight coefficient, PR is the PageRank value of the web page, is the PageRank value of page A, and the content quality score is the score calculated based on the NLP analysis results. is the probability of randomly jumping to a page, N is the total number of web pages; d is the damping coefficient; is the sum of the PageRank values ​​passed from all pages pointing to page A, weighted by the quality and quantity of the links; (1-d) is the probability that the user will continue to click on the link instead of randomly jumping; (1+α)×CQ(A) is the direct contribution of the content quality score to the PageRank value, adjusted by the weight coefficient α; InLinks(A) is the sum of the PageRank values ​​passed from all pages pointing to page A; is a page pointing to page A The PageRank value, Is the page The number of links pointing to other pages.

5. The method for identifying malicious websites based on PageRank algorithm according to claim 4, characterized in that: In S42, user behavior data analysis: collecting and analyzing the multi-dimensional behavior data of users on the web page, building a user behavior score model, and calculating the specific process of the user behavior score is: The multi-dimensional behavior data of users on web pages includes: visits, nighttime behavior, association analysis, number of visits, historical tags and user reviews; By setting the weight coefficient, the user behavior score is combined with the PageRank value to generate a ranking index. The calculation formula of the ranking index is as follows: CRS(A)=β×PR(A)+(1-β)×UB(A); Among them, CRS(A) is the final ranking indicator of page A; β is the weight coefficient; PR(A) is the PageRank value of page A, and UB(A) is the user behavior score of page A, which is calculated based on user behavior data, as follows: ； Where n is the number of user behavior indicators; is the weight of the i-th user behavior indicator, is the score of page A on the i-th user behavior indicator.

6. The method for identifying malicious websites based on PageRank algorithm according to claim 5, characterized in that: In S43, the time decay factor is calculated: the specific process of calculating the time decay score according to the time information of the web page is: ； Among them, PR(A) is the PageRank value of page A; is the time decay factor of page A, which is calculated based on the online time and content update status of page A; It is page T i is the inverse of the time decay factor; d is the damping coefficient; InLinks(A) is the sum of the PageRank values ​​passed from all pages pointing to page A; Is a page pointing to page A The PageRank value, It is a page The number of links pointing to other pages; The calculation formula of the time decay factor τ is: τ = e - λ × Δt; Among them, λ is the decay rate, e is the base of the natural logarithm, and Δt is the difference between the page online time and the current time.

7. The method for identifying malicious websites based on PageRank algorithm according to claim 1, characterized in that: In S5, the specific process of building an intelligent analysis model is as follows: Combining natural language processing technology, machine learning and time decay mechanism, comprehensively considering the characteristics of multi-dimensional feature data, selecting classifiers, and building intelligent judgment models; S51: Data preprocessing: Standardize the extracted feature data; clean the collected URL samples to ensure the accuracy of the data set; S52: Model training and validation: Divide the experimental data set into training set and test set; Use K-fold cross validation method to evaluate the model; S53: Comprehensive analysis: Based on the optimized PageRank algorithm, calculate the comprehensive PageRank value of each URL.

Images