A website detection method and device, a storage medium and an electronic device

Abstract

Embodiments of the present specification disclose a website detection method and device, a storage medium and an electronic device. The method comprises: obtaining a plurality of websites to be detected, determining a risk seed webpage according to sample webpage content in the plurality of websites to be detected; determining a webpage path structure of the risk seed webpage, determining a webpage risk matching rule based on the webpage path structure; performing webpage risk detection on the plurality of websites to be detected based on the risk matching rule, and obtaining risk website data. By using the above technical solution, the risk seed webpage is determined through sample webpage content identification, and then the matching rule is generated based on the path structure to perform detection on the full webpage, thereby realizing batch delineation of the risk webpage, breaking through the lag of manual rule update and the limitation of piece-by-piece detection of the stored webpage, and improving the coverage range and processing efficiency of the website risk detection.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a website detection method, apparatus, storage medium, and electronic device. Background Technology

[0002] With the widespread use of search engines in internet information retrieval, the quality and security of search results directly impact user experience and trust in search engines, placing higher demands on the ability to identify and filter malicious content in search results. Attackers exploit website system vulnerabilities to inject illegal and harmful content into legitimate websites. These compromised web pages are easily crawled and indexed by search engines, polluting search results.

[0003] Traditional website risk detection typically relies on manual summarization of URL characteristics and configuration of filtering blacklists. Rule updates struggle to keep pace with evolving attack patterns, resulting in high maintenance costs. Even when some detection solutions employ text classification models to identify webpage content, they are prone to identification biases when facing different types of websites. Furthermore, they can only detect one crawled and stored webpage at a time, failing to automatically generate reusable matching rules for batch targeting of all URLs. This limits the detection efficiency and coverage of search engine content quality assurance systems. Summary of the Invention

[0004] This specification provides a website detection method, apparatus, storage medium, and electronic device, the technical solutions of which are as follows: Firstly, embodiments of this specification provide a website detection method, the method comprising: Multiple websites to be tested are obtained, and risk seed web pages are determined based on the sample web page content from the multiple websites to be tested. Determine the webpage path structure of the risk seed webpage, and determine the webpage risk matching rules based on the webpage path structure; Based on the risk matching rules, webpage risk detection is performed on the multiple websites to be detected to obtain risk website data.

[0005] Secondly, embodiments of this specification provide a website detection device, the device comprising: The acquisition module is used to acquire multiple websites to be detected and determine risk seed web pages based on the sample web page content of the multiple websites to be detected. The determination module is used to determine the webpage path structure of the risk seed webpage and determine the webpage risk matching rules based on the webpage path structure; The detection module is used to perform webpage risk detection on the multiple websites to be detected based on the risk matching rules, and obtain risk website data.

[0006] In one feasible implementation, the risk seed webpage includes a first risk seed webpage and a second risk seed webpage, and the step of determining the risk seed webpage from the sample webpages based on the risk probability includes: A first risk probability threshold is obtained, the number of sample web pages in each website whose risk probability is greater than or equal to the first risk probability threshold is detected, and the multiple websites to be detected are divided into risk websites and abnormal websites according to the number of sample web pages. When the website to be detected is classified as a risky website, a second risk probability threshold is obtained, and a first risk seed webpage is determined from the sample webpages based on the second risk probability threshold. When the website to be detected is classified as an abnormal website, a third risk probability threshold is obtained, and a second risk seed webpage is determined from the sample webpages based on the third risk probability threshold. The number of sample web pages on the risky website is greater than the number of sample web pages on the abnormal website; the second risk probability threshold is greater than the third risk probability threshold, and the third risk probability threshold is greater than or equal to the first risk probability threshold.

[0007] In one feasible implementation, the device is further used for: Obtain the risk quantity threshold and risk ratio threshold for abnormal websites; Determine the risk target webpages hit by the webpage risk matching rules corresponding to the second risk seed webpage and the coverage of the risk target webpages, and determine the purity ratio of high-confidence risk webpages among the risk target webpages based on the risk target webpages; When the coverage quantity is greater than or equal to the risk quantity threshold, and the purity ratio is greater than or equal to the risk ratio threshold, the webpage risk matching rule corresponding to the second risk seed webpage is retained; When the coverage quantity is less than the risk quantity threshold, and / or the purity ratio is less than the risk ratio threshold, the webpage risk matching rule corresponding to the second risk seed webpage is discarded.

[0008] In one feasible implementation, the webpage risk matching rule includes at least one of a first webpage risk matching rule and a second webpage risk matching rule, and the step of determining the webpage risk matching rule based on the webpage path structure includes: Extract first-level path segments from the path structure of the risk seed webpage, determine target prefixes that meet preset coverage conditions based on the first-level path segments, determine the risk matching rules for the first webpage based on the target prefixes, and / or, Based on the path structure of the risk seed webpage, path segment mapping is performed on the risk seed webpage to obtain the path skeleton of the risk seed webpage, and the risk matching rule of the second webpage is determined based on the path skeleton.

[0009] In one feasible implementation, the step of performing path segment mapping processing on the risk seed webpage based on its path structure to obtain the path skeleton of the risk seed webpage includes: Based on the path structure of the risk seed webpage, the path of the risk seed webpage is divided into multiple path segments. Identify the character composition features of each of the path segments; Based on the preset mapping relationship between character composition features and structural tags, the path segment is converted into the corresponding structural tag; The converted structural tags are concatenated according to the path structure to obtain the path skeleton of the risk seed webpage.

[0010] In one feasible implementation, webpage risk detection is performed on the plurality of websites to be detected based on the risk matching rules to obtain risky website data, including: The paths of all web pages of the multiple websites to be tested are matched with the web page risk matching rules using regular expressions to obtain risk candidate web pages, and the number of risk candidate web pages for each website to be tested is counted. Obtain the risk coverage threshold for each website, and determine the risk target webpage based on the risk coverage threshold and the number of risk candidate webpages; Risk website data is generated based on the risk target webpage.

[0011] In one feasible implementation, determining the risk target webpage based on the risk coverage threshold and the number of risk candidate webpages includes: The number of risky candidate web pages in each website is sequentially checked to see if it exceeds the risk coverage threshold. If the number of risk candidate web pages on the current website is greater than or equal to the risk coverage threshold, then the risk candidate web pages will be used as risk target web pages on the current website. If the number of risk candidate web pages on the current website is less than the risk coverage threshold, then the risk seed web page of the current website is determined and used as the risk target web page of the current website.

[0012] Thirdly, this specification provides a computer storage medium storing at least one instruction adapted for loading by a processor and executing method steps of one or more embodiments of this specification.

[0013] Fourthly, embodiments of this specification provide an electronic device that may include: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the above-described method steps.

[0014] The beneficial effects of the technical solutions provided in some embodiments of this specification include at least the following: In one or more embodiments of this specification, multiple websites to be detected are acquired, and risk seed webpages are determined based on the sample webpage content from these websites. The webpage path structure of the risk seed webpages is determined, and webpage risk matching rules are determined based on this structure. Webpage risk detection is then performed on the multiple websites to be detected based on these risk matching rules to obtain risk website data. By employing this technical solution, risk seed webpages are identified through sample webpage content recognition, and then matching rules are generated based on the path structure to perform detection on all webpages. This achieves batch identification of risky webpages, overcoming the limitations of slow manual rule updates and the need for line-by-line detection of stored webpages, thereby improving the coverage and processing efficiency of website risk detection. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating a website detection method provided in the embodiments of this specification; Figure 2 This is a flowchart illustrating another website detection method provided in the embodiments of this specification; Figure 3 This is a schematic diagram illustrating an example of the path skeleton determination method provided in the embodiments of this specification; Figure 4 This is an example diagram illustrating the method for determining risky website data provided in the embodiments of this specification; Figure 5 This is a flowchart illustrating another website detection method provided in the embodiments of this specification; Figure 6 This is a schematic diagram of the structure of a website detection device provided in the embodiments of this specification; Figure 7 This is a schematic diagram of the structure of an electronic device provided in the embodiments of this specification. Detailed Implementation

[0017] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0018] In the description of this specification, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In the description of this specification, it should be noted that, unless otherwise expressly specified and limited, "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. Those skilled in the art can understand the specific meaning of the above terms in this specification based on the specific circumstances. Furthermore, in the description of this specification, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0019] The present specification will now be described in detail with reference to specific embodiments.

[0020] In one embodiment, such as Figure 1 As shown, a website detection method is proposed. This method can be implemented using a computer program and can run on a website detection device based on the von Neumann architecture. The computer program can be integrated into an application or run as a standalone utility application. The website detection device can be a terminal device, including but not limited to: personal computers, tablets, handheld devices, in-vehicle devices, wearable devices, computing devices, or other processing devices connected to a wireless modem. In different networks, the terminal device can be called by different names, such as: user equipment, access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication device, user agent or user equipment, cellular phone, cordless phone, terminal device in 5G networks or future evolved networks, etc.

[0021] Specifically, the website's detection methods include: S101: Obtain multiple websites to be detected, and determine risk seed web pages based on the sample web page content of the multiple websites to be detected; The websites to be tested can be various types of websites that require risk content screening, such as government portals, official websites of higher education institutions, websites of public institutions, and websites of social organizations.

[0022] Sample web pages can be a subset of web pages randomly selected from the full set of web pages indexed by the website to be tested. For example, 500 web pages can be randomly selected from all web pages of a website using a reservoir sampling algorithm. The content of the sample web pages can be the text information contained within the web pages, such as the page title, body paragraphs, and navigation bar text.

[0023] Risk-seeking web pages are web pages that have been initially assessed as potentially containing risky content, such as those containing harmful information.

[0024] For example, a pre-trained risk text classification model can be used to process the content of each sample webpage and output the risk probability corresponding to each sample webpage. The risk text classification model can be fine-tuned based on a pre-trained Chinese language model. After obtaining the risk probability, sample webpages with a risk probability greater than a preset probability threshold are identified as risk seed webpages. Taking a university website as an example, 500 sample webpages are sampled through a reservoir sampling method. After evaluation by the risk text classification model, 120 sample webpages are found to have a risk probability exceeding the preset probability threshold, and these 120 sample webpages are identified as risk seed webpages. By automatically completing the risk assessment and seed selection of sample webpages through the text classification model, the reliance on manual review of each webpage is reduced, providing a reliable initial input for subsequent rule generation.

[0025] S102: Determine the webpage path structure of the risk seed webpage, and determine the webpage risk matching rules based on the webpage path structure; A webpage path structure can be the hierarchical path portion following the domain name in a Uniform Resource Locator (URL), such as a combination of multi-level directories and filenames separated by forward slashes after the domain name in a webpage address.

[0026] Webpage risk matching rules can be regular expressions used to identify risky webpages in batches across the entire set of webpages, such as regular expressions that start with a specific prefix and match any subsequent characters.

[0027] For example, the webpage path structure of a identified risk seed webpage can be analyzed, and webpage risk matching rules can be determined using at least one of prefix extraction and path segment mapping methods. In the prefix extraction method, the first-level path segments in each risk seed webpage path are extracted, the frequency of each first-level path segment in the risk seed webpage is counted, and path segments whose coverage meets a preset condition are determined as target prefixes. A regular expression corresponding to the target prefix is ​​then generated as the webpage risk matching rule. In the path segment mapping method, the path of the risk seed webpage is divided into multiple path segments, the character composition features of each path segment are identified, and the path segments are converted into corresponding structure tags according to the mapping relationship between character composition features and structural tags. The converted structure tags are then concatenated according to their original hierarchical order to obtain a path structure signature, and webpage risk matching rules are generated based on the path structure signature. The mapping relationship between character composition features and structure tags can include: pure numeric strings are mapped to numeric tags, pure letter strings exceeding a preset length are mapped to long letter tags, mixed strings containing numbers and letters are mapped to mixed tags, strings containing special characters are mapped to special character tags, and pure letter strings with a length not exceeding a preset value retain their original values. Taking a university website as an example, one hundred paths in the risk-prone webpages begin with the directory segment "nwzr" and one hundred and twenty paths begin with the directory segment "krug". Regular expressions are generated using prefix extraction and used as the webpage risk matching rules. By automatically summarizing the path structure patterns of risk-prone webpages to generate matching rules, there is no need to manually write and maintain filtering blacklists, reducing the operational costs of rule generation.

[0028] S103: Based on the risk matching rules, perform webpage risk detection on the multiple websites to be detected to obtain risk website data.

[0029] Risk website data may include a risk webpage path structure list obtained by summarizing the webpage path structure of the risk target webpage, a risk site list generated by summarizing the website identifiers to be detected corresponding to the risk target webpage, a risk rule list generated by summarizing the risk matching rules used to match the risk target webpage, and information such as the risk probability corresponding to the risk target webpage.

[0030] For example, the established webpage risk matching rules can be applied to all webpages included in the website to be detected. Regular expression matching is performed on the path portion of the Uniform Resource Locator (URL) of all webpages, and webpages matching any of the established risk matching rules are identified as risk candidate webpages. The number of risk candidate webpages in each website to be detected is counted. When the number of risk candidate webpages is greater than or equal to a preset coverage threshold, the risk candidate webpage is used as the risk target webpage and output as risk website data; when the number of risk candidate webpages is less than the preset coverage threshold, the identified risk seed webpages are directly used as risk target webpages and output as risk website data. By extending the matching rules derived from a small number of risk seed webpages to the entire webpage for batch detection, the limitations of page-by-page content recognition in terms of coverage are overcome, improving the overall efficiency and recall capability of risk webpage detection.

[0031] In the embodiments of this specification, multiple websites to be detected are acquired, and risk seed webpages are determined based on the sample webpage content from these websites. The webpage path structure of the risk seed webpages is determined, and webpage risk matching rules are determined based on this structure. Based on these risk matching rules, webpage risk detection is performed on the multiple websites to be detected, resulting in risk website data. By employing this technical solution, risk seed webpages are identified through sample webpage content recognition, and then matching rules are generated based on the path structure to perform detection on all webpages. This achieves batch identification of risky webpages, overcoming the limitations of slow manual rule updates and the need for line-by-line detection of stored webpages, thus improving the coverage and processing efficiency of website risk detection.

[0032] Please see Figure 2 , Figure 2 This is a flowchart illustrating another embodiment of a website detection method proposed in this specification. Specifically: S201: Obtain multiple websites to be detected, divide sample web pages in the multiple websites to be detected into multiple semantic blocks according to the page content based on the risk text classification model, score each semantic block to obtain the block score of each semantic block, and determine the risk probability of each web page based on the block score. Semantic blocks can be several text segments obtained by dividing the page text content of a sample webpage according to semantic relevance. For example, the main text of a webpage can be divided into multiple text blocks according to natural paragraphs, heading levels, or preset fixed character lengths, and each text block is a semantic block.

[0033] The block score is a numerical value output by the risk text classification model after evaluating a single semantic block. It is used to characterize the probability that the corresponding semantic block contains risky content. For example, if a semantic block is evaluated by the model and outputs a value of 0.97, it indicates that the corresponding semantic block is highly likely to contain risky content.

[0034] For example, multiple sample web pages from websites to be detected can be selected. The text content of each sample web page is divided according to a preset segmentation rule to obtain multiple semantic blocks corresponding to each sample web page. Each semantic block is sequentially input into a pre-trained risk text classification model, which evaluates each semantic block and outputs a block score. After obtaining all block scores for the same sample web page, the risk probability of the corresponding sample web page is determined based on all block scores. The method for determining the risk probability based on block scores can include using the maximum value among all block scores as the risk probability of the corresponding sample web page, or it can include weighted summation of all block scores to obtain the risk probability of the corresponding sample web page. Taking a sample web page from a university website as an example, the page text content is segmented into five semantic blocks according to natural paragraphs. After evaluation by the risk text classification model, block scores of 0.98, 0.95, 0.42, 0.87, and 0.91 are obtained respectively. The maximum value of these five block scores, 0.98, is taken as the risk probability of the corresponding sample web page. By conducting fine-grained semantic block-level assessments of page content, the possibility of interference with overall risk assessment caused by a large amount of irrelevant content mixed in with the webpage text is reduced, and the accuracy of risk probability calculation for a single sample webpage is improved.

[0035] The training process of the risk text classification model is explained below: Model creation: Create an initial risk text classification model for webpage risk detection scenarios based on machine learning models; Sample data acquisition: Acquire a large amount of sample data, which consists of sample web pages containing page content. The page content of the sample web pages has been divided into multiple semantic blocks. Sample data annotation: Based on the needs of webpage risk content identification scenarios, an expert service is introduced to manually annotate the sample data with corresponding sample tags. The sample tags include risk tags for each semantic block of each sample data and risk probability tags for the whole page. Model training process: Input sample data into the initial risk text classification model for at least one round of model training to obtain the predicted risk probability. Based on the predicted risk probability and the sample risk probability label, use the model loss function to determine the model loss value. Based on the model loss value, adjust the model parameters of the initial risk text classification model until the model training termination condition is met to obtain the risk text classification model.

[0036] Optionally, the conditions for ending model training may include, for example, the value of the loss function being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold. Specific conditions for ending model training can be determined based on the actual situation and are not specifically limited here.

[0037] It should be noted that the machine learning models involved in one or more embodiments of this specification include, but are not limited to, fitting of one or more of the following machine learning models: Convolutional Neural Network (CNN), Deep Neural Network (DNN), Recurrent Neural Networks (RNN), Embedding, Gradient Boosting Decision Tree (GBDT), and Logistic Regression (LR).

[0038] S202: Determine risk seed web pages from the sample web pages based on the risk probability.

[0039] For example, the risk probability of each sample webpage can be compared with a preset probability threshold, and sample webpages with a risk probability greater than or equal to the preset probability threshold can be identified as risk seed webpages. The preset probability threshold can be a value pre-set according to the actual detection accuracy requirements. By setting a probability threshold to filter risk probabilities, the confidence level of risk seed webpages is ensured, while the input on which subsequent rule generation depends is controlled within a reasonable scale, reducing the interference of invalid data on subsequent processing.

[0040] In one feasible implementation, the risk seed webpage includes a first risk seed webpage and a second risk seed webpage, and S202 includes the following steps: S11: Obtain a first risk probability threshold, detect the number of sample web pages in each website whose risk probability is greater than or equal to the first risk probability threshold, and classify the plurality of websites to be detected into risky websites and abnormal websites according to the number of sample web pages; the number of sample web pages of the risky websites is greater than the number of sample web pages of the abnormal websites. The first risk probability threshold can be a basic probability value used to initially measure the overall risk level of the website to be tested, such as 0.95.

[0041] A risky website can be a website to be tested that has a large number of web pages with a risk probability greater than or equal to the first risk probability threshold. For example, a website may have 200 web pages with a risk probability greater than or equal to 0.95 out of 500 sample web pages.

[0042] An abnormal website can be a website to be detected that has a small number of web pages with a risk probability greater than or equal to the first risk probability threshold, but still exhibits certain abnormal behavior. For example, a website may have only 15 web pages with a risk probability greater than or equal to 0.95 out of 500 sample web pages.

[0043] The number of sample web pages can be the total number of sample web pages on a website to be tested whose risk probability is greater than or equal to the first risk probability threshold.

[0044] For example, a preset first risk probability threshold can be obtained. Multiple websites to be tested are traversed, and the number of sample web pages with a risk probability greater than or equal to the first risk probability threshold in each website is counted. This number of sample web pages is compared with a preset threshold value. Websites with a number of sample web pages greater than or equal to the threshold value are classified as risk websites, while those with a number of sample web pages less than the threshold value are classified as abnormal websites. Taking a government website and an educational website as examples, the first risk probability threshold is set to 0.95, and the threshold value is set to 50. If 200 out of 500 sample web pages on the government website have a risk probability greater than or equal to 0.95 (200 greater than 50), then the government website is classified as a risk website. Similarly, if 15 out of 500 sample web pages on the educational website have a risk probability greater than or equal to 0.95 (15 less than 50), then the educational website is classified as an abnormal website. By classifying the websites to be tested according to their risk level, a classification basis is provided for subsequent differentiated screening of risk seed web pages, avoiding misjudgments or omissions caused by applying a uniform standard to all websites.

[0045] S12: When the website to be detected is classified as a risky website, a second risk probability threshold is obtained, and a first risk seed webpage is determined from the sample webpages based on the second risk probability threshold; The first risk seed webpage can be a high-confidence risk seed webpage selected from risky websites. The second risk probability threshold can be a relatively strict probability screening value set for the risky website, such as 0.995.

[0046] For example, when a website to be detected is classified as a risky website, a second risk probability threshold is obtained. Sample web pages with a risk probability greater than or equal to the second risk probability threshold are identified as first risk seed web pages. Since risky web pages constitute a relatively high proportion of risky websites, the overall confidence level of the risk probability output by the risk text classification model is high. Therefore, a relatively strict second risk probability threshold is used for screening to ensure the accuracy of the first risk seed web pages. Continuing with the example of a government website, the second risk probability threshold is set to 0.995. Of the 200 sample web pages on an education website with a risk probability greater than or equal to 0.95, 180 sample web pages have a risk probability greater than or equal to 0.995. These 180 sample web pages are identified as first risk seed web pages. By using a strict probability threshold to screen risky websites, the high confidence level of the first risk seed web pages is ensured, providing a reliable input basis for subsequent rule generation.

[0047] S13: When the website to be detected is classified as an abnormal website, a third risk probability threshold is obtained, and a second risk seed webpage is determined from the sample webpages based on the third risk probability threshold; the second risk probability threshold is greater than the third risk probability threshold, and the third risk probability threshold is greater than or equal to the first risk probability threshold.

[0048] The second risk seed webpage can be a risk seed webpage selected from abnormal websites. The third risk probability threshold can be a relatively lenient probability screening value set for abnormal websites, such as 0.98.

[0049] For example, when a website to be detected is classified as an anomalous website, a third risk probability threshold is obtained. Sample web pages with a risk probability greater than or equal to the third risk probability threshold are identified as second risk seed web pages. Since the number of risky web pages on anomalous websites is relatively small, using the same strict threshold as for high-risk websites might result in too few selected risk seed web pages, insufficient to support effective rule generation. Therefore, a more lenient third risk probability threshold is used to broaden the recall scope. Continuing with the example of an educational website, the third risk probability threshold is set to 0.98. Of the 15 sample web pages on the educational website with a risk probability greater than or equal to 0.95, 12 sample web pages have a risk probability greater than or equal to 0.98. These 12 sample web pages are identified as second risk seed web pages. By using a lenient probability threshold for anomalous websites, as many risk seed web pages as possible are obtained despite the limited number of risky web pages, ensuring sufficient input data for subsequent rule generation. The second risk probability threshold is greater than the third risk probability threshold, and the third risk probability threshold is greater than or equal to the first risk probability threshold. The relationship between these thresholds ensures that the screening criteria for risky websites are stricter than those for abnormal websites, while the screening criteria for abnormal websites are not lower than the basic judgment criteria. This approach controls the overall risk of misjudgment while taking into account the recall process.

[0050] In one feasible implementation, the method further includes: obtaining a risk quantity threshold and a risk ratio threshold for abnormal websites; determining the risk target webpages matched by the webpage risk matching rule corresponding to the second risk seed webpage and the coverage quantity of the risk target webpages, and determining the purity ratio of high-confidence risk webpages among the risk target webpages based on the risk target webpages; retaining the webpage risk matching rule corresponding to the second risk seed webpage when the coverage quantity is greater than or equal to the risk quantity threshold and the purity ratio is greater than or equal to the risk ratio threshold; and discarding the webpage risk matching rule corresponding to the second risk seed webpage when the coverage quantity is less than the risk quantity threshold and / or the purity ratio is less than the risk ratio threshold.

[0051] The risk quantity threshold can be the minimum number of web pages required to determine whether the risk matching rule covers a sufficient range of web pages, such as 20. The risk ratio threshold can be the minimum percentage of risky web pages covered by the risk matching rule to determine whether the proportion of risky web pages meets the standard, such as 0.75.

[0052] The target webpage for risk can be the webpage selected after the webpage risk matching rule hits all webpages. The coverage count can be the total number of webpages hit by the webpage risk matching rule across all webpages on the corresponding abnormal website. High-confidence risk webpages can be webpages with a risk probability greater than or equal to a preset confidence threshold, which can be 0.995. The purity ratio can be the proportion of high-confidence risk webpages among the target webpages.

[0053] For example, the risk quantity threshold and risk ratio threshold corresponding to each abnormal website can be obtained. The webpage risk matching rule corresponding to the second risk seed webpage can be applied to all webpages of the corresponding abnormal website. Regular expression matching is performed on the path part of the URL of all webpages. Webpages that match the webpage risk matching rule are identified as risk target webpages, and the coverage of risk target webpages is counted. Among the risk target webpages, high-confidence risk webpages with a risk probability greater than or equal to a preset confidence threshold are selected. The ratio of the number of high-confidence risk webpages to the coverage quantity is calculated, and this ratio is determined as the purity ratio. For each abnormal website, the coverage quantity is compared with the risk quantity threshold, and the purity ratio is compared with the risk ratio threshold. When the coverage quantity is greater than or equal to the risk quantity threshold, and the purity ratio is greater than or equal to the risk ratio threshold, the webpage risk matching rule corresponding to the second risk seed webpage of the current abnormal website is retained. When the coverage quantity is less than the risk quantity threshold, and / or the purity ratio is less than the risk ratio threshold, the webpage risk matching rule corresponding to the second risk seed webpage of the current abnormal website is discarded.

[0054] For example, taking an educational website as an example, after the website is classified as an abnormal website, a webpage risk matching rule is generated based on the second risk seed webpage. This rule is then applied to all webpages on the website for matching. It matches 30 risk target webpages, covering 30 pages, which is greater than the risk quantity threshold of 20. Among these 30 risk target webpages, 25 have a risk probability greater than or equal to 0.995, and the purity ratio (25 divided by 30) is approximately 0.83, greater than the risk ratio threshold of 0.75. Both conditions are met, and the corresponding webpage risk matching rule is retained. Taking another abnormal website as an example, the webpage risk matching rule only matches 12 risk target webpages in the entire website, covering 12 pages, which is less than the risk quantity threshold of 20. Therefore, the corresponding webpage risk matching rule is discarded. Because the abnormal website used a relatively lenient third risk probability threshold when screening the second risk seed webpage, there is a possibility that some normal webpages may be mistakenly identified as risk seed webpages. By performing dual verification on the generated webpage risk matching rules in terms of coverage quantity and purity ratio, rules with insufficient coverage or insufficient proportion of risk webpages are discarded, ensuring the accuracy of the final webpage risk matching rules and reducing the risk of mistakenly targeting normal webpages.

[0055] S203: Determine the webpage path structure of the risk seed webpage, and determine the webpage risk matching rules based on the webpage path structure; For details, please refer to the method steps in other embodiments of this specification, which will not be repeated here.

[0056] In one feasible implementation, the webpage risk matching rule includes at least one of a first webpage risk matching rule and a second webpage risk matching rule. Alternatively, the first webpage risk matching rule and the second webpage risk matching rule can be extracted separately and their union taken as the webpage risk matching rule to complement each other and cover blind spots. S203 includes the following steps: S21: Extract a first-level path segment from the path structure of the risk seed webpage, determine a target prefix that meets a preset coverage condition based on the first-level path segment, determine the risk matching rule for the first webpage based on the target prefix, and / or, The first-level webpage risk matching rule can be a regular expression generated based on prefix extraction to match risky webpages. The first-level path segment can be the first directory segment immediately following the domain name in the webpage path; for example, "abc" in the path ` / abc / def / ghi` is a first-level path segment. The preset coverage condition can be the minimum coverage requirement that the target prefix must meet, for example, covering no fewer than 5 risk seed webpages. The target prefix can be the first-level path segment of the risk seed webpage that meets the preset coverage condition.

[0057] For example, the first-level path segments in the path structure of each risk seed webpage can be extracted, and the frequency of each first-level path segment appearing in all risk seed webpages can be counted. First-level path segments with a frequency greater than or equal to a preset coverage threshold are identified as target prefixes. Redundancy is removed from the target prefixes; when multiple target prefixes have an inclusion relationship, the longer and more specific target prefix is ​​retained first. A first-page risk matching rule is generated based on the target prefix. The regular expression form of the first-page risk matching rule can be something that starts with the target prefix and matches any subsequent character, for example: ^ / {prefix}. Taking an educational website as an example, among the risk seed webpages, 100 paths have the first-level path segment "nwzr" and 120 paths have the first-level path segment "krug". Both 100 and 120 are greater than the preset coverage threshold of 5. Therefore, "nwzr" and "krug" are determined as target prefixes, and the corresponding first webpage risk matching rule "^ / nwzr" is generated. $ and ^ / krug. By extracting frequently occurring first-level path segments from risk seed web pages to generate matching rules, it is possible to quickly cover scenarios where attack content is concentrated in a single top-level directory, and the rule generation process requires no manual intervention.

[0058] S22: Based on the path structure of the risk seed webpage, perform path segment mapping processing on the risk seed webpage to obtain the path skeleton of the risk seed webpage, and determine the risk matching rule of the second webpage based on the path skeleton.

[0059] The second webpage risk matching rule can be a regular expression generated based on path segment mapping to match risky webpages. Path segment mapping processing can be the process of converting each path segment in the path into a corresponding structural tag according to its character composition characteristics. The path skeleton can be the path structure representation obtained by concatenating the various structural tags in the original hierarchical order after path segment mapping processing of the risk seed webpage.

[0060] For example, the path of a risk seed webpage can be divided into multiple path segments according to the forward slashes in the path. The character composition features of each path segment can be identified. Based on a preset mapping relationship between character composition features and structural tags, each path segment can be converted into a corresponding structural tag. The converted structural tags are then concatenated in their original hierarchical order to obtain a path skeleton. A second webpage risk matching rule is then generated based on the path skeleton. For example, the conversion can be performed according to the mapping relationship in the following table:

[0061] Table 1 Mapping Relationship Conversion Table Table 1 illustrates the core transformation logic of URL path skeleton signature, primarily used to automatically extract structural features of hacker attack templates from discovered risky URLs. In actual website operation, normal URL paths typically contain fixed directory names and variable business parameters, such as article numbers or timestamps. When hackers mass-inject malicious web pages, they often use fixed attack templates to generate a large number of URLs. These URLs have the same directory structure, with only randomly generated characters or incrementing numbers in certain path segments. To categorize URLs with the same attack template and generate batch matching rules, it is necessary to generalize the URL paths, eliminating the differences caused by variable parameters while retaining fixed structural features.

[0062] For purely numeric strings, such as "12345," these characters typically correspond to dynamically changing business parameters in a URL, such as article numbers, page identifiers, or timestamps. Mapping these purely numeric strings to numeric tags effectively masks changes in the specific values. For purely alphabetic strings no longer than 10 characters, such as "nfzw" or "krug," these characters are usually fixed directory names or category identifiers preset by website developers. Preserving the original values ​​allows for precise anchoring of specific directory structures used by attackers. For purely alphabetic strings longer than 10 characters, such as "abcdefghijk," these long strings are often meaningless characters or hash checksums randomly generated by hacker tools. Mapping these long purely alphabetic strings to long alphabetic tags can normalize various types of random long text. For mixed strings containing numbers and letters, such as "abc123" or "news2024," these characters are common in mixed parameters with year identifiers or specific business numbers. Mapping these mixed strings to mixed tags can cover such composite variables. For strings containing special characters, such as abc-def_g, these characters usually contain hyphens or underscores and are often used in specific naming conventions or encryption parameters. Mapping special strings to special character tags can accommodate complex character combinations.

[0063] By using path segment mapping, the differences caused by different page content in the path are eliminated, while the hierarchical structure of the path and fixed directory names are preserved. This allows the generated second webpage risk matching rules to cover risky webpages scattered across multiple levels of paths, complementing the first webpage risk matching rules and improving the overall rule coverage.

[0064] In one feasible implementation, S22 includes the following steps: Please see Figure 3 , Figure 3 This is an example diagram illustrating the path skeleton determination method provided in this manual. Specifically: S221: Based on the path structure of the risk seed webpage, divide the path of the risk seed webpage into multiple path segments; Path splitting is the process of dividing the URL of a risk seed webpage, specifically the path portion after the domain name, into multiple independent path segments using forward slashes as separators. The forward slash separator is the slash symbol used to delineate directory levels within the path. For example, the path / nfzw / 12345 / abcdefghijk.html, after splitting using forward slash separators, results in three path segments: nfzw, 12345, and abcdefghijk.html.

[0065] For example, the URL of a risk-seeding webpage can be obtained, the path portion after the domain name can be extracted, and the path portion can be segmented according to the forward slash separator to obtain multiple independent path segments. When the path portion begins with a forward slash, the leading forward slash is removed before segmentation. Taking a risk-seeding webpage on an educational website as an example, the path portion after the domain name in the Uniform Resource Locator is / nfzw / 12345 / abcdefghijk.html. After removing the leading forward slash and segmenting according to the forward slash separator, three path segments are obtained: nfzw, 12345, and abcdefghijk.html. By performing structured segmentation of the path, independent processing units are provided for subsequent segment-by-segment identification of character composition features.

[0066] S222: Identify the character composition features of each of the path segments; For example, it is possible to iterate through all the characters contained in each path segment, determine whether the path segment consists entirely of numbers, whether it consists entirely of letters, whether it contains both numbers and letters, and whether it contains special characters, and determine the character composition characteristics of the path segment based on the determination results. Special characters can be characters other than numbers and letters, such as hyphens, underscores, periods, etc.

[0067] S223: Based on the preset mapping relationship between character composition features and structural markers, the path segment is converted into the corresponding structural marker; For example, the character composition features of each path segment obtained in the aforementioned character composition feature recognition can be acquired. These features are then matched with a preset mapping relationship to determine the corresponding structural marker for each path segment, and the path segment is replaced with the corresponding structural marker. For path segments containing webpage extensions, the main body is converted according to the mapping relationship, while the webpage extension retains its original value. Taking a risk seed webpage on an educational website as an example, the aforementioned path segmentation yields three path segments: nfzw, 12345, and abcdefghijk.html. Path segment nfzw is a pure letter string with a length of 4, less than 10, and retains its original value; path segment 12345 is a pure numeric string and is converted to a numeric marker; the main body of path segment abcdefghijk.html, abcdefghijk, is a pure letter string with a length of 11, greater than 10, and is converted to a long letter marker, retaining the .html extension. The conversion result is the long letter marker .html. By converting each path segment into a structural tag through a preset mapping relationship, the character differences caused by variable content such as article number and random string in the path are eliminated, the hierarchical structure of the path and fixed directory name are preserved, and a structural foundation is provided for the subsequent generation of universal matching rules.

[0068] S224: The converted structural tags are spliced ​​together according to the path structure to obtain the path skeleton of the risk seed webpage.

[0069] For example, the structural tags obtained after the transformation of each path segment in the aforementioned mapping transformation can be obtained. Following the hierarchical order in the original path from front to back, each structural tag is concatenated using a forward slash separator to obtain the path skeleton of the risk seed webpage. Continuing with the previous example, the results of the three path segments after transformation are nfzw, numeric tags, and long letter tags.html. Concatenating them using forward slashes in the original order yields the path skeleton / nfzw / numeric tags / long letter tags.html. Taking another risk seed webpage as an example, with the path / nfzw / 67890 / zyxwvutsrq.html, after the aforementioned path segmentation, character composition feature recognition, and mapping transformation, the path segment nfzw retains its original value, the path segment 67890 is converted to numeric tags, and the main part of the path segment zyxwvutsrq.html, which is a pure letter string longer than 10 characters, is converted to long letter tags while retaining the extension. The resulting path skeleton is also / nfzw / numeric tags / long letter tags.html. By concatenating structural tags into a path skeleton according to the original hierarchical order, risky URLs with different surface characters can be categorized into the same path skeleton as long as they follow the same attack template. This provides a unified and reusable structural input for generating risk matching rules for the second webpage based on the path skeleton.

[0070] S204: Based on the risk matching rules, perform webpage risk detection on the multiple websites to be detected to obtain risk website data.

[0071] For details, please refer to the method steps in other embodiments of this specification, which will not be repeated here.

[0072] In one feasible implementation, S204 includes the following steps: Please see Figure 4 , Figure 4 This is an example diagram illustrating the method for determining risky website data provided in this manual. Specifically: S31: Perform regular expression matching between the paths of all web pages of the multiple websites to be detected and the web page risk matching rules to obtain risk candidate web pages, and count the number of risk candidate web pages for each website to be detected. The full set of web pages refers to all web pages on a specific website to be tested, indexed in the search engine's database. Regular expression matching refers to the operation of matching text strings using regular expressions to determine whether a target string matches a preset character combination pattern. Risk candidate web pages are web pages initially selected from the full set after being matched by the web page risk matching rules. The number of risk candidate web pages represents the total number of web pages on a specific website to be tested that have been matched by the web page risk matching rules.

[0073] For example, multiple websites to be detected can be retrieved, and the path portion of the URLs of these websites can be extracted. This path portion is then matched with the webpage risk matching rules generated in the rule generation process using regular expressions. Webpages matching any one of these rules are identified as risk candidate webpages, and the number of risk candidate webpages for each website is counted. Taking an educational website as an example, the search engine index contains 50,000 webpages. Using risk matching rules generated from risk seed webpages from multiple websites, regular expression matching is performed on the path portions of these 50,000 webpages, resulting in 3,000 matching webpages. These 3,000 webpages are then identified as risk candidate webpages, bringing the total number of risk candidate webpages to 3,000. By applying webpage risk matching rules to all webpages for batch regular expression matching, the limitation of only identifying content-by-content on crawled sample webpages is overcome, extending the detection scope to all webpages in the search engine index.

[0074] S32: Obtain the risk coverage threshold for each website, and determine the risk target webpage based on the risk coverage threshold and the number of risk candidate webpages; The risk coverage threshold can be the minimum number of candidate risk web pages used to determine whether the number of pages is sufficient to support batch output, such as 20 pages.

[0075] For example, the risk coverage threshold for each website to be tested can be obtained. The number of risk candidate web pages for each website can be compared with the risk coverage threshold, and the target web pages can be determined based on the comparison results. When the number of risk candidate web pages is greater than or equal to the risk coverage threshold, it indicates that the web page risk matching rule has sufficient coverage in the corresponding website to be tested, and the risk candidate web pages are determined as target web pages. When the number of risk candidate web pages is less than the risk coverage threshold, it indicates that the web page risk matching rule has limited coverage in the corresponding website to be tested, and the risk seed web pages obtained in the aforementioned seed screening are directly used as target web pages. Continuing with the example of an educational website, the risk coverage threshold is set to 20, and the number of risk candidate web pages for the educational website is 3000. Since 3000 is greater than 20, the 3000 risk candidate web pages are determined as target web pages. Taking another website to be tested as an example, the web page risk matching rule only hits 8 risk candidate web pages in the entire set of web pages. Since 8 is less than the risk coverage threshold of 20, the risk seed web pages obtained in the aforementioned seed screening are directly used as target web pages. By introducing a risk coverage threshold to determine the number of risk candidate web pages, batch matching results are output when the coverage is sufficient to expand the detection coverage, and the risk seed web pages are reverted to ensure the accuracy of the output results when the coverage is insufficient, thus balancing the recall capability and accuracy of the detection.

[0076] In one feasible implementation, S32 includes the following steps: S321: Sequentially detect whether the number of risk candidate web pages in each website is greater than the risk coverage threshold; For example, the number of risk candidate web pages for each website to be detected, obtained from the aforementioned full-page regular expression matching, and the aforementioned risk coverage threshold can be acquired. Following the traversal order of the websites to be detected, the number of risk candidate web pages is compared with the risk coverage threshold one by one to determine whether the number of risk candidate web pages is greater than or equal to the risk coverage threshold. Taking an educational website and a public institution website as examples, with a risk coverage threshold set to 20, the number of risk candidate web pages for the educational website is 3000, and the number of risk candidate web pages for the technical website is 8. The 3000 are compared with 20, and the 8 are compared with 20, yielding the respective comparison results. By comparing the number of risk candidate web pages for each website to be detected with the risk coverage threshold one by one, a basis for determining differentiated risk target web pages for different websites is provided.

[0077] S322: If the number of risk candidate web pages on the current website is greater than or equal to the risk coverage threshold, then the risk candidate web pages are used as risk target web pages on the current website. For example, when the number of risk candidate pages on the current website is greater than or equal to the risk coverage threshold, it indicates that the aforementioned webpage risk matching rules have sufficient coverage across all webpages on the current website, and the risk candidate pages are directly used as the risk target pages for the current website. For instance, if an educational website has 3000 risk candidate pages and a risk coverage threshold of 20, and 3000 is greater than or equal to 20, then these 3000 risk candidate pages are identified as the risk target pages for that educational website. By directly outputting batch matching results when the coverage is sufficient, the detection capability is expanded from a small number of risk seed pages to the entire webpage level, improving the detection coverage of risky webpages.

[0078] S323: If the number of risk candidate web pages on the current website is less than the risk coverage threshold, then the risk seed web page of the current website is determined, and the risk seed web page is used as the risk target web page of the current website.

[0079] For example, when the number of risk candidate web pages on the current website is less than the risk coverage threshold, it indicates that the aforementioned web page risk matching rules have limited coverage across all web pages on the current website. In this case, the risk seed web pages obtained from the aforementioned seed screening are directly used as the risk target web pages for the current website. For instance, if an academic website has 8 risk candidate web pages and a risk coverage threshold of 20, since 8 is less than 20, the risk seed web pages corresponding to a public institution website obtained from the aforementioned seed screening are retrieved and directly used as the risk target web pages for that public institution website. By falling back to the risk seed web pages when the coverage is insufficient, low-quality output caused by limited coverage of matching rules is avoided, ensuring the accuracy of the risk target web pages.

[0080] S33: Generate risk website data based on the risk target webpage.

[0081] For example, identified risky target web pages from various websites to be tested can be retrieved, and their URLs, domain names, and corresponding risk matching rules can be linked and recorded to form risky website data. For instance, if 200 risky target web pages are identified in website A and 50 in website B, their URLs, domain names, and matching risk rules are linked and recorded to generate risky website data containing 250 risky web page records. By generating risky website data based on these risky target web pages, the risky web page information scattered across various websites to be tested is unified and integrated, facilitating batch processing of risky web pages by search engines and improving the efficiency of search result content quality assurance.

[0082] Please see Figure 5 , Figure 5This is a flowchart illustrating another embodiment of a website detection method proposed in this specification. Specifically: URLs of official websites such as government and educational websites are used as the initial processing objects. A reservoir sampling method is used to randomly sample URLs from each site, with a maximum of 500 URLs sampled per site. The sampled URLs are input into a hacked text classification model based on a robust optimized bidirectional encoder representation pre-training method (corresponding to Roberta in the diagram) for individual scoring, outputting the probability of each URL being hacked. Based on the scoring results, a site-level judgment is performed, classifying sites into strongly hacked sites (the aforementioned risky websites) and weakly abnormal sites (the aforementioned abnormal websites), while normal sites are skipped. For sites determined to be hacked, a prefix method (the aforementioned corresponding first webpage risk matching rule) and a skeleton signature method (the aforementioned corresponding...) are used. The second method combines webpage risk matching rules to extract a two-layer pattern (Pattern in the diagram, i.e., matching rules). The extracted matching rules are then extended to match all URLs of the site, covering all site URLs, to achieve batch identification from a small number of seed URLs to all hacked URLs of the site. Finally, three types of detection results are output in parallel: a list of hacked URLs for deletion (sending risky website data to relevant parties for deletion processing) and anti-crawling (prohibiting crawler access and inclusion), a list of hacked sites for site seed library accumulation, and a list of hacked patterns for anti-crawling processing, thus forming a complete closed loop from URL sampling, model scoring, site judgment, rule extraction, full-scale extended matching to result output.

[0083] In the embodiments of this specification, multiple websites to be detected are acquired, and risk seed webpages are determined based on the sample webpage content from these websites. The webpage path structure of the risk seed webpages is determined, and webpage risk matching rules are determined based on this structure. Based on these risk matching rules, webpage risk detection is performed on the multiple websites to be detected, resulting in risk website data. By employing this technical solution, risk seed webpages are identified through sample webpage content recognition, and then matching rules are generated based on the path structure to perform detection on all webpages. This achieves batch identification of risky webpages, overcoming the limitations of slow manual rule updates and the need for line-by-line detection of stored webpages, thus improving the coverage and processing efficiency of website risk detection.

[0084] The following will combine Figure 6 This specification provides a detailed description of the website detection device provided in the embodiments. It should be noted that... Figure 6 The website detection device shown is used to execute this instruction manual. Figures 1-5 The methods shown in the embodiments are illustrated for ease of explanation, showing only the parts related to the embodiments of this specification. For specific technical details not disclosed, please refer to this specification. Figures 1-5 The example shown.

[0085] Please see Figure 6 This diagram illustrates the structure of a website detection device according to an embodiment of this specification. The website detection device 1 can be implemented as all or part of a user terminal through software, hardware, or a combination of both. According to some embodiments, the website detection device 1 includes an acquisition module 11, a determination module 12, and a detection module 13, specifically used for: The acquisition module 11 is used to acquire multiple websites to be detected and determine risk seed web pages based on the sample web page content of the multiple websites to be detected. The determining module 12 is used to determine the webpage path structure of the risk seed webpage and determine the webpage risk matching rules based on the webpage path structure; The detection module 13 is used to perform webpage risk detection on the multiple websites to be detected based on the risk matching rules, and obtain risk website data.

[0086] In one feasible implementation, determining the risk seed webpage based on the sample webpage content from the plurality of websites to be detected includes: Based on the risk text classification model, the sample web pages in the multiple websites to be detected are divided into multiple semantic blocks according to the page content. Each semantic block is scored to obtain a block score, and the risk probability of each web page is determined based on the block score. Risk seed web pages are determined from the sample web pages based on the risk probability.

[0087] In one feasible implementation, the risk seed webpage includes a first risk seed webpage and a second risk seed webpage, and the step of determining the risk seed webpage from the sample webpages based on the risk probability includes: A first risk probability threshold is obtained, the number of sample web pages in each website whose risk probability is greater than or equal to the first risk probability threshold is detected, and the multiple websites to be detected are divided into risk websites and abnormal websites according to the number of sample web pages. When the website to be detected is classified as a risky website, a second risk probability threshold is obtained, and a first risk seed webpage is determined from the sample webpages based on the second risk probability threshold. When the website to be detected is classified as an abnormal website, a third risk probability threshold is obtained, and a second risk seed webpage is determined from the sample webpages based on the third risk probability threshold. The number of sample web pages on the risky website is greater than the number of sample web pages on the abnormal website; the second risk probability threshold is greater than the third risk probability threshold, and the third risk probability threshold is greater than or equal to the first risk probability threshold.

[0088] In one feasible implementation, the device is further used for: Obtain the risk quantity threshold and risk ratio threshold for abnormal websites; Determine the risk target webpages hit by the webpage risk matching rules corresponding to the second risk seed webpage and the coverage of the risk target webpages, and determine the purity ratio of high-confidence risk webpages among the risk target webpages based on the risk target webpages; When the coverage quantity is greater than or equal to the risk quantity threshold, and the purity ratio is greater than or equal to the risk ratio threshold, the webpage risk matching rule corresponding to the second risk seed webpage is retained; When the coverage quantity is less than the risk quantity threshold, and / or the purity ratio is less than the risk ratio threshold, the webpage risk matching rule corresponding to the second risk seed webpage is discarded.

[0089] In one feasible implementation, the webpage risk matching rule includes at least one of a first webpage risk matching rule and a second webpage risk matching rule, and the step of determining the webpage risk matching rule based on the webpage path structure includes: Extract first-level path segments from the path structure of the risk seed webpage, determine target prefixes that meet preset coverage conditions based on the first-level path segments, determine the risk matching rules for the first webpage based on the target prefixes, and / or, Based on the path structure of the risk seed webpage, path segment mapping is performed on the risk seed webpage to obtain the path skeleton of the risk seed webpage, and the risk matching rule of the second webpage is determined based on the path skeleton.

[0090] In one feasible implementation, the step of performing path segment mapping processing on the risk seed webpage based on its path structure to obtain the path skeleton of the risk seed webpage includes: Based on the path structure of the risk seed webpage, the path of the risk seed webpage is divided into multiple path segments. Identify the character composition features of each of the path segments; Based on the preset mapping relationship between character composition features and structural tags, the path segment is converted into the corresponding structural tag; The converted structural tags are concatenated according to the path structure to obtain the path skeleton of the risk seed webpage.

[0091] In one feasible implementation, webpage risk detection is performed on the plurality of websites to be detected based on the risk matching rules to obtain risky website data, including: The paths of all web pages of the multiple websites to be tested are matched with the web page risk matching rules using regular expressions to obtain risk candidate web pages, and the number of risk candidate web pages for each website to be tested is counted. Obtain the risk coverage threshold for each website, and determine the risk target webpage based on the risk coverage threshold and the number of risk candidate webpages; Risk website data is generated based on the risk target webpage.

[0092] In one feasible implementation, determining the risk target webpage based on the risk coverage threshold and the number of risk candidate webpages includes: The number of risky candidate web pages in each website is sequentially checked to see if it exceeds the risk coverage threshold. If the number of risk candidate web pages on the current website is greater than or equal to the risk coverage threshold, then the risk candidate web pages will be used as risk target web pages on the current website. If the number of risk candidate web pages on the current website is less than the risk coverage threshold, then the risk seed web page of the current website is determined and used as the risk target web page of the current website.

[0093] It should be noted that the website detection device provided in the above embodiments is only illustrated by the division of the above functional modules when executing the website detection method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the website detection device and the website detection method embodiments provided in the above embodiments belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.

[0094] The example numbers in this specification are for descriptive purposes only and do not represent the superiority or inferiority of the examples.

[0095] In the embodiments of this specification, multiple websites to be detected are acquired, and risk seed webpages are determined based on the sample webpage content from these websites. The webpage path structure of the risk seed webpages is determined, and webpage risk matching rules are determined based on this structure. Based on these risk matching rules, webpage risk detection is performed on the multiple websites to be detected, resulting in risk website data. By employing this technical solution, risk seed webpages are identified through sample webpage content recognition, and then matching rules are generated based on the path structure to perform detection on all webpages. This achieves batch identification of risky webpages, overcoming the limitations of slow manual rule updates and the need for line-by-line detection of stored webpages, thus improving the coverage and processing efficiency of website risk detection.

[0096] This specification also provides a computer storage medium that can store multiple instructions adapted to be loaded and executed by a processor as described above. Figures 1-5 The website detection method described in the illustrated embodiment can be found in the following document for a detailed execution process: Figures 1-5The specific details of the illustrated embodiments will not be elaborated here.

[0097] This specification also provides a computer program product that stores at least one instruction, said at least one instruction being loaded and executed by the processor as described above. Figures 1-5 The website detection method described in the illustrated embodiment can be found in the following document for a detailed execution process: Figures 1-5 The specific details of the illustrated embodiments will not be elaborated here.

[0098] Please refer to Figure 7 This is a structural block diagram of an electronic device provided in an embodiment of this specification. The electronic device in this specification may include one or more of the following components: a processor 1010, a memory 1020, an input device 1030, an output device 1040, and a bus 1050. The processor 1010, memory 1020, input device 1030, and output device 1040 may be connected to each other via the bus 1050.

[0099] Processor 1010 may include one or more processing cores. Processor 1010 connects to various parts of the electronic device using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 1020, and by calling data stored in memory 1020. Optionally, processor 1010 may be implemented using at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). Processor 1010 may integrate one or more of a central processing unit (CPU), graphics processing unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 1010 and may be implemented separately through a communication chip.

[0100] The memory 1020 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 1020 may include non-transitory computer-readable storage medium. The memory 1020 may be used to store instructions, programs, code, code sets, or instruction sets.

[0101] The input device 1030 is used to receive input instructions or data, and includes, but is not limited to, a keyboard, mouse, camera, microphone, or touch device. The output device 1040 is used to output instructions or data, and includes, but is not limited to, a display device and a speaker. In this embodiment, the input device 1030 can be a temperature sensor for acquiring the operating temperature of the electronic device. The output device 1040 can be a speaker for outputting audio signals.

[0102] In addition, those skilled in the art will understand that the structure of the electronic device shown in the above figures does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the electronic device may also include radio frequency circuits, input units, sensors, audio circuits, wireless fidelity (WIFI) modules, power supplies, Bluetooth modules, etc., which will not be described in detail here.

[0103] In the embodiments of this specification, the executing entity for each step can be the electronic device described above. Optionally, the executing entity for each step can be the operating system of the electronic device. The operating system can be Android, iOS, or other operating systems; this specification does not limit this.

[0104] exist Figure 7 In the electronic device, the processor 1010 can be used to call a program stored in the memory 1020 and execute it to implement the website detection method as described in the various method embodiments of this specification.

[0105] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory, or random access memory, etc.

[0106] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in the embodiments of this specification are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the object characteristics, interactive behavior characteristics, and user information involved in this specification were all obtained under full authorization.

[0107] The above-disclosed embodiments are merely preferred embodiments of this specification and should not be construed as limiting the scope of this specification. Therefore, any equivalent variations made in accordance with the claims of this specification shall still fall within the scope of this specification.

Claims

1. A website detection method, characterized in that, The method includes: Multiple websites to be tested are obtained, and risk seed web pages are determined based on the sample web page content from the multiple websites to be tested. Determine the webpage path structure of the risk seed webpage, and determine the webpage risk matching rules based on the webpage path structure; Based on the risk matching rules, webpage risk detection is performed on the multiple websites to be detected to obtain risk website data.

2. The method according to claim 1, characterized in that, The step of determining risk seed webpages based on sample webpage content from the plurality of websites to be detected includes: Based on the risk text classification model, the sample web pages in the multiple websites to be detected are divided into multiple semantic blocks according to the page content. Each semantic block is scored to obtain a block score, and the risk probability of each web page is determined based on the block score. Risk seed web pages are determined from the sample web pages based on the risk probability.

3. The method according to claim 2, characterized in that, The risk seed webpage includes a first risk seed webpage and a second risk seed webpage. The step of determining the risk seed webpage from the sample webpages based on the risk probability includes: A first risk probability threshold is obtained, the number of sample web pages in each website whose risk probability is greater than or equal to the first risk probability threshold is detected, and the multiple websites to be detected are divided into risk websites and abnormal websites according to the number of sample web pages. When the website to be detected is classified as a risky website, a second risk probability threshold is obtained, and a first risk seed webpage is determined from the sample webpages based on the second risk probability threshold. When the website to be detected is classified as an abnormal website, a third risk probability threshold is obtained, and a second risk seed webpage is determined from the sample webpages based on the third risk probability threshold. The number of sample web pages on the risky website is greater than the number of sample web pages on the abnormal website; the second risk probability threshold is greater than the third risk probability threshold, and the third risk probability threshold is greater than or equal to the first risk probability threshold.

4. The method according to claim 3, characterized in that, The method further includes: Obtain the risk quantity threshold and risk ratio threshold for abnormal websites; Determine the risk target webpages hit by the webpage risk matching rules corresponding to the second risk seed webpage and the coverage of the risk target webpages, and determine the purity ratio of high-confidence risk webpages among the risk target webpages based on the risk target webpages; When the coverage quantity is greater than or equal to the risk quantity threshold, and the purity ratio is greater than or equal to the risk ratio threshold, the webpage risk matching rule corresponding to the second risk seed webpage is retained; When the coverage quantity is less than the risk quantity threshold, and / or the purity ratio is less than the risk ratio threshold, the webpage risk matching rule corresponding to the second risk seed webpage is discarded.

5. The method according to claim 1, characterized in that, The webpage risk matching rules include at least one of a first webpage risk matching rule and a second webpage risk matching rule, and the step of determining the webpage risk matching rules based on the webpage path structure includes: Extract first-level path segments from the path structure of the risk seed webpage, determine target prefixes that meet preset coverage conditions based on the first-level path segments, determine the risk matching rules for the first webpage based on the target prefixes, and / or, Based on the path structure of the risk seed webpage, path segment mapping is performed on the risk seed webpage to obtain the path skeleton of the risk seed webpage, and the risk matching rule of the second webpage is determined based on the path skeleton.

6. The method according to claim 5, characterized in that, The step of mapping the path segments of the risk seed webpage based on its path structure to obtain its path skeleton includes: Based on the path structure of the risk seed webpage, the path of the risk seed webpage is divided into multiple path segments. Identify the character composition features of each of the path segments; Based on the preset mapping relationship between character composition features and structural tags, the path segment is converted into the corresponding structural tag; The converted structural tags are concatenated according to the path structure to obtain the path skeleton of the risk seed webpage.

7. The method according to claim 1, characterized in that, Based on the risk matching rules, webpage risk detection is performed on the multiple websites to be detected to obtain risky website data, including: The paths of all web pages of the multiple websites to be tested are matched with the web page risk matching rules using regular expressions to obtain risk candidate web pages, and the number of risk candidate web pages for each website to be tested is counted. Obtain the risk coverage threshold for each website, and determine the risk target webpage based on the risk coverage threshold and the number of risk candidate webpages; Risk website data is generated based on the risk target webpage.

8. The method according to claim 7, characterized in that, The step of determining the risk target webpage based on the risk coverage threshold and the number of risk candidate webpages includes: The number of risky candidate web pages in each website is sequentially checked to see if it exceeds the risk coverage threshold. If the number of risk candidate web pages on the current website is greater than or equal to the risk coverage threshold, then the risk candidate web pages will be used as risk target web pages on the current website. If the number of risk candidate web pages on the current website is less than the risk coverage threshold, then the risk seed web page of the current website is determined and used as the risk target web page of the current website.

9. A website detection device, characterized in that, The device includes: The acquisition module is used to acquire multiple websites to be detected and determine risk seed web pages based on the sample web page content of the multiple websites to be detected. The determination module is used to determine the webpage path structure of the risk seed webpage and determine the webpage risk matching rules based on the webpage path structure; The detection module is used to perform webpage risk detection on the multiple websites to be detected based on the risk matching rules, and obtain risk website data.

10. The apparatus according to claim 9, characterized in that, The step of determining risk seed webpages based on sample webpage content from the plurality of websites to be detected includes: Based on the risk text classification model, the sample web pages in the multiple websites to be detected are divided into multiple semantic blocks according to the page content. Each semantic block is scored to obtain a block score, and the risk probability of each web page is determined based on the block score. Risk seed web pages are determined from the sample web pages based on the risk probability.

11. The apparatus according to claim 10, characterized in that, The risk seed webpage includes a first risk seed webpage and a second risk seed webpage. The step of determining the risk seed webpage from the sample webpages based on the risk probability includes: A first risk probability threshold is obtained, the number of sample web pages in each website whose risk probability is greater than or equal to the first risk probability threshold is detected, and the multiple websites to be detected are divided into risk websites and abnormal websites according to the number of sample web pages. When the website to be detected is classified as a risky website, a second risk probability threshold is obtained, and a first risk seed webpage is determined from the sample webpages based on the second risk probability threshold. When the website to be detected is classified as an abnormal website, a third risk probability threshold is obtained, and a second risk seed webpage is determined from the sample webpages based on the third risk probability threshold. The number of sample web pages on the risky website is greater than the number of sample web pages on the abnormal website; the second risk probability threshold is greater than the third risk probability threshold, and the third risk probability threshold is greater than or equal to the first risk probability threshold.

12. The apparatus according to claim 11, characterized in that, The device is also used for: Obtain the risk quantity threshold and risk ratio threshold for abnormal websites; Determine the risk target webpages hit by the webpage risk matching rules corresponding to the second risk seed webpage and the coverage of the risk target webpages, and determine the purity ratio of high-confidence risk webpages among the risk target webpages based on the risk target webpages; When the coverage quantity is greater than or equal to the risk quantity threshold, and the purity ratio is greater than or equal to the risk ratio threshold, the webpage risk matching rule corresponding to the second risk seed webpage is retained; When the coverage quantity is less than the risk quantity threshold, and / or the purity ratio is less than the risk ratio threshold, the webpage risk matching rule corresponding to the second risk seed webpage is discarded.

13. The apparatus according to claim 9, characterized in that, The webpage risk matching rules include at least one of a first webpage risk matching rule and a second webpage risk matching rule, and the step of determining the webpage risk matching rules based on the webpage path structure includes: Extract first-level path segments from the path structure of the risk seed webpage, determine target prefixes that meet preset coverage conditions based on the first-level path segments, determine the risk matching rules for the first webpage based on the target prefixes, and / or, Based on the path structure of the risk seed webpage, path segment mapping is performed on the risk seed webpage to obtain the path skeleton of the risk seed webpage, and the risk matching rule of the second webpage is determined based on the path skeleton.

14. The apparatus according to claim 13, characterized in that, The step of mapping the path segments of the risk seed webpage based on its path structure to obtain its path skeleton includes: Based on the path structure of the risk seed webpage, the path of the risk seed webpage is divided into multiple path segments. Identify the character composition features of each of the path segments; Based on the preset mapping relationship between character composition features and structural tags, the path segment is converted into the corresponding structural tag; The converted structural tags are concatenated according to the path structure to obtain the path skeleton of the risk seed webpage.

15. The apparatus according to claim 9, characterized in that, Based on the risk matching rules, webpage risk detection is performed on the multiple websites to be detected to obtain risky website data, including: The paths of all web pages of the multiple websites to be tested are matched with the web page risk matching rules using regular expressions to obtain risk candidate web pages, and the number of risk candidate web pages for each website to be tested is counted. Obtain the risk coverage threshold for each website, and determine the risk target webpage based on the risk coverage threshold and the number of risk candidate webpages; Risk website data is generated based on the risk target webpage.

16. The apparatus according to claim 15, characterized in that, The step of determining the risk target webpage based on the risk coverage threshold and the number of risk candidate webpages includes: The number of risky candidate web pages in each website is sequentially checked to see if it exceeds the risk coverage threshold. If the number of risk candidate web pages on the current website is greater than or equal to the risk coverage threshold, then the risk candidate web pages will be used as risk target web pages on the current website. If the number of risk candidate web pages on the current website is less than the risk coverage threshold, then the risk seed web page of the current website is determined and used as the risk target web page of the current website.

17. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions adapted for loading by a processor and executing the steps of the method as described in any one of claims 1 to 8.

18. An electronic device, characterized in that, include: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the steps of the method as claimed in any one of claims 1 to 8.

Images