Website processing method, apparatus, device, and medium

Abstract

Embodiments of the present application provide a website processing method, device, equipment and medium, relate to the technical field of Internet, the method is applied to electronic equipment, and low-quality root domain name data is maintained in the electronic equipment. The method comprises the following steps: obtaining a uniform resource locator of a webpage to be crawled based on a dark web search engine crawler, determining whether the root domain name of the uniform resource locator matches the low-quality root domain name data, if yes, rejecting to crawl the webpage to be crawled, and if not, processing the webpage to be crawled, thereby realizing the early identification of low-quality webpages, reducing the invalid collection of low-quality webpages, and reducing resource waste.

Description

Technical Field

[0001] This invention relates to the field of Internet technology, and more specifically, to a website processing method, apparatus, device, and medium. Background Technology

[0002] With the development of the internet, various websites are widely used in people's lives and work, and page ranking is a crucial function for these websites. Currently, there are many methods for page ranking. For example, PageRank, also known as Google's left-side ranking or PR, is an algorithm used by Google to rank web pages in its search engine results. Essentially, PageRank is an algorithm that roughly analyzes the importance of web pages based primarily on the number and quality of hyperlinks between them. Its basic assumption is that more important pages tend to be referenced more by other pages (or that other pages contain more hyperlinks to those pages). It interprets a link from page A to page B as "page A voting for page B," and determines the ranking of the voted page based on the source of the vote (or even the source of the source, i.e., the page linking to page A) and the ranking of the voted page. Simply put, a high-ranking page can boost other low-ranking pages. Google's search engine uses it to analyze the relevance and importance of web pages, and it is frequently used in search engine optimization (SEO) as one of the factors for evaluating the effectiveness of web page optimization. However, this method suffers from the ineffective collection of a large number of low-quality web pages, resulting in a waste of server resources. Summary of the Invention

[0003] One of the objectives of this invention includes, for example, providing a website processing method, apparatus, device, and medium to at least partially reduce invalid collection of low-quality web pages and mitigate resource waste.

[0004] The embodiments of the present invention can be implemented as follows:

[0005] In a first aspect, the present invention provides a website processing method applied to an electronic device, wherein the electronic device maintains low-quality root domain name data, the method comprising:

[0006] The Uniform Resource Locator (URL) of the webpage to be crawled is obtained based on the dark web search engine crawler;

[0007] Determine whether the root domain name of the Uniform Resource Locator matches the low-quality root domain name data;

[0008] If so, then refuse to crawl the webpage to be crawled;

[0009] If not, then process the webpage to be crawled.

[0010] In an optional implementation, the electronic device also maintains low-quality keyword check data and weight data for each domain name, and the processing of the webpage to be crawled includes:

[0011] Determine whether the Uniform Resource Locator (URL) of the webpage to be crawled, or whether the request headers and content match the low-quality keyword check data;

[0012] If so, the root domain of the Uniform Resource Locator is updated to the low-quality root domain data, and the weight of the domain of the webpage to be crawled is reduced.

[0013] If not, then crawl the website content of the webpage to be crawled.

[0014] In an optional implementation, after crawling the website content of the webpage to be crawled, the method further includes:

[0015] Determine whether the website content contains external dark web links;

[0016] If so, for each external dark web link, determine whether it is a target external dark web link whose root domain of the Uniform Resource Locator matches the low-quality root domain data;

[0017] Count the number of target external dark web links in the website content;

[0018] The weight of the domain name of the webpage to be crawled is adjusted based on the number of external dark web links to the target.

[0019] In an optional implementation, adjusting the weight of the domain name of the webpage to be crawled based on the number of external dark web links includes:

[0020] Determine whether the number of the target external dark web links exceeds a set threshold;

[0021] If so, reduce the weight of the domain name of the webpage to be crawled.

[0022] In an optional implementation, after crawling the website content of the webpage to be crawled, the method further includes:

[0023] Determine whether the website content contains external dark web links;

[0024] If so, determine the weighted average of the weights for all the aforementioned external dark web links;

[0025] The weight of the domain name of the webpage to be crawled is adjusted based on the weighted average.

[0026] In an optional implementation, adjusting the weight of the domain name of the webpage to be crawled based on the weighted average includes:

[0027] Determine whether the weighted average is less than a set weight threshold;

[0028] If so, reduce the weight of the domain name of the webpage to be crawled.

[0029] In an optional implementation, the method further includes:

[0030] Determine whether the weight data of each domain name is less than the low quality threshold;

[0031] If so, update the root domain of the domain whose weight data is less than the low-quality threshold to the low-quality root domain data.

[0032] In a second aspect, embodiments of the present invention provide a website processing device applied to an electronic device, wherein the electronic device maintains low-quality root domain name data, and the website processing device includes:

[0033] The information acquisition module is used to obtain the Uniform Resource Locators (URLs) of the web pages to be crawled based on the dark web search engine crawler;

[0034] The information processing module is used to determine whether the root domain of the Uniform Resource Locator matches the low-quality root domain data; if so, the crawling of the webpage to be crawled is rejected; if not, the webpage to be crawled is processed.

[0035] Thirdly, the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the website processing method described in any of the foregoing embodiments.

[0036] Fourthly, the present invention provides a computer-readable storage medium comprising a computer program, wherein the computer program, when executed, controls the electronic device containing the computer-readable storage medium to perform the website processing method described in any of the foregoing embodiments.

[0037] The beneficial effects of the embodiments of the present invention include, for example, maintaining low-quality root domain data, filtering low-quality web pages before web page processing, rejecting the crawling of low-quality web pages that match the low-quality root domain data, thereby avoiding invalid collection of low-quality websites and greatly reducing the waste of resources in the process of processing dark web websites. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 The diagram illustrates an application scenario provided by an embodiment of the present invention.

[0040] Figure 2 This is a schematic flowchart of a website processing method provided by an embodiment of the present invention.

[0041] Figure 3 The second schematic diagram illustrates a website processing method provided by an embodiment of the present invention.

[0042] Figure 4 The third illustration shows a flowchart of a website processing method provided by an embodiment of the present invention.

[0043] Figure 5 The fourth illustration shows a flowchart of a website processing method provided by an embodiment of the present invention.

[0044] Figure 6 The fifth illustration shows a flowchart of a website processing method provided by an embodiment of the present invention.

[0045] Figure 7 The sixth illustration shows a flowchart of a website processing method provided by an embodiment of the present invention.

[0046] Figure 8 An exemplary structural block diagram of a website processing apparatus provided in an embodiment of the present invention is shown.

[0047] Icons: 100 - Electronic device; 110 - Memory; 120 - Processor; 130 - Communication module; 140 - Website processing device; 141 - Information acquisition module; 142 - Information processing module. Detailed Implementation

[0048] Today, search engine ranking rules refer to the methods by which search engines determine the position of a webpage in search results based on a series of algorithms and standards. Here are some common search engine ranking rules and factors:

[0049] Keyword relevance: Search engines evaluate the relevance of web pages to user search queries. Factors such as the frequency, placement, and density of relevant keywords on a web page can affect ranking.

[0050] Webpage quality and content: Search engines evaluate the quality and relevance of webpages, including the originality, usefulness, information richness, and user experience factors such as page loading speed and mobile friendliness.

[0051] External links and citations: Search engines evaluate a webpage's external links and citations. Links from high-quality and relevant websites can improve a webpage's ranking.

[0052] User behavior signals: Search engines evaluate user behavior signals such as click-through rate, dwell time, and bounce rate. If a user clicks and interacts frequently on a webpage, the search engine may consider that webpage to be a relevant and valuable answer to the user's query.

[0053] Website authority and credibility: Search engines assess a website's authority and credibility. This includes the website's domain age, domain history, and anti-phishing and anti-fraud signals.

[0054] Social media signals: Search engines may assess the impact of social media on web pages. For example, sharing, likes, and comments on social media can affect a web page's ranking.

[0055] Search engine ranking rules are complex and frequently updated and adjusted. Different search engines may have different ranking algorithms and factor weights. Search engine operators typically keep their ranking rules confidential to prevent abuse and manipulation. Therefore, specific ranking rules may vary and change over time and with technological advancements.

[0056] Research has revealed that because registering .onion domains on the dark web is cost-free, many low-value websites increase their PageRank by registering multiple dark web domains and subdomains. This aims to improve their ranking on dark web search engines (such as Ahmia and Fresh Onions), most of which use the PageRank algorithm. Studies show that over 90% of dark web websites engage in this excessive search engine optimization (SEO). These websites have massive amounts of page data, often containing tens of thousands of low-value domains per page, leading to a significant waste of server resources when crawling them.

[0057] The principle of SEO is to improve a website's organic ranking within relevant search engines by utilizing their rules.

[0058] Based on the above research, it is clear that the processing methods for dark web sites need to be optimized. For example, a quality assessment index system for Tor dark web sites could be constructed, determining the assessment indicators and their corresponding weights. Based on each assessment value and its corresponding weight, the importance of Tor dark web sites could be determined, thereby assessing the quality of the dark web sites. However, assessing site quality by collecting content from existing sites is extremely wasteful of server resources, especially given that over 90% of dark web sites are excessively SEO-driven. Furthermore, the rich data structures of these websites mean that current technologies, such as aesthetic evaluation, usability evaluation, multimedia support evaluation, and content richness evaluation, cannot effectively and accurately determine their quality.

[0059] Therefore, this invention provides a dark web website processing solution that identifies low-value dark web websites, enabling reliable processing of dark web websites and providing conditions for subsequent reliable evaluation of dark web websites.

[0060] In this embodiment, the dark web can only be accessed by setting software, setting authorization, or setting computer configurations.

[0061] The shortcomings of the above solutions are the result of the inventors' practical experience and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the embodiments of the present invention in the following text should be considered as contributions made by the inventors during the invention process.

[0062] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0063] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0064] It should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0065] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0066] It should be noted that, where there is no conflict, the features in the embodiments of the present invention can be combined with each other.

[0067] Please refer to Figure 1 This is a block diagram of an electronic device 100 provided in this embodiment. The electronic device 100 in this embodiment can be a web server, processing device, processing platform, etc., capable of data interaction and processing. The electronic device 100 includes a memory 110, a processor 120, and a communication module 130. The memory 110, processor 120, and communication module 130 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0068] The memory 110 is used to store programs or data. The memory 110 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0069] The processor 120 is used to read / write data or programs stored in the memory 110 and to perform corresponding functions.

[0070] The communication module 130 is used to establish a communication connection between the electronic device 100 and other communication terminals through the network, and to send and receive data through the network.

[0071] It should be understood that, Figure 1 The structure shown is only a schematic diagram of the electronic device 100. The electronic device 100 may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown. Figure 1 The components shown can be implemented using hardware, software, or a combination thereof.

[0072] Please refer to the following: Figure 2 This is a flowchart illustrating a website processing method provided in an embodiment of the present invention, which can be derived from... Figure 1 The electronic device 100 performs the operation, for example, by a processor 120 within the electronic device 100, which maintains low-quality root domain name data. The website processing method includes steps S110, S120, S130, and S140.

[0073] S110, obtain the Uniform Resource Locator (URL) of the webpage to be crawled based on the dark web search engine crawler.

[0074] S120, determine whether the root domain name of the Uniform Resource Locator matches the low-quality root domain name data. If yes, proceed to S130; if no, proceed to S140.

[0075] S130, refuse to crawl the webpage to be crawled.

[0076] S140, Process the webpage to be crawled.

[0077] Low-quality root domain data can be predefined or obtained through big data collection and analysis. For example, users can define some root domains as low-quality root domain data, or low-quality dark web websites can be identified by periodically or in real-time collecting and analyzing big data from various dark web websites, and their root domains can be included in the low-quality root domain data.

[0078] Low-quality root domain data can exist in various forms, such as lists, databases, datasets, and text. For example, low-quality root domain data can be a list of low-quality root domains, such as `root_domain_list`.

[0079] In one implementation, low-quality root domain data can be a Redis collection object, such as a deduplicated list of root domains. Accordingly, the `redis sismember` command can be used to determine if a root domain exists in the Redis collection object, and the `smember` function can be used to view the specific root domains contained within the Redis collection object.

[0080] Low-quality web pages are filtered based on low-quality root domain data, and low-quality web pages that match the low-quality root domain data are rejected from crawling, thereby avoiding invalid collection of low-quality websites and reducing the waste of resources in the process of processing dark web websites.

[0081] Please refer to the following: Figure 3 To further improve the comprehensiveness of low-quality webpage screening and to facilitate convenient and reliable ranking of each webpage, the electronic device can also maintain low-quality keyword check data and weight data for each domain name. Accordingly, the processing of the webpage to be crawled in S140 may include S141, S142, and S143.

[0082] S141, determine whether the Uniform Resource Locator (URL) of the webpage to be crawled, or the request headers and content, matches the low-quality keyword check data. If yes, proceed to S142; if no, proceed to S143.

[0083] S142, update the root domain of the Uniform Resource Locator to the low-quality root domain data, and reduce the weight of the domain of the webpage to be crawled.

[0084] S143, crawl the website content of the webpage to be crawled.

[0085] The low-quality keyword check data can take various forms, such as lists, databases, datasets, and text. For example, low-quality keyword check data can be a low-quality keyword check rule (keyword_rule).

[0086] For example, low-quality keyword checking data can include, but is not limited to, regular expressions such as ^https?\:\ / \ / .*(sex|girl|gay|porn|incest|fucking|child).*[\w\-\.]+\.onion, etc.

[0087] Low-quality keyword check data can be predefined or obtained through big data collection and analysis. For example, users can define certain characters as low-quality keyword check data, or low-quality dark web websites can be identified by periodically or in real-time collecting and analyzing big data from various dark web websites, and characters with high overlap from low-quality dark web websites can be included in the low-quality keyword check data.

[0088] The weight data for each domain can correspond to the ranking of the dark web pages associated with that domain. For example, the weight data can include the weight value corresponding to each domain address, such as 0 to 100. By dynamically adjusting the weight value of each domain, the ranking of the dark web pages associated with each domain can be obtained based on the weight value.

[0089] The weight data can take various forms, such as lists, databases, datasets, and text. For example, the weight data can be a domain weight table (domain_weight, including domain addresses and weights from 0 to 100).

[0090] To further improve the rationality of the domain name weight data, please refer to the following: Figure 4 In one implementation, after crawling the website content of the webpage to be crawled in S143, it may also include S144, S145, S146 and S147.

[0091] S144, Determine whether there are external dark web links in the website content. If so, proceed to S145 to S147.

[0092] S145, for each of the external dark web links, determine whether it is a target external dark web link whose root domain of the Uniform Resource Locator matches the low-quality root domain data.

[0093] S146, Count the number of target external dark web links in the website content.

[0094] S147, Adjust the weight of the domain name of the webpage to be crawled based on the number of external dark web links to the target.

[0095] In this method, a predefined threshold can be set to determine whether the number of external dark web links to the target exceeds the threshold. If so, the weight of the domain name of the webpage to be crawled is reduced, thereby achieving a reasonable adjustment of the weight data.

[0096] For web pages containing external dark web links, the number of target external dark web links matching the root domain of the web page with low-quality root domain data is determined to adjust the domain weight of the web page to be crawled. This links the weight of each web page to be crawled with the number of low-quality web page links it includes. For web pages containing a large number of target external dark web links (e.g., exceeding a set threshold), the domain weight is reduced to further improve the rationality of the weight data and avoid the proliferation of low-quality web pages.

[0097] Understandably, for web pages with a small number of external dark web links, such as less than a certain threshold, their domain authority can be increased.

[0098] Please refer to the following: Figure 5 In another implementation, after crawling the website content of the webpage to be crawled in S143, it may also include S148, S149 and S150.

[0099] S148, Determine whether there are external dark web links in the website content. If so, proceed to S149 and S150.

[0100] S149, For all the aforementioned external dark web links, determine the weighted average of their weights.

[0101] S150, Adjust the weight of the domain name of the webpage to be crawled based on the weighted average.

[0102] In this process, a low-quality threshold can be predefined to determine whether the weighted average is less than the set weight threshold. If so, the weight of the domain name of the webpage to be crawled is reduced.

[0103] For web pages with external dark web links, the weighted average of the weights of all external dark web links is determined to adjust the weight of the domain of the web page to be crawled. This links the weight of each web page to be crawled with the proportion of low-quality web page links included. For web pages with a large proportion of low-quality target external dark web links (weighted average less than the set weight threshold), the weight of their domains is reduced to further improve the rationality of the weight data.

[0104] Understandably, for web pages with a low proportion of low-quality target dark web links (e.g., less than a certain threshold), their domain weight can be increased.

[0105] Based on the above approach, a more accurate and reasonable weight ranking can be obtained by analyzing whether the webpage itself is of low quality and whether the external dark web links it includes are of low quality. Please refer to [the relevant documentation / references]. Figure 6 Website processing methods may also include S160 and S170.

[0106] S160: Determine if the weight data of each domain name is less than the low-quality threshold. If so, proceed to S170.

[0107] S170, update the root domains of domains whose weight data is less than the low-quality threshold to the low-quality root domain data.

[0108] By dynamically updating low-quality root domain data, the accuracy and timeliness of identifying low-quality web pages are improved, thereby effectively avoiding invalid data collection from low-quality websites and greatly reducing resource waste.

[0109] To more clearly explain the implementation principle of this embodiment, please refer to [the relevant documentation / reference]. Figure 7 Taking the low-quality root domain data as the low-quality root domain list root_domain_list, the low-quality keyword check data as the low-quality keyword check rules, and the weight data as the domain weight table domain_weight as an example, the overall implementation process is illustrated.

[0110] Website handling methods include:

[0111] Electronic devices obtain the URL of the webpage to be crawled based on a dark web search engine crawler, and check whether the root domain of the URL exists in the low-quality root domain list root_domain_list.

[0112] If the URL exists, the website content of the webpage to be crawled will not be crawled; otherwise, the URL will be checked to see if it matches the low-quality keyword check rules.

[0113] The low-quality keyword detection rule is a regular expression matching rule, such as ^https?\:\ / \ / .*(sex|girl|gay|porn|incest|fucking|child).*[\w\-\.]+\.onion, which matches URLs containing keywords such as sex and girl. This matching rule is dynamically updated based on the actual content.

[0114] If a match is found, the root domain of the URL is updated to the list of low-quality root domains, and the domain's weight is reduced; otherwise, the website content of the URL is crawled.

[0115] The number of external dark web links from the crawled website content in the low-quality root domain list (root_domain_list) is calculated and denoted as LowCount. If LowCount is greater than the threshold A (e.g., more than 10 low-quality root domains), the domain weight is reduced; otherwise, the process ends normally.

[0116] The weighted average of external dark web links in the crawled website content is calculated in the domain_weight table and denoted as AvgWeight. If AvgWeight is less than the weight threshold B, such as less than 50, the domain weight is reduced; otherwise, the process ends normally.

[0117] Check if the request headers and content returned by the website match the low-quality keyword check rule keyword_rule. If they match, update the list of low-quality root domains; otherwise, terminate normally.

[0118] The keyword_rule matching analysis is performed based on the request header and content. Compared with the URL-based matching, there are some new matching rules. For example, if the response header matches the ^"\w{8}- regular expression rule and the content contains the porn keyword, then this check rule will be matched.

[0119] Regularly check the domain weight table for domains that are below the low-quality threshold C and add them to the list of low-quality root domains.

[0120] To perform the corresponding steps in the above embodiments and various possible methods, an implementation of a website processing apparatus is given below. Please refer to... Figure 8 , Figure 8 This is a functional block diagram of a website processing device 140 provided in an embodiment of the present invention. The website processing device 140 can be applied to... Figure 1 The electronic device 100 shown maintains low-quality root domain name data. It should be noted that the website processing device 140 provided in this embodiment has the same basic principle and technical effects as the method embodiments described above. For the sake of brevity, any parts not mentioned in this embodiment can be referred to the corresponding content in the above embodiments. The website processing device 140 includes an information acquisition module 141 and an information processing module 142.

[0121] Among them, the information acquisition module 141 is used to obtain the Uniform Resource Locator (URL) of the webpage to be crawled based on the dark web search engine crawler.

[0122] The information processing module 142 is used to determine whether the root domain of the Uniform Resource Locator matches the low-quality root domain data; if yes, then the crawling of the webpage to be crawled is rejected; if no, then the webpage to be crawled is processed.

[0123] Based on the above, embodiments of the present invention also provide a computer-readable storage medium, the computer-readable storage medium including a computer program, wherein the computer program, when running, controls the electronic device where the computer-readable storage medium is located to execute the above-described website processing method.

[0124] By employing the above-described scheme in this embodiment of the invention, low-quality websites are effectively filtered before content crawling by maintaining low-quality root domain data and low-quality keyword check data, thus avoiding invalid collection of low-quality websites and greatly reducing server resource waste. Furthermore, by calculating the number of external dark web links in the website content and the weighted average of these links, the website domain weight is scored, achieving a ranking of dark web site quality. This further prevents the proliferation of low-quality web pages and improves the rationality of the weight ranking.

[0125] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0126] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0127] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0128] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A website processing method, characterized in that, Applied to electronic devices, the electronic devices maintain low-quality root domain name data, low-quality keyword check data, and weight data of each domain name. The method includes: The Uniform Resource Locator (URL) of the webpage to be crawled is obtained based on the dark web search engine crawler; Determine whether the root domain name of the Uniform Resource Locator matches the low-quality root domain name data; If so, then refuse to crawl the webpage to be crawled; If not, then the webpage to be crawled will be processed, including: Determine whether the Uniform Resource Locator (URL) of the webpage to be crawled, or whether the request headers and content match the low-quality keyword check data; If so, the root domain of the Uniform Resource Locator is updated to the low-quality root domain data, and the weight of the domain of the webpage to be crawled is reduced. If not, then crawl the website content of the webpage to be crawled; Determine whether the website content contains external dark web links; If so, for each external dark web link, determine whether it is a target external dark web link whose root domain of the Uniform Resource Locator matches the low-quality root domain data; Count the number of target external dark web links in the website content; Adjust the weight of the domain name of the webpage to be crawled based on the number of external dark web links to the target; or, Determine whether the website content contains external dark web links; If so, determine the weighted average of the weights for all the aforementioned external dark web links; The weight of the domain name of the webpage to be crawled is adjusted based on the weighted average.

2. The website processing method according to claim 1, characterized in that, The step of adjusting the domain name weight of the webpage to be crawled based on the number of external dark web links includes: Determine whether the number of the target external dark web links exceeds a set threshold; If so, reduce the weight of the domain name of the webpage to be crawled.

3. The website processing method according to claim 2, characterized in that, The adjustment of the weight of the domain name of the webpage to be crawled based on the weighted average includes: Determine whether the weighted average is less than a set weight threshold; If so, reduce the weight of the domain name of the webpage to be crawled.

4. The website processing method according to any one of claims 1 to 3, characterized in that, The method further includes: Determine whether the weight data of each domain name is less than the low quality threshold; If so, update the root domain of the domain whose weight data is less than the low-quality threshold to the low-quality root domain data.

5. A website processing device, characterized in that, An electronic device is used to maintain low-quality root domain name data, low-quality keyword check data, and weight data for each domain name. The website processing device includes: The information acquisition module is used to obtain the Uniform Resource Locators (URLs) of the web pages to be crawled based on the dark web search engine crawler; An information processing module is used to determine whether the root domain of the Uniform Resource Locator (URL) matches the low-quality root domain data; if yes, then the crawling of the webpage to be crawled is rejected; if no, the webpage to be crawled is processed, including: determining whether the URL of the webpage to be crawled, or the request header and content, matches the low-quality keyword check data; if yes, then the root domain of the URL is updated to the low-quality root domain data, and the weight of the domain of the webpage to be crawled is reduced; if no, then the website content of the webpage to be crawled is crawled; the website content is determined. Does the website contain external dark web links? If so, for each external dark web link, determine whether it is a target external dark web link whose root domain matches the low-quality root domain data; count the number of target external dark web links in the website content; adjust the weight of the domain of the webpage to be crawled based on the number of target external dark web links; or, determine whether the website contains external dark web links; if so, for all external dark web links, determine the weighted average of their weights; adjust the weight of the domain of the webpage to be crawled based on the weighted average.

6. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the website processing method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a computer program that, when executed, controls the electronic device containing the computer-readable storage medium to perform the website processing method according to any one of claims 1 to 4.

Images