Abnormal link processing method, device, computer equipment and storage medium

Abstract

The embodiments of the present application disclose an abnormal link processing method, apparatus, computer equipment and storage medium; the embodiments of the present application can obtain a search link to be detected; perform content search based on the search link to obtain a web page structure corresponding to each search link; parse the web page structure to obtain descriptive information of the search link in at least one content dimension; for the descriptive information of each content dimension, use a corresponding abnormal link detection strategy to perform abnormality detection on the search link to obtain an abnormality detection result for each content dimension; and block the search link based on the abnormality detection result, thereby improving the accuracy of processing the abnormal link.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, computer device and storage medium for processing abnormal links. Background Art

[0002] Abnormal links may include links with abnormal conditions. For example, the abnormal link may include a link where the server address has been changed, resulting in an inability to find the corresponding address. For another example, an abnormal link may include a link that cannot correctly display the corresponding web page content. When a user uses a search engine to search for content, if the search results contain too many abnormal links, it will affect the search results and thus affect the user's experience of using the search engine. Through practice with the prior art, the inventors of this application found that the existing methods for processing abnormal links have the problem of low accuracy. Summary of the Invention

[0003] The embodiments of the present application provide a method, apparatus, computer device, and storage medium for processing abnormal links, which can improve the accuracy of processing abnormal links.

[0004] The present invention provides a method for handling abnormal links, including:

[0005] Get the search link to be detected;

[0006] Performing content search based on the search link to obtain a web page structure corresponding to each search link;

[0007] Parsing the webpage structure to obtain description information of the search link in at least one content dimension;

[0008] For the description information of each content dimension, a corresponding abnormal link detection strategy is used to perform abnormality detection on the search link to obtain an abnormality detection result for each content dimension;

[0009] The search link is blocked based on the abnormality detection result.

[0010] Accordingly, an embodiment of the present application further provides an abnormal link processing device, comprising:

[0011] An acquisition unit, used for acquiring a search link to be detected;

[0012] A content search unit, configured to perform content search based on the search links and obtain a web page structure corresponding to each search link;

[0013] a parsing unit, configured to parse the webpage structure to obtain description information of the search link in at least one content dimension;

[0014] An anomaly detection unit is used to perform an anomaly detection on the search link using a corresponding anomaly link detection strategy based on the description information of each content dimension, and obtain an anomaly detection result for each content dimension;

[0015] A blocking unit is used to block the search link based on the abnormality detection result.

[0016] In one embodiment, the anomaly detection unit includes:

[0017] A filtering content detection subunit, configured to perform filtering content detection on the original web page content;

[0018] A first parsing subunit is configured to parse the original web page content to obtain the main content of the original web page content when the original web page content does not include the preset filtering content;

[0019] The content detection subunit is used to perform content detection on the main content to obtain the abnormality detection result.

[0020] In one embodiment, the content detection subunit includes:

[0021] A text detection module, configured to perform text detection on the main content;

[0022] The abnormal keyword detection module is used to perform abnormal keyword detection on the text content when it is detected that the main content includes text content, and obtain the abnormal detection result.

[0023] In one embodiment, the abnormal keyword detection module includes:

[0024] A word segmentation submodule, configured to perform word segmentation processing on the text content to obtain at least one text subword;

[0025] A keyword matching submodule is used to match the text subwords with preset abnormal keywords to obtain keyword matching results;

[0026] The result generation submodule is used to generate the anomaly detection result based on the keyword matching result.

[0027] In one embodiment, the result generation submodule is used to:

[0028] When the text subword does not match the preset abnormal keyword, semantic extraction is performed on the text content to obtain semantic features of the text content;

[0029] Performing forward coding and backward coding on the semantic features respectively to obtain forward coding information corresponding to the forward coding and backward coding information corresponding to the backward coding;

[0030] Fusing the forward coded information and the backward coded information to obtain fused coded information;

[0031] The abnormality probability of the search link is calculated based on the fused encoded information to obtain the abnormality detection result.

[0032] In one embodiment, the abnormal keyword detection module further includes:

[0033] Keyword acquisition submodule, used to obtain initial abnormal keywords;

[0034] An expansion submodule, configured to expand the initial abnormal keywords to obtain expanded abnormal keywords;

[0035] An abnormal link search submodule is used to search for abnormal links based on the expanded abnormal keywords to obtain abnormal link search results;

[0036] A screening submodule is configured to screen out the preset abnormal keywords from the expanded abnormal keywords based on the abnormal link search results.

[0037] In one embodiment, the content detection subunit further includes:

[0038] An image detection module, configured to perform image detection on the main content when it is detected that the main content does not include text content;

[0039] a character recognition module, configured to perform character recognition on the image content when detecting that the main content includes image content;

[0040] The abnormal character detection module is used to perform abnormal character detection on the character information when character information of the image content is recognized, and obtain the abnormal character detection result.

[0041] In one embodiment, the content detection subunit further includes:

[0042] A clustering processing module, configured to perform clustering processing on the original web page content to obtain a target cluster corresponding to the original web page content when it is detected that the main content does not include image content;

[0043] A first calculation module, configured to calculate the similarity between objects in the target cluster;

[0044] A judgment module is used to judge whether the search link is an abnormal link based on the similarity.

[0045] In one embodiment, the anomaly detection unit includes:

[0046] A second parsing subunit is configured to parse the link content to obtain domain names of the link content at different levels;

[0047] a clustering processing subunit, configured to perform clustering processing on the link content based on the domain names of the link content at different levels, to obtain a target domain name cluster corresponding to the link content;

[0048] The similarity determination subunit is configured to perform similarity determination on the target domain name cluster and generate the anomaly detection result based on the determination result.

[0049] In one embodiment, the cluster processing subunit includes:

[0050] a second calculation module, configured to calculate a distance between the domain name and a plurality of preset domain name clusters;

[0051] A determination module is configured to determine a target domain name cluster for the domain name clustering from the plurality of preset domain name clusters based on the distance.

[0052] In one embodiment, the anomaly detection unit includes:

[0053] A status code matching subunit, configured to match the link status code with a preset abnormal status code to obtain a status code matching result;

[0054] A mapping subunit is used to map the matching result to the corresponding anomaly detection result.

[0055] The present application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method provided in various optional embodiments of the above-mentioned aspect.

[0056] Correspondingly, an embodiment of the present application further provides a storage medium, which stores instructions, and when the instructions are executed by a processor, implements any abnormal link processing method provided in the embodiments of the present application.

[0057] The embodiments of the present application can obtain a search link to be detected; perform a content search based on the search link to obtain a web page structure corresponding to each search link; parse the web page structure to obtain descriptive information of the search link in at least one content dimension; use a corresponding abnormal link detection strategy to perform abnormality detection on the search link for the descriptive information of each content dimension to obtain an abnormality detection result for each content dimension; and block the search link based on the abnormality detection result, thereby improving the accuracy of processing abnormal links. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0059] Figure 1 This is a scenario diagram of the abnormal link processing method provided in an embodiment of the present application;

[0060] Figure 2 Schematic diagram of the process of handling abnormal links provided by the embodiment of the present application;

[0061] Figure 3 This is a schematic diagram of a web page structure provided by an embodiment of the present application;

[0062] Figure 4 is a schematic diagram of a scene of image content provided by an embodiment of the present application;

[0063] Figure 5 This is another flowchart of the abnormal link processing method provided in an embodiment of the present application;

[0064] Figure 6 This is another scenario diagram of the abnormal link processing method provided in an embodiment of the present application;

[0065] Figure 7 Schematic diagram of the structure of the abnormal link processing device provided in an embodiment of the present application;

[0066] Figure 8 It is a structural diagram of the computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. However, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0068] The embodiments of the present application provide a method for handling abnormal links. The abnormal link handling method can be performed by an abnormal link handling device, which can be integrated into a computer device. The computer device can include at least one of a terminal and a server. That is, the abnormal link handling method provided in the embodiments of the present application can be performed by a terminal, a server, or a terminal and a server that can communicate with each other.

[0069] The terminals may be smartphones, tablets, laptops, personal computers (PCs), smart homes, wearable electronic devices, VR / AR devices, in-vehicle computers, etc. The servers may be intercommunication servers or backend servers between multiple heterogeneous systems, independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and big data and artificial intelligence platforms.

[0070] In one embodiment, if Figure 1 The abnormal link processing device can be integrated into a computer device such as a terminal or server to implement the abnormal link processing method proposed in the embodiment of the present application. Specifically, the computer device can obtain a search link to be detected; perform a content search based on the search link to obtain a web page structure corresponding to each search link; parse the web page structure to obtain descriptive information of the search link in at least one content dimension; for the descriptive information of each content dimension, use a corresponding abnormal link detection strategy to perform an abnormality detection on the search link to obtain an abnormality detection result for each content dimension; and block the search link based on the abnormality detection result.

[0071] The following are detailed descriptions of each embodiment. It should be noted that the description order of the following embodiments does not limit the preferred order of the embodiments.

[0072] The embodiments of the present application will be described from the perspective of an abnormal link processing device, which can be integrated into a computer device, which can be a server, a terminal or other device.

[0073] like Figure 2 A method for handling abnormal links is provided, and the specific process includes:

[0074] 101. Obtain the search link to be detected.

[0075] With the development of information technology and the internet, people around the world are no longer isolated and separated, but are now united through information technology and the internet. For example, the internet allows people to search for information from all over the world. However, when users search for information on their computers, if the returned links contain too many abnormal links, the user experience will be affected.

[0076] In one embodiment, the abnormal link processing method proposed in the embodiment of the present application can be embedded in a search engine. When a user searches for content through the search engine, the abnormal link processing method proposed in the embodiment of the present application can be executed.

[0077] For example, when a user searches for information through a computer device, the computer device may receive a search trigger instruction and obtain at least one search link based on the search trigger instruction.

[0078] The search link may include a connection relationship from a web page to a target, for example, the search link may be a hyperlink or the like.

[0079] In one embodiment, the search link has various presentation forms. For example, the search link can be a text or an image, etc. When the user clicks on the linked text or image, the current page will jump to the target page connected to the search link.

[0080] In one embodiment, at least one search link obtained by the search may contain an abnormal link. An abnormal link includes a link with an abnormal condition. For example, the abnormal link may include a link where the server address has been changed, resulting in an inability to find the corresponding address. For another example, the abnormal link may include a link that cannot correctly display the corresponding webpage content, etc.

[0081] For example, abnormal links may include broken links. A broken link refers to a link where the server address has changed and the current address location cannot be found. Broken links may include protocol broken links and content broken links. Protocol links may include broken links that are clearly indicated by the TCP protocol status / HTTP protocol status of the page, such as the common 404, 403, and 503 statuses. Content broken links may include pages where the server returns a normal status but the content has been changed to information pages that do not exist, have been deleted, require permission, or are unrelated to the original content.

[0082] When at least one of the search links obtained contains an abnormal link, if the user clicks on the abnormal link but the corresponding content is not displayed, the user's content search experience will be affected. For example, a user uses a search website to search for information, and the search results display several search links. Among these search links, if an abnormal link is included, the user's experience of using the search website will be affected. For another example, a user uses the search function of a social software to search for information, and the search results display several search links. Among these search links, if an abnormal link is included, the user's experience of using the search function of the social software will be affected.

[0083] In search applications, some of the search results users receive are invalid or expired. Identifying and cleaning these invalid or expired results requires the platform to be able to identify the content of web links. Identifying broken web links is a crucial foundation for the platform's survival. The ability to identify and filter broken web link content directly impacts the entire platform ecosystem and user experience. The platform needs to handle various anomalies and special cases when detecting broken web links, such as when crawled results differ from the actual page, when the target web page is all images without text, or when the target website has reversed crawling.

[0084] Therefore, the abnormal link processing method proposed in the embodiment of the present application can effectively detect abnormal links in search links, thereby improving the accuracy of processing abnormal links and improving the user's search experience.

[0085] 102. Perform content search based on the search link to obtain the webpage structure corresponding to the search link.

[0086] In one embodiment, after the search link is obtained, a content search may be performed based on the search link to obtain a webpage structure corresponding to the webpage content of each search content.

[0087] The content search may include a process of obtaining a web page structure of a search link.

[0088] Wherein, the webpage structure may include information describing the content of the webpage and how the content is distributed. Through the webpage structure, the abnormal link processing device can know what the webpage content corresponding to the search link is and the layout of these webpage contents in the webpage.

[0089] In one embodiment, the webpage structure can be represented in a variety of ways. For example, the webpage structure can be represented using Hypertext Markup Language (HTML). Another example is the Extensible Hypertext Markup Language (XHTML), and so on.

[0090] For example, when the web page structure is expressed in HTML, it can be expressed as follows Figure 3 shown. Figure 3 The 001 in the figure may be a schematic example of the network structure corresponding to the search link.

[0091] In one embodiment, the webpages mentioned in the embodiments of the present application may include various webpages, such as webpages on computers, webpages on mobile phones, webpages on tablets, webpages on various IoT devices, and so on.

[0092] In one embodiment, there are multiple ways to perform content search based on search links to obtain the webpage structure corresponding to each search link.

[0093] For example, a crawler may be used to crawl the web page structure corresponding to the search link and the web page content.

[0094] For another example, the web page structure corresponding to the web page content of the search link can be obtained by using the developer mode of the web page.

[0095] In one embodiment, a large number of exposure logs and click logs may be obtained, and then after crawling the webpage structure content via a crawler platform, external features of the webpage structure content may be counted, wherein the external features may represent descriptive information of the webpage structure content.

[0096] 103. Analyze the webpage structure to obtain description information of the search link in at least one content dimension.

[0097] In one embodiment, after obtaining the webpage structure corresponding to the search link, the webpage structure may be parsed to obtain the content corresponding to the search link.

[0098] For example, as shown in the figure, the information in the network structure can be the content corresponding to the search link. When the user clicks the search link, the information in the network structure will be displayed in the form of web page content. Figure 3 As shown, when the web page structure is as Figure 3 When 001 is shown in , the corresponding information can be as follows Figure 3 As shown in Figure 002.

[0099] In one embodiment, the information in the network structure may include various contents. Therefore, the information in the network structure may be divided into description information in different content dimensions.

[0100] The description information may include information describing whether the search link is likely to be an abnormal link. For example, the description information may include original web page content, link content, link status code, and the like.

[0101] The link status code may include information that can describe the search link status. For example, the link status code may include an HTTP (HTTP Status Code) status code, etc. For example, the link status code may include a 200 status code, a 202 status code, a 404 status code, a 403 status code, or a 503 status code, etc.

[0102] Among them, the 200 status code can indicate that the webpage request was successful and the expected response headers or data body will be returned with the response. The 202 status code can indicate that the server has received the webpage request but has not yet processed it. The 404 status code can indicate that the webpage or file corresponding to the search link cannot be found. The 403 status code can indicate that the webpage resource is unavailable. The 503 status code can indicate that there are no relevant results for the webpage.

[0103] In one embodiment, a search link is essentially the address of a webpage, and thus the webpage content can be obtained through the search link. The link content may include the address representation of the search link. For example, the link content may be a website address, etc. For example, the link content of the search link may be "https: / / www.XXX.com". For another example, the link content of the search link may be "https: / / www.XXX.com / item / xx.459253", etc.

[0104] In one embodiment, the original web page content may include content other than link content and link status code in the web page structure. For example, the original web page content may include text content, image content, video content, animation content, table content, etc. in the web page.

[0105] In one embodiment, there are multiple ways to parse the web page structure to obtain description information of the search link in at least one content dimension.

[0106] For example, description information of different content dimensions may have corresponding identifiers. Therefore, description information of the search link in at least one content dimension may be obtained by identifying the identifiers carried by the web page structure.

[0107] For another example, when the web page structure is represented by HTML, tags in the HTML can be identified to obtain descriptive information of the search link in at least one content dimension.

[0108] In one embodiment, parsing the webpage structure may obtain description information of the search link in at least one content dimension, or may obtain description information of the search link in multiple content dimensions.

[0109] For example, when parsing a webpage structure, the description information obtained may only include the link content. For another example, when parsing a webpage structure, the description information obtained may only include the link content and the link status code. For another example, when parsing a webpage structure, the description information obtained may only include the link content and the original webpage content. For another example, when parsing a webpage structure, the description information obtained may include the link content, the original webpage content, the link status code, and so on.

[0110] 104. For the description information of each content dimension, use the corresponding abnormal link detection strategy to perform abnormal detection on the search link to obtain the abnormal detection result of each content dimension.

[0111] In one embodiment, after obtaining the description information of the search link in at least one content dimension, the search link may be subjected to anomaly detection using a corresponding abnormal link detection strategy for the description information of each content dimension to obtain an anomaly detection result for each content dimension.

[0112] The abnormal link detection strategy includes rules that need to be followed when performing abnormality detection on search links based on description information.

[0113] In one embodiment, description information of different content dimensions corresponds to different abnormal link detection strategies. For example, when the description information includes link content and a link status code, the link content may correspond to one abnormal link detection strategy, while the link status code may correspond to another abnormal link detection strategy. For another example, when the description information includes link content, a link status code, and the original webpage content, the link content may correspond to abnormal link detection strategy A, the link status code may correspond to abnormal link detection strategy B, and the original webpage content may correspond to abnormal link detection strategy C.

[0114] In one embodiment, when the description content is the original webpage content, it can be determined whether the original webpage content contains content that needs to be filtered. When the original webpage content does not include the filtered content, the original webpage content is detected to determine whether the search link is an abnormal link. Specifically, the step of "for each content dimension description information, using the corresponding abnormal link detection strategy to perform abnormality detection on the search link to obtain an abnormality detection result for each content dimension" may include:

[0115] Perform filtering content detection on the original web page content;

[0116] When the original web page content does not include the preset filtered content, the original web page content is parsed to obtain the main content of the original web page content;

[0117] Perform content detection on the main content to obtain abnormal detection results.

[0118] When the original webpage content contains filtered content, the original webpage content is not checked, and other methods are used to determine whether the search link is an abnormal link. For example, when the original webpage content is empty, it means that the original webpage content has no substantive content. If the original webpage content is still checked to determine whether the search link is an abnormal link, the error rate of the abnormality detection of the search link will be increased.

[0119] Therefore, when the description information is the original web page content, the original web page content can be first subjected to a filter content detection. When the original web page content does not include the preset filter content, the original web page content can be detected to determine whether the search link is an abnormal link. For example, it can be determined whether the original web page content is empty. If the original web page content is empty, other abnormal link detection strategies can be used to detect abnormalities in the search link. If the original web page content is not empty, the original web page content can be detected to determine whether the search link is an abnormal link.

[0120] In one embodiment, there are multiple ways to detect the original webpage content to determine whether the original webpage content includes the preset filtered content. For example, the original webpage content can be traversed. For another example, when the original webpage content is described in HTML, the HTML tags in the original webpage content can be detected to determine whether the webpage content includes the preset filtered content.

[0121] In one embodiment, when the original webpage content includes preset filtering content, other abnormal link detection strategies may be adopted to perform abnormality detection on the search link.

[0122] In one embodiment, when the original webpage content does not include the preset filtering content, whether the search link is an abnormal link can be determined based on the main content of the original webpage content. For example, when the original webpage content is not empty content, whether the search link is an abnormal link can be determined based on the main content of the original webpage content. Therefore, when the original webpage content does not include the preset filtering content, the original webpage content can be parsed to obtain the main content of the original webpage content.

[0123] The main content may include the content of the original web page content that has substantial meaning. For example, the main content may include text content, image content, video content, animation content, table content, etc. in the original web page content.

[0124] In one embodiment, when the search link is an abnormal link, the original web page content corresponding to the search link often has relevant text prompts. For example, when the search link is a dead link, the original web page content of the search link may have relevant text "Sorry, the resource you are accessing does not exist." Therefore, when performing content detection on the main content, it is possible to first determine whether the main content contains text content. When the main content includes text content, it is possible to determine whether the search link is an abnormal link based on the text content in the main content. Specifically, the step of "performing content detection on the main content to obtain an abnormality detection result" may include:

[0125] Perform text detection on the main content;

[0126] When it is detected that the main content includes text content, abnormal keyword detection is performed on the text content to obtain an abnormal detection result.

[0127] In one embodiment, the purpose of performing text detection on the main content is to determine whether the main content contains text content. There are multiple ways to perform text detection on the main content.

[0128] For example, when the original web page content is described in HTML language, it is possible to determine whether the main content includes text content by detecting the HTML tags of the main content. For example, when the HTML tags of the main content include text tags, it means that the main content may include text content. Among them, the text tags may include tags that act on text. For example, the text tags may include 、 <em>、 and < / em> <h1> <em> etc. HTML tags

[0129] In one embodiment, when the main content includes text content, it can be determined whether the text content contains abnormal keywords. When the text content includes abnormal keywords, it indicates that the search link may be an abnormal link. Therefore, when it is detected that the main content includes main content, abnormal keyword detection can be performed on the text content to obtain abnormal detection results. Specifically, the step of "performing abnormal keyword detection on the text content to obtain abnormal detection results" can include:

[0130] Perform word segmentation on the text content to obtain at least one text subword;

[0131] Match the text subwords with the preset abnormal keywords to obtain keyword matching results;

[0132] Generate anomaly detection results based on keyword matching results.

[0133] Text subwords may include words that make up the text content. For example, if the text content is "Maybe the URL is incorrect," the text subwords may include "maybe," "is," "URL," and "incorrect." For another example, if the text content is "The content of the URL was deleted," the text subwords may include "URL," "of," "content," "was," "deleted," and so on.

[0134] In one embodiment, there are multiple ways to segment the text content to obtain at least one text subword. For example, the text content can be segmented using a segmentation tool such as Han Language Processing (HanLp), Jieba Library, Language Technology Platform (LTP), or QQSeg to obtain at least one text subword.

[0135] In one embodiment, after obtaining at least one text subword, the text subword can be matched with a preset abnormal keyword to obtain a keyword matching result, and an abnormality detection result is generated based on the keyword matching result. The preset abnormal keyword can include a word pre-stored in the abnormal link processing device.

[0136] In one embodiment, before storing the preset abnormal keywords in the abnormal link processing device, the abnormal link processing device may obtain the initial abnormal keywords and generate the preset abnormal keywords based on the initial abnormal keywords. Specifically, before the step of "matching the text subwords with the preset abnormal keywords", the following steps may be included:

[0137] Get the initial exception keyword;

[0138] Expand the initial abnormal keywords to obtain expanded abnormal keywords;

[0139] Perform abnormal link search based on the expanded abnormal keywords to obtain abnormal link search results;

[0140] Based on the abnormal link search results, preset abnormal keywords are filtered out from the expanded abnormal keywords.

[0141] The initial abnormal keywords may include abnormal keywords that have not been expanded.

[0142] For example, the initial abnormal keywords may include abnormal keywords generated by human experience. For example, abnormal keywords may be manually annotated on the text content corresponding to the known abnormal links, and the annotated abnormal keywords are used as the initial abnormal keywords.

[0143] For another example, the initial abnormal keywords may include abnormal keywords obtained by performing data mining on known abnormal links.

[0144] In one embodiment, since the initial abnormal keywords are all obtained through known abnormal links, their generalization is relatively poor. If new abnormal keywords appear in new known abnormal links, it is necessary to manually annotate or data mine the known abnormal links, which will affect the reliability and efficiency of abnormal link detection. Therefore, after obtaining the initial abnormal keywords, the initial abnormal keywords can be expanded to obtain expanded abnormal keywords; abnormal link searches can be performed based on the expanded abnormal keywords to obtain abnormal link search results; based on the abnormal link search results, preset abnormal keywords are filtered out from the expanded abnormal keywords, thereby improving the generalization of the preset abnormal keywords. Therefore, when using the preset abnormal keywords to determine whether the search link is an abnormal link, due to the high generalization of the preset abnormal keywords, the accuracy of determining whether the search link is an abnormal link will be improved, thereby improving the accuracy of abnormal link processing.

[0145] In one embodiment, the initial abnormal keywords can be expanded in a variety of ways to obtain expanded abnormal keywords. For example, synonyms and antonyms of the initial abnormal keywords can be searched for, and the keywords obtained from the search can be used as the expanded abnormal keywords. In another example, similar words with similar expressions to the initial abnormal keywords can be manually expanded, and the expanded similar words can be used as the expanded abnormal keywords, and so on.

[0146] In one embodiment, after obtaining the expanded abnormal keywords, the expanded abnormal keywords need to be verified. This verification determines whether the expanded abnormal keywords can search for abnormal links, thereby improving the accuracy and reliability of the preset abnormal keywords. Therefore, an abnormal link search can be performed based on the expanded abnormal keywords to obtain abnormal link search results. Then, based on the abnormal link search results, the preset abnormal keywords are filtered out from the expanded abnormal keywords.

[0147] For example, the expanded abnormal keyword can be used to search and obtain at least one keyword search link based on the expanded abnormal keyword search. Then, it can be determined whether the keyword search link is a dead link, thereby obtaining abnormal link search results for the expanded abnormal keyword.

[0148] The abnormal link search results may include abnormal links that can be searched through the expanded abnormal keywords, and the number of abnormal links that are searched.

[0149] Next, preset abnormal keywords can be screened from the expanded abnormal keywords based on the abnormal link search results. For example, the abnormal link search results can first be used to determine whether abnormal links can be found using the expanded abnormal keywords. If the expanded abnormal keywords can find abnormal links, the number of abnormal links that can be found using the expanded abnormal keywords can be determined. If the number of abnormal links is greater than or equal to a preset threshold, the expanded abnormal keywords can be used as the preset abnormal keywords. If the expanded abnormal keywords cannot find abnormal links, or if the number of abnormal links found is less than the preset threshold, the expanded abnormal keywords are not used as the preset keywords.

[0150] In one embodiment, in addition to using the expanded abnormal keywords as the preset abnormal keywords, the initial abnormal keywords can also be used as the preset abnormal keywords. That is, when matching text subwords with the preset abnormal keywords, the preset abnormal keywords include the initial abnormal keywords and the screened expanded abnormal keywords.

[0151] In one embodiment, after obtaining the preset abnormal keywords, the text subwords can be matched with the preset abnormal keywords to obtain keyword matching results, and then an abnormality detection result can be generated based on the keyword matching results. In other words, it can be detected whether the text subwords of the text content include the preset abnormal keywords. If the text subwords include the preset abnormal keywords, an abnormality detection result can be generated indicating that the search link is an abnormal link. If the text subwords do not include the preset abnormal keywords, an abnormality detection result can be generated indicating that the search link is not an abnormal link.

[0152] The anomaly detection result is used to indicate whether the search link is an abnormal link.

[0153] In one embodiment, the anomaly detection result can be expressed in various forms. For example, a "1" anomaly detection result can indicate that the search link is an anomaly link, while a "0" anomaly detection result can indicate that the search link is not an anomaly link. For another example, a "True" anomaly detection result can indicate that the search link is an anomaly link, while a "False" anomaly detection result can indicate that the search link is not an anomaly link, and so on.

[0154] In one embodiment, to improve the accuracy of abnormal link detection, when keyword detection of text content indicates that the search link is not an abnormal link, semantic features of the text content may be further extracted, and based on the semantic features of the text content, whether the search link is an abnormal link may be determined. Specifically, the step of "generating anomaly detection results based on keyword matching results" may include:

[0155] When the text subwords do not match the preset abnormal keywords, semantic extraction is performed on the text content to obtain the semantic features of the text content;

[0156] Perform forward coding and backward coding on the semantic features respectively to obtain forward coding information corresponding to the forward coding and backward coding information corresponding to the backward coding;

[0157] Fusing the forward coded information and the backward coded information to obtain fused coded information;

[0158] The anomaly probability of the search link is calculated based on the fused encoded information to obtain the anomaly detection result.

[0159] Semantic features may include information describing the meaning expressed by text content.

[0160] In one embodiment, there are multiple ways to perform semantic extraction on text content to obtain semantic features of the text content. For example, a convolution operation can be performed on the text content to obtain semantic features of the text content. In another example, the text content can be sampled according to a preset step size and semantically transformed on the sampled text content to obtain semantic features.

[0161] In one embodiment, in order to improve the accuracy of detection, after obtaining the semantic features, the semantic features may be forward encoded and backward encoded to obtain forward encoding information corresponding to the forward encoding and backward encoding information corresponding to the backward encoding.

[0162] Forward encoding can include encoding semantic features from front to back. For example, if the semantic feature is "content is empty," forward encoding can mean encoding the semantic features in the order of "content is empty." Backward encoding can mean encoding the semantic features in the order of "empty is content."

[0163] In one embodiment, after the forward coding information and the backward coding information are obtained, the forward coding information and the backward coding information may be fused to obtain fused coding information.

[0164] For example, the forward coded information and the backward coded information may be added to obtain the fused coded information. For another example, the forward coded information and the backward coded information may be concatenated to obtain the fused coded information.

[0165] By forward encoding and backward encoding the semantic features, we can take into account both the actual semantics of the text content and the reverse semantics of the text content, so that the information content of the fused encoded information is higher, and the accuracy of calculating the abnormal probability of the search link based on the fused encoded information can be improved.

[0166] In one embodiment, after obtaining the fused coded information, the abnormality probability of the search link may be calculated based on the fused coded information, and the abnormality detection result may be obtained based on the abnormality probability.

[0167] For example, the abnormality probability may be compared with a preset abnormality probability discrimination threshold, and the abnormality detection result may be determined based on the comparison result.

[0168] For example, the abnormal probability threshold can be set to 90%. When the abnormal probability is greater than or equal to 90%, the search link can be an abnormal link. When the abnormal probability is less than 90%, the search link is not an abnormal link.

[0169] For another example, other methods may be used to perform semantic detection on the text content, and determine whether the search link is an abnormal link based on the detection result.

[0170] For example, neural networks such as the Long Short-Term Memory (LSTM) or Bi-directional Long Short-Term Memory (BISTIM) can be used to perform semantic detection on text content, and determine whether the search content is an abnormal link based on the detection results.

[0171] In one embodiment, when it is detected that the main content does not include text content, it can be determined whether the search link is an abnormal link through other types of main content. Figure 4 As shown, when the search link is an abnormal link, the webpage content corresponding to the search link may indicate that the link is an abnormal link through an image. Therefore, it is possible to determine whether the search link is an abnormal link based on the image content. Specifically, the method proposed in the embodiment of the present application also includes:

[0172] When it is detected that the main content does not include text content, performing image detection on the main content;

[0173] When it is detected that the main content includes image content, character recognition is performed on the image content;

[0174] When character information of the image content is recognized, abnormal character detection is performed on the character information to obtain an abnormality detection result.

[0175] In one embodiment, when it is detected that the subject does not include subject content, image detection can be performed on the subject content to determine whether the subject content includes image content. There are multiple ways to perform image detection on the subject content.

[0176] For example, when the original web page content is described in HTML language, it is possible to determine whether the main content includes image content by detecting the HTML tags of the main content. For example, when the HTML tags of the image content include image tags, it means that the main content may include image content. The image content may include tags that function for images. For example, the image tag may be Tags, etc.

[0177] In one embodiment, when it is detected that the main content includes image content, character recognition may be performed on the image content. Character recognition may include determining whether the image content includes character information. Character information may include text, punctuation marks, numbers, and other symbols.

[0178] By performing character recognition on the image content, it can be determined whether the image content contains character information. When the image content contains character information, abnormal character detection can be performed on the character information to obtain an abnormality detection result.

[0179] In one embodiment, there are multiple ways to perform character recognition on image content. For example, optical character recognition (OCR) can be used to perform character recognition on image content, thereby determining whether the image content contains character information. For another example, a deep learning model can be used to perform character recognition on image content. For example, a deep learning model such as convolutional neural networks (CNN) or deep convolutional neural networks (DCNN) can be used to perform character recognition on image content.

[0180] In one embodiment, when it is recognized that the image content includes character information, abnormal character detection may be performed on the character information to obtain an abnormality detection result.

[0181] The character information can be subjected to anomaly detection in a manner similar to detecting abnormal keywords in text content. For example, the character information can be segmented, and then the segmented character information can be matched with preset abnormal keywords, and an anomaly detection result can be generated based on the matching result. If the character information is detected as not being an abnormal link, semantic recognition can be performed on the character information to further detect whether the search link is an abnormal link.

[0182] In one embodiment, when the main content of the original webpage content does not include text content or image content, other methods can be used to detect the original webpage content to determine whether the search link is an abnormal link. Specifically, the abnormal link processing method proposed in this embodiment of the application may also include:

[0183] When it is detected that the main content does not include image content, clustering is performed on the original web page content to obtain a target cluster corresponding to the original web page content;

[0184] Calculate the similarity between objects in the target cluster;

[0185] Determine whether the search link is an abnormal link based on similarity.

[0186] In one embodiment, when it is detected that the main content does not include image content and text content, the original web page content may be clustered as a whole to obtain a target cluster corresponding to the original web page content.

[0187] Clustering is the process of dividing a collection of physical or abstract objects into clusters of similar objects. A cluster is a set of data objects that are similar to objects in the same cluster but different from objects in other clusters.

[0188] There are many ways to cluster the original web page content to obtain the target cluster corresponding to the original web page content. For example, the K-Means Clustering Algorithm (K-MEANS) or the random selection-based clustering algorithm (CLARANS) can be used to cluster the original web page to obtain the target cluster corresponding to the original web page. For another example, the balanced iterative reduction and clustering using hierarchies (BIRCH), the divisive hierarchical clustering (DHC), or the agglomerative hierarchical clustering (AHC) can be used to cluster the original web page to obtain the target cluster corresponding to the original web page.

[0189] In one embodiment, when a search link is an abnormal link, the objects in the target cluster corresponding to the original webpage content will exhibit a high degree of similarity. Therefore, after determining the target cluster corresponding to the original webpage content, the similarity between the objects in the target cluster can be calculated, and based on the similarity, whether the search link is an abnormal link can be determined.

[0190] In this embodiment of the present application, the objects in the target cluster may include original web page contents of different search links that are clustered together.

[0191] In one embodiment, the similarity between objects in the target cluster can be calculated using methods such as Euclidean distance or Pearson correlation coefficient. When the similarity between objects in the target cluster is greater than or equal to a preset similarity threshold, the search link can be determined to be an abnormal link. When the similarity between objects in the target cluster is less than the preset similarity threshold, the search link can be determined to be not an abnormal link.

[0192] For example, the preset similarity threshold may be set to 60%. When the similarity between objects in the target cluster is greater than or equal to 60%, it may be determined that the search link is an abnormal link.

[0193] In one embodiment, when the description information is link content, an abnormal link detection strategy corresponding to the link content may be used to detect the link information, thereby determining whether the search link is an abnormal link based on the abnormal detection result. Specifically, the step of "performing an abnormality detection on the search link using the corresponding abnormal link detection strategy for the description information of each content dimension to obtain an abnormality detection result for each content dimension" may include:

[0194] Parse the link content to obtain the domain names of the link content at different levels;

[0195] Clustering the link content based on the domain names at different levels of the link content to obtain the target domain name cluster corresponding to the link content;

[0196] Perform similarity judgment on the target domain name cluster and generate anomaly detection results based on the judgment results.

[0197] A domain name is a name for a computer or group of computers on the internet, consisting of a series of names separated by dots. It is used to locate the computer during data transmission. Domain names can be mapped to IP (Internet Protocol) addresses, making internet access more convenient.

[0198] In one embodiment, domain names may be divided into different levels according to the location range represented by the domain name, for example, domain names may be divided into root domain names, first-level domain names, and second-level domain names, etc.

[0199] The root domain name is the highest level domain name node, and general link content carries the root domain name. For example, the root domain name can include ".root", etc.

[0200] The domain name below the root domain name is the first-level domain name. For example, the first-level domain name may include ".com", ".org", ".net", ".cn", etc.

[0201] The domain name below the first-level domain name is the second-level domain name. For example, the second-level domain name can include "web1.web.cn", "web2.web1.web.cn", etc.

[0202] In one embodiment, the link content may consist of domain names. For example, the link content may be "host name.second-level domain name.first-level domain name.root domain name." For example, when the link content is "www.web1.com.root," ".web1" may be the second-level domain name, ".com" may be the first-level domain name, and ".root" may be the root domain name. Therefore, the link content may be parsed to obtain domain names at different levels of the link content. The searched link may then be judged based on the domain names at different levels to determine whether it is an abnormal link.

[0203] In one embodiment, the link content may be traversed and then the link content obtained through the traversal may be matched with the preset level domain name identification information to obtain the domain names of the link content at different levels.

[0204] For example, the link content is "www.web1.com.root". By traversing the link content, the link contents are obtained as ".www", ".web1", ".com", and ".root". Then, the link content can be matched with the preset level domain name identification information to obtain domain names at different levels.

[0205] In one embodiment, after obtaining the domain names of the link contents at different levels, the link contents may be clustered based on the domain names of the link contents at different levels to obtain target domain name clusters corresponding to the link contents.

[0206] There are many methods to cluster domain names at different levels.

[0207] For example, algorithms such as K-MEANS, CLARANS, or BIRCH can be used to cluster domain names at different levels to obtain target domain name clusters corresponding to link content.

[0208] For another example, the distance between a domain name and a predetermined domain name in a plurality of predetermined domain name clusters may be calculated, and then a target domain name cluster for domain name clustering may be determined from the plurality of predetermined domain name clusters based on the distance. Specifically, the step of "clustering the link content based on the domain names at different levels of the link content to obtain a target domain name cluster corresponding to the link content" may include:

[0209] Calculating the distance between the domain name and a plurality of preset domain names in the preset domain name clusters;

[0210] A target domain name cluster for domain name clustering is determined from multiple preset domain name clusters based on the distance.

[0211] There are many ways to calculate the distance between a domain name and the preset domain names in the multiple preset domain name clusters. For example, the distance between a domain name and the preset domain names in the multiple preset domain name clusters can be calculated using methods such as Euclidean distance or Pearson correlation coefficient.

[0212] Then, a target domain name cluster for domain name clustering can be determined from the multiple preset domain name clusters based on the distance. For example, the preset domain name cluster with the smallest distance can be selected as the target domain name cluster.

[0213] For example, when the link content includes a first-level domain name and a second-level domain name, the distance between the first-level domain name of the link content and a preset domain name in multiple preset domain name clusters can be calculated. Furthermore, the distance between the second-level domain name of the link content and a preset domain name in multiple preset domain name clusters can also be calculated. These two distances are then added together, and based on the resulting distance, a target domain name cluster for domain name clustering is determined from the multiple preset domain name clusters.

[0214] For example, the link content includes a first-level domain name a1 and a second-level domain name a2. Multiple preset domain name clusters include a preset domain name cluster b1, a preset domain name cluster b2, and a preset domain name cluster b3. Then, the distance c11 between the preset domain name clusters in a1 and b1 can be calculated; the distance c12 between the preset domain name clusters in a1 and b2 can be calculated; and the distance c13 between the preset domain name clusters in a1 and b3 can be calculated. Similarly, the distance c21 between the preset domain name clusters in a2 and b1 can be calculated; the distance c22 between the preset domain name clusters in a2 and b2 can be calculated; and the distance c23 between the preset domain name clusters in a2 and b3 can be calculated. Then, c11 and c21 can be added to obtain c1; c12 and c22 can be added to obtain c2; and c13 and c23 can be added to obtain c3. Then, based on c1, c2, and c3, a target domain name cluster for domain name clustering can be determined from the multiple preset domain name clusters. For example, when c1 is the smallest, preset domain name cluster b1 can be determined as the target domain name cluster.

[0215] In one embodiment, when the description information includes a link status code, the link status code may be detected using an abnormal link detection strategy corresponding to the link status code, thereby determining whether the search link is an abnormal link result based on the abnormal detection result. Specifically, the step of "performing an abnormality detection on the search link using the corresponding abnormal link detection strategy for the description information of each content dimension to obtain an abnormality detection result for each content dimension" may include:

[0216] Match the link status code with the preset exception status code to obtain a status code matching result;

[0217] Map the matching results to the corresponding anomaly detection results.

[0218] The preset abnormal status codes may include status codes such as 404, 403 and 405, which indicate that abnormalities exist in the web page content.

[0219] In one embodiment, the link status code can be matched with a preset abnormal status code to determine whether the link status code is the preset abnormal status code. When the link status code is the preset abnormal status code, it indicates that the search link is an abnormal link. When the link status code is not the preset abnormal status code, it indicates that the search link is not an abnormal link.

[0220] 105. Block search links based on abnormal detection results.

[0221] In one embodiment, after obtaining the abnormality detection results for each content dimension, the search link may be blocked based on the abnormality detection results. For example, when a search link is detected to be an abnormal link, the abnormal search link may be blocked.

[0222] Among them, the blocking process may include clearing abnormal links. By blocking the search links, when the computer device displays the search results to the user, the user will not obtain abnormal links, thereby improving the user's search experience.

[0223] In one embodiment, when the abnormal link processing device detects that a search link is an abnormal link, it can add an abnormality flag to the abnormal search link. Then, when the search link is displayed to the user, the computer device can block the abnormal search link based on the abnormality flag, so that the abnormal link is not included in the search links displayed to the user.

[0224] The embodiment of the present application proposes a method for processing abnormal links, which includes: obtaining a search link to be detected; performing a content search based on the search link to obtain a web page structure corresponding to the search link; parsing the web page structure to obtain descriptive information of the search link in at least one content dimension; using the corresponding abnormal link detection strategy for the descriptive information of each content dimension to perform abnormality detection on the search link to obtain an abnormality detection result for each content dimension; and banning the search link based on the abnormality detection result. The embodiment of the present application performs abnormality detection on the search link based on the descriptive information on different content dimensions, thereby making it possible to determine whether the search link is an abnormal link in multiple content dimensions. By determining whether the search link is an abnormal link in multiple content dimensions, the detection range of the search link can be expanded to avoid missing search links that are abnormal links when detecting abnormal links, thereby improving the accuracy of detecting the search link.

[0225] When detecting text content, keyword detection can be performed on the text content first to determine whether the search link is an abnormal link. If the keyword detection of the text content shows that the search link is not an abnormal link, semantic detection can be performed on the text content to determine whether the search link is an abnormal link. By performing keyword detection and semantic detection on the text content, the scope of text content detection can be expanded, thereby improving the accuracy of text detection and the accuracy of determining whether the search link is an abnormal link.

[0226] Furthermore, the embodiments of the present application can also use text clustering to target semantically non-semantic or garbled text content, thereby filtering out frequently occurring text clusters and identifying instances of anti-crawl, false positives, or missed calls. For unrecognizable original webpage content, content clustering can be used to identify abnormal links, effectively addressing instances of website anti-crawl and webpage structural anomalies.

[0227] For example, for websites with anti-crawl functionality, the webpage structure found is often abnormal, for example, there are many duplicate contents, or the structure of the searched webpage does not match the content of its links. In this case, through the embodiments of the present application, it is possible to detect whether the search link corresponding to the website is a real abnormal link or is just a reason for anti-crawl, thereby improving the accuracy of abnormal link detection.

[0228] The method described in the above embodiment will be further described in detail below with examples.

[0229] The embodiment of the present application will take the abnormal link processing method integrated on a computer device as an example to introduce the embodiment method of the present application.

[0230] In one embodiment, if Figure 5 As shown in FIG, a method for handling abnormal links, the specific process is as follows:

[0231] 201. The computer device obtains a search link to be detected.

[0232] For example, the computer device obtains multiple search links to be detected.

[0233] 202. The computing device performs a content search based on the search link to obtain a webpage structure corresponding to the search link.

[0234] For example, the computer device may perform a content search on each search link to obtain a webpage structure corresponding to each search link.

[0235] 203. The computer device parses the webpage structure to obtain description information of the search link in at least one content dimension.

[0236] For example, the computer device may parse the webpage structure of each search link to obtain description information of each search link in at least one content dimension.

[0237] The description information on at least one content dimension may include original web page content, link content, and link status code.

[0238] Different search links may have description information in different content dimensions.

[0239] For example, some search links may be parsed to obtain link content and link status codes. For another example, some search links may be parsed to obtain link content and original web page content. For another example, some search links may be parsed to obtain link content, link status codes, and original web page content.

[0240] 204. The computer device performs an anomaly detection on the search link using a corresponding abnormal link detection strategy based on the description information of each content dimension, and obtains an anomaly detection result for each content dimension.

[0241] For example, Figure 6 As shown, when the description information of the search link includes link content, link status code and original web page content, the link content can correspond to one abnormal link detection strategy, the link status code can correspond to another abnormal link detection strategy, and the original web page content can correspond to a third abnormal link detection strategy.

[0242] The link status code can be matched against a preset abnormal status code. If the link status code matches the preset abnormal status code, the search link corresponding to the link status code is an abnormal link. If the link status code does not match the preset abnormal status code, the search link corresponding to the link status code is not an abnormal link.

[0243] The link content can be parsed to obtain domain names at different levels of the link content, and then the domain names at different levels are clustered, and the clustering results are used to determine whether the search link is an abnormal link.

[0244] For example, the first-level domain names of the link content and the second-level domain names of the link content can be clustered. Then, whether the search link is an abnormal link can be determined based on the clustering results of the first-level domain names and the clustering results of the second-level domain names.

[0245] Wherein, for the original web page content, it can be first determined whether the content of the original web page content is empty content. When the original web page content is not empty content, the original web page content can be subjected to text parsing to determine whether the original web page content contains text content.

[0246] When the original webpage contains text, the text can be checked for unusual keywords to determine whether the search link corresponding to the text is an unusual link. If the unusual keyword detection indicates that the text link is not an unusual link, the text can be converted into a vector representation using the word2vec method. This vector representation is then incorporated into the BILSTM model to determine whether the search link is an unusual link. If the text content determines that the search link is an unusual link, the search link can be blocked. By using the BILSTM model to judge text content, longer-range dependencies and bidirectional semantic dependencies can be better captured.

[0247] In one embodiment, when the original webpage content does not contain text content, it can be determined whether the original webpage content contains image content. If the original webpage content contains image content, the image content can be subjected to OCR analysis to obtain text characters corresponding to the image content.

[0248] Then, abnormal keyword detection can be performed on the text characters and the BILSTM model can be used to judge the text characters to determine whether the search link is an abnormal link.

[0249] When the original webpage content does not have text content and image content, the original webpage content can be regarded as a whole, and then clustered to obtain a clustering result. Then, based on the clustering result, it can be determined whether the search link is an abnormal link.

[0250] 205. The computer device blocks the search link based on the abnormality detection result.

[0251] In an embodiment of the present application, a computer device obtains a search link to be detected; performs a content search based on the search link to obtain a web page structure corresponding to the search link; the computer device parses the web page structure to obtain descriptive information of the search link in at least one content dimension; for the descriptive information of each content dimension, the computer device uses a corresponding abnormal link detection strategy to perform abnormality detection on the search link to obtain an abnormality detection result for each content dimension; and blocks the search link based on the abnormality detection result. The embodiment of the present application performs abnormality detection on the search link based on the descriptive information on different content dimensions, thereby making it possible to determine whether the search link is an abnormal link in multiple content dimensions. By determining whether the search link is an abnormal link in multiple content dimensions, the detection range of the search link can be expanded, avoiding the omission of search links that are abnormal links when detecting abnormal links, thereby improving the accuracy of detecting the search link.

[0252] In order to better implement the abnormal link processing method provided in the embodiment of the present application, an abnormal link processing device is also provided in one embodiment. The abnormal link processing device can be integrated into a computer device. The meanings of the terms herein are the same as those in the abnormal link processing method described above. For specific implementation details, please refer to the description in the method embodiment.

[0253] In one embodiment, an abnormal link processing device is provided. The abnormal link processing device can be integrated into a computer device, such as Figure 7 As shown, the abnormal link processing device includes: an acquisition unit 301, a content search unit 302, a parsing unit 303, an abnormality detection unit 304 and a blocking unit 305:

[0254] An acquisition unit 301 is used to acquire a search link to be detected;

[0255] A content search unit 302 is configured to perform a content search based on the search link to obtain a web page structure corresponding to each search link;

[0256] The parsing unit 303 is configured to parse the webpage structure to obtain description information of the search link in at least one content dimension;

[0257] Anomaly detection unit 304 is used to perform anomaly detection on the search link using a corresponding abnormal link detection strategy based on the description information of each content dimension, and obtain an anomaly detection result for each content dimension;

[0258] The blocking unit 305 is configured to block the search link based on the abnormality detection result.

[0259] In one embodiment, the anomaly detection unit 304 includes:

[0260] A filtering content detection subunit, configured to perform filtering content detection on the original web page content;

[0261] A first parsing subunit is configured to parse the original web page content to obtain the main content of the original web page content when the original web page content does not include the preset filtering content;

[0262] The content detection subunit is used to perform content detection on the main content to obtain the abnormality detection result.

[0263] In one embodiment, the content detection subunit includes:

[0264] A text detection module, configured to perform text detection on the main content;

[0265] The abnormal keyword detection module is used to perform abnormal keyword detection on the text content when it is detected that the main content includes text content, and obtain the abnormal detection result.

[0266] In one embodiment, the abnormal keyword detection module includes:

[0267] A word segmentation submodule, configured to perform word segmentation processing on the text content to obtain at least one text subword;

[0268] A keyword matching submodule is used to match the text subwords with preset abnormal keywords to obtain keyword matching results;

[0269] The result generation submodule is used to generate the anomaly detection result based on the keyword matching result.

[0270] In one embodiment, the result generation submodule is used to:

[0271] When the text subword does not match the preset abnormal keyword, semantic extraction is performed on the text content to obtain semantic features of the text content;

[0272] Performing forward coding and backward coding on the semantic features respectively to obtain forward coding information corresponding to the forward coding and backward coding information corresponding to the backward coding;

[0273] Fusing the forward coded information and the backward coded information to obtain fused coded information;

[0274] The abnormality probability of the search link is calculated based on the fused encoded information to obtain the abnormality detection result.

[0275] In one embodiment, the abnormal keyword detection module further includes:

[0276] Keyword acquisition submodule, used to obtain initial abnormal keywords;

[0277] An expansion submodule, configured to expand the initial abnormal keywords to obtain expanded abnormal keywords;

[0278] An abnormal link search submodule is used to search for abnormal links based on the expanded abnormal keywords to obtain abnormal link search results;

[0279] A screening submodule is configured to screen out the preset abnormal keywords from the expanded abnormal keywords based on the abnormal link search results.

[0280] In one embodiment, the content detection subunit further includes:

[0281] An image detection module, configured to perform image detection on the main content when it is detected that the main content does not include text content;

[0282] a character recognition module, configured to perform character recognition on the image content when detecting that the main content includes image content;

[0283] The abnormal character detection module is used to perform abnormal character detection on the character information when character information of the image content is recognized, and obtain the abnormal character detection result.

[0284] In one embodiment, the content detection subunit further includes:

[0285] A clustering processing module, configured to perform clustering processing on the original web page content to obtain a target cluster corresponding to the original web page content when it is detected that the main content does not include image content;

[0286] A first calculation module, configured to calculate the similarity between objects in the target cluster;

[0287] A judgment module is used to judge whether the search link is an abnormal link based on the similarity.

[0288] In one embodiment, the anomaly detection unit includes:

[0289] A second parsing subunit is configured to parse the link content to obtain domain names of the link content at different levels;

[0290] a clustering processing subunit, configured to perform clustering processing on the link content based on the domain names of the link content at different levels, to obtain a target domain name cluster corresponding to the link content;

[0291] The similarity determination subunit is configured to perform similarity determination on the target domain name cluster and generate the anomaly detection result based on the determination result.

[0292] In one embodiment, the cluster processing subunit includes:

[0293] a second calculation module, configured to calculate a distance between the domain name and a plurality of preset domain name clusters;

[0294] A determination module is configured to determine a target domain name cluster for the domain name clustering from the plurality of preset domain name clusters based on the distance.

[0295] In one embodiment, the anomaly detection unit includes:

[0296] A status code matching subunit, configured to match the link status code with a preset abnormal status code to obtain a status code matching result;

[0297] A mapping subunit is used to map the matching result to the corresponding anomaly detection result.

[0298] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can be found in the previous method embodiments and will not be repeated here.

[0299] The above abnormal link processing device can improve the reliability of detecting abnormal links.

[0300] The embodiment of the present application also provides a computer device, which may include a terminal or a server. For example, the computer device may be used as an abnormal link processing terminal, which may be a mobile phone, a tablet computer, etc.; for another example, the computer device may be a server, such as an abnormal link processing server. Figure 8 As shown, it shows a schematic diagram of the structure of the terminal involved in the embodiment of the present application, specifically:

[0301] The computer device may include one or more processing core processors 401, one or more computer readable storage media memories 402, a power supply 403, an input unit 404 and other components. Those skilled in the art will understand that Figure 8 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.

[0302] Processor 401 is the control center of the computer device. It utilizes various interfaces and lines to connect the various components of the entire computer device. By running or executing software programs and / or modules stored in memory 402 and accessing data stored in memory 402, it performs various functions of the computer device and processes data, thereby performing overall testing of the computer device. Optionally, processor 401 may include one or more processing cores; preferably, processor 401 may integrate an application processor and a modem processor, wherein the application processor primarily processes the operating system, user interfaces, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 401.

[0303] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0304] The computer device also includes a power supply 403 for supplying power to various components. Preferably, the power supply 403 can be logically connected to the processor 401 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 403 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0305] The computer device may further include an input unit 404, which may be configured to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0306] Although not shown, the computer device may further include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the computer device will load the executable files corresponding to one or more application processes into the memory 402 according to the following instructions, and the processor 401 will run the application stored in the memory 402 to implement various functions as follows:

[0307] Get the search link to be detected;

[0308] Performing content search based on the search link to obtain a web page structure corresponding to each search link;

[0309] Parsing the webpage structure to obtain description information of the search link in at least one content dimension;

[0310] For the description information of each content dimension, a corresponding abnormal link detection strategy is used to perform abnormality detection on the search link to obtain an abnormality detection result for each content dimension;

[0311] The search link is blocked based on the abnormality detection result.

[0312] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0313] According to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various optional implementations of the above embodiments.

[0314] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by a computer program, or by controlling related hardware through a computer program. The computer program may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0315] To this end, an embodiment of the present application further provides a storage medium storing a computer program that can be loaded by a processor to execute the steps of any of the abnormal link processing methods provided in the embodiments of the present application. For example, the computer program can execute the following steps:

[0316] Get the search link to be detected;

[0317] Performing content search based on the search link to obtain a web page structure corresponding to each search link;

[0318] Parsing the webpage structure to obtain description information of the search link in at least one content dimension;

[0319] For the description information of each content dimension, a corresponding abnormal link detection strategy is used to perform abnormality detection on the search link to obtain an abnormality detection result for each content dimension;

[0320] The search link is blocked based on the abnormality detection result.

[0321] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0322] Since the computer program stored in the storage medium can execute the steps in any one of the abnormal link processing methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the abnormal link processing methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0323] The above is a detailed introduction to the abnormal link processing method, device, computer equipment and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application. < / em> < / h1>

Claims

1. A method for handling abnormal links, characterized in that: include: Get the search link to be detected; Performing a content search based on the search link to obtain a webpage structure corresponding to the search link; Parsing the webpage structure to obtain description information of the search link in at least one content dimension, wherein the description information in at least one content dimension includes original webpage content; Performing filtering content detection on the original webpage content; When the original webpage content does not include the preset filtering content, parsing the original webpage content to obtain the main content of the original webpage content; Performing text detection on the main content; When it is detected that the main content includes text content, performing word segmentation processing on the text content to obtain at least one text subword; Get the initial exception keyword; Expanding the initial abnormal keywords to obtain expanded abnormal keywords; Perform abnormal link search based on the expanded abnormal keywords to obtain abnormal link search results; Filtering preset abnormal keywords from the expanded abnormal keywords based on the abnormal link search results; Matching the text subwords with the preset abnormal keywords to obtain keyword matching results; Generating an anomaly detection result based on the keyword matching result, including: when the text subword does not match the preset abnormal keyword, performing semantic extraction on the text content to obtain semantic features of the text content, performing forward encoding and backward encoding on the semantic features respectively to obtain forward encoding information corresponding to the forward encoding and backward encoding information corresponding to the backward encoding, fusing the forward encoding information and the backward encoding information to obtain fused encoding information, and calculating the abnormality probability of the search link based on the fused encoding information to obtain the anomaly detection result; Wherein, generating the abnormality detection result based on the keyword matching result further includes: when the text subword includes the preset abnormal keyword, generating an abnormality detection result that the search link is an abnormal link; When it is detected that the main content does not include text content, performing image detection on the main content; When it is detected that the main content includes image content, character recognition is performed on the image content, and when character information of the image content is recognized, abnormal character detection is performed on the character information to obtain the abnormality detection result; When it is detected that the main content does not include image content, clustering is performed on the original webpage content to obtain a target cluster corresponding to the original webpage content, similarities between objects in the target cluster are calculated, and based on the similarities, whether the search link is an abnormal link is determined; The search link is blocked based on the abnormality detection result.

2. The method according to claim 1, characterized in that The method further comprises: When it is detected that the main content does not include text content, performing image detection on the main content; When it is detected that the main content includes image content, performing character recognition on the image content; When character information of the image content is recognized, abnormal character detection is performed on the character information to obtain the abnormality detection result.

3. The method according to claim 2, characterized in that The method further comprises: When it is detected that the main content does not include image content, clustering is performed on the original web page content to obtain a target cluster corresponding to the original web page content; Calculating the similarity between objects in the target cluster; It is determined whether the search link is an abnormal link based on the similarity.

4. The method according to claim 1, wherein The description information on at least one content dimension includes link content; for the description information of each content dimension, anomaly detection is performed on the search link using a corresponding abnormal link detection strategy to obtain an anomaly detection result for each content dimension, including: Parsing the link content to obtain domain names of the link content at different levels; Clustering the link content based on domain names at different levels of the link content to obtain a target domain name cluster corresponding to the link content; A similarity determination is performed on the target domain name cluster, and the anomaly detection result is generated based on the determination result.

5. The method according to claim 4, characterized in that The clustering of the link contents based on the domain names at different levels of the link contents to obtain target domain name clusters corresponding to the link contents includes: Calculating the distance between the domain name and a plurality of preset domain name clusters; A target domain name cluster for domain name clustering is determined from the plurality of preset domain name clusters based on the distance.

6. The method according to claim 1, wherein The description information on at least one content dimension includes a link status code; for the description information of each content dimension, a corresponding abnormal link detection strategy is used to perform abnormality detection on the search link, and an abnormality detection result for each content dimension is obtained, including: Matching the link status code with a preset abnormal status code to obtain a status code matching result; The matching result is mapped to the corresponding anomaly detection result.

7. An abnormal link processing device, characterized in that: include: An acquisition unit, used for acquiring a search link to be detected; A content search unit, configured to perform a content search based on the search link and obtain a webpage structure corresponding to the search link; a parsing unit, configured to parse the webpage structure to obtain description information of the search link in at least one content dimension, wherein the description information in at least one content dimension includes original webpage content; The anomaly detection unit is used to perform anomaly detection on the search link using a corresponding anomaly link detection strategy for the description information of each content dimension to obtain an anomaly detection result for each content dimension, including: performing filtered content detection on the original web page content; when the original web page content does not include preset filtered content, parsing the original web page content to obtain the main content in the original web page content; performing text detection on the main content; when it is detected that the main content includes text content, performing word segmentation processing on the text content to obtain at least one text subword, obtaining initial abnormal keywords; expanding the initial abnormal keywords to obtain expanded abnormal keywords; performing abnormal link search based on the expanded abnormal keywords to obtain abnormal link search results; filtering out preset abnormal keywords from the expanded abnormal keywords based on the abnormal link search results; matching the text subwords with the preset abnormal keywords to obtain keyword matching results; generating the anomaly detection result based on the keyword matching results. Wherein, the generating of the anomaly detection result based on the keyword matching result includes: when the text subword and the preset abnormal keyword do not match, performing semantic extraction on the text content to obtain the semantic features of the text content, performing forward encoding and backward encoding on the semantic features respectively to obtain forward encoding information corresponding to the forward encoding and backward encoding information corresponding to the backward encoding, fusing the forward encoding information and the backward encoding information to obtain fused encoding information, calculating the abnormal probability of the search link based on the fused encoding information to obtain the anomaly detection result; when the text subword includes the preset abnormal keyword, generating an anomaly detection result that the search link is an abnormal link, When it is detected that the main content does not include text content, performing image detection on the main content, When it is detected that the main content includes image content, character recognition is performed on the image content, and when character information of the image content is recognized, abnormal character detection is performed on the character information to obtain the abnormality detection result. When it is detected that the main content does not include image content, clustering is performed on the original webpage content to obtain a target cluster corresponding to the original webpage content, similarities between objects in the target cluster are calculated, and based on the similarities, whether the search link is an abnormal link is determined; A blocking unit is used to block the search link based on the abnormality detection result.

8. A computer device, characterized in that: It comprises a memory and a processor; the memory stores an application program, and the processor is used to run the application program in the memory to execute the operations in the abnormal link processing method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps of the abnormal link processing method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the abnormal link processing method according to any one of claims 1 to 6 are implemented.

Images