Webpage analysis identification method and device, and electronic device
By acquiring webpage data, preprocessing it, and matching it with FOFA rules, and using the AC algorithm to construct an automaton, the limitations of existing malicious URL identification technologies are solved. This achieves efficient and accurate malicious URL identification and dynamic updates, adapting to different types of malicious spoofing methods.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NAT COMP NETWORK & INFORMATION SECURITY MANAGEMENT CENT
- Filing Date
- 2024-10-09
- Publication Date
- 2026-07-21
AI Technical Summary
Existing methods for identifying malicious websites are limited to specific types and cannot adapt to new malicious websites or spoofing methods, resulting in the need for frequent adjustments to identification strategies and significant limitations.
By obtaining the source code data and image data of web page URLs, preprocessing them, and then matching and analyzing them with preset FOFA rules, the AC algorithm is used to build an automaton for matching. Combined with the database, malicious URLs are identified, adapting to different types of malicious spoofing methods.
It improves the efficiency and accuracy of malicious URL identification, reduces reliance on specific types of URLs, can dynamically update identification methods to maintain timeliness, and enhances security through blacklists and alert information.
Smart Images
Figure CN119475330B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of network security technology, specifically to a webpage analysis and identification method, device, and electronic device. Background Technology
[0002] With the rapid development of the internet, malicious websites and phishing websites are emerging in large numbers, posing a serious threat to users' information security and privacy. Related technologies typically employ specific analysis and identification strategies for specific types of malicious websites. For example, blacklist checks compare specific types of websites with pre-collected blacklists of malicious websites for detection; or domain name analysis methods analyze domain registration information and DNS records to determine their credibility. However, blacklist detection methods cannot be used for domain name analysis, exhibiting significant limitations. Therefore, when new malicious websites or phishing techniques emerge, it is often necessary to readjust the identification and analysis strategies, resulting in considerable limitations. Summary of the Invention
[0003] This application provides a webpage analysis and identification method, apparatus, and electronic device that can adapt to different types of malicious spoofing methods without relying on a specific type of URL.
[0004] The technical solution of this application embodiment is as follows:
[0005] In a first aspect, embodiments of this application provide a webpage analysis and identification method, the method comprising:
[0006] Obtain the URL of the webpage to be tested;
[0007] The webpage content to be detected is crawled based on the URL of the webpage to be detected, and the webpage content to be detected includes source code data and image data;
[0008] The source code data and the image data are preprocessed to obtain the structural feature data of the page, and the structural feature data of the page is matched one by one with the keywords of the preset FOFA rules to obtain the correspondence relationship;
[0009] The FOFA rules are loaded into the analysis module of the preset AC algorithm, and the structural feature data is used as input data. The analysis module is used to match and analyze the input data with the key values corresponding to the keywords, and output the matching rule identifier.
[0010] Based on the matching rule identifier, relevant description information of the rule identifier is matched from a preset database, and it is determined whether the URL is malicious based on the relevant description information.
[0011] In the above technical solution, the URL of the webpage to be detected is first obtained to provide data support for subsequent webpage detection. The content of the webpage to be detected, including source code data and image data, is crawled based on the URL. This webpage information is used for subsequent data preprocessing to obtain structural feature data for analysis and identification. The source code data and image data are preprocessed separately to obtain the page's structural feature data, which is then matched one-to-one with keywords of preset FOFA rules to establish a correspondence. Since the webpage development format is the same and does not depend on specific types of URLs, matching with preset FOFA rules reduces the limitations of specific types and facilitates subsequent analysis. The FOFA rules are loaded into the analysis module of the preset AC algorithm, using the structural feature data as input. The analysis module matches the input data with the key values corresponding to the keywords, outputting matching rule identifiers. Using the AC algorithm for analysis allows for adaptation to different types of malicious spoofing methods. Based on the matching rule identifiers, relevant descriptive information is matched from a preset database to determine whether the URL is malicious, improving parsing efficiency and accuracy.
[0012] In some embodiments of this application, loading the FOFA rule into the analysis module of a preset AC algorithm, using the structural feature data as input data, and using the analysis module to perform matching analysis between the input data and the key values corresponding to the keywords, and outputting matching rule identifiers, includes:
[0013] Key features are extracted from the FOFA rules to obtain the pattern string;
[0014] Insert the pattern string into the Trie tree of the AC algorithm, and set the fail pointer corresponding to the pattern string to construct the AC automaton;
[0015] The analysis module extracts the associated data from the input data that is related to the pattern string, and performs matching analysis between the output data of the AC automaton and the associated data to output the matching rule identifier.
[0016] In the above technical solution, corresponding association features are extracted from the keyword fields in the FOFA rules to obtain a pattern string. This pattern string is then inserted into the Trie tree of the AC algorithm. Following the order and format of the pattern string, characters are inserted sequentially into the Trie tree, with a fail pointer set for each node during insertion. This allows for tree traversal via the fail pointer, thus constructing the AC automaton. The analysis module then extracts the association data related to the pattern string from the input data and receives the output data from the AC automaton for analysis and matching to obtain the matching rule identifier. By employing the AC algorithm for analysis, it can adapt to different types of malicious spoofing methods.
[0017] In some embodiments of this application, the step of using the analysis module to extract the associated data in the input data that is related to the pattern string, and performing matching analysis between the output data of the AC automaton and the associated data to output the matching rule identifier includes:
[0018] The analysis module is used to capture the position information of the pattern string in the output data of the AC automaton;
[0019] Based on the location information, the FOFA rule corresponding to the associated data is determined, and the matching rule identifier is obtained.
[0020] In the above technical solution, the output data of the AC automaton is captured through the interface of the analysis module for data analysis. Specifically, the position information of the pattern string in the output data is captured, and this position information can be obtained through the position indicated by the fail pointer to obtain the position information of the matching string. By obtaining the position information, the position of the key feature in the FOFA rules can be determined, and each FOFA rule can be matched at the corresponding position to obtain the matching rule identifier. By using the AC algorithm for analysis, it can adapt to different types of malicious spoofing methods.
[0021] In some embodiments of this application, the preprocessing of the source code data and the image data to obtain the structural feature data of the page includes:
[0022] The image data is processed using the phash algorithm to generate phash data.
[0023] The source code data is transformed into DOM structure data to generate code hash values;
[0024] The phash data and the code hash value are stored to obtain the structural feature data of the page.
[0025] In the above technical solution, image data is generated using the image processing function integrated into the phash algorithm in the Pillow library, resulting in phash data. The HTML and JavaScript in the source code are parsed, and the built-in DOM parser is used to transform them into DOM structure data, generating code hash values. The phash data and code hash values are stored separately in a file according to the generated hash format, yielding the page's structural feature data, which can then be used for subsequent analysis and identification of page content.
[0026] In some embodiments of this application, the relevant descriptive information includes the victim and victim classification;
[0027] The step of matching relevant descriptive information of the rule identifier from a preset database according to the matching rule identifier includes:
[0028] Extract the rule content corresponding to the FOFA rule corresponding to the rule identifier;
[0029] The rule content is compared one by one with the rule information in the database to obtain the victim and the victim classification to which the victim belongs.
[0030] In the above technical solution, a string extraction algorithm is used to extract the rule content corresponding to the FOFA rule corresponding to the rule identifier. The extracted rule content is then compared one by one with the rule information in the database. This allows for the matching of all information of the FOFA rule corresponding to the rule content in the database, resulting in the victim and the victim category to which the victim belongs. The victim indicates content that cannot be contained in the webpage, and the victim category to which the victim belongs is illustrated in the diagram, so as to facilitate the subsequent identification of malicious URLs.
[0031] In some embodiments of this application, determining whether a URL is malicious based on the relevant descriptive information includes:
[0032] If the relevant description information matches a preset specific keyword, it is determined to be a malicious URL;
[0033] If none of the relevant descriptive information matches the preset specific keywords, the URL is determined to be a non-malicious URL.
[0034] In the above technical solution, if the relevant description information matches the preset specific keywords, that is, if content that cannot be included is matched, it is determined to be a malicious website; if the relevant description information does not match the preset specific keywords, that is, if no content that cannot be included is matched, it is determined to be a non-malicious website, thus enabling accurate identification of the website address to be detected.
[0035] In some embodiments of this application, after determining whether a URL is malicious based on the relevant descriptive information, the method further includes:
[0036] If the URL is identified as malicious, the URL of the webpage to be detected corresponding to the malicious URL is added to the blacklist, and an alarm message is sent to notify the user or administrator.
[0037] In the above technical solution, since malicious URLs can easily deceive people or infect their electronic devices with malware, once a malicious URL is identified, the corresponding webpage URL to be detected is added to a blacklist to prevent accidental clicks. Furthermore, an alert message is sent to the user or administrator to serve as a warning and prevent them from being scammed.
[0038] Secondly, embodiments of this application provide a webpage analysis and identification device, the device comprising:
[0039] The data acquisition module is used to obtain the URL of the webpage to be detected;
[0040] The webpage acquisition module is used to crawl the content of the webpage to be detected based on the URL of the webpage to be detected. The webpage content includes source code data and image data.
[0041] The data processing module is used to preprocess the source code data and the image data respectively to obtain the structural feature data of the page, and to match the structural feature data of the page with the keywords of the preset FOFA rules to obtain the correspondence relationship;
[0042] The analysis and matching module is used to load the FOFA rules into the analysis module of the preset AC algorithm, and use the structural feature data as input data. The analysis module performs matching analysis on the input data and the key values corresponding to the keywords, and outputs the matching rule identifier.
[0043] The analysis and identification module is used to match relevant description information of the matching rule identifier from a preset database, and determine whether it is a malicious website based on the relevant description information.
[0044] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, a user interface, a communication bus, and a network interface. The processor, the memory, the user interface, and the network interface are respectively connected to the communication bus. The memory is used to store instructions. The user interface and the network interface are used to communicate with other devices. The processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method described in any one of the first aspects.
[0045] Fourthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed, perform the method described in any one of the methods provided in the first aspect above.
[0046] In summary, one or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:
[0047] 1. This method first obtains the URL of the webpage to be detected, providing data support for subsequent webpage detection. Then, it crawls the content of the webpage to be detected, including source code data and image data. This webpage information is used for subsequent data preprocessing to obtain structural feature data for analysis and identification. The source code data and image data are preprocessed separately to obtain the page's structural feature data, which is then matched one-to-one with keywords in the preset FOFA rules to establish a correspondence. Since the webpage development format is the same, it does not depend on specific types of URLs. By matching it with the preset FOFA rules, the limitations of specific types are reduced, facilitating subsequent analysis. The FOFA rules are loaded into the preset AC algorithm analysis module, and the structural feature data is used as input data. The analysis module matches the input data with the key values corresponding to the keywords, outputting matching rule identifiers. By using the AC algorithm for analysis, it can adapt to different types of malicious spoofing methods. Based on the matching rule identifiers, relevant descriptive information of the rule identifiers is matched from a preset database. Based on the relevant descriptive information, it determines whether it is a malicious URL, improving parsing efficiency and accuracy. Therefore, it effectively solves the problem of related technologies requiring readjustment of identification and analysis strategies and having significant limitations.
[0048] 2. FOFA rules can be dynamically updated to maintain the accuracy and timeliness of the identification method.
[0049] 3. Adding users to a blacklist and notifying them or the administrator can serve as a warning. Attached Figure Description
[0050] Figure 1 This is one of the flowcharts illustrating a webpage analysis and recognition method provided in an embodiment of this application;
[0051] Figure 2 yes Figure 1 A flowchart illustrating a sub-step of step S300;
[0052] Figure 3 yes Figure 1 A flowchart illustrating another sub-step of step S400;
[0053] Figure 4 yes Figure 3A flowchart illustrating a sub-step of step S430 in the middle section;
[0054] Figure 5 yes Figure 1 A flowchart illustrating a sub-step of step S500;
[0055] Figure 6 This is a schematic diagram of the structure of a webpage analysis and recognition device provided in one embodiment of this application;
[0056] Figure 7 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation
[0057] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.
[0058] In the description of the embodiments of this application, the words "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design that is described as "for example" or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options. Rather, the use of the words "for example" or "for instance" is intended to present the relevant concepts in a specific manner.
[0059] In the description of the embodiments of this application, the term "multiple" means two or more. For example, multiple systems means two or more systems, and multiple screen terminals means two or more screen terminals. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.
[0060] This application provides a webpage analysis and identification method, apparatus, electronic device, and readable storage medium. The method first obtains the URL of the webpage to be detected, providing data support for subsequent webpage detection. It then crawls the content of the webpage to be detected based on the URL. The webpage content includes source code data and image data. By obtaining webpage information, subsequent data preprocessing is performed to obtain structural feature data for analysis and identification. The source code data and image data are preprocessed separately to obtain the page's structural feature data. This structural feature data is then matched one-to-one with keywords of preset FOFA rules to obtain a correspondence. Since webpages have the same development format and do not depend on specific types of URLs, the correspondence with preset FOFA rules reduces the limitations of specific types and facilitates subsequent analysis. The FOFA rules are loaded into the analysis module of a preset AC algorithm, and the structural feature data is used as input data. The analysis module matches the input data with the key values corresponding to the keywords, outputting matching rule identifiers. By using the AC algorithm for analysis, it can adapt to different types of malicious spoofing methods. Based on the matching rule identifiers, relevant descriptive information of the rule identifiers is matched from a preset database. Based on the relevant descriptive information, it is determined whether the URL is malicious, improving parsing efficiency and accuracy.
[0061] It should be noted that this webpage analysis and identification method can be applied when users visit webpages. It can analyze webpage content and domain information to identify malicious webpages, and then block and warn users. It can also be applied to search engines to prevent users from being deceived in search results. It can also be used for network security monitoring, mobile applications, etc. It is not dependent on specific types of URLs and can adapt to different types of malicious imitation methods.
[0062] The technical solutions provided in the embodiments of this application will be further described below with reference to the accompanying drawings.
[0063] Reference Figure 1 , Figure 1 This is a schematic flowchart of the webpage analysis and recognition method provided in this application embodiment. The webpage analysis and recognition method is applied to a webpage analysis and recognition device, and is executed by a processor in an electronic device or a readable storage medium. The webpage analysis and recognition method includes steps S100, S200, S300, S400, and S500.
[0064] Step S100: Obtain the URL of the webpage to be detected.
[0065] In one embodiment, the URL of the webpage to be detected includes the protocol, server name, domain name, path, port number, and query parameters; some may also include anchor points. The URL of the webpage to be detected can be obtained by configuring an interface in the browser and reading the URL information from the browser's address bar during a search. Alternatively, the URL can be obtained from bookmarks or favorites. Obtaining the URL of the webpage to be detected provides data support for subsequent webpage analysis and identification.
[0066] Step S200: Crawl the content of the webpage to be detected based on the URL of the webpage to be detected. The content of the webpage to be detected includes source code data and image data.
[0067] In one embodiment, the webpage content to be detected includes source code data and image data. The source code data includes style code data and HTML code data. In the source code data of the webpage design, style data includes JavaScript code, CSS code, PHP language code, etc., while HTML code data includes the header, body, and footer, etc., and includes numerous tags. Image data can also be inserted into the HTML code data. The webpage content to be detected may also consist only of source code data or only of image data. Based on the URL of the webpage to be detected, a preset crawling algorithm is used to crawl the content of the webpage to be detected. By obtaining the above webpage content, subsequent analysis can be performed to determine whether the webpage content to be detected is malicious.
[0068] It should be noted that there can be one or more URLs of the webpage to be detected. When there are multiple URLs, each URL corresponds to one webpage content to be detected, which can realize the analysis and identification of webpages corresponding to multiple URLs.
[0069] Step S300: Preprocess the source code data and image data respectively to obtain the structural feature data of the page, and match the structural feature data of the page with the keywords of the preset FOFA rules to obtain the correspondence relationship.
[0070] In one embodiment, preprocessing includes data cleaning and data format conversion. Source code data and image data are preprocessed separately. Specifically, source code data undergoes decommenting and data format conversion. Image data undergoes size conversion, pixel conversion, and format conversion. These preprocessing steps yield the page's structural feature data. FOFA is a secure web search engine. FOFA rules are rules conforming to the FOFA syntax of this search engine, and each FOFA rule includes multiple different keywords. The page's structural feature data includes hash sequences and their corresponding values. Keywords are hash sequences. The page's structural feature data is mapped one-to-one with the keywords of the preset FOFA rules to obtain the correspondence. For example, an FOFA rule could be (title=="Login"&&body="Open an Account Now")||(title=="Member Login"&&body="Open an Account Now")&&body!="Securities Company A". This rule means that websites whose title is "Login" and whose text content contains "Open an Account Now", or websites whose title is "Member Login" and whose text content contains "Open an Account Now", but whose text information does not contain "Securities Company A", can be matched with this rule. The keywords in this rule are title, body, etc. Specifically, the keywords in the page's structural feature data are matched one-to-one with the keywords in the FOFA rule. Since the webpage development format is the same, it does not depend on a specific type of URL. By matching it with the preset FOFA rule, the limitations of specific types are reduced, facilitating subsequent analysis.
[0071] For example, the FOFA rule can also be specifically defined as: Rule content: js_name="static / js / jiami.js"||js_md5="e4adae8a67a04492b39c42ebcafd2417"; Victim: None. Example 2, the FOFA rule can also be specifically defined as: Rule content: body="B Securities Company"&&body="Username cannot be empty"&&body="js / demo-1.js"&&body="Online Customer Service"; Victim: B Securities Company. Example 3, the FOFA rule can also be specifically defined as: Rule content: url=".gov.cn"; Victim: Government website.
[0072] like Figure 2 As shown, the source code data and image data are preprocessed separately to obtain the structural feature data of the page, including but not limited to the following steps:
[0073] Step S310: Use the phash algorithm to generate data from the image data to obtain phash data.
[0074] In some possible embodiments of this application, the phash algorithm is a method for generating hash values for image content based on visual similarity rather than pixel-level exact match. Image data can be generated from image data using image processing functions integrated with the phash algorithm in the Pillow library to obtain phash data, which in turn yields structural feature data for the page. Alternatively, image data can be generated from image data using image processing functions integrated with the phash algorithm in the CV library to obtain phash data, which in turn yields structural feature data for the page.
[0075] Step S320: Perform DOM structure data transformation on the source code data to generate code hash values.
[0076] In some possible embodiments of this application, the HTML, JavaScript, etc. in the source code are parsed, and the DOM parser inside is used to transform them into DOM structure data to generate code hash values so that the structural feature data of the page can be obtained later.
[0077] Step S330: Store the phash data and code hash value to obtain the structural feature data of the page.
[0078] In some possible embodiments of this application, the phash data and code hash values are stored separately in a single file according to the generated hash format, resulting in structural feature data of the page. This structural feature data can then be used for subsequent analysis and identification of the page content. It should be noted that since both phash data and code hash values are represented by hashes, they are stored in the same format, can be stored in the same file, and are read in the same way.
[0079] Step S400: Load the FOFA rules into the preset AC algorithm analysis module, and use the structural feature data as input data. Use the analysis module to match and analyze the input data with the key values corresponding to the keywords, and output the matching rule identifier.
[0080] In one embodiment, loading FOFA rules into the analysis module of a preset AC algorithm combines FOFA rules with the AC algorithm, improving processing efficiency and accuracy. After combination, structural feature data is used as input data, and the analysis module matches the input data with the key values corresponding to the keywords, outputting matching rule identifiers. This approach can adapt to different types of malicious spoofing methods, automatically analyzing and improving processing efficiency.
[0081] like Figure 3As shown, FOFA rules are loaded into the pre-defined AC algorithm's analysis module, and structural feature data is used as input data. The analysis module performs matching analysis between the input data and the key values corresponding to the keywords, and outputs matching rule identifiers, including but not limited to the following steps:
[0082] Step S410: Extract key features from the FOFA rules to obtain the pattern string.
[0083] In some possible embodiments of this application, according to the FOFA rules, the FOFA rules include protocol type, port number, key features in the webpage content, etc. Key features are extracted from the FOFA rules, and these key features may include protocol, title, domain name, field features corresponding to the title, field features corresponding to the body, field features corresponding to images, and field features of related information, etc. Corresponding associated features are extracted through the corresponding keyword fields in the FOFA rules to obtain a pattern string, which can then be used to construct an automaton.
[0084] Step S420: Insert the pattern string into the Trie tree of the AC algorithm and set the fail pointer corresponding to the pattern string to construct the AC automaton.
[0085] In some possible embodiments of this application, the pattern string is inserted into the Trie tree of the AC algorithm. First, a tree structure with an empty root is constructed. Then, characters from the pattern string are inserted into the Trie tree sequentially according to their order. A fail pointer is set for each node during insertion, allowing tree traversal via the fail pointer, thus constructing the AC automaton. The AC automaton can automatically traverse strings, improving processing efficiency.
[0086] Step S430: Use the analysis module to extract the associated data in the input data that is related to the pattern string, and perform matching analysis between the output data of the AC automaton and the associated data to output the matching rule identifier.
[0087] like Figure 4 As shown, the analysis module extracts the associated data from the input data that is related to the pattern string, and performs matching analysis between the output data of the AC automaton and the associated data, outputting the matching rule identifier, including but not limited to the following steps:
[0088] Step S431: Use the analysis module to capture the position information of the pattern string in the output data of the AC automaton.
[0089] In some possible embodiments of this application, the analysis module is equipped with an interface capable of receiving the output data of the AC automaton after matching a pattern string, and automatically analyzing the output data. The analysis module can also load all FOFA rules for analysis. Based on the output data of the AC automaton constructed in step S420 after traversing the entire tree, the analysis module's interface is used to capture the output data of the AC automaton for data analysis. Specifically, the position information of the pattern string in the output data is captured. This position information can be obtained through the position indicated by the fail pointer to obtain the position information of the matched string. This position information is then used for subsequent matching rule identification.
[0090] Step S432: Based on the location information, determine the FOFA rule corresponding to the associated data and obtain the matching rule identifier.
[0091] In some possible embodiments of this application, based on the location information obtained in step S431, the position of the key feature in the FOFA rules can be determined, and each FOFA rule can be matched at the corresponding position to obtain the matching rule identifier. For example, if the analysis module loads 10 FOFA rules, and based on the location information and the structural feature data of the page, it can match 3 of the rules, then the rule identifiers of these 3 matching rules are output for subsequent determination of whether it is a malicious website.
[0092] Step S500: Based on the matching rule identifier, match the relevant description information of the rule identifier from the preset database, and determine whether it is a malicious website based on the relevant description information.
[0093] In one embodiment, the matching rule identifier obtained in step S400 can identify a specific FOFA rule. A preset database stores all information about FOFA rules, including other descriptive information. For example, this other descriptive information may include the victim, victim classification, and whether the URL of the webpage to be detected in the rule has been identified as a malicious URL. The relevant descriptive information of the rule identifier is matched from the preset database. This relevant descriptive information records the matching and non-matching content of the rule. Based on the relevant descriptive information, it can be determined whether the URL is malicious, thereby achieving accurate webpage identification.
[0094] In one embodiment, the relevant descriptive information includes the victim and victim classification. For example... Figure 5 As shown, based on the matching rule identifier, relevant descriptive information of the rule identifier is matched from a preset database, including but not limited to the following steps:
[0095] Step S510: Extract the rule content corresponding to the FOFA rule corresponding to the rule identifier.
[0096] In some possible embodiments of this application, the FOFA rule corresponding to the matching rule identifier is selected, and the rule content corresponding to the FOFA rule corresponding to the rule identifier is extracted using a string extraction algorithm, so as to obtain relevant description information and thus determine whether it is a malicious website.
[0097] Step S520: Compare the rule content with the rule information in the database one by one to obtain the victim and the victim classification to which the victim belongs.
[0098] In some possible embodiments of this application, the rule content extracted in step S510 is compared one by one with the rule information in the database, thereby matching all the information of the FOFA rule corresponding to the rule content in the database, obtaining the victim and the victim category to which the victim belongs. The victim indicates content that cannot be contained in the webpage, and the victim category to which the victim belongs is an illustration, so as to determine the malicious URL in the future. For example, the FOFA rule can be (title=="Login"&&body="Open an account now")||(title=="Member login"&&body="Open an account now")&&body!="Securities Company A"). The relevant description information matched in the database shows that the victim of this rule is "Securities Company A", the victim category is "finance and securities", and the corresponding intent is "fraud".
[0099] For example, the rule information in the database can be: Rule content: js_name="static / js / jiami.js"||js_md5="e4adae8a67a04492b39c42ebcafd2417"; Victim: None; Victim category: Gambling; Intent: Online gambling. The rule information can also be specifically defined as: Rule content: body="B Securities Company"&&body="Username cannot be empty"&&body="js / demo-1.js"&&body="Online Customer Service"; Victim: B Securities Company; Victim category: Finance and Securities; Intent: Fraud. The rule information can also be specifically defined as: Rule content: url=".gov.cn"; Victim: Government website; Victim category: Government; Intent: Gray market traffic redirection.
[0100] In another embodiment, determining whether a URL is malicious based on relevant descriptive information includes, but is not limited to: determining a URL as malicious if the relevant descriptive information matches preset specific keywords; and determining a URL as non-malicious if none of the relevant descriptive information matches preset specific keywords.
[0101] Specifically, if the relevant description information matches preset keywords, i.e., if it matches content that cannot be included, it is determined to be a malicious URL. For example, the FOFA rule could be `title=="Login"&&body="Open an Account Now")||(title=="Member Login"&&body="Open an Account Now")&&body!="A Securities Company"`. If the relevant description information matches the victim as "A Securities Company", the victim category as "finance and securities", and the corresponding intent as "fraud", then the URL to be detected is a malicious URL. If none of the relevant description information matches the preset keywords, i.e., if no content that cannot be included is matched, it is determined to be a non-malicious URL. In the above rule, if the relevant description information matches the victim as not "A Securities Company", the victim category as "finance and securities", and the corresponding intent as "investment", then the URL to be detected is a non-malicious URL, thus accurately identifying the URL to be detected.
[0102] In one embodiment, after determining whether a webpage is a malicious URL based on relevant descriptive information, the webpage analysis and identification method further includes, but is not limited to: if a malicious URL is determined to be a malicious URL, adding the URL of the webpage to be detected corresponding to the malicious URL to a blacklist and sending an alarm message, the alarm message being used to notify the user or administrator.
[0103] Specifically, because malicious websites can easily deceive people or infect their electronic devices with malware, once a website is identified as malicious, its corresponding URL is added to a blacklist to prevent accidental clicks. Furthermore, an alert is sent to the user or administrator to serve as a warning and prevent them from being scammed.
[0104] like Figure 6As shown, this application embodiment provides a webpage analysis and recognition device 100. This device 100 acquires the URL of a webpage to be detected through a data acquisition module 110, providing data support for subsequent webpage detection. Then, the webpage acquisition module 120 crawls the content of the webpage to be detected based on the URL. The content includes source code data and image data. By acquiring webpage information, subsequent data preprocessing is performed to obtain structural feature data for analysis and recognition. The data processing module 130 then preprocesses the source code data and image data respectively to obtain the page's structural feature data. The page's structural feature data is then matched one-to-one with keywords of a preset FOFA rule to obtain a correspondence. Since the webpage... The development format is consistent and does not depend on specific URL types. By corresponding to preset FOFA rules, the limitations of specific types are reduced, facilitating subsequent analysis. The analysis matching module 140 loads the FOFA rules into the preset AC algorithm analysis module, using structural feature data as input data. The analysis module matches the input data with the key values corresponding to the keywords, outputting the matching rule identifier. By using the AC algorithm for analysis, it can adapt to different types of malicious spoofing methods. The analysis identification module 150 matches the relevant descriptive information of the matching rule identifier from the preset database, and determines whether it is a malicious URL based on the relevant descriptive information, improving parsing efficiency and accuracy.
[0105] It should be noted that the data acquisition module 110 is connected to the web page acquisition module 120, the web page acquisition module 120 is connected to the data processing module 130, the data processing module 130 is connected to the analysis and matching module 140, and the analysis and matching module 140 is connected to the analysis and identification module 150. The aforementioned webpage analysis and recognition method is applied to a webpage analysis and recognition device 100. The device 100 obtains the URL of the webpage to be detected, providing data support for subsequent webpage detection. It then crawls the content of the webpage to be detected based on the URL. The content includes source code data and image data. By obtaining webpage information, it performs subsequent data preprocessing to obtain structural feature data for analysis and recognition. The source code data and image data are preprocessed separately to obtain the page's structural feature data. This structural feature data is then matched one-to-one with the keywords of preset FOFA rules to establish a correspondence. Since webpages have the same development format and do not depend on specific types of URLs, the correspondence with preset FOFA rules reduces the limitations of specific types and facilitates subsequent analysis. The FOFA rules are loaded into the analysis module of a preset AC algorithm, using the structural feature data as input. The analysis module matches the input data with the key values corresponding to the keywords, outputting matching rule identifiers. By using the AC algorithm, it can adapt to different types of malicious spoofing methods. Based on the matching rule identifiers, relevant descriptive information of the rule identifiers is matched from a preset database. Based on the relevant descriptive information, it determines whether the URL is malicious, improving parsing efficiency and accuracy.
[0106] It should also be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.
[0107] This application also discloses an electronic device. (See reference...) Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. The electronic device 500 may include: at least one processor 501, at least one network interface 504, a user interface 503, a memory 505, and at least one communication bus 502.
[0108] The communication bus 502 is used to enable communication between these components.
[0109] The user interface 503 may include a display screen and a camera. Optionally, the user interface 503 may also include a standard wired interface and a wireless interface.
[0110] The network interface 504 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).
[0111] The processor 501 may include one or more processing cores. The processor 501 connects to various parts of the server using various interfaces and lines, and performs various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 505, and by calling data stored in memory 505. Optionally, the processor 501 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 501 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content to be displayed on the screen; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 501 and may be implemented as a separate chip.
[0112] The memory 505 may include random access memory (RAM) or read-only memory. Optionally, the memory 505 may include a non-transitory computer-readable storage medium. The memory 505 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 505 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 505 may also be at least one storage device located remotely from the aforementioned processor 501. (Refer to...) Figure 7The memory 505, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application program for web page analysis and recognition.
[0113] exist Figure 7 In the illustrated electronic device 500, the user interface 503 is mainly used to provide an input interface for the user and to acquire user input data; while the processor 501 can be used to call an application program storing a web page analysis and recognition method in the memory 505. When executed by one or more processors 501, the electronic device 500 performs one or more methods as described in the above embodiments. It should be noted that, for the foregoing method embodiments, for the sake of simplicity, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.
[0114] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.
[0115] In the various embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between apparatuses or units may be electrical or other forms.
[0116] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0117] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0118] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, portable hard drives, magnetic disks, or optical disks.
[0119] The above are merely exemplary embodiments of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Other embodiments of this disclosure will readily conceive of those skilled in the art upon consideration of the specification and the disclosure of practical truths.
[0120] This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.
Claims
1. A webpage analysis and recognition method, characterized in that, The method includes: Obtain the URL of the webpage to be tested; The webpage content to be detected is crawled based on the URL of the webpage to be detected, and the webpage content to be detected includes source code data and image data; The image data is processed using the phash algorithm to generate phash data. The source code data is then transformed into DOM structure data to generate code hash values. The phash data and the code hash values are stored to obtain the structural feature data of the webpage. The structural feature data of the webpage is then mapped one-to-one with the keywords of the preset FOFA rules to obtain the correspondence. Here, the phash algorithm is a hash value used to generate image content, and the FOFA rules are rules that conform to the FOFA syntax of the web search engine. The FOFA rules are loaded into the analysis module of the preset AC algorithm, and the structural feature data is used as input data. The analysis module is used to match and analyze the input data with the key values corresponding to the keywords, and output the matching rule identifier. Specifically, the FOFA rules are loaded into the analysis module of a preset AC algorithm, and the structural feature data is used as input data. The analysis module performs matching analysis between the input data and the key values corresponding to the keywords, and outputs matching rule identifiers, including: Key features are extracted from the FOFA rules to obtain the pattern string; Insert the pattern string into the Trie tree of the AC algorithm, and set the fail pointer corresponding to the pattern string to construct the AC automaton; The analysis module is used to extract the associated data with the pattern string from the input data, and the output data of the AC automaton is matched and analyzed with the associated data to output the matching rule identifier. The process involves using an analysis module to extract associated data from the input data that is related to the pattern string, and performing matching analysis between the output data of the AC automaton and the associated data to output a matching rule identifier. This includes: using the analysis module to capture the position information of the pattern string in the output data of the AC automaton; determining the position of the FOFA rule corresponding to the key feature based on the position information; matching each FOFA rule at the position to determine the FOFA rule corresponding to the associated data, and obtaining the matching rule identifier. Based on the matching rule identifier, relevant description information of the rule identifier is matched from a preset database, and it is determined whether the URL is malicious based on the relevant description information.
2. The method according to claim 1, characterized in that, The relevant descriptive information includes the victim and victim classification; The step of matching relevant descriptive information of the rule identifier from a preset database according to the matching rule identifier includes: Extract the rule content corresponding to the FOFA rule corresponding to the rule identifier; The rule content is compared one by one with the rule information in the database to obtain the victim and the victim classification to which the victim belongs.
3. The method according to claim 1, characterized in that, The step of determining whether a URL is malicious based on the relevant descriptive information includes: If the relevant description information matches a preset specific keyword, it is determined to be a malicious URL; If none of the relevant descriptive information matches the preset specific keywords, the URL is determined to be a non-malicious URL.
4. The method according to claim 3, characterized in that, After determining whether a URL is malicious based on the relevant descriptive information, the method further includes: If the URL is identified as malicious, the URL of the webpage to be detected corresponding to the malicious URL is added to the blacklist, and an alarm message is sent to notify the user or administrator.
5. A webpage analysis and recognition device, characterized in that, The device includes: The data acquisition module (110) is used to acquire the URL of the webpage to be detected; The webpage acquisition module (120) is used to crawl the content of the webpage to be detected based on the URL of the webpage to be detected. The webpage content includes source code data and image data. The data processing module (130) is used to generate data from the image data using the phash algorithm to obtain phash data, perform DOM structure data conversion on the source code data to generate code hash values, store the phash data and the code hash values to obtain the structural feature data of the webpage, and match the structural feature data of the webpage with the keywords of the preset FOFA rules to obtain the correspondence relationship. The phash algorithm is a hash value used to generate image content, and the FOFA rules are rules that conform to the FOFA syntax of the web search engine. The analysis and matching module (140) is used to load the FOFA rules into the analysis module of the preset AC algorithm, and use the structural feature data as input data. The analysis module performs matching analysis on the input data and the key values corresponding to the keywords, and outputs a matching rule identifier. Loading the FOFA rules into the analysis module of the preset AC algorithm, and using the structural feature data as input data, and performing matching analysis on the input data and the key values corresponding to the keywords, and outputting a matching rule identifier includes: extracting key features from the FOFA rules to obtain a pattern string; inserting the pattern string into the Trie tree of the AC algorithm, and setting the fail index corresponding to the pattern string. A needle is used to construct an AC automaton; an analysis module is used to extract associated data from the input data that is related to the pattern string, and the output data of the AC automaton is matched and analyzed with the associated data to output a matching rule identifier; wherein, the process of extracting associated data from the input data that is related to the pattern string and matching and analyzing the output data of the AC automaton with the associated data to output a matching rule identifier includes: using the analysis module to capture the position information of the pattern string in the output data of the AC automaton; determining the position of the FOFA rule corresponding to the key feature based on the position information; matching each FOFA rule at the position to determine the FOFA rule corresponding to the associated data, and obtaining the matching rule identifier; The analysis and identification module (150) is used to match the relevant description information of the rule identifier from a preset database according to the matching rule identifier, and determine whether it is a malicious website based on the relevant description information.
6. An electronic device, characterized in that, The device includes a processor (501), a memory (505), a user interface (503), a communication bus (502), and a network interface (504). The processor (501), the memory (505), the user interface (503), and the network interface (504) are respectively connected to the communication bus (502). The memory (505) is used to store instructions. The user interface (503) and the network interface (504) are used to communicate with other devices. The processor (501) is used to execute the instructions stored in the memory (505) so that the electronic device (500) performs the method as described in any one of claims 1-4.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed, perform the method as described in any one of claims 1-4.