Webpage analysis and identification method and device and electronic equipment

Abstract

The invention provides a webpage analysis and identification method and device and electronic equipment, and relates to the field of network security. The method comprises the following steps: acquiring a website of a webpage to be detected; crawling to-be-detected webpage content according to the website of the to-be-detected webpage, wherein the to-be-detected webpage content comprises source code data and picture data; preprocessing the source code data and the picture data to obtain structural feature data of the page, and enabling the structural feature data of the page to be in one-to-one correspondence with keywords of a preset FOFA rule to obtain a corresponding relation; loading the FOFA rule to an analysis module of a preset AC algorithm, taking the structural feature data as input data, performing matching analysis on the input data and a key value corresponding to the keyword by using the analysis module, and outputting a matched rule identifier; according to the matched rule identifier, the related description information of the rule identifier is matched from the preset database, whether the website is the malicious website or not is determined according to the related description information, and the method can adapt to different types of websites.

Description

Technical Field

[0001] The present application relates to the field of network security technology, and specifically to a web page analysis and identification method, device and electronic equipment. Background Art

[0002] With the rapid development of the Internet, malicious URLs and counterfeit web pages emerge in an endless stream, posing a serious threat to users' information security and privacy. In the related art, specific analysis and identification strategies are usually adopted for specific types of malicious URLs. For example, blacklist checks are adopted to compare specific types of URLs with pre-collected malicious URL blacklists for detection; or domain name analysis methods are used to analyze domain registration information, resolution records, etc. to determine their credibility. The blacklist detection of the above method cannot be used for domain name analysis and has strong limitations. Therefore, when new malicious URLs or counterfeiting methods appear, it is often necessary to readjust the identification and analysis strategies, which has great limitations. Summary of the invention

[0003] The present application provides a web page analysis and identification method, device and electronic device, which are independent of a specific type of website address and can adapt to different types of malicious counterfeiting methods.

[0004] The technical solution of the embodiment of the present application is as follows: In a first aspect, an embodiment of the present application provides a web page analysis and identification method, the method comprising: Get the URL of the web page to be detected; Crawl the webpage content to be detected according to the webpage URL to be detected, wherein the webpage content to be detected includes source code data and image data; Preprocessing the source code data and the image data respectively to obtain structural feature data of the page, and making one-to-one correspondence between the structural feature data of the page and the keywords of the preset FOFA rule to obtain a corresponding relationship; Loading the FOFA rule into the analysis module of the preset AC algorithm, and taking the structural feature data as input data, using the analysis module to match and analyze the input data with the key value corresponding to the keyword, and outputting a matching rule identifier; According to the matched rule identifier, relevant description information of the rule identifier is matched from a preset database, and whether it is a malicious website is determined according to the relevant description information.

[0005] In the above technical scheme, the URL of the web page to be detected is first obtained to provide data support for subsequent web page detection; the web page content to be detected is crawled according to the web page URL to be detected, and the web page content to be detected includes source code data and image data. By obtaining web page information, data preprocessing is performed subsequently to obtain structural feature data for analysis and identification; the source code data and image data are preprocessed respectively to obtain the structural feature data of the page, and the structural feature data of the page is matched one by one with the keywords of the preset FOFA rule to obtain a corresponding relationship. Since the web page development format is the same and does not depend on a specific type of URL, the limitation of the specific type is reduced by matching with the preset FOFA rule, which is convenient for subsequent analysis; the FOFA rule is loaded into the analysis module of the preset AC algorithm, and the structural feature data is used as input data. The analysis module is used to match and analyze the input data with the key value corresponding to the keyword, and the matching rule identifier is output. By using the AC algorithm for analysis, it can adapt to different types of malicious counterfeiting methods; according to the matched rule identifier, the relevant description information of the rule identifier is matched from the preset database, and whether it is a malicious URL is determined according to the relevant description information, thereby improving the parsing efficiency and accuracy.

[0006] In some embodiments of the present application, the FOFA rule is loaded into the analysis module of the preset AC algorithm, and the structural feature data is used as input data, and the input data is matched and analyzed with the key value corresponding to the keyword by the analysis module, and the matching rule identifier is output, including: Extract key features from the FOFA rules to obtain a pattern string; Insert the pattern string into the Trie tree of the AC algorithm, set the fail pointer corresponding to the pattern string, and construct an AC automaton; The analysis module is used to extract the associated data associated with the pattern string from the input data, and the output data of the AC automaton is matched and analyzed with the associated data to output the matching rule identifier.

[0007] In the above technical solution, the corresponding associated features are extracted through the corresponding keyword fields in the FOFA rules to obtain the pattern string, and the pattern string is inserted into the Trie tree of the AC algorithm. According to the sequential format of the pattern string, the characters of the pattern string are inserted into the Trie tree in sequence, and the fail pointer of each node is set during the insertion, so that the tree can be traversed through the fail pointer, thereby constructing an AC automaton; then the analysis module is used to extract the associated data associated with the pattern string in the input data, and the output data of the AC automaton is received for analysis and matching to obtain the matching rule identifier. By using the AC algorithm for analysis, it can adapt to different types of malicious counterfeiting methods.

[0008] In some embodiments of the present application, the extracting associated data associated with the pattern string from the input data by using the analysis module, performing matching analysis on the output data of the AC automaton and the associated data, and outputting the matching rule identifier includes: Using the analysis module to capture the position information of the pattern string in the output data of the AC automaton; The FOFA rule corresponding to the associated data is determined according to the location information, and the matching rule identifier is obtained.

[0009] In the above technical solution, the output data of the AC automaton is captured by the interface of the analysis module for data analysis. Specifically, by capturing the position information of the pattern string in the output data, the position information can be used to obtain the position information of the matching string through the indicated position of the fail pointer. By obtaining the position information, the position of the key feature in the FOFA rule can be determined, and each FOFA rule can be matched at the corresponding position to obtain the matching rule identifier. By using the AC algorithm for analysis, it can adapt to different types of malicious counterfeiting methods.

[0010] In some embodiments of the present application, the source code data and the image data are preprocessed separately to obtain structural feature data of the page, including: Using a phash algorithm to generate data for the image data to obtain phash data; Performing DOM structure data conversion on the source code data to generate a code hash value; The phash data and the code hash value are stored to obtain structural feature data of the page.

[0011] In the above technical solution, the image data is generated by the image processing function integrated by the phash algorithm in the pillow library to obtain phash data. The HTML, JavaScript, etc. in the source code are parsed, and the DOM structure data is converted using the internal DOM parser to generate a code hash value. The phash data and the code hash value are stored in a file according to the generated hash format to obtain the structural feature data of the page, so that the structural feature data of the page can be used to analyze and identify the page content later.

[0012] In some embodiments of the present application, the relevant description information includes the victim and the classification of the victim; The matching of the relevant description information of the rule identifier from a preset database according to the matched rule identifier includes: Extracting the rule content corresponding to the FOFA rule corresponding to the rule identifier; The rule content is compared one by one with the rule information in the database to obtain the victim and the victim classification to which the victim belongs.

[0013] In the above technical scheme, a string extraction algorithm is used to extract the rule content corresponding to the FOFA rule corresponding to the rule identifier, and the extracted rule content is compared one by one with the rule information in the database, so as to match all the information of the FOFA rule corresponding to the rule content in the database, and obtain the victim and the victim classification to which the victim belongs, the content that cannot be included in the victim indication web page, and the victim classification indication map to which the victim belongs, so as to subsequently determine the malicious URL.

[0014] In some embodiments of the present application, determining whether the website is a malicious website according to the relevant description information includes: When the relevant description information matches a preset specific keyword, the website is determined to be a malicious website; When the related description information does not match the preset specific keywords, it is determined to be a non-malicious website.

[0015] In the above technical solution, when the relevant description information meets the preset specific keywords, that is, it matches the content that cannot be included, and it is determined to be a malicious URL; when the relevant description information does not meet the preset specific keywords, that is, it does not match the content that cannot be included, it is determined to be a non-malicious URL, and the URL of the web page to be detected can be accurately identified.

[0016] In some embodiments of the present application, after determining whether the website is a malicious website according to the relevant description information, the method further includes: In the case where it is determined to be the malicious website, the web page URL to be detected corresponding to the malicious website is added to the blacklist, and an alarm message is sent, wherein the alarm message is used to prompt the user or administrator.

[0017] In the above technical solution, since malicious URLs can easily deceive people or cause the electronic devices they use to be poisoned, when a malicious URL is determined, the URL of the web page to be detected corresponding to the malicious URL is added to the blacklist to avoid missed clicks. In addition, an alarm message is sent to the user or administrator to serve as a warning to avoid being deceived.

[0018] In a second aspect, an embodiment of the present application provides a web page analysis and identification device, the device comprising: A data acquisition module is used to obtain the URL of the web page to be detected; A webpage acquisition module, used to crawl the webpage content to be detected according to the URL of the webpage to be detected, wherein the webpage content includes source code data and image data; A data processing module, used to pre-process the source code data and the image data respectively to obtain structural feature data of the page, and to correspond the structural feature data of the page to the keywords of the preset FOFA rule one by one to obtain a corresponding relationship; An analysis and matching module, used to load the FOFA rule into an analysis module of a preset AC algorithm, and use the structural feature data as input data, use the analysis module to match and analyze the input data with the key value corresponding to the keyword, and output a matching rule identifier; The analysis and identification module is used to match the relevant description information of the rule identifier from a preset database according to the matched rule identifier, and determine whether it is a malicious website according to the relevant description information.

[0019] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, a user interface, a communication bus and a network interface, wherein the processor, the memory, the user interface and the network interface are respectively connected to the communication bus, the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes any one of the methods provided in the first aspect.

[0020] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, and when the instructions are executed, any one of the methods provided in the first aspect is executed.

[0021] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. The method first obtains the URL of the web page to be detected to provide data support for subsequent web page detection; crawls the content of the web page to be detected according to the URL of the web page to be detected, and the content of the web page to be detected includes source code data and image data. By obtaining web page information, data preprocessing is performed to obtain structural feature data for analysis and identification; the source code data and image data are preprocessed respectively to obtain the structural feature data of the page, and the structural feature data of the page is matched one by one with the keywords of the preset FOFA rules to obtain a corresponding relationship. Since the web page development format is the same and does not depend on a specific type of URL, the limitations of the specific type are reduced by matching with the preset FOFA rules, which is convenient for subsequent analysis; the FOFA rules are loaded into the analysis module of the preset AC algorithm, and the structural feature data is used as input data. The analysis module matches the input data with the key value corresponding to the keyword and outputs the matching rule identifier. By using the AC algorithm for analysis, it can adapt to different types of malicious counterfeiting methods; according to the matching rule identifier, the relevant description information of the rule identifier is matched from the preset database, and whether it is a malicious URL is determined according to the relevant description information, thereby improving the parsing efficiency and accuracy. Therefore, the problem that the identification and analysis strategies need to be readjusted in the related technology and the limitations are large is effectively solved.

[0022] 2. FOFA rules can be updated dynamically to maintain the accuracy and timeliness of the identification method.

[0023] 3. By adding to the blacklist and notifying the user or administrator, it can serve as a warning. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is one of the flow charts of a web page analysis and identification method provided by an embodiment of the present application; Figure 2 yes Figure 1 A schematic flow chart of a sub-step of step S300; Figure 3 yes Figure 1 Another sub-step flow chart of step S400; Figure 4 yes Figure 3 A schematic flow chart of a sub-step of step S430; Figure 5 yes Figure 1 A schematic flow chart of a sub-step of step S500; Figure 6 It is a structural diagram of a web page analysis and identification device provided by an embodiment of the present application; Figure 7 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to enable technicians in this field to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0026] In the description of the embodiments of the present application, words such as "for example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "for example" or "for example" is intended to present related concepts in a specific way.

[0027] In the description of the embodiments of the present application, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0028] The embodiment of the present application provides a web page analysis and identification method, device, electronic device and readable storage medium. The web page analysis and identification method first obtains the URL of a web page to be detected to provide data support for subsequent web page detection; crawls the web page content to be detected according to the web page URL to be detected, and the web page content to be detected includes source code data and image data, and obtains web page information so as to perform subsequent data preprocessing to obtain structural feature data for analysis and identification; preprocesses the source code data and image data respectively to obtain the structural feature data of the page, and corresponds the structural feature data of the page with the keywords of the preset FOFA rule one by one to obtain a corresponding relationship. Since the web page development format is the same and does not depend on a specific type of URL, by corresponding with the preset FOFA rule, the limitation of the specific type is reduced, and subsequent analysis is convenient; the FOFA rule is loaded into the analysis module of the preset AC algorithm, and the structural feature data is used as input data, and the input data is matched and analyzed with the key value corresponding to the keyword by the analysis module, and the matching rule identifier is output. By adopting the AC algorithm for analysis, it can adapt to different types of malicious counterfeiting means; according to the matched rule identifier, the relevant description information of the rule identifier is matched from the preset database, and whether it is a malicious URL is determined according to the relevant description information, thereby improving the parsing efficiency and accuracy.

[0029] It should be noted that when the web page analysis and identification method is applied to users visiting web pages, it can analyze the web page content and domain name information to identify malicious web pages, and then intercept and warn them; it can also be applied to search engines to avoid being defrauded when searching results; it can also be used for network security monitoring, mobile applications, etc., and it can be independent of specific types of URLs and adapt to different types of malicious counterfeiting methods.

[0030] The technical solution provided in the embodiments of the present application is further described below in conjunction with the accompanying drawings.

[0031] Reference Figure 1 , Figure 1 The web page analysis and identification method is applied to a web page analysis and identification device, and is executed by a processor in an electronic device or a readable storage medium. The web page analysis and identification method includes steps S100, S200, S300, S400, and S500.

[0032] Step S100, obtaining the URL of the web page to be detected.

[0033] In one embodiment, the webpage URL to be detected is a protocol, a server name, a domain name, a path, a port number, a query parameter, and some may also include an anchor point, etc. The webpage URL to be detected can be obtained by setting an interface in the browser and reading the URL information in the address bar of the browser using the set interface when searching in the browser. The webpage URL to be detected can also be obtained by obtaining the URL information in the favorites or bookmarks. By obtaining the webpage URL to be detected, data support is provided for subsequent webpage analysis and identification.

[0034] Step S200, crawling the content of the web page to be detected according to the URL of the web page to be detected, where the content of the web page to be detected includes source code data and image data.

[0035] In one embodiment, the web page content to be detected includes source code data and image data, the source code data includes style code data and HTML code data, in the source code data of the web page design, the style data includes JavaScript code, CSS code, PHP language code, etc., the HTML code data includes the head, body and footer, etc., and includes more tags, and image data can also be inserted into the HTML code data. The web page content to be detected may also include only source code data, or only image data. According to the URL of the web page to be detected, the web page content to be detected is crawled using a preset crawling algorithm, and the above web page content is obtained so as to analyze whether the web page content to be detected is a malicious web page.

[0036] It should be noted that the web page URL to be detected can be one or more. When there are multiple web page URLs to be detected, each web page URL to be detected corresponds to a web page content to be detected, and analysis and identification of web pages corresponding to multiple URLs can be achieved.

[0037] Step S300 , preprocessing the source code data and the image data respectively to obtain structural feature data of the page, and making one-to-one correspondence between the structural feature data of the page and the keywords of the preset FOFA rule to obtain a corresponding relationship.

[0038] In one embodiment, the preprocessing includes data cleaning and data format conversion. Data preprocessing is performed on the source code data and the image data respectively, specifically, the source code data is de-annotated, and the source code data can also be converted into a data format. The image data is converted in size, the pixels of the image are converted, and the format of the image data is converted. The structural feature data of the page is obtained by performing data preprocessing respectively. FOFA is an information security network search engine. The FOFA rules are corresponding rules that conform to the FOFA syntax of the search engine. The FOFA rules include multiple different keywords. The structural feature data of the page includes a hash sequence and its corresponding value. The keyword is a hash sequence. The structural feature data of the page is matched one by one with the keywords of the preset FOFA rules to obtain a corresponding relationship. Exemplarily, the FOFA rule can be (title=="Login"&&body="Open an account now")||(title=="Member Login"&&body="Open an account now")&&body!="A Securities Company", and the rule is interpreted as `Websites with the title "Login" and the text content of the website containing "Open an account now", or the title "Member Login" and the text content of the website containing "Open an account now", and the website text information does not contain "A Securities Company"` can match this rule, and the keywords of the above rules are title, body, etc. Specifically, the keywords in the structural feature data of the page are matched one by one with the keywords in the FOFA rule. Since the web page development format is the same and does not depend on a specific type of URL, the limitations of the specific type are reduced by matching with the preset FOFA rules, which is convenient for subsequent analysis.

[0039] Exemplarily, the FOFA rule can also be specifically defined as: rule content: js_name="static / js / jiami.js"||js_md5="e4adae8a67a04492b39c42ebcafd2417"; victim: none. Example 2, the FOFA rule can also be specifically defined as: rule content: body="B Securities Company"&&body="Username cannot be empty"&&body="js / demo-1.js"&&body="Online Customer Service"; victim: B Securities Company. Example 3, the FOFA rule can also be specifically defined as: rule content: url=".gov.cn"; victim: government official website.

[0040] like Figure 2 As shown, the source code data and the image data are preprocessed respectively to obtain the structural feature data of the page, including but not limited to the following steps: Step S310, using the phash algorithm to generate data for the image data to obtain phash data.

[0041] In some possible embodiments of the present application, the phash algorithm is a method for generating a hash value of image content based on the visual similarity of the image content rather than pixel-level exactness. The image data can be generated by the image processing function integrated by the phash algorithm in the pillow library to obtain phash data, so as to obtain the structural feature data of the page. The image data can also be generated by the image processing function integrated by the phash algorithm in the CV library to obtain phash data, so as to obtain the structural feature data of the page.

[0042] Step S320, converting the source code data into DOM structure data to generate a code hash value.

[0043] In some possible embodiments of the present application, the HTML, JavaScript, etc. in the source code are parsed, and the internal DOM parser is used to convert them into DOM structure data to generate a code hash value so as to subsequently obtain the structural feature data of the page.

[0044] Step S330, storing the phash data and the code hash value to obtain the structural feature data of the page.

[0045] In some possible embodiments of the present application, the phash data and the code hash value are stored in a file according to the generated hash format, and the structural feature data of the page is obtained, so that the structural feature data of the page can be used to analyze and identify the content of the page. It should be noted that since the phash data and the code hash value are both represented by hash, the storage format is the same, they can be stored in one file, and the method of reading data is also the same.

[0046] Step S400, loading the FOFA rules into the analysis module of the preset AC algorithm, and taking the structural feature data as input data, using the analysis module to match and analyze the input data with the key values ​​corresponding to the keywords, and outputting the matching rule identifier.

[0047] In one embodiment, the FOFA rule is loaded into the analysis module of the preset AC algorithm, and the FOFA rule can be combined with the AC algorithm to improve processing efficiency and accuracy. After the combination, the structural feature data is used as input data, and the analysis module is used to match and analyze the input data with the key value corresponding to the keyword, and the matching rule identifier is output, which can adapt to different types of malicious counterfeiting methods and automatically analyze to improve processing efficiency.

[0048] like Figure 3 As shown, the FOFA rule is loaded into the analysis module of the preset AC algorithm, and the structural feature data is used as input data. The analysis module is used to match and analyze the input data with the key values ​​corresponding to the keywords, and the matching rule identifier is output, including but not limited to the following steps: Step S410: extract key features from the FOFA rules to obtain a pattern string.

[0049] In some possible embodiments of the present application, according to the FOFA rule, the FOFA rule includes the protocol type, port number, key features in the webpage content, etc. The key features are extracted from the FOFA rule, and the key features may include the protocol, title, domain name, field features corresponding to the title, field features corresponding to the body, field features corresponding to the picture, field features of related information, etc. The corresponding associated features are extracted through the corresponding keyword fields in the FOFA rule to obtain a pattern string, so that the pattern string can be used to construct an automaton later.

[0050] Step S420, inserting the pattern string into the Trie tree of the AC algorithm, and setting the fail pointer corresponding to the pattern string, to construct an AC automaton.

[0051] In some possible embodiments of the present application, the pattern string is inserted into the Trie tree of the AC algorithm, and a tree structure with an empty root is first constructed. The characters of the pattern string are inserted into the Trie tree in sequence according to the sequential format of the pattern string, and the fail pointer of each node is set during the insertion, so that the tree is traversed through the fail pointer, thereby constructing an AC automaton. The AC automaton can automatically traverse the string, thereby improving processing efficiency.

[0052] Step S430: extract associated data associated with the pattern string from the input data using the analysis module, perform matching analysis on the output data of the AC automaton and the associated data, and output a matching rule identifier.

[0053] like Figure 4 As shown, the analysis module is used to extract the associated data associated with the pattern string in the input data, and the output data of the AC automaton is matched and analyzed with the associated data, and the matching rule identifier is output, including but not limited to the following steps: Step S431: using the analysis module to capture the position information of the pattern string in the output data of the AC automaton.

[0054] In some possible embodiments of the present application, the analysis module is provided with an interface capable of receiving the output data of the AC automaton after matching the pattern string, and automatically analyzing the output data. The analysis module can also load all FOFA rules for analysis. The AC automaton constructed according to step S420 traverses the entire tree and outputs data, and the output data of the AC automaton is captured using the interface of the analysis module for data analysis. Specifically, by capturing the position information of the pattern string in the output data, the position information can be used to obtain the position information of the matching string through the indicated position of the fail pointer, and the position information is obtained for subsequent matching rule identification.

[0055] Step S432: Determine the FOFA rule corresponding to the associated data according to the location information, and obtain a matching rule identifier.

[0056] In some possible embodiments of the present application, the position of the key feature in the FOFA rule can be determined according to the position information obtained in step S431, and each FOFA rule is matched at the corresponding position to obtain a matching rule identifier. Exemplarily, 10 FOFA rules are loaded in the analysis module, and according to the position information and the structural feature data of the page, 3 of the rules can be matched, and the rule identifiers of the 3 rules are output to determine whether it is a malicious URL later.

[0057] Step S500, according to the matched rule identifier, relevant description information of the matching rule identifier is obtained from a preset database, and whether it is a malicious website is determined according to the relevant description information.

[0058] In one embodiment, according to the matching rule identifier obtained in step S400, the rule identifier can identify a specific FOFA rule. All information of the FOFA rule is stored in a preset database, including some other description information. For example, the other description information may include the victim, the victim classification, whether the web page URL to be detected in the rule has been identified as a malicious URL, etc. The relevant description information of the matching rule identifier is matched from the preset database. The relevant description information records the rule matching content and the rule non-matching content. According to the relevant description information, it can be determined whether it is a malicious URL, thereby achieving accurate web page identification.

[0059] In one embodiment, the relevant description information includes the victim and the victim classification. Figure 5 As shown, according to the matched rule identifier, the relevant description information of the matched rule identifier is matched from the preset database, including but not limited to the following steps: Step S510: extracting the rule content corresponding to the FOFA rule corresponding to the rule identifier.

[0060] In some possible embodiments of the present application, the FOFA rule corresponding to the rule identifier is selected according to the matching rule identifier, and the rule content corresponding to the FOFA rule corresponding to the rule identifier is extracted using a string extraction algorithm, so as to subsequently obtain relevant description information to determine whether it is a malicious URL.

[0061] Step S520, comparing the rule content with the rule information in the database one by one to obtain the victim and the victim classification to which the victim belongs.

[0062] In some possible embodiments of the present application, the rule content extracted in step S510 is compared one by one with the rule information in the database, so as to match all the information of the FOFA rule corresponding to the rule content in the database, and obtain the victim and the victim classification to which the victim belongs, the content that cannot be included in the victim indication webpage, and the victim classification indication diagram to which the victim belongs, so as to facilitate the subsequent determination of malicious URLs. Exemplarily, the FOFA rule can be (title=="Login"&&body="Open an account now")||(title=="Member Login"&&body="Open an account now")&&body!="A Securities Company", and the relevant description information matched in the database shows that the victim of this rule is "A Securities Company", the victim classification is "Financial Securities", and the corresponding intention is "Fraud".

[0063] Exemplarily, the rule information in the database may be: rule content: js_name="static / js / jiami.js"||js_md5="e4adae8a67a04492b39c42ebcafd2417"; victim: none; victim classification: gambling; intention: online gambling. The rule information may also be specifically defined as: rule content: body="B Securities Company"&&body="Username cannot be empty"&&body="js / demo-1.js"&&body="Online Customer Service"; victim: B Securities Company; victim classification: financial securities; intention: fraud. The rule information may also be specifically defined as: rule content: url=".gov.cn"; victim: government official website; victim classification: government; intention: gray industry diversion.

[0064] In another embodiment, whether it is a malicious URL is determined based on relevant description information, including but not limited to: when the relevant description information meets the preset specific keywords, it is determined to be a malicious URL; when the relevant description information does not meet the preset specific keywords, it is determined to be a non-malicious URL.

[0065] Specifically, when the relevant description information meets the preset specific keywords, that is, it matches the content that cannot be included, and it is determined to be a malicious URL. Exemplarily, the FOFA rule can be title=="Login"&&body="Open an account now")||(title=="Member Login"&&body="Open an account now")&&body!="A Securities Company", the relevant description information matches that the victim is "A Securities Company", the victim's classification is "financial securities", and the corresponding intention is "fraud", then the web page URL to be detected is a malicious URL. When the relevant description information does not meet the preset specific keywords, that is, it does not match the content that cannot be included, it is determined to be a non-malicious URL. In the above rules, the relevant description information matches that the victim is not "A Securities Company", the victim's classification is "financial securities", and the corresponding intention is "investment", then the web page URL to be detected is a non-malicious URL, and the web page URL to be detected can be accurately identified.

[0066] In one embodiment, after determining whether it is a malicious URL based on relevant description information, the web page analysis and identification method also includes but is not limited to: if it is determined to be a malicious URL, the web page URL to be detected corresponding to the malicious URL is added to a blacklist and an alarm message is sent, and the alarm message is used to prompt the user or administrator.

[0067] Specifically, since malicious URLs can easily deceive people or cause their electronic devices to be infected, if a malicious URL is determined, the URL of the web page to be detected corresponding to the malicious URL will be added to the blacklist to avoid missed clicks. In addition, an alarm message will be sent to the user or administrator to serve as a warning to avoid being deceived.

[0068] like Figure 6 As shown, the embodiment of the present application provides a web page analysis and identification device 100, which obtains the URL of the web page to be detected through the data acquisition module 110 to provide data support for subsequent web page detection; then crawls the content of the web page to be detected according to the URL of the web page to be detected through the web page acquisition module 120, and the content of the web page to be detected includes source code data and image data, and obtains the web page information so as to perform subsequent data preprocessing to obtain structural feature data for analysis and identification; then uses the data processing module 130 to preprocess the source code data and the image data respectively to obtain the structural feature data of the page, and the structural feature data of the page is matched one by one with the keywords of the preset FOFA rule to obtain the corresponding relationship, because the web page The development format is the same and does not depend on a specific type of URL. By corresponding to the preset FOFA rules, the limitations of specific types are reduced, which facilitates subsequent analysis. The analysis and matching module 140 is used to load the FOFA rules into the analysis module of the preset AC algorithm, and the structural feature data is used as input data. The analysis module is used to match and analyze the input data with the key values ​​corresponding to the keywords, and the matching rule identifier is output. By using the AC algorithm for analysis, it can adapt to different types of malicious counterfeiting methods. The analysis and identification module 150 matches the relevant description information of the rule identifier from the preset database according to the matched rule identifier, and determines whether it is a malicious URL according to the relevant description information, thereby improving the parsing efficiency and accuracy.

[0069] It should be noted that the data acquisition module 110 is connected to the web page acquisition module 120 , the web page acquisition module 120 is connected to the data processing module 130 , the data processing module 130 is connected to the analysis and matching module 140 , and the analysis and matching module 140 is connected to the analysis and identification module 150 . The web page analysis and identification method is applied to a web page analysis and identification device 100. The web page analysis and identification device 100 obtains the URL of the web page to be detected to provide data support for subsequent web page detection; crawls the web page content to be detected according to the web page URL to be detected, and the web page content to be detected includes source code data and image data, and obtains web page information so as to perform subsequent data preprocessing to obtain structural feature data for analysis and identification; preprocesses the source code data and image data respectively to obtain the structural feature data of the page, and corresponds the structural feature data of the page with the keywords of the preset FOFA rule one by one to obtain a corresponding relationship. Since the web page development format is the same and does not depend on a specific type of URL, by corresponding with the preset FOFA rule, the limitation of the specific type is reduced, which is convenient for subsequent analysis; loads the FOFA rule into the analysis module of the preset AC algorithm, and uses the structural feature data as input data, uses the analysis module to match and analyze the input data with the key value corresponding to the keyword, and outputs the matching rule identifier. By using the AC algorithm for analysis, it can adapt to different types of malicious counterfeiting methods; according to the matching rule identifier, matches the relevant description information of the rule identifier from the preset database, and determines whether it is a malicious URL according to the relevant description information, thereby improving the parsing efficiency and accuracy.

[0070] It should also be noted that: when the device provided in the above embodiment realizes its function, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0071] The present application also discloses an electronic device. Figure 7 , Figure 7 The electronic device 500 may include: at least one processor 501 , at least one network interface 504 , a user interface 503 , a memory 505 , and at least one communication bus 502 .

[0072] The communication bus 502 is used to realize the connection and communication between these components.

[0073] The user interface 503 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 503 may also include a standard wired interface and a wireless interface.

[0074] The network interface 504 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0075] Among them, the processor 501 may include one or more processing cores. The processor 501 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 505, and calling data stored in the memory 505. Optionally, the processor 501 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 501 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 501, and it can be implemented separately through a chip.

[0076] Among them, the memory 505 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 505 includes a non-transitory computer-readable storage medium. The memory 505 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 505 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 505 may also be optionally at least one storage device located away from the aforementioned processor 501. Refer to Figure 7, the memory 505 as a computer storage medium may include an operating system, a network communication module, a user interface module and an application program of a web page analysis and recognition method.

[0077] exist Figure 7 In the electronic device 500 shown, the user interface 503 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 501 can be used to call an application program storing a web page analysis and recognition method in the memory 505, and when executed by one or more processors 501, the electronic device 500 executes one or more methods in the above-mentioned embodiments. It should be noted that for the aforementioned method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0078] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0079] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0080] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0081] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0082] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, mobile hard drives, magnetic disks or optical disks.

[0083] The above are only exemplary embodiments of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification and the truth of practice, those skilled in the art will easily think of other embodiments of the present disclosure.

[0084] This application is intended to cover any variation, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the art not described in the present disclosure. The description and examples are to be regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A web page analysis and identification method, characterized in that: The method comprises: Get the URL of the web page to be detected; Crawl the webpage content to be detected according to the webpage URL to be detected, wherein the webpage content to be detected includes source code data and image data; Preprocessing the source code data and the image data respectively to obtain structural feature data of the page, and making one-to-one correspondence between the structural feature data of the page and the keywords of the preset FOFA rule to obtain a corresponding relationship; Loading the FOFA rule into the analysis module of the preset AC algorithm, and taking the structural feature data as input data, using the analysis module to match and analyze the input data with the key value corresponding to the keyword, and outputting a matching rule identifier; According to the matched rule identifier, relevant description information of the rule identifier is matched from a preset database, and whether it is a malicious website is determined according to the relevant description information.

2. The method according to claim 1, characterized in that The FOFA rule is loaded into the analysis module of the preset AC algorithm, and the structural feature data is used as input data, and the input data is matched and analyzed with the key value corresponding to the keyword by the analysis module, and the matching rule identification is output, including: Extract key features from the FOFA rules to obtain a pattern string; Insert the pattern string into the Trie tree of the AC algorithm, set the fail pointer corresponding to the pattern string, and construct an AC automaton; The analysis module is used to extract the associated data associated with the pattern string from the input data, and the output data of the AC automaton is matched and analyzed with the associated data to output the matching rule identifier.

3. The method according to claim 2, characterized in that The extracting associated data associated with the pattern string from the input data by using the analysis module, performing matching analysis on the output data of the AC automaton and the associated data, and outputting the matching rule identifier, comprises: Using the analysis module to capture the position information of the pattern string in the output data of the AC automaton; The FOFA rule corresponding to the associated data is determined according to the location information, and the matching rule identifier is obtained.

4. The method according to claim 1, characterized in that: The preprocessing of the source code data and the image data respectively to obtain structural feature data of the page includes: Using a phash algorithm to generate data for the image data to obtain phash data; Performing DOM structure data conversion on the source code data to generate a code hash value; The phash data and the code hash value are stored to obtain structural feature data of the page.

5. The method according to claim 1, characterized in that The relevant description information includes the victim and the classification of the victim; The matching of the relevant description information of the rule identifier from a preset database according to the matched rule identifier includes: Extracting the rule content corresponding to the FOFA rule corresponding to the rule identifier; The rule content is compared one by one with the rule information in the database to obtain the victim and the victim classification to which the victim belongs.

6. The method according to claim 1, characterized in that The determining whether the website is malicious according to the relevant description information includes: When the relevant description information matches a preset specific keyword, it is determined to be a malicious URL; In the case that none of the related description information matches the preset specific keywords, it is determined to be a non-malicious URL.

7. The method according to claim 6, characterized in that After determining whether the website is a malicious website according to the relevant description information, the method further includes: In the case where it is determined to be the malicious website, the web page URL to be detected corresponding to the malicious website is added to the blacklist, and an alarm message is sent, wherein the alarm message is used to prompt the user or administrator.

8. A web page analysis and identification device, characterized in that: The device comprises: A data acquisition module (110), used to acquire the URL of the web page to be detected; A webpage acquisition module (120), used for crawling the content of the webpage to be detected according to the URL of the webpage to be detected, wherein the webpage content includes source code data and image data; A data processing module (130) is used to pre-process the source code data and the image data respectively to obtain structural feature data of the page, and to correspond the structural feature data of the page to the keywords of the preset FOFA rule one by one to obtain a corresponding relationship; An analysis and matching module (140) is used to load the FOFA rule into an analysis module of a preset AC algorithm, and use the structural feature data as input data, use the analysis module to perform matching analysis on the input data and the key value corresponding to the keyword, and output a matching rule identifier; The analysis and identification module (150) is used to match the relevant description information of the rule identifier from a preset database according to the matched rule identifier, and determine whether it is a malicious website according to the relevant description information.

9. An electronic device, characterized in that: The electronic device (500) comprises a processor (501), a memory (505), a user interface (503), a communication bus (502) and a network interface (504), wherein the processor (501), the memory (505), the user interface (503) and the network interface (504) are respectively connected to the communication bus (502), the memory (505) is used to store instructions, the user interface (503) and the network interface (504) are used to communicate with other devices, and the processor (501) is used to execute the instructions stored in the memory (505) so that the electronic device (500) executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 7 is performed.

Images