Creation method, detection method, device, electronic device and readable medium
By converting words and paths in web page content into binary encoding and using BERT neural network for training, the problem of generating web domain names in web penetration testing is solved, and the efficiency and accuracy of the test is improved.
Patent Information
- Application Number
- CN202210266340.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-17
- Publication Date
- 2025-05-16
- Estimated Expiration
- 2042-03-17
AI Technical Summary
When conducting web penetration testing, how to effectively generate web domain names has become an urgent problem.
By obtaining the web page content of multiple web pages, the first word and lower path in the web page content are converted into binary encoding based on the full-link neural network and sigmoid function, and these encodings are input into the BERT neural network for training to generate a web path.
It realizes the generation of web paths through neural networks, and improves the efficiency and accuracy of information collection in web penetration testing.
Smart Images

Figure CN114611043B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of penetration testing, and in particular to a creation method, a detection method, a device, an electronic device and a readable medium. Background Art
[0002] Penetration testing is to completely simulate the attack techniques and vulnerability discovery techniques that hackers may use, conduct in-depth detection of the security of the target system, and discover the most vulnerable links of the system. Penetration testing can intuitively let managers know the problems facing their network. Among them, when developing a Web-based system, it is already a necessary process to conduct Web penetration testing on the system. Web penetration testing mainly tests the security of Web applications and corresponding software and hardware equipment configurations. Because Web penetration testing simulates the behavior of hackers, usually the infiltrators are in a state of complete ignorance of the system, and conduct black box security testing on the Web system. Web attacks are not random attacks without purpose, but often attacks with certain established goals. This usually goes through the following 5 steps: information collection, scanning, attacking, implanting backdoors, eliminating traces, etc. Among them, Web domain names are a very important part of information collection. Therefore, when conducting Web penetration testing, how to generate Web domain names becomes a problem that needs to be solved urgently. Summary of the invention
[0003] In view of this, the main purpose of the present invention is to provide a creation method, a detection method, a device, an electronic device and a readable medium.
[0004] To achieve the above-mentioned purpose, the technical solution of the present invention is implemented as follows: a method for creating a neural network, comprising the following steps: obtaining web page contents corresponding to multiple web pages, and performing the following processing on each web page content: generating a binary code C1 corresponding to the web page based on all the first words in the web page content, obtaining the binary code C2 corresponding to all subordinate paths in the web page, and then mapping the binary code C1 to the probability of occurrence of each first word in the web page content based on a fully linked neural network and a sigmoid function; creating a BERT neural network, and performing the following processing on each web page: inputting the binary code C1 and the binary code C2 of the web page into the BERT neural network, and training the BERT neural network.
[0005] As an improvement of an embodiment of the present invention, the "obtaining the binary code C2 corresponding to all subordinate paths in the web page" specifically includes: performing the following processing on each subordinate path in the web page content: using the character " / " to divide the subordinate path into a number of second words, generating binary codes C3 corresponding to the number of second words, and the length of the binary code C3 is fixed; performing a bit-by-bit OR operation on the binary codes C3 corresponding to all subordinate paths in the web page content, thereby obtaining the binary code C2.
[0006] As an improvement of the embodiment of the present invention, the “generating the binary code C1 corresponding to the webpage based on all the first words in the webpage content” specifically includes: obtaining a preset vocabulary W1, W2, ..., W N , create a binary number B = b1b2...b of length N N , when W i When b is one of all the first words, i =1, otherwise, b i =0; where N and i are natural numbers, 1≤i≤N; binary number B=b1b2...b N is the binary code C1 corresponding to the webpage.
[0007] As an improvement of the embodiment of the present invention, the "generating binary codes C3 corresponding to a plurality of second words" specifically includes: obtaining a preset vocabulary W1, W2, ..., W N , create a binary number C = c1c2...c with a length of N N , when W i When c is one of the plurality of second words, i =1, otherwise, c i =0; where N and i are natural numbers, 1≤i≤N; binary number C=c1c2...c N It is the binary code C3 corresponding to the webpage.
[0008] As an improvement of the embodiment of the present invention, the loss function of the BERT neural network is: L = ∑ i -[y i ·log(pi)+(1-yi)yi·log(1-pi)].
[0009] The embodiment of the present invention also provides a neural network creation device, comprising the following modules: a first content acquisition module, used to acquire web page content corresponding to multiple web pages, and perform the following processing on each web page content: generate a binary code C1 corresponding to the web page based on all first words in the web page content, acquire the binary code C2 corresponding to all subordinate paths in the web page, and then map the binary code C1 to the probability of occurrence of each first word in the web page content based on a fully linked neural network and a sigmoid function; a network training module, used to create a BERT neural network, and perform the following processing on each web page: input the binary code C1 and the binary code C2 of the web page into the BERT neural network, and train the BERT neural network.
[0010] An embodiment of the present invention further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program executes the above-mentioned creation method when executed.
[0011] An embodiment of the present invention further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the creation method as described above.
[0012] An embodiment of the present invention also provides a Web path detection method, comprising the following steps: obtaining web page content corresponding to a web page, generating a binary code C3 corresponding to the web page based on all words in the web page content; executing the above-mentioned creation method to generate a BERT neural network, and inputting the binary code C3 into the BERT neural network to obtain a number of subordinate paths.
[0013] An embodiment of the present invention also provides a Web path detection device, comprising the following modules: a second content acquisition module, used to acquire web page content corresponding to a web page, and generate a binary code C3 corresponding to the web page based on all words in the web page content; an execution module, used to execute the above-mentioned construction method to generate a BERT neural network, and input the binary code C3 into the BERT neural network to obtain a number of subordinate paths.
[0014] The creation method, detection method, device, electronic device and readable medium provided by the embodiment of the present invention have the following advantages: The embodiment of the present invention discloses a creation method, detection method, device, electronic device and readable medium, and the creation method includes the following steps: obtaining web page contents corresponding to multiple web pages, and performing the following processing on each web page content: generating a binary code C1 corresponding to the web page based on all the first words in the web page content, obtaining the binary code C2 corresponding to all the subordinate paths in the web page, and then mapping the binary code C1 to the probability of occurrence of each first word in the web page content based on a fully linked neural network and a sigmoid function; creating a BERT neural network, and performing the following processing on each web page: inputting the binary code C1 and the binary code C2 of the web page into the BERT neural network, and training the BERT neural network. Thus, a Web path can be generated. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 A schematic diagram of a process for creating a neural network provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0016] The present invention will be described in detail below in conjunction with the embodiments shown in the accompanying drawings. However, the embodiments do not limit the present invention, and any structural, methodological, or functional changes made by a person skilled in the art based on the embodiments are all within the protection scope of the present invention.
[0017] The following description and accompanying drawings fully illustrate the specific embodiments of this article so that those skilled in the art can practice them. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. The scope of the embodiments of this article includes the entire scope of the claims, as well as all available equivalents of the claims. Herein, the terms "first", "second", etc. are only used to distinguish one element from another, without requiring or implying any actual relationship or order between these elements. In fact, the first element can also be called the second element, and vice versa. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the structure, device or equipment including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also include elements inherent to such structure, device or equipment. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the structure, device or equipment including the elements. Each embodiment is described in a progressive manner herein, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other.
[0018] The terms "longitudinal", "lateral", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. in this document indicate the orientation or position relationship based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing this document and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In the description of this document, unless otherwise specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a mechanical connection or an electrical connection, it can also be the internal communication of two elements, it can be a direct connection, or it can be an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.
[0019] Embodiment 1 of the present invention provides a method for creating a neural network, such as Figure 1 As shown, the following steps are included:
[0020] Step 101: obtaining webpage contents corresponding to a plurality of webpages, and performing the following processing on each webpage content: generating a binary code C1 corresponding to the webpage based on all first words in the webpage content, obtaining a binary code C2 corresponding to all subordinate paths in the webpage, and then mapping the binary code C1 to the probability of occurrence of each first word in the webpage content based on a fully linked neural network and a sigmoid function;
[0021] Here, in practice, each web page has its corresponding web page content (the web page content can be in various languages). If the content is in English, the English words (i.e., the first word) can be obtained by removing the spaces and punctuation marks. If the content is in Chinese or other languages, semantic analysis and phrase segmentation are required to segment the phrases (i.e., the first word). Similarly, it is understandable that a web page often contains several hyperlinks, some of which are subordinate paths of the web page.
[0022] In order to facilitate the processing of the neural network, all first words may be processed to generate a binary code C1 corresponding to the web page, and all subordinate paths may be processed to generate a corresponding binary code C2.
[0023] Step 102: Create a BERT (Bidirectional Encoder Representation from Transformers) neural network, and perform the following processing on each web page: input the binary code C1 and the binary code C2 of the web page into the BERT neural network, and train the BERT neural network.
[0024] The binary code C1 and the binary code C2 are input into the BERT neural network to train the BERT neural network.
[0025] In practice, most of the subordinate web paths are relevant to the website content. For example, the web paths under the sports news section of the website are likely to contain various sports-related words. The BERT neural network can model the web page content and the corresponding sub-paths, and can effectively extract the potential language feature information between the web page content and the sub-paths, which greatly helps the subsequent path detection work.
[0026] In this embodiment, the "obtaining the binary codes C2 corresponding to all subordinate paths in the web page" specifically includes: performing the following processing on each subordinate path in the web page content: using the character " / " to divide the subordinate path into a number of second words, generating a binary code C3 corresponding to the second words, and the length of the binary code C3 is fixed; performing a bitwise OR operation on the binary code C3 corresponding to all subordinate paths in the web page content, thereby obtaining the binary code C2. Here, the subordinate path is usually composed of the character " / " and a number of words, so the character " / " can be used to divide the subordinate path into a number of second words.
[0027] In this embodiment, the “generating the binary code C1 corresponding to the webpage based on all the first words in the webpage content” specifically includes: obtaining a preset vocabulary W1, W2, ..., W N , create a binary number B = b1b2...b of length N N , when W i When b is one of all the first words, i =1, otherwise, b i =0; where N and i are natural numbers, 1≤i≤N; binary number B=b1b2...b N is the binary code C1 corresponding to the web page. Here, the preset word lists W1, W2, ..., W N It can be set according to actual needs, for example, the corresponding preset word lists W1, W2, ..., W N It's usually different.
[0028] In this embodiment, the "generating binary codes C3 corresponding to a plurality of second words" specifically includes: obtaining a preset vocabulary W1, W2, ..., W N , create a binary number C = c1c2...c with a length of N N , when W i When c is one of the plurality of second words, i =1, otherwise, c i =0; where N and i are natural numbers, 1≤i≤N; binary number C=c1c2...c N It is the binary code C3 corresponding to the webpage.
[0029] In this embodiment, the loss function of the BERT neural network is: L = ∑ i -[y i ·log(p i )+(1-y i )y i ·log(1-pi)].
[0030] The output of the model is the probability of each word in the vocabulary: for example, if our vocabulary has a total of 10,000 words, then the output of the model is a 10,000-dimensional vector, where each dimension represents the probability of each word appearing.
[0031] Each web page content in the dataset corresponds to a label. For example, the web page content of www.sin.com.cn corresponds to the labels sports.sin.com.cn and http: / / blog.sin.com.cn. First, the web page content passes through the model and finally gets the output: a 10,000-dimensional vector. Then, each component of the vector is mapped to a value between (0, 1) through the sigmoid function.
[0032] Because the labels are sports.sin.com.cn and ht: / / blog.sin.com.cn, the words sport and blog are in the vocabulary. So we perform an OR operation on the one-hot vectors corresponding to the two words sport and blog to get the label (also a 10,000-dimensional vector, in which only the index positions of the two words sport and blog are 1, and the rest are all 0). Now we have two 10,000-dimensional vectors, where i is the index of each dimension of the two 10,000-dimensional vectors, yi represents the index of the label at that position (either 1 or 0), and pi represents the index of the model output vector at that position (a value between 0 and 1).
[0033] Embodiment 2 of the present invention provides a device for creating a neural network, comprising the following modules:
[0034] A first content acquisition module is used to acquire webpage contents corresponding to a plurality of webpages, and perform the following processing on each webpage content: based on all first words in the webpage content, a binary code C1 corresponding to the webpage is generated, and binary codes C2 corresponding to all subordinate paths in the webpage are acquired, and then, based on a fully linked neural network and a sigmoid function, the binary code C1 is mapped to the probability of occurrence of each first word in the webpage content;
[0035] The network training module is used to create a BERT neural network and perform the following processing on each web page: input the binary code C1 and the binary code C2 of the web page into the BERT neural network and train the BERT neural network.
[0036] A third embodiment of the present invention provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program executes the creation method in the first embodiment when executed.
[0037] Embodiment 4 of the present invention provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the creation method as in Embodiment 1.
[0038] Embodiment 5 of the present invention provides a Web path detection method, comprising the following steps:
[0039] Step 1: Obtain webpage content corresponding to a webpage, and generate a binary code C3 corresponding to the webpage based on all words in the webpage content;
[0040] Step 2: Execute the creation method in Example 1 to generate a BERT neural network, and input the binary code C3 into the BERT neural network to obtain several lower-level paths.
[0041] During the inventor's invention process, at least the following experiments were performed:
[0042] Experiment 1, website: https: / / jandan.net / ;
[0043] Category: Discovery / Daily;
[0044] Multiple sub-paths:
[0045] news=>https: / / jandan.net / news,
[0046] app=>https: / / jandan.net / app,
[0047] more=>https: / / jandan.net / more.
[0048] Detection process:
[0049] Step 1: Crawl the homepage content of the website, including relevant links and corresponding text;
[0050] Step 2: The BERT model extracts keywords from the homepage content and learns and predicts possible subpaths.
[0051] (1) The above website category belongs to "Life", which can be predicted to contain sub-paths such as news and other commonly used life sub-paths;
[0052] (2) The page content contains "APP", "download" and other content, so it can be predicted that it may contain sub-paths such as app and download;
[0053] (3) The keyword "more" appears frequently in the page content, so it can be predicted that the page may contain the more subpath;
[0054] Step 3: Verify whether the possible sub-paths in step 2 actually exist and that the homepage link does not contain these sub-paths;
[0055] Step 4: Get multiple subordinate path results: news, app, more.
[0056] Experiment 2, website: https: / / www.ifanr.com;
[0057] Category: Lifestyle / Shopping;
[0058] Multiple sub-paths:
[0059] news=>https: / / www.ifanr.com / news,
[0060] app=>https: / / www.ifanr.com / app,
[0061] coolbuy=>https: / / www.ifanr.com / coolbuy,
[0062] video=>https: / / www.ifanr.com / video.
[0063] Detection process:
[0064] Step 1: Crawl the homepage content of the website, including relevant links and corresponding text;
[0065] Step 2: The BERT model extracts keywords from the content of the home page for learning and prediction to obtain possible sub-paths.
[0066] (1) The above website category belongs to "lifestyle", and it can be predicted that it may contain common lifestyle sub-paths such as news.
[0067] (2) The page content contains "Download the client" and other content, and it can be predicted that it may contain sub-paths such as app and download.
[0068] (3) The keywords "Playthings Magazine" and "Video" appear in the navigation bar of the page content, and it can be predicted that it may contain sub-paths such as coolbuy and video.
[0069] Step 3: Verify whether the possible sub-paths in Step 2 actually exist, and the home page link does not contain these sub-paths.
[0070] Step 4: Obtain multiple lower-level path results: news, app, coolbuy, video.
[0071] Experiment 3. Website: https: / / www.qmpython.com / ;
[0072] Category: Blog;
[0073] Multiple lower-level paths:
[0074] api / articles => https: / / www.qmpython.com / api / articles,
[0075] articles => https: / / www.qmpython.com / articles,
[0076] api => https: / / www.qmpython.com / api,
[0077] admin => https: / / www.qmpython.com / admin,
[0078] Detection process:
[0079] Step 1: Crawl the content of the website home page, including relevant links and corresponding text.
[0080] Step 2: The BERT model extracts keywords from the content of the home page for learning and prediction to obtain possible sub-paths.
[0081] (1) The above website category belongs to "blog category", which can be predicted to contain common blog category sub-paths such as articles, categories, top, and rank;
[0082] (2) Blogs are usually built with a backend management system, and it is predicted that there may be sub-paths such as admin, xadmin, byadmin, backoffice, and system;
[0083] Step 3: Verify whether the possible sub-paths in step 2 actually exist and that the homepage link does not contain these sub-paths. Multiple sub-path results are obtained: admin, articles;
[0084] Step 4: During the verification process, it was found that the webpage content returned by https: / / www.qmpython.com / admin includes the api and api / articles sub-paths;
[0085] Step 5: Verify again whether it really exists.
[0086] Step 6: Get the final multiple subordinate path results: admin, articles, api, api / articles.
[0087] Experiment 4, website: http: / / www.cyai.com / ;
[0088] Category: Company website;
[0089] Multiple sub-paths:
[0090] admin=>http: / / www.cyai.com / admin.
[0091] Detection process:
[0092] Step 1: Crawl the homepage content of the website, including relevant links and corresponding text;
[0093] Step 2: The BERT model extracts keywords from the homepage content and learns and predicts possible subpaths;
[0094] (1) The above website category belongs to "official website category", and it is predicted that there may be sub-paths such as news, products, and apps;
[0095] (2) The webpage content contains the keyword "Django", which means that the website is built with the Django framework and may expose the backend management system. It is predicted that there may be sub-paths such as admin, xadmin, byadmin, backoffice, and system.
[0096] Step 3: Verify whether the possible sub-paths in step 2 actually exist and whether the homepage link does not contain these sub-paths;
[0097] Step 4: Get multiple subordinate path results: admin.
[0098] Experiment 5, website: https: / / www.iplaysoft.com / ;
[0099] Category: Software Resources;
[0100] Multiple sub-paths:
[0101] wp-login.php=>https: / / www.iplaysoft.com / wp-login.php;
[0102] Detection process:
[0103] Step 1: Crawl the homepage content of the website, including relevant links and corresponding text;
[0104] Step 2: The BERT model extracts keywords from the homepage content and learns and predicts possible subpaths.
[0105] (1) The above website category belongs to "software", and it is predicted that there may be sub-paths such as free and os;
[0106] (2) The webpage content contains keywords such as "wordpress" and "php". The website may be built with the thinkphp framework, which may expose the backend management system. It is predicted that there may be sub-paths such as admin, admin / login, and wp-login.php.
[0107] Step 3: Verify whether the possible sub-paths in step 2 actually exist and whether the homepage link does not contain these sub-paths;
[0108] Step 4: Get multiple subordinate path results: wp-login.php;
[0109] Experiment 6
[0110] Website: https: / / www.gamersky.com / ;
[0111] Category: Games;
[0112] Multiple sub-paths:
[0113] console / ps4 / =>https: / / www.gamersky.com / console / ps4 / .
[0114] Detection process:
[0115] Step 1: Crawl the homepage content of the website, including relevant links and corresponding text;
[0116] Step 2: The BERT model extracts keywords from the homepage content and learns and predicts possible subpaths.
[0117] (1) The above website category belongs to "games", which can be predicted to contain sub-paths such as game, which are common game sub-paths;
[0118] (2) The keyword "PS5" appears frequently in the page content, so it can be predicted that it may contain the ps5 subpath;
[0119] (3) The crawler finds that the webpage content contains the subpath console / ps5 / and predicts that there may be subpaths console / ps4 and console / ps3;
[0120] Step 3: Verify whether the possible sub-paths in step 2 actually exist and that the homepage link does not contain these sub-paths;
[0121] Step 4: Get multiple subordinate path results: console / ps4.
[0122] Embodiment 5 of the present invention provides a Web path detection device, including the following modules:
[0123] A second content acquisition module, used to acquire webpage content corresponding to a webpage, and generate a binary code C3 corresponding to the webpage based on all words in the webpage content;
[0124] An execution module is used to execute the creation method in Example 1 to generate a BERT neural network, and input the binary code C3 into the BERT neural network to obtain a number of lower-level paths.
[0125] It should be understood that although this specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each implementation mode may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.
[0126] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. Any equivalent implementation methods or changes that do not deviate from the technical spirit of the present invention should be included in the scope of protection of the present invention.
Claims
1. A method for creating a neural network, characterized in that: The following steps are involved: Obtain web page contents corresponding to a plurality of web pages, and perform the following processing on each web page content: generate a binary code C1 corresponding to the web page based on all first words in the web page content, obtain binary codes C2 corresponding to all subordinate paths in the web page, and then map the binary code C1 to the probability of occurrence of each first word in the web page content based on a fully linked neural network and a sigmoid function; Create a BERT neural network and perform the following processing on each web page: input the binary code C1 and the binary code C2 of the web page into the BERT neural network and train the BERT neural network; The "obtaining the binary codes C2 corresponding to all subordinate paths in the webpage" specifically includes: The following processing is performed on each subordinate path in the webpage content: the subordinate path is divided into a plurality of second words using the character " / ", and a binary code C3 corresponding to the plurality of second words is generated, and the length of the binary code C3 is fixed; Performing a bitwise OR operation on the binary codes C3 corresponding to all subordinate paths in the webpage content, thereby obtaining a binary code C2; The “generating the binary code C1 corresponding to the webpage based on all the first words in the webpage content” specifically includes: Get the preset word list W1, W2, ..., W N , create a binary number B = b1b2...b of length N N , when W i When b is one of all the first words, i =1, otherwise, b i =0; where N and i are natural numbers, 1≤i≤N; binary number B = b1 b2...b N is the binary code C1 corresponding to the webpage; The "generating binary codes C3 corresponding to a plurality of second words" specifically includes: Get the preset word list W1, W2, ..., W N , create a binary number C = c1c2...c with a length of N N , when W i When c is one of the plurality of second words, i =1, otherwise, c i =0; where N and i are natural numbers, 1≤i≤N; binary number C = c1 c2...c N is the binary code C3.
2. The creation method according to claim 1, characterized in that: The loss function of the BERT neural network is: L=∑ i -[y i ·log(p i )+(1-y i )y i ·log(1-p i )]; where i is the index, y i is the position of the label at that index, p i The position of this index in the model output vector.
3. A device for creating a neural network, characterized in that: Includes the following modules: A first content acquisition module is used to acquire webpage contents corresponding to a plurality of webpages, and perform the following processing on each webpage content: based on all first words in the webpage content, a binary code C1 corresponding to the webpage is generated, and binary codes C2 corresponding to all subordinate paths in the webpage are acquired, and then, based on a fully linked neural network and a sigmoid function, the binary code C1 is mapped to the probability of occurrence of each first word in the webpage content; A network training module is used to create a BERT neural network and perform the following processing on each web page: input the binary code C1 and the binary code C2 of the web page into the BERT neural network to train the BERT neural network; The "obtaining the binary codes C2 corresponding to all subordinate paths in the webpage" specifically includes: The following processing is performed on each subordinate path in the webpage content: the subordinate path is divided into a plurality of second words using the character " / ", and a binary code C3 corresponding to the plurality of second words is generated, and the length of the binary code C3 is fixed; Performing a bitwise OR operation on the binary codes C3 corresponding to all subordinate paths in the webpage content, thereby obtaining a binary code C2; The “generating the binary code C1 corresponding to the webpage based on all the first words in the webpage content” specifically includes: Get the preset word list W1, W2, ..., W N , create a binary number B = b1b2...b of length N N , when W i When b is one of all the first words, i =1, otherwise, b i =0; where N and i are natural numbers, 1≤i≤N; binary number B = b1 b2...b N is the binary code C1 corresponding to the webpage; The "generating binary codes C3 corresponding to a plurality of second words" specifically includes: Get the preset word list W1, W2, ..., W N , create a binary number C = c1c2...c with a length of N N , when W i When c is one of the plurality of second words, i =1, otherwise, c i =0; where N and i are natural numbers, 1≤i≤N; binary number C = c1 c2...c N is the binary code C3.
4. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program executes the creation method according to claim 1 when executed.
5. A computer-readable storage medium, characterized in that: Computer executable instructions are stored, and the computer executable instructions are used to execute the creation method according to claim 1.
6. A Web path detection method, characterized in that: The following steps are involved: Obtain webpage content corresponding to the webpage, and generate a binary code C4 corresponding to the webpage based on all words in the webpage content; Execute the creation method of claim 1 to generate a BERT neural network, and input the binary code C4 into the BERT neural network to obtain several lower-level paths.
7. A Web path detection device, characterized in that: Includes the following modules: A second content acquisition module, used to acquire webpage content corresponding to a webpage, and generate a binary code C4 corresponding to the webpage based on all words in the webpage content; An execution module is used to execute the creation method of claim 1 to generate a BERT neural network, and input the binary code C4 into the BERT neural network to obtain a plurality of lower-level paths.
Citation Information
Patent Citations
Intelligent webpage information extraction method based on BERT and LSTM
CN111581476A
Internet negative information monitoring method based on Bert model
CN113065348A