Domain name crawling method and device, and network equipment

Abstract

Embodiments of the present application provide a domain name crawling method and device and network equipment, the method comprising: obtaining a domain name to be crawled; crawling the domain name to be crawled through a preset crawler engine to obtain first domain name data, the preset crawler engine being one of a plurality of crawler engines; extracting webpage features from the first domain name data to obtain webpage information features; inputting the webpage information features into a recurrent neural network for domain name classification to obtain classification results, each classification result indicating the use of a different type of crawler engine. The above scheme not only saves cost and time, but also alleviates the situation of excessive website service access pressure caused by high-frequency repeated crawling.

Description

Technical Field

[0001] This invention relates to the field of Internet technology, and in particular to a domain name crawling method, apparatus, and network equipment. Background Technology

[0002] With the rapid development of the internet, the World Wide Web (WWW) has generated massive amounts of web page data and resources, often containing illegal and inappropriate content. To ensure users can browse healthy and legitimate data and content, the industry typically uses web crawler technology to process web page content by first crawling and then monitoring it.

[0003] Currently, traditional web crawlers typically use a high-frequency, repetitive scanning method based on rounds, depths, and cycles to crawl content, which can easily lead to problems such as excessive business access pressure on websites. Summary of the Invention

[0004] This invention provides a domain name crawling method, apparatus, and network device to solve the problem that high-frequency repeated crawling methods in the prior art can easily cause excessive business access pressure on websites.

[0005] In a first aspect, embodiments of the present invention provide a domain name crawling method, including:

[0006] Obtain the domain name to be crawled;

[0007] The domain name to be crawled is crawled using a preset crawling engine to obtain the first domain name data. The preset crawling engine is one of a variety of crawling engines.

[0008] Webpage feature extraction is performed on the first domain name data to obtain webpage information features;

[0009] The webpage information features are input into a recurrent neural network for domain name classification, and the classification results are obtained. Each classification result indicates the use of a different type of crawler engine.

[0010] Optionally, after inputting the webpage information features into a recurrent neural network for domain name classification and obtaining the classification result, the method further includes:

[0011] Based on the classification results, the first queue to be crawled and scanned corresponding to the target crawler engine is obtained, wherein the target crawler engine is at least one of the multiple crawler engines indicated by the classification results.

[0012] The domain name to be crawled is added to the first queue to be crawled, thus obtaining the second queue to be crawled;

[0013] The domains in the second queue to be crawled are crawled sequentially to obtain the crawling results.

[0014] Optionally, the step of extracting webpage features from the first domain name data to obtain webpage information features includes:

[0015] Obtain the domain depth of the domain to be crawled and the first number of Uniform Resource Locators (URLs) contained in the domain to be crawled;

[0016] Based on the domain name depth and the first quantity, obtain the URL frequency vector matrix and the URL content information matrix;

[0017] Calculate the webpage information vector matrix based on the URL frequency vector matrix and the URL content information matrix;

[0018] The webpage information vector matrix is ​​input into a deep learning model for calculation to obtain webpage information features.

[0019] Optionally, obtaining the URL frequency vector matrix based on the domain name depth and the first quantity includes:

[0020] Get the first occurrence count of the j-th URL at the i-th depth of the domain to be crawled, and the first total occurrence count of all URLs at the i-th depth, where i is an integer from 1 to m, and m is the total depth of the domain to be crawled.

[0021] Based on the first number and the first total number, calculate the first frequency of the j-th URL at the i-th layer depth;

[0022] Based on the first frequency, the domain name depth, and the first quantity, obtain the URL frequency vector matrix.

[0023] Optionally, the URL frequency vector matrix is ​​represented by the following formula:

[0024]

[0025]

[0026] Where m represents the total depth of the domain name;

[0027] n represents the first quantity;

[0028] i represents the i-th layer depth of the domain name to be crawled;

[0029] j represents the j-th URL at the i-th depth of the domain name to be crawled;

[0030] A ij Indicates the first frequency;

[0031] n ij Indicates the first number;

[0032] ∑ k n ik This represents the first total number;

[0033] k represents the total number of URLs at the i-th depth.

[0034] Optionally, obtaining the URL content information matrix based on the domain name depth and the first quantity includes:

[0035] Get the second number of times the content of the j-th URL appears at the i-th depth of the domain to be crawled, and the second total number of times all URL content appears at the i-th depth, where i is an integer from 1 to m, and m is the total number of depths of the domain to be crawled;

[0036] Based on the second number and the second total number, calculate the second frequency of the j-th URL content at the i-th layer depth;

[0037] A URL content information matrix is ​​obtained based on the second frequency, the domain name depth, and the first quantity.

[0038] Optionally, the URL content information matrix is ​​represented by the following formula:

[0039]

[0040]

[0041] Where m represents the total depth of the domain name;

[0042] n represents the first quantity;

[0043] i represents the i-th layer depth of the domain name to be crawled;

[0044] j represents the j-th URL at the i-th depth of the domain name to be crawled;

[0045] B ij Indicates the second frequency;

[0046] n ij Indicates the second number;

[0047] ∑ k n ik This represents the second total;

[0048] k represents the total number of URLs at the i-th depth.

[0049] Optionally, calculating the webpage information vector matrix based on the URL frequency vector matrix and the URL content information matrix includes:

[0050] The webpage information vector matrix is ​​obtained by multiplying the URL frequency vector matrix and the URL content information matrix.

[0051] Secondly, embodiments of the present invention also provide a network device, including a memory, a transceiver, and a processor:

[0052] Memory is used to store computer programs; transceiver is used to send and receive data under the control of the processor; processor is used to read the computer programs from memory and perform the following operations:

[0053] Obtain the domain name to be crawled;

[0054] The domain name to be crawled is crawled using a preset crawling engine to obtain the first domain name data. The preset crawling engine is one of a variety of crawling engines.

[0055] Webpage feature extraction is performed on the first domain name data to obtain webpage information features;

[0056] The webpage information features are input into a recurrent neural network for domain name classification, and the classification results are obtained. Each classification result indicates the use of a different type of crawler engine.

[0057] Optionally, after inputting the webpage information features into a recurrent neural network for domain name classification and obtaining the classification result, the processor is further configured to:

[0058] Based on the classification results, the first queue to be crawled and scanned corresponding to the target crawler engine is obtained, wherein the target crawler engine is at least one of the multiple crawler engines indicated by the classification results.

[0059] The domain name to be crawled is added to the first queue to be crawled, thus obtaining the second queue to be crawled;

[0060] The domains in the second queue to be crawled are crawled sequentially to obtain the crawling results.

[0061] Optionally, when the processor extracts webpage features from the first domain name data to obtain webpage information features, it is specifically used for:

[0062] Obtain the domain depth of the domain to be crawled and the first number of Uniform Resource Locators (URLs) contained in the domain to be crawled;

[0063] Based on the domain name depth and the first quantity, obtain the URL frequency vector matrix and the URL content information matrix;

[0064] Calculate the webpage information vector matrix based on the URL frequency vector matrix and the URL content information matrix;

[0065] The webpage information vector matrix is ​​input into a deep learning model for calculation to obtain webpage information features.

[0066] Optionally, when the processor obtains the URL frequency vector matrix based on the domain name depth and the first quantity, it is specifically used for:

[0067] Get the first occurrence count of the j-th URL at the i-th depth of the domain to be crawled, and the first total occurrence count of all URLs at the i-th depth, where i is an integer from 1 to m, and m is the total depth of the domain to be crawled.

[0068] Based on the first number and the first total number, calculate the first frequency of the j-th URL at the i-th layer depth;

[0069] Based on the first frequency, the domain name depth, and the first quantity, obtain the URL frequency vector matrix.

[0070] Optionally, the URL frequency vector matrix is ​​represented by the following formula:

[0071]

[0072]

[0073] Where m represents the total depth of the domain name;

[0074] n represents the first quantity;

[0075] i represents the i-th layer depth of the domain name to be crawled;

[0076] j represents the j-th URL at the i-th depth of the domain name to be crawled;

[0077] A ij Indicates the first frequency;

[0078] n ij Indicates the first number;

[0079] ∑ k n ik This represents the first total number;

[0080] k represents the total number of URLs at the i-th depth.

[0081] Optionally, when the processor obtains the URL content information matrix based on the domain name depth and the first quantity, it is specifically used for:

[0082] Get the second number of times the content of the j-th URL appears at the i-th depth of the domain to be crawled, and the second total number of times all URL content appears at the i-th depth, where i is an integer from 1 to m, and m is the total number of depths of the domain to be crawled;

[0083] Based on the second number and the second total number, calculate the second frequency of the j-th URL content at the i-th layer depth;

[0084] A URL content information matrix is ​​obtained based on the second frequency, the domain name depth, and the first quantity.

[0085] Optionally, the URL content information matrix is ​​represented by the following formula:

[0086]

[0087]

[0088] Where m represents the total depth of the domain name;

[0089] n represents the first quantity;

[0090] i represents the i-th layer depth of the domain name to be crawled;

[0091] j represents the j-th URL at the i-th depth of the domain name to be crawled;

[0092] B ij Indicates the second frequency;

[0093] n ij Indicates the second number;

[0094] ∑ k n ik This represents the second total;

[0095] k represents the total number of URLs at the i-th depth.

[0096] Optionally, when the processor calculates the webpage information vector matrix based on the URL frequency vector matrix and the URL content information matrix, it specifically performs the following:

[0097] The webpage information vector matrix is ​​obtained by multiplying the URL frequency vector matrix and the URL content information matrix.

[0098] Thirdly, embodiments of the present invention also provide a domain name crawling device, comprising:

[0099] The first acquisition module is used to acquire the domain name to be crawled / scanned.

[0100] The first crawling module is used to crawl the domain name to be crawled using a preset crawling engine to obtain the first domain name data. The preset crawling engine is one of a variety of crawling engines.

[0101] The first extraction module is used to extract webpage features from the first domain name data to obtain webpage information features;

[0102] The first classification module is used to input the web page information features into a recurrent neural network to classify the domain names and obtain classification results. Each classification result indicates the use of a different type of crawler engine.

[0103] Fourthly, embodiments of the present invention also provide a processor-readable storage medium storing a computer program for causing the processor to execute the aforementioned domain name crawling method.

[0104] In this embodiment of the invention, by acquiring the domain name to be crawled, the domain name is crawled using a preset crawling engine among various crawling engines to obtain first domain name data. That is, a rapid coarse crawl is performed using the preset crawling engine to obtain coarse crawled data, namely, the first domain name data. By extracting webpage features from the first domain name data, webpage information features are obtained. The webpage information features are input into a recurrent neural network for domain name classification to obtain classification results. Each classification result indicates the use of a different type of crawling engine. That is, the first domain name data is first classified and analyzed to obtain classification results. Targeted crawling is then performed based on the classification results. This not only saves costs and time but also alleviates the excessive website access pressure caused by high-frequency repetitive crawling methods. Attached Figure Description

[0105] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0106] Figure 1 This is a flowchart of the domain name crawling method provided in an embodiment of the present invention;

[0107] Figure 2 This is a schematic diagram of a recurrent neural network structure provided in an embodiment of the present invention;

[0108] Figure 3 This is a structural block diagram of the domain name crawling device provided in an embodiment of the present invention;

[0109] Figure 4 This is a structural block diagram of a network device provided in an embodiment of the present invention. Detailed Implementation

[0110] In this embodiment of the invention, the term "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0111] In the embodiments of this application, the term "multiple" refers to two or more, and other quantifiers are similar.

[0112] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0113] Specifically, such as Figure 1 As shown, this embodiment of the invention provides a domain name crawling method, which may specifically include:

[0114] Step 101: Obtain the domain name to be crawled / scanned.

[0115] Specifically, before obtaining the domain name to be crawled, a web crawler engine matrix is ​​constructed. This web crawler engine matrix includes, but is not limited to, the following types of crawler engines: low-level high-frequency crawler engine, deep low-frequency crawler engine, infinite-depth crawler engine, broken link rescanning crawler engine, Internet Protocol (IP) proxy crawler engine, programming language (JavaScript, JS) crawler engine, Internet Protocol Version 6 (IPv6) crawler engine, Baidu User Agent (UA) crawler engine, and external link polling scanning engine, etc.

[0116] It should be noted that the domain name to be crawled can also be the Uniform Resource Locator (URL) to be crawled.

[0117] Step 102: The domain name to be crawled is crawled using a preset crawler engine to obtain the first domain name data. The preset crawler engine is one of a variety of crawler engines.

[0118] Specifically, a pre-set crawler engine is used to perform a coarse crawl of the domain name to be crawled, thereby obtaining the first domain name data. This first domain name data can contain one domain name or URL data, or it can contain multiple domain names or URL data.

[0119] Each crawler engine has a corresponding crawling queue, including a regular task crawling queue and a limited task crawling queue, which distinguishes between regular crawling tasks and urgent crawling tasks. You can determine which queue to use for crawling as needed, dynamically adjust crawling tasks, and make reasonable use of crawling resources.

[0120] It should be noted that coarse crawling is the process of adding the domain name to be crawled to the crawling queue corresponding to the preset crawling engine, starting the preset crawling engine to crawl and obtain the first domain name data.

[0121] Preferably, the preset crawler engine can be a low-level, high-frequency crawler engine. Using a low-level, high-frequency crawler engine for coarse crawling can complete the crawling quickly in a short time.

[0122] Step 103: Extract webpage features from the first domain name data to obtain webpage information features.

[0123] Specifically, webpage features are extracted from each domain name in the first domain name data obtained from the coarse crawling. The main focus is on extracting features of the text information in the crawled webpages, such as: the body text of the webpage, the title of the webpage, the description of the webpage, the position of the URL in the domain name, the total number of domain names, etc., thereby obtaining the corresponding webpage information features.

[0124] The first domain name data can also be the first URL data. The URL page's title, description, and keywords can all be extracted from the corresponding tag pairs. For example, the URL page title can be extracted from... <title>...< / title> The description information of the URL webpage can be extracted from the tag pairs. <meta name＝”description”> Keyword information from the URL of a webpage can be extracted from tag pairs. <meta name＝”keywords”> Extracted from tags. The position of the URL within the domain can be obtained from the crawl logs.

[0125] Step 104: Input the webpage information features into a recurrent neural network for domain name classification to obtain classification results. Each classification result indicates the use of a different type of crawler engine.

[0126] Specifically, the web page information features obtained in step 103 are input into the recurrent neural network classification model for domain name classification prediction or URL classification prediction. After the recurrent neural network classification model performs classification prediction, the classification result is output. Different classification results indicate the use of different types of crawler engines, that is, different classification results indicate that they are added to the crawling queues corresponding to different types of crawler engines.

[0127] like Figure 2As shown, the above recurrent neural network classification model consists of an input layer (X), a hidden layer (S), a recurrent layer (W), and an output layer (O). Web page information features are input into the input layer (X), classification analysis is performed in the hidden layer (S) by fusing the data in the recurrent layer (W), and finally the predicted classification result is output in the output layer (O).

[0128] Specifically, the webpage information features are input into the input layer (X) of the recurrent neural network classification model to obtain the webpage information feature vector:

[0129]

[0130]

[0131] Among them, w j This indicates the default central webpage;

[0132] w represents w j The sequence of web pages surrounding the central web page;

[0133] w′ represents non-w j The sequence of web pages surrounding the central web page;

[0134] P(w|w j ) indicates the default central webpage w j The probability of the surrounding webpage w appearing;

[0135] D represents the set of all domains to be crawled and scanned;

[0136] C represents the set of all known URLs under a given domain;

[0137] j represents the set of natural numbers;

[0138] e represents the natural constant;

[0139] e′ represents the negative natural constant;

[0140] exp represents an exponential function with base e.

[0141] V represents the dimension of D;

[0142] T represents the transpose operation.

[0143] After obtaining the webpage information feature vector using the above formula, the webpage information feature vector is input into the hidden layer (S). The hidden layer is a fully connected neural network with a dimension of a*b, where a is the depth of the domain name and b is the number of URLs contained in the domain name.

[0144] While the hidden layer is being computed, data from the recurrent layer (W) is input into the hidden layer. The data in the recurrent layer is the state information stored from the previous time step. The calculation formula for the fusion of the hidden layer and the recurrent layer is as follows:

[0145] h t =tanh(x) t w ab +h t-1 w ab +g)

[0146] tanh(X) = 2*sigmoid(X)

[0147] Among them, h t This represents the result of merging the hidden layer and the recurrent layer at the current moment;

[0148] w ab The weight matrix has dimensions a*b and is initially set to random values ​​between [0, 1]. The weight matrix represents the parameters obtained by the recurrent neural network after each learning iteration. Through continuous self-adjustment, the classification results can be made more accurate.

[0149] g represents the bias term, with dimensions a*b. For ease of calculation, the initial value can be set to 1. The bias term can accelerate the fitting of recurrent neural networks.

[0150] x t This indicates the characteristics of the webpage information entered at the current moment;

[0151] h t-1 This represents the state information stored at the previous moment;

[0152] 'a' indicates the domain depth;

[0153] b indicates the number of URLs contained in the domain name;

[0154] tanh represents the activation function;

[0155] The sigmoid function represents the nonlinear action function of a neuron.

[0156] t represents the current time.

[0157] Finally, the classification output is completed in the output layer (O). The classification formula of the output layer is as follows:

[0158] q t =softmax(h t w ab +g)

[0159]

[0160] Where, q t Indicates the classification result;

[0161] w ab Represents the weight matrix;

[0162] g represents the bias term;

[0163] softmax represents the activation function;

[0164] z i This represents the output value of the i-th node;

[0165] z C Represents the set of output values ​​for all nodes;

[0166] C represents the number of output nodes, i.e. the number of categories.

[0167] c represents the c-th node;

[0168] e This means converting the output value of the i-th node to a range of [0,1].

[0169] Similarly, convert the output values ​​of all nodes to the range [0,1].

[0170] The softmax function can be used to convert the output values ​​of multi-class classification into a probability distribution ranging from [0,1] to 1. The classification categories are currently defined as follows: low-level high-frequency type, deep-level low-frequency type, infinite-depth type, broken link type, IP proxy type, JS type, IPv6 type, Baidu UserAgent type, external link polling type, etc. Categories can be added or removed as needed.

[0171] In the above embodiments of the present invention, by obtaining the domain name to be crawled, the domain name is crawled using a preset crawling engine among various crawling engines to obtain first domain name data, that is, by quickly and coarsely crawling using the preset crawling engine to obtain coarse crawled data, namely, the first domain name data. By extracting web page features from the first domain name data, web page information features are obtained. The web page information features are input into a recurrent neural network for domain name classification to obtain classification results. Each classification result indicates the use of a different type of crawling engine. That is, the first domain name data is first classified and analyzed to obtain classification results. Targeted crawling is then performed based on the classification results. This not only saves costs and time but also alleviates the excessive website access pressure caused by high-frequency repetitive crawling methods.

[0172] As an optional embodiment, after step 104 inputs the webpage information features into a recurrent neural network for domain name classification and obtains the classification result, the method further includes:

[0173] Based on the classification results, the first queue to be crawled and scanned corresponding to the target crawler engine is obtained, wherein the target crawler engine is at least one of the multiple crawler engines indicated by the classification results.

[0174] The domain name to be crawled is added to the first queue to be crawled, thus obtaining the second queue to be crawled;

[0175] The domains in the second queue to be crawled are crawled sequentially to obtain the crawling results.

[0176] Specifically, based on the classification results, the target crawler engine indicated by the classification results is obtained, and the first crawling queue corresponding to the target crawler engine is obtained. The domain name to be crawled is added to the first crawling queue, thereby updating the first crawling queue to the second crawling queue. Priorities can be set as needed, and crawling is performed sequentially according to the priority order in the second crawling queue. In this way, the crawler engine of the corresponding category will crawl and resolve the domain name of the corresponding category more accurately and quickly.

[0177] It should be noted that for URLs with lower depth in the domain, because the number of underlying URLs is relatively small and the probability of finding illegal data is higher, using a low-level high-frequency crawler engine as the target crawler engine for high-frequency crawling can enhance the high-quality detection of low-level URL data.

[0178] For URLs with high depth in a domain, since there are relatively more high-level URLs and the probability of finding illegal data is moderate, using a deep, low-frequency crawler engine as the target crawler engine for low-frequency crawling can enhance the high-quality detection of deep URL data.

[0179] For URLs with very high depth in a domain, because the number of such URLs is large and uncontrollable, and the probability of discovering illegal data is low, an infinite-depth crawler engine is used as the target crawler engine to perform infinite-depth crawling. This can enhance the discovery of an unlimited number of URLs under the domain and enable reasonable monitoring.

[0180] When the URLs crawled show error network status codes such as 400 or 500, the crawler engine that re-scans invalid links is used as the target crawler engine for re-scanning to ensure that this type of data is not missed.

[0181] When crawling a domain and encountering a situation where the target server mistakenly identifies the crawler's behavior as an attack (e.g., when a crawler accesses a batch of URLs and a large number of URLs under that domain show a network status of 600), resulting in the crawler's IP being blocked, an IP proxy crawler engine is used as the target crawler engine. This means using a valid IP to obtain data under the corresponding domain, thereby ensuring that this data is not missed during the crawling process.

[0182] For JS domain types appearing in the domain name, a JS crawler engine is used as the target crawler engine to scan for this type of domain in order to complete the monitoring of illegal data.

[0183] For IPv6 domain names, an IPv6 crawler engine is used as the target crawler engine to scan the domain names and monitor for illegal data.

[0184] For domain types that cannot be scanned using Google UserAgent, Baidu UserAgent crawler engine is used as the target crawler engine for scanning to complete the monitoring of illegal data.

[0185] For external links, an external link polling and scanning engine is used as the target crawler engine for long-term polling and monitoring to achieve more efficient monitoring of illegal data.

[0186] As an optional embodiment, step 103 extracts webpage features from the first domain name data to obtain webpage information features, including:

[0187] Obtain the domain depth of the domain to be crawled and the first number of Uniform Resource Locators (URLs) contained in the domain to be crawled;

[0188] Based on the domain name depth and the first quantity, obtain the URL frequency vector matrix and the URL content information matrix;

[0189] Calculate the webpage information vector matrix based on the URL frequency vector matrix and the URL content information matrix;

[0190] The webpage information vector matrix is ​​input into a deep learning model for calculation to obtain webpage information features.

[0191] Specifically, the domain depth of the domain to be crawled and the first number of Uniform Resource Locators (URLs) contained in the domain are obtained. A URL frequency vector matrix and a URL content information matrix are calculated using the domain depth and the first number. Then, a webpage information vector matrix is ​​calculated using the URL frequency vector matrix and the URL content information matrix. The obtained webpage information vector matrix is ​​input into a deep learning model for calculation to obtain the output webpage information features. Compared with traditional models, this method incorporates content information by initially fusing the URL frequency vector matrix and the URL content information matrix. This preserves both the content information of the URL and the position information of the URL in the domain. At the same time, because the two matrices are very sparse, the sparsity problem of the matrix is ​​also solved after the operation.

[0192] For example, a domain's location structure is as follows: www.a.com|www.a.com / b|www.a.com / c|www.a.com / d|www.a.com / e; its content information structure is as follows: Sports Portal|Basketball|NBA|Rockets|Player News. Assuming the selected URL with the location structure is "www.a.com / b", the following scenarios are possible:

[0193] www.a.com - Sports Portal | www.a.com / b - Basketball;

[0194] www.a.com / c·NBA|www.a.com / b·Basketball;

[0195] www.a.com / d·Rockets | www.a.com / b·Basketball;

[0196] www.a.com / e·Player News | www.a.com / b·Basketball

[0197] Assuming the context prediction window is 1, then only "www.a.com·sports portal|www.a.com / b·basketball" is the correct sample structure among these cases. In fact, this means calculating the maximum value of the probability P(www.a.com·sports portal|www.a.com / b·basketball)*P(www.a.com / c·NBA|www.a.com / b·basketball)*P(www.a.com / d·Rockets|www.a.com / b·basketball)*P(www.a.com / e·player news|www.a.com / b·basketball). This probability value is the webpage feature information of the URL "www.a.com / b".

[0198] As an optional embodiment, the step of obtaining the URL frequency vector matrix based on the domain name depth and the first quantity specifically includes:

[0199] Get the first occurrence count of the j-th URL at the i-th depth of the domain to be crawled, and the first total occurrence count of all URLs at the i-th depth, where i is an integer from 1 to m, and m is the total depth of the domain to be crawled.

[0200] Based on the first number and the first total number, calculate the first frequency of the j-th URL at the i-th layer depth;

[0201] Based on the first frequency, the domain name depth, and the first quantity, obtain the URL frequency vector matrix.

[0202] Specifically, the number of times the j-th URL appears at the i-th depth of the domain to be crawled is obtained, i.e., the first count, and the total number of all URLs appearing at the i-th depth of the domain to be crawled is obtained, i.e., the first total. The frequency of the j-th URL appearing at the i-th depth of the domain to be crawled is calculated using the first count and the first total, i.e., the first frequency. The URL frequency vector matrix is ​​obtained using the first frequency, the domain depth, and the first count.

[0203] Furthermore, the URL frequency vector matrix is ​​represented by the following formula:

[0204]

[0205]

[0206] Where m represents the total depth of the domain name;

[0207] n represents the first quantity;

[0208] i represents the i-th layer depth of the domain name to be crawled;

[0209] j represents the j-th URL at the i-th depth of the domain name to be crawled;

[0210] A ij Indicates the first frequency;

[0211] n i j represents the first number;

[0212] ∑ k n ik This represents the first total number;

[0213] k represents the second number of URLs at the i-th depth.

[0214] As an optional embodiment, the step of obtaining the URL content information matrix based on the domain name depth and the first quantity specifically includes:

[0215] Get the second number of times the content of the j-th URL appears at the i-th depth of the domain to be crawled, and the second total number of times all URL content appears at the i-th depth, where i is an integer from 1 to m, and m is the total number of depths of the domain to be crawled;

[0216] Based on the second number and the second total number, calculate the second frequency of the j-th URL content at the i-th layer depth;

[0217] A URL content information matrix is ​​obtained based on the second frequency, the domain name depth, and the first quantity.

[0218] Specifically, the number of times the j-th URL content appears at the i-th depth of the domain to be crawled is obtained, i.e., the second count, and the total number of times all URL content appears at the i-th depth of the domain to be crawled is obtained, i.e., the second total. The frequency of the j-th URL content at the i-th depth of the domain to be crawled is calculated using the second count and the second total, i.e., the second frequency. The URL content information matrix is ​​obtained using the second frequency, the domain depth, and the first count.

[0219] Furthermore, the URL content information matrix is ​​represented by the following formula:

[0220]

[0221]

[0222] Where m represents the total depth of the domain name;

[0223] n represents the first quantity;

[0224] i represents the i-th layer depth of the domain name to be crawled;

[0225] j represents the j-th URL at the i-th depth of the domain name to be crawled;

[0226] B ij Indicates the second frequency;

[0227] n ij Indicates the second number;

[0228] ∑ k n ik This represents the second total;

[0229] k represents the total number of URLs at the i-th depth.

[0230] As an optional embodiment, calculating the webpage information vector matrix based on the URL frequency vector matrix and the URL content information matrix includes:

[0231] The webpage information vector matrix is ​​obtained by multiplying the URL frequency vector matrix and the URL content information matrix.

[0232] Specifically, the URL frequency vector matrix and the URL content information matrix are initially fused by dot product calculation. This preserves both the content information of the URL and the position information of the URL in the domain name. At the same time, because the two matrices are very sparse, the sparsity problem of the matrix is ​​also solved after the operation.

[0233] In summary, this invention, through a pre-set crawler engine performing a coarse crawl of the domain name to be crawled, and then extracting features from the coarse crawl data, can abstract and combine key information such as the main text, title, description, URL position within the domain, and total number of domains of web pages that appear irregular, generating web page information features. This allows for more effective capture of URL features and more efficient sending of URL classification results to the corresponding crawler engine, making reasonable and effective use of crawling resources and efficiently and accurately extracting sufficient data from the massive amount of web page data on the Internet. Secondly, by using dot product calculations to initially fuse the URL frequency vector matrix and the URL content information matrix, both the content information and the position information of the URL within the domain are preserved. Furthermore, because both matrices are very sparse, the sparsity problem is also solved after the operation. Moreover, to meet targeted scanning needs, the concept of a crawler matrix allows for targeted crawling of the domain name to be crawled based on the classification results, and the number of crawler engine types can be expanded or reduced as needed according to project requirements, making it more versatile.

[0234] The domain name crawling and scanning method provided by the embodiments of the present invention has been described above. The domain name crawling and scanning device provided by the embodiments of the present invention will be described below with reference to the accompanying drawings.

[0235] like Figure 3 As shown, this embodiment of the invention also provides a domain name crawling device 300, comprising:

[0236] The first acquisition module 301 is used to acquire the domain name to be crawled / scanned.

[0237] The first crawling module 302 is used to crawl the domain name to be crawled using a preset crawling engine to obtain the first domain name data. The preset crawling engine is one of a variety of crawling engines.

[0238] The first extraction module 303 is used to extract web page features from the first domain name data to obtain web page information features;

[0239] The first classification module 304 is used to input the web page information features into a recurrent neural network for domain name classification and obtain classification results. Each classification result indicates the use of a different type of crawler engine.

[0240] In the above embodiments of the present invention, by obtaining the domain name to be crawled, the domain name is crawled using a preset crawling engine among various crawling engines to obtain first domain name data, that is, by quickly and coarsely crawling using the preset crawling engine to obtain coarse crawled data, namely, the first domain name data. By extracting web page features from the first domain name data, web page information features are obtained. The web page information features are input into a recurrent neural network for domain name classification to obtain classification results. Each classification result indicates the use of a different type of crawling engine. That is, the first domain name data is first classified and analyzed to obtain classification results. Targeted crawling is then performed based on the classification results. This not only saves costs and time but also alleviates the excessive website access pressure caused by high-frequency repetitive crawling methods.

[0241] Optionally, the device further includes:

[0242] The second acquisition module is used to acquire the first queue to be crawled corresponding to the target crawler engine according to the classification result, wherein the target crawler engine is at least one of the multiple crawler engines indicated by the classification result.

[0243] The first processing module is used to add the domain name to be crawled to the first queue to be crawled, thereby obtaining the second queue to be crawled.

[0244] The second crawling module is used to crawl the domain names in the second crawling queue in sequence to obtain the crawling results.

[0245] Optionally, the first extraction module 303 is specifically used for:

[0246] Obtain the domain depth of the domain to be crawled and the first number of Uniform Resource Locators (URLs) contained in the domain to be crawled;

[0247] Based on the domain name depth and the first quantity, obtain the URL frequency vector matrix and the URL content information matrix;

[0248] Calculate the webpage information vector matrix based on the URL frequency vector matrix and the URL content information matrix;

[0249] The webpage information vector matrix is ​​input into a deep learning model for calculation to obtain webpage information features.

[0250] Optionally, when the first extraction module 303 obtains the URL frequency vector matrix based on the domain name depth and the first quantity, it is specifically used for:

[0251] Get the first occurrence count of the j-th URL at the i-th depth of the domain to be crawled, and the first total occurrence count of all URLs at the i-th depth, where i is an integer from 1 to m, and m is the total depth of the domain to be crawled.

[0252] Based on the first number and the first total number, calculate the first frequency of the j-th URL at the i-th layer depth;

[0253] Based on the first frequency, the domain name depth, and the first quantity, obtain the URL frequency vector matrix.

[0254] Optionally, the URL frequency vector matrix is ​​represented by the following formula:

[0255]

[0256]

[0257] Where m represents the total depth of the domain name;

[0258] n represents the first quantity;

[0259] i represents the i-th layer depth of the domain name to be crawled;

[0260] j represents the j-th URL at the i-th depth of the domain name to be crawled;

[0261] A ij Indicates the first frequency;

[0262] n ij Indicates the first number;

[0263] ∑ k n ik This represents the first total number;

[0264] k represents the total number of URLs at the i-th depth.

[0265] Optionally, when the first extraction module 303 obtains the URL content information matrix based on the domain name depth and the first quantity, it is specifically used for:

[0266] Get the second number of times the content of the j-th URL appears at the i-th depth of the domain to be crawled, and the second total number of times all URL content appears at the i-th depth, where i is an integer from 1 to m, and m is the total number of depths of the domain to be crawled;

[0267] Based on the second number and the second total number, calculate the second frequency of the j-th URL content at the i-th layer depth;

[0268] A URL content information matrix is ​​obtained based on the second frequency, the domain name depth, and the first quantity.

[0269] Optionally, the URL content information matrix is ​​represented by the following formula:

[0270]

[0271]

[0272] Where m represents the total depth of the domain name;

[0273] n represents the first quantity;

[0274] i represents the i-th layer depth of the domain name to be crawled;

[0275] j represents the j-th URL at the i-th depth of the domain name to be crawled;

[0276] B ij Indicates the second frequency;

[0277] n ij Indicates the second number;

[0278] ∑ k n ik This represents the second total;

[0279] k represents the total number of URLs at the i-th depth.

[0280] Optionally, when the first extraction module 303 calculates the webpage information vector matrix based on the URL frequency vector matrix and the URL content information matrix, it is specifically used for:

[0281] The webpage information vector matrix is ​​obtained by multiplying the URL frequency vector matrix and the URL content information matrix.

[0282] It should be noted that the division of units in the embodiments of this application is illustrative and only represents one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0283] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0284] In summary, this invention, through a pre-set crawler engine performing a coarse crawl of the domain name to be crawled, and then extracting features from the coarse crawl data, can abstract and combine key information such as the main text, title, description, URL position within the domain, and total number of domains of web pages that appear irregular, generating web page information features. This allows for more effective capture of URL features and more efficient sending of URL classification results to the corresponding crawler engine, making reasonable and effective use of crawling resources and efficiently and accurately extracting sufficient data from the massive amount of web page data on the Internet. Secondly, by using dot product calculations to initially fuse the URL frequency vector matrix and the URL content information matrix, both the content information and the position information of the URL within the domain are preserved. Furthermore, because both matrices are very sparse, the sparsity problem is also solved after the operation. Moreover, to meet targeted scanning needs, the concept of a crawler matrix allows for targeted crawling of the domain name to be crawled based on the classification results, and the number of crawler engine types can be expanded or reduced as needed according to project requirements, making it more versatile.

[0285] It should be noted that the apparatus provided in this embodiment of the invention can implement all the method steps implemented in the above method embodiment and can achieve the same technical effect. Therefore, the parts and beneficial effects that are the same as those in the method embodiment will not be described in detail here.

[0286] like Figure 4 As shown, embodiments of the present invention also provide a network device, including a memory 420, a transceiver 410, and a processor 400:

[0287] Memory 420 is used to store computer programs;

[0288] Transceiver 410 is used to send and receive data under the control of the processor;

[0289] Processor 400 is used to read computer programs from memory and perform the following operations:

[0290] Obtain the domain name to be crawled;

[0291] The domain name to be crawled is crawled using a preset crawling engine to obtain the first domain name data. The preset crawling engine is one of a variety of crawling engines.

[0292] Webpage feature extraction is performed on the first domain name data to obtain webpage information features;

[0293] The webpage information features are input into a recurrent neural network for domain name classification, and the classification results are obtained. Each classification result indicates the use of a different type of crawler engine.

[0294] Optionally, after inputting the webpage information features into a recurrent neural network for domain name classification and obtaining the classification result, the processor 400 is further configured to:

[0295] Based on the classification results, the first queue to be crawled and scanned corresponding to the target crawler engine is obtained, wherein the target crawler engine is at least one of the multiple crawler engines indicated by the classification results.

[0296] The domain name to be crawled is added to the first queue to be crawled, thus obtaining the second queue to be crawled;

[0297] The domains in the second queue to be crawled are crawled sequentially to obtain the crawling results.

[0298] Optionally, when the processor 400 extracts webpage features from the first domain name data to obtain webpage information features, it is specifically used for:

[0299] Obtain the domain depth of the domain to be crawled and the first number of Uniform Resource Locators (URLs) contained in the domain to be crawled;

[0300] Based on the domain name depth and the first quantity, obtain the URL frequency vector matrix and the URL content information matrix;

[0301] Calculate the webpage information vector matrix based on the URL frequency vector matrix and the URL content information matrix;

[0302] The webpage information vector matrix is ​​input into a deep learning model for calculation to obtain webpage information features.

[0303] Optionally, when the processor 400 obtains the URL frequency vector matrix based on the domain name depth and the first quantity, it is specifically used for:

[0304] Get the first occurrence count of the j-th URL at the i-th depth of the domain to be crawled, and the first total occurrence count of all URLs at the i-th depth, where i is an integer from 1 to m, and m is the total depth of the domain to be crawled.

[0305] Based on the first number and the first total number, calculate the first frequency of the j-th URL at the i-th layer depth;

[0306] Based on the first frequency, the domain name depth, and the first quantity, obtain the URL frequency vector matrix.

[0307] Optionally, the URL frequency vector matrix is ​​represented by the following formula:

[0308]

[0309]

[0310] Where m represents the total depth of the domain name;

[0311] n represents the first quantity;

[0312] i represents the i-th layer depth of the domain name to be crawled;

[0313] j represents the j-th URL at the i-th depth of the domain name to be crawled;

[0314] A ij Indicates the first frequency;

[0315] n ij Indicates the first number;

[0316] ∑ k n ik This represents the first total number;

[0317] k represents the total number of URLs at the i-th depth.

[0318] Optionally, when the processor 400 obtains the URL content information matrix based on the domain name depth and the first quantity, it is specifically used for:

[0319] Get the second number of times the content of the j-th URL appears at the i-th depth of the domain to be crawled, and the second total number of times all URL content appears at the i-th depth, where i is an integer from 1 to m, and m is the total number of depths of the domain to be crawled;

[0320] Based on the second number and the second total number, calculate the second frequency of the j-th URL content at the i-th layer depth;

[0321] A URL content information matrix is ​​obtained based on the second frequency, the domain name depth, and the first quantity.

[0322] Optionally, the URL content information matrix is ​​represented by the following formula:

[0323]

[0324]

[0325] Where m represents the total depth of the domain name;

[0326] n represents the first quantity;

[0327] i represents the i-th layer depth of the domain name to be crawled;

[0328] j represents the j-th URL at the i-th depth of the domain name to be crawled;

[0329] B ij Indicates the second frequency;

[0330] n ij Indicates the second number;

[0331] ∑ k n ik This represents the second total;

[0332] k represents the total number of URLs at the i-th depth.

[0333] Optionally, when the processor 400 calculates the webpage information vector matrix based on the URL frequency vector matrix and the URL content information matrix, it specifically performs the following:

[0334] The webpage information vector matrix is ​​obtained by multiplying the URL frequency vector matrix and the URL content information matrix.

[0335] Among them, Figure 4In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 400) and memory (memory 420). The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 410 can be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over transmission media, including wireless channels, wired channels, optical fibers, etc. The processor 400 is responsible for managing the bus architecture and general processing, and the memory 420 can store data used by the processor 400 during operation.

[0336] The processor 400 can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a complex programmable logic device (CPLD). The processor can also adopt a multi-core architecture.

[0337] The processor executes any of the domain name crawling methods provided in this application embodiment by calling a computer program stored in memory, according to the obtained executable instructions. The processor and memory can also be physically separated.

[0338] It should be noted that the network device provided in this embodiment of the invention can implement all the method steps implemented in the above domain name crawling method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0339] Embodiments of the present invention also provide a processor-readable storage medium storing a computer program for causing the processor to execute the above-described domain name crawling method.

[0340] The processor-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0341] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0342] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-executable instructions. These computer-executable instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0343] These processor-executable instructions may also be stored in a processor-readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the processor-readable memory produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0344] These processors can execute instructions that can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0345] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A domain name crawling method, characterized in that, The method includes: Obtain the domain name to be crawled; The domain name to be crawled is crawled using a preset crawling engine to obtain the first domain name data. The preset crawling engine is one of a variety of crawling engines. Webpage feature extraction is performed on the first domain name data to obtain webpage information features; The webpage information features are input into a recurrent neural network for domain name classification, and the classification results are obtained. Each classification result indicates the use of a different type of crawler engine. The step of extracting webpage features from the first domain name data to obtain webpage information features includes: Obtain the domain depth of the domain to be crawled and the first number of Uniform Resource Locators (URLs) contained in the domain to be crawled; Based on the domain name depth and the first quantity, obtain the URL frequency vector matrix and the URL content information matrix; Calculate the webpage information vector matrix based on the URL frequency vector matrix and the URL content information matrix; The webpage information vector matrix is ​​input into a deep learning model for calculation to obtain webpage information features; The step of obtaining the URL frequency vector matrix based on the domain name depth and the first quantity includes: Get the first occurrence count of the j-th URL at the i-th depth of the domain to be crawled, and the first total occurrence count of all URLs at the i-th depth, where i is an integer from 1 to m, and m is the total depth of the domain to be crawled. Based on the first number and the first total number, calculate the first frequency of the j-th URL at the i-th layer depth; Based on the first frequency, the domain name depth, and the first quantity, obtain the URL frequency vector matrix; The step of obtaining the URL content information matrix based on the domain name depth and the first quantity includes: Get the second number of times the content of the j-th URL appears at the i-th depth of the domain to be crawled, and the second total number of times all URL content appears at the i-th depth, where i is an integer from 1 to m, and m is the total number of depths of the domain to be crawled; Based on the second number and the second total number, calculate the second frequency of the j-th URL content at the i-th layer depth; A URL content information matrix is ​​obtained based on the second frequency, the domain name depth, and the first quantity.

2. The method according to claim 1, characterized in that, After inputting the webpage information features into a recurrent neural network for domain name classification and obtaining the classification result, the method further includes: Based on the classification results, the first queue to be crawled and scanned corresponding to the target crawler engine is obtained, wherein the target crawler engine is at least one of the multiple crawler engines indicated by the classification results. The domain name to be crawled is added to the first queue to be crawled, thus obtaining the second queue to be crawled; The domains in the second queue to be crawled are crawled sequentially to obtain the crawling results.

3. The method according to claim 1, characterized in that, The URL frequency vector matrix is ​​represented by the following formula: Where m represents the total depth of the domain name; n represents the first quantity; i represents the i-th layer depth of the domain name to be crawled; j represents the j-th URL at the i-th depth of the domain name to be crawled; Indicates the first frequency; Indicates the first number; This represents the first total number; k represents the total number of URLs at the i-th depth.

4. The method according to claim 1, characterized in that, The URL content information matrix is ​​represented by the following formula: Where m represents the total depth of the domain name; n represents the first quantity; i represents the i-th layer depth of the domain name to be crawled; j represents the j-th URL at the i-th depth of the domain name to be crawled; Indicates the second frequency; Indicates the second number; This represents the second total; k represents the total number of URLs at the i-th depth.

5. The method according to claim 1, characterized in that, The step of calculating the webpage information vector matrix based on the URL frequency vector matrix and the URL content information matrix includes: The webpage information vector matrix is ​​obtained by multiplying the URL frequency vector matrix and the URL content information matrix.

6. A network device, characterized in that, Includes memory, transceiver, and processor: A memory for storing computer programs; a transceiver for sending and receiving data under the control of the processor; and a processor for reading the computer programs from the memory and performing the following operations: Obtain the domain name to be crawled; The domain name to be crawled is crawled using a preset crawling engine to obtain the first domain name data. The preset crawling engine is one of a variety of crawling engines. Webpage feature extraction is performed on the first domain name data to obtain webpage information features; The webpage information features are input into a recurrent neural network for domain name classification, and the classification results are obtained. Each classification result indicates the use of a different type of crawler engine. When the processor extracts webpage features from the first domain name data to obtain webpage information features, it is specifically used for: Obtain the domain depth of the domain to be crawled and the first number of Uniform Resource Locators (URLs) contained in the domain to be crawled; Based on the domain name depth and the first quantity, obtain the URL frequency vector matrix and the URL content information matrix; Calculate the webpage information vector matrix based on the URL frequency vector matrix and the URL content information matrix; The webpage information vector matrix is ​​input into a deep learning model for calculation to obtain webpage information features; When the processor obtains the URL frequency vector matrix based on the domain name depth and the first quantity, it is specifically used for: Get the first occurrence count of the j-th URL at the i-th depth of the domain to be crawled, and the first total occurrence count of all URLs at the i-th depth, where i is an integer from 1 to m, and m is the total depth of the domain to be crawled. Based on the first number and the first total number, calculate the first frequency of the j-th URL at the i-th layer depth; Based on the first frequency, the domain name depth, and the first quantity, obtain the URL frequency vector matrix; When the processor obtains the URL content information matrix based on the domain name depth and the first quantity, it is specifically used for: Get the second number of times the content of the j-th URL appears at the i-th depth of the domain to be crawled, and the second total number of times all URL content appears at the i-th depth, where i is an integer from 1 to m, and m is the total number of depths of the domain to be crawled; Based on the second number and the second total number, calculate the second frequency of the j-th URL content at the i-th layer depth; A URL content information matrix is ​​obtained based on the second frequency, the domain name depth, and the first quantity.

7. The network device according to claim 6, characterized in that, After inputting the webpage information features into a recurrent neural network to classify domain names and obtain the classification results, the processor is further configured to: Based on the classification results, the first queue to be crawled and scanned corresponding to the target crawler engine is obtained, wherein the target crawler engine is at least one of the multiple crawler engines indicated by the classification results. The domain name to be crawled is added to the first queue to be crawled, thus obtaining the second queue to be crawled; The domains in the second queue to be crawled are crawled sequentially to obtain the crawling results.

8. The network device according to claim 6, characterized in that, The URL frequency vector matrix is ​​represented by the following formula: Where m represents the total depth of the domain name; n represents the first quantity; i represents the i-th layer depth of the domain name to be crawled; j represents the j-th URL at the i-th depth of the domain name to be crawled; Indicates the first frequency; Indicates the first number; This represents the first total number; k represents the total number of URLs at the i-th depth.

9. The network device according to claim 6, characterized in that, The URL content information matrix is ​​represented by the following formula: Where m represents the total depth of the domain name; n represents the first quantity; i represents the i-th layer depth of the domain name to be crawled; j represents the j-th URL at the i-th depth of the domain name to be crawled; Indicates the second frequency; Indicates the second number; This represents the second total; k represents the total number of URLs at the i-th depth.

10. The network device according to claim 6, characterized in that, When the processor calculates the webpage information vector matrix based on the URL frequency vector matrix and the URL content information matrix, it specifically performs the following tasks: The webpage information vector matrix is ​​obtained by multiplying the URL frequency vector matrix and the URL content information matrix.

11. A domain name crawling device, characterized in that, include: The first acquisition module is used to acquire the domain name to be crawled / scanned. The first crawling module is used to crawl the domain name to be crawled using a preset crawling engine to obtain the first domain name data. The preset crawling engine is one of a variety of crawling engines. The first extraction module is used to extract webpage features from the first domain name data to obtain webpage information features; The first classification module is used to input the web page information features into a recurrent neural network to classify the domain names and obtain classification results. Each classification result indicates the use of a different type of crawler engine. The first extraction module is specifically used for: Obtain the domain depth of the domain to be crawled and the first number of Uniform Resource Locators (URLs) contained in the domain to be crawled; Based on the domain name depth and the first quantity, obtain the URL frequency vector matrix and the URL content information matrix; Calculate the webpage information vector matrix based on the URL frequency vector matrix and the URL content information matrix; The webpage information vector matrix is ​​input into a deep learning model for calculation to obtain webpage information features; When the first extraction module obtains the URL frequency vector matrix based on the domain name depth and the first quantity, it is specifically used for: Get the first occurrence count of the j-th URL at the i-th depth of the domain to be crawled, and the first total occurrence count of all URLs at the i-th depth, where i is an integer from 1 to m, and m is the total depth of the domain to be crawled. Based on the first number and the first total number, calculate the first frequency of the j-th URL at the i-th layer depth; Based on the first frequency, the domain name depth, and the first quantity, obtain the URL frequency vector matrix; When the first extraction module obtains the URL content information matrix based on the domain name depth and the first quantity, it is specifically used for: Get the second number of times the content of the j-th URL appears at the i-th depth of the domain to be crawled, and the second total number of times all URL content appears at the i-th depth, where i is an integer from 1 to m, and m is the total number of depths of the domain to be crawled; Based on the second number and the second total number, calculate the second frequency of the j-th URL content at the i-th layer depth; A URL content information matrix is ​​obtained based on the second frequency, the domain name depth, and the first quantity.

12. A processor-readable storage medium, characterized in that, The processor-readable storage medium stores a computer program that causes the processor to execute the domain name crawling method according to any one of claims 1 to 5.

Images