Data crawler method and device, electronic equipment, storage medium and program product

Abstract

The application provides a data crawler method and device, electronic equipment, a storage medium and a program product, and relates to the technical field of communication. The method comprises the following steps: receiving a detection task request, wherein the detection task request comprises a to-be-detected object; performing crawling on the to-be-detected object to obtain crawler data corresponding to at least one first address and at least one first intermediate address; performing iteration on the at least one first intermediate address to obtain at least one second intermediate address, wherein the at least one second intermediate address is an expanded address; performing crawling on the at least one second intermediate address to obtain crawler data corresponding to at least one second address; and outputting the crawler data corresponding to the at least one first address and the crawler data corresponding to the at least one second address. The application can improve the crawling effect.

Description

Technical Field

[0001] This invention relates to the field of communication technology, specifically to a data crawler method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] Web crawling is a crucial technology for information gathering on the internet, enabling the rapid extraction of web page content. Common web crawling techniques typically involve parsing HyperText Markup Language (HTML) structures to obtain web page addresses, and then extracting the content from those addresses. However, in these techniques, web pages often hide the structural details of the addresses, resulting in a limited number of web page addresses obtained. This makes it difficult to extract substantial data from a limited set of addresses, leading to poor crawling performance.

[0003] It is evident that the related technologies suffer from the problem of insufficient data obtained by web crawlers, resulting in poor crawling performance. Summary of the Invention

[0004] This invention provides a data crawler method, apparatus, electronic device, storage medium, and program product to solve the problem in related technologies where crawlers obtain relatively little data, resulting in poor crawling performance.

[0005] To solve the above problems, the present invention is implemented as follows:

[0006] In a first aspect, embodiments of the present invention provide a data crawler method, comprising:

[0007] Receive a detection task request, the detection task request including the object to be detected;

[0008] The object to be detected is crawled to obtain crawler data corresponding to at least one first address and at least one first intermediate address;

[0009] The at least one first intermediate address is traversed to obtain at least one second intermediate address, wherein the at least one second intermediate address is the extended address;

[0010] The crawler performs a crawling operation on the at least one second intermediate address to obtain crawler data corresponding to the at least one second address;

[0011] Output the crawler data corresponding to the at least one first address and the crawler data corresponding to the at least one second address.

[0012] Secondly, embodiments of the present invention also provide a data crawler device, comprising:

[0013] The receiving module is used to receive a detection task request, wherein the detection task request includes an object to be detected;

[0014] The first crawler module is used to crawl the object to be detected and obtain crawler data corresponding to at least one first address and at least one first intermediate address.

[0015] The traversal module is used to traverse the at least one first intermediate address to obtain at least one second intermediate address, wherein the at least one second intermediate address is an extended address;

[0016] The second crawler module is used to crawl the at least one second intermediate address to obtain crawler data corresponding to the at least one second address.

[0017] The output module is used to output the crawler data corresponding to the at least one first address and the crawler data corresponding to the at least one second address.

[0018] Thirdly, embodiments of the present invention also provide an electronic device, including a transceiver and a processor.

[0019] The transceiver is used to receive a detection task request, which includes the object to be detected.

[0020] The processor is used to crawl the object to be detected and obtain crawler data corresponding to at least one first address and at least one first intermediate address;

[0021] The processor is configured to traverse the at least one first intermediate address to obtain at least one second intermediate address, wherein the at least one second intermediate address is an extended address;

[0022] The processor is configured to crawl the at least one second intermediate address to obtain crawler data corresponding to the at least one second address.

[0023] The transceiver is used to output the crawler data corresponding to the at least one first address and the crawler data corresponding to the at least one second address.

[0024] Fourthly, embodiments of the present invention provide an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, it implements the steps of the data crawler method described in the first aspect.

[0025] Fifthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data crawler method described in the first aspect.

[0026] In a sixth aspect, the present invention also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps in the data crawler method described in the first aspect above.

[0027] In this embodiment of the invention, a detection task request is received, the detection task request including an object to be detected; a web crawler is performed on the object to be detected to obtain crawler data corresponding to at least one first address and at least one first intermediate address; the at least one first intermediate address is traversed to obtain at least one second intermediate address, the at least one second intermediate address being an extended address; a web crawler is performed on the at least one second intermediate address to obtain crawler data corresponding to the at least one second address; and the crawler data corresponding to the at least one first address and the crawler data corresponding to the at least one second address are output. Thus, by traversing the first intermediate address to obtain the extended second intermediate address, and then performing a web crawler on the second intermediate address to obtain the crawler data corresponding to the second address, more crawler data is obtained, thereby improving the data crawling effect. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a flowchart of a data crawler method provided in an embodiment of the present invention;

[0030] Figure 2 This is a structural diagram of the address detection system provided in an embodiment of the present invention;

[0031] Figure 3 This is a structural diagram of the centralized dispatch center provided in an embodiment of the present invention;

[0032] Figure 4 This is a structural diagram of the crawler engine provided in an embodiment of the present invention;

[0033] Figure 5 This is a structural diagram of the traversal engine provided in an embodiment of the present invention;

[0034] Figure 6 This is a flowchart illustrating the process of the address detection system executing a detection task request provided in an embodiment of the present invention.

[0035] Figure 7 This is a flowchart illustrating the process of a crawler engine executing a detection task request, as provided in an embodiment of the present invention.

[0036] Figure 8 This is a flowchart illustrating the process of the traversal engine executing a detection task request, as provided in an embodiment of the present invention.

[0037] Figure 9 This is a structural diagram of a data crawler device provided in an embodiment of the present invention;

[0038] Figure 10 This is a structural diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] This invention provides a data crawler method, apparatus, electronic device, storage medium, and program product to solve the problem of poor crawling performance caused by insufficient data obtained by crawlers in related technologies.

[0041] Please see Figure 1 , Figure 1 This is a flowchart of a data crawler method provided in an embodiment of the present invention, such as... Figure 1 As shown, it includes the following steps:

[0042] Step 101: Receive a detection task request, wherein the detection task request includes the object to be detected.

[0043] The aforementioned detection task request can be a service request sent by a terminal, which is the terminal used by the user who needs to perform the detection task. The user sends the detection task request to the server through the terminal, and the server executes the detection task request after receiving it. Alternatively, the detection task request can be a request that the user configures directly on the detection device. After the user configures the detection task request on the detection device, the detection device directly executes the detection task request. The detection task request includes a detection object, which can be a set of web pages or a set of addresses, etc., and data related to the detection object is obtained by executing the detection task.

[0044] It should be noted that in this invention, the detection task request is received through a server, which can be an address detection system, such as... Figure 2 As shown, the address detection system receives and executes detection task requests. Specifically, the address detection system includes components such as a front-end display service component, a centralized scheduling center, a business processing service component, a microservice framework, a crawler engine, and a traversal engine. These components are used to receive and execute detection task requests.

[0045] The aforementioned front-end display service component is used to support the visualization display function of the address detection system. It primarily implements visual configuration functions such as detection task configuration and scheduling configuration, allowing users to determine detection task requests through this component. The aforementioned business processing service component is the core service component of the address detection system. It provides corresponding backend interfaces to the front-end display service component and other service attachments such as the centralized scheduling center, and is responsible for processing and running various business logics of the address detection system. The aforementioned centralized scheduling center is responsible for scheduling the crawler engine and traversal engine to run detection tasks and managing these tasks. The aforementioned crawler engine and traversal engine are used to obtain crawled data and addresses.

[0046] In some implementations, users configure detection tasks by displaying service components on the front end, selecting information such as detection start time, traversal directory list, crawling depth, and resources to be detected. The detection tasks are then distributed to the crawling engine and traversal engine through a centralized scheduling center. After execution, the collected crawling data is collected and relevant indicators are statistically analyzed. Finally, the data is displayed on the front end of the service component, thus realizing data crawling.

[0047] Furthermore, the various steps in the embodiments of the present invention are as follows: Figure 3 The centralized scheduling center shown implements the scheduling mechanism. It schedules the crawler engine and traversal engine to execute detection tasks and manages the lifecycle of these tasks. Specifically, the centralized scheduling center includes an Application Programming Interface (API) module, a configuration module, a statistics module, and a scheduling module.

[0048] The API module provides API interfaces to collect information reported by various crawler and traversal engines, including processor and memory usage information of the servers hosting the crawler and traversal engines, and the progress of the current detection tasks. It also uses the API module to request information from the centralized scheduling center. The configuration module provides configuration functions, including detection task configuration, detection rate control configuration, crawler depth configuration, and traversal directory and filename configuration. The statistics module is responsible for statistics on traversal directories and filenames, detection task data, and address data, such as calculating the CPU, memory, and execution time consumed by each task. The scheduling module executes detection tasks according to the scheduling configuration, monitors the lifecycle of detection tasks, and collects statistics on task status, execution time, and detection results.

[0049] Step 102: Perform a web crawler operation on the object to be detected to obtain at least one crawler data corresponding to a first address and at least one first intermediate address.

[0050] The aforementioned at least one first address is a valid address obtained through web crawling, and the at least one first address is an address related to the object to be detected. The crawling data is data obtained by crawling the first address, including related resources of the object to be detected, such as the host resources, web application resources, and other address resources of the object to be detected. The aforementioned first intermediate address is an address that may contain more data. By using the first intermediate address, more crawling data can be obtained, thereby improving the crawling effect.

[0051] In some implementations, crawling the target object can be done through methods such as... Figure 4 The crawler engine implementation shown is as follows. Specifically, the crawler engine includes a union module, a crawler parsing module, a splitting and classification module, and a filtering and verification module.

[0052] The aforementioned joint module is used to provide API interfaces, receive joint call requests and process related requests (such as requests to detect task execution), obtain server status information of the crawler engine, and call different modules in the crawler engine according to the request content; at the same time, the joint module is also used to report parameters such as the crawler engine's execution task information and server usage information, so that the centralized scheduling center can schedule different components.

[0053] The aforementioned parsing crawler module is used to parse the HTML file in the object to be detected, and sequentially discover the Uniform Resource Locator (URL) information of the valid address (i.e., the first address) and the first intermediate address contained in the HTML file. The first intermediate address can be the hypertext reference (href) binding address of the a tag, the source (src) binding address of the script tag, and other addresses obtained by regular expression matching.

[0054] The aforementioned splitting and classification module is mainly used to further split and classify the first address obtained by the crawler. It mainly includes the main body, directory (including single-level and multi-level directories), file name, and parameter parts. The first address is stored as a result through classification, and the first intermediate address is reserved for subsequent modules to process.

[0055] The aforementioned filtering and verification module is used to filter and verify the validity of addresses to obtain relevant preliminary results. It removes addresses that contain various blacklists, regular invalid addresses, and addresses that partially conflict with the domain name. Based on the filtered addresses, it makes concurrent requests and obtains verification to obtain the final valid first address.

[0056] Step 103: Traverse the at least one first intermediate address to obtain at least one second intermediate address, wherein the at least one second intermediate address is the extended address.

[0057] The second intermediate address mentioned above is the address obtained after traversing the first intermediate address. The second intermediate address is obtained by expanding the first intermediate address. By expanding the first intermediate address to obtain the second intermediate address, the number of addresses that may be able to obtain data is increased, thereby increasing the amount of data obtained by the crawler.

[0058] In some implementations, at least one first intermediate address can be traversed, and a second intermediate address can be obtained for each first intermediate address.

[0059] In some implementations, traversing the at least one first intermediate address can specifically be done through methods such as... Figure 5 The traversal engine implementation shown includes a union module, a core traversal module, a dictionary management module, and a recursive processing module.

[0060] The joint module is used to call the API interface provided by the crawler engine, receive and process the address resources (such as the first intermediate address) passed by the crawler engine, and periodically obtain the status information of the server where the traversal engine is located, the execution information of the traversal task, etc.

[0061] The core traversal module is used to traverse and process the discovery of the second intermediate address, and starts the traversal process by parsing traversal parameters and initializing configuration.

[0062] The dictionary management module is used to manage and load dictionary files, such as reading directories and filenames from dictionary files, so that the core traversal module can combine with the dictionary module to generate actual URL requests, such as combining entries in the dictionary with the base URL to form a complete request path; wherein, the dictionary file includes at least one of the following: directory dictionary, filename dictionary, backup file dictionary, and backend entry dictionary;

[0063] The recursive processing module is used to handle the recursive scanning logic, so that the discovered directories will be further scanned to find more subdirectories and files, thereby obtaining more second intermediate addresses.

[0064] Step 104: Crawl the at least one second intermediate address to obtain crawler data corresponding to the at least one second address.

[0065] The aforementioned second address is the address from which crawler data is obtained from the second intermediate address. The crawler data corresponding to the second address is additional data obtained through expansion. It should be noted that crawling the at least one second intermediate address is done in the same way as crawling the at least one first address; both methods can obtain the crawler data corresponding to the address. In some embodiments, a crawler engine is used to crawl at least one second intermediate address and at least one first address to obtain the corresponding crawler data.

[0066] Step 105: Output the crawler data corresponding to the at least one first address and the crawler data corresponding to the at least one second address.

[0067] In some implementations, the process of executing a detection task request through an address detection system is as follows: Figure 6 As shown, it includes the following steps:

[0068] a. The scheduling module of the centralized scheduling center initiates the detection task. This detection task is one that meets the activation conditions. These conditions can be that the resources required by the detection task are less than the remaining hardware resources of the current address detection system. The resources required by the detection task are estimated using an algorithm based on the object to be detected. The remaining hardware resources of the current address detection system are obtained by scanning the hardware of the address detection system.

[0069] b. Send detection task requests to the crawler engine via the API interface. Before sending the detection task request, first parse the detection task to obtain the object to be tested and the detection configuration, then verify the validity of the detection task request, and add valid detection task requests to the message queue in sequence to wait for sending and execution.

[0070] c. The crawler engine retrieves the detection task requests from the message queue and begins execution.

[0071] d. After execution, the crawler engine actively sends the first intermediate address to the traversal engine.

[0072] e. The traversal engine traverses the first intermediate address.

[0073] f. The traversal engine sends an extended second intermediate address to the crawler engine.

[0074] g. The traversal engine sends a traversal completion message to the statistics module of the centralized scheduling center.

[0075] h. After the crawler engine completes crawling the second intermediate address, it sends crawling completion information to the statistics module of the centralized scheduling center.

[0076] i. After receiving the task completion information, the statistics module of the centralized scheduling center stores the task completion information, including time, completion status, and the engine that executed the detection task request.

[0077] In this embodiment of the invention, a detection task request is received, the detection task request including an object to be detected; a web crawler is performed on the object to be detected to obtain crawler data corresponding to at least one first address and at least one first intermediate address; the at least one first intermediate address is traversed to obtain at least one second intermediate address, the at least one second intermediate address being an extended address; a web crawler is performed on the at least one second intermediate address to obtain crawler data corresponding to the at least one second address; and the crawler data corresponding to the at least one first address and the crawler data corresponding to the at least one second address are output. Thus, by traversing the first intermediate address to obtain the extended second intermediate address, and then performing a web crawler on the second intermediate address to obtain the crawler data corresponding to the second address, more crawler data is obtained, thereby improving the data crawling effect.

[0078] In one embodiment, the step of crawling the object to be detected to obtain crawler data corresponding to at least one first address and at least one first intermediate address includes:

[0079] The object to be detected is parsed using Hypertext Markup Language (HTML) to obtain at least one first address;

[0080] The at least one first address is split to obtain a main part, a directory part, a file part, and a parameter part;

[0081] The at least one first intermediate address is generated based on the main body, the directory, and the file.

[0082] In this embodiment of the invention, the object to be detected is parsed using Hypertext Markup Language (HTML) to obtain at least one first address; the at least one first address is then split into a main body, a directory, a file, and a parameter; and at least one first intermediate address is generated based on the main body, the directory, and the file. Thus, by generating a first intermediate address from a first address, a second intermediate address can be derived from the first intermediate address, thereby obtaining more addresses from which data can be crawled.

[0083] It should be noted that the first address includes a main body, a directory, a file, and a parameter. Among the different addresses related to the object to be detected, the main body, directory, or file usually have matching or association. By generating at least one first intermediate address based on the main body, the directory, and the file, and expanding the first intermediate address to obtain a second intermediate address that is still related to the object to be detected, the data obtained by the crawler from the second intermediate address is still related to the object to be detected, thus improving the accuracy of the crawler.

[0084] In one embodiment, when the at least one address includes at least two first addresses, after performing Hypertext Markup Language (HTML) parsing on the object to be detected to obtain the at least one first address, the method further includes:

[0085] The at least two first addresses are filtered based on preset rules to obtain the filtered first addresses;

[0086] The step of splitting the at least one first address to obtain a main body, a directory, a file, and a parameter part includes:

[0087] The filtered first address is split to obtain the main body part, the directory part, the file part, and the parameter part;

[0088] The preset rules include at least one of the following:

[0089] Filter out duplicate addresses;

[0090] Filter addresses included in the preset list.

[0091] It should be noted that some of the addresses obtained through the crawler are not very valuable or even irrelevant to the resources to be crawled. These include a large number of public network addresses, a small number of intranet addresses, Content Delivery Network (CDN) addresses, and a large number of static resource Cascading Style Sheets (CSS) file addresses. These addresses are quite numerous, so it is necessary to remove these worthless addresses to improve the efficiency of the subsequent traversal engine and reduce resource consumption to process more valuable addresses.

[0092] In this embodiment of the invention, the at least two first addresses are filtered based on preset rules to obtain filtered first addresses, making the first addresses non-repeating and accessible, thereby improving the security and efficiency of the crawler.

[0093] The preset list includes addresses that are either blacklisted or specific addresses. Blacklisted addresses are those deemed too risky to access, while specific addresses are those that cannot be crawled for security and confidentiality reasons. This filtering process using the preset list ensures that the first set of filtered addresses are all accessible.

[0094] In one embodiment, when the preset rule includes filtering duplicate addresses, filtering the at least two first addresses based on the preset rule to obtain the filtered first addresses includes:

[0095] Calculate the first distance corresponding to the main body portion, the second distance corresponding to the directory portion, and the third distance corresponding to the file portion between the target first address and other first addresses, wherein the target first address is one of the at least two first addresses;

[0096] The first distance, the second distance, and the third distance are weighted to obtain the target similarity score between the target first address and the other first addresses;

[0097] If the target similarity score is less than a preset threshold, delete the target first address from the at least two first addresses.

[0098] In this embodiment of the invention, a first distance corresponding to the main body portion, a second distance corresponding to the directory portion, and a third distance corresponding to the file portion are calculated between the target first address and other first addresses. The target first address is one of the at least two first addresses. The first distance, the second distance, and the third distance are weighted to obtain a target similarity score between the target first address and the other first addresses. If the target similarity score is less than a preset threshold, the target first address is deleted from the at least two first addresses. Thus, by calculating the target similarity score between the target first address and other first addresses, it is determined whether the target first address needs to be deleted, thereby deleting addresses with high similarity scores from at least two first addresses, ensuring that there are no duplicate addresses in the filtered first addresses.

[0099] The target similarity score can be expressed by the following formula:

[0100]

[0101] In the formula, S represents the target similarity score, a, b, and c are weighting coefficients, d1 is the first distance, d2 is the second distance, and d3 is the third distance. In one implementation, a is greater than 1, and b and c are less than 1.

[0102] Thus, if the target similarity score S is less than the preset threshold, the target first address is considered to be duplicated with other first addresses. At this time, the target first address is deleted to remove the duplicate first address.

[0103] In other implementations, the distance between the target first address and other first addresses can also be calculated using the Minkowski metric, as shown in the following formula:

[0104]

[0105] Where d represents the distance between addresses, x represents the target first address, y represents other first addresses, and p represents a variable parameter.

[0106] In one embodiment, calculating the first distance corresponding to the main portion, the second distance corresponding to the directory portion, and the third distance corresponding to the file portion between the target first address and other first addresses includes at least one of the following:

[0107] If the target first address matches the target portion of the other first addresses, the distance corresponding to the target portion is set to a first preset value, where the target portion is the main body portion, the directory portion, or the file portion;

[0108] If the target first address does not match the target portion of the other first addresses, the distance corresponding to the target portion is set to a second preset value.

[0109] In this embodiment of the invention, when the target first address matches the target portion of another first address, the distance corresponding to the target portion is set to a first preset value, where the target portion is the main body portion, the directory portion, or the file portion; when the target first address does not match the target portion of another first address, the distance corresponding to the target portion is set to a second preset value. Thus, by pre-setting the distances corresponding to different portions, the pre-set distances are directly used when calculating the distance between the target first address and other first addresses, improving the accuracy of the calculation and enabling a quick determination of whether the target first address overlaps with other first addresses.

[0110] Specifically, the distance between the target first address and other first addresses can be calculated using the following formula:

[0111]

[0112] x and y are the components of the target's first address and other first addresses, respectively, and d 拆分部分 Let n be the first distance, the second distance, or the third distance, where n is the first set value and m is the second set value. That is, if the split parts are exactly the same, the distance between the two split parts is n; if the split parts are completely different, the distance between the two split parts is 1+m.

[0113] In some implementations, the following algorithm can be used to delete duplicate first addresses.

[0114] Algorithm 1 Address Removal Algorithm

[0115] Input: The target first address X = [X1, X2, X3...] and a list of other first addresses Y = [Y1, Y2, Y3...]

[0116] Output: The first address after filtering, Z = [Z1, Z2, Z3...]

[0117] Let i = 0, j = 0

[0118] for iin len(X)do

[0119] for j in len(Y)do

[0120] #Splitting the main body, directory, and file portions of the Xi and Xj addresses

[0121] #Calculate the distance d for each part, and assign the values ​​1, 0.1, and 0.1 to abc respectively.

[0122] Si = s(i,j)

[0123] If (Si>100)

[0124] #Reserved Address

[0125] ELSE

[0126] #Remove Address

[0127] Thus, the algorithm described above is used to delete duplicate first addresses.

[0128] In some implementations, the process by which the crawler engine executes a detection task request is as follows: Figure 7 As shown, the specific steps include the following:

[0129] a. The crawler engine's combined module performs HTML parsing to obtain crawler tasks, and then sends the crawler tasks to the parsing crawler module.

[0130] b. After receiving the crawling task, the crawling module of the crawling engine executes the crawling task according to the predefined rules, obtains the first address, and sends the first address to the splitting and classification module.

[0131] c. After receiving the first address, the splitting and classification module of the crawler engine splits the first address into the main body, directory, file and parameter parts, and generates the first intermediate address.

[0132] d. The crawler engine's splitting and classification module sends the classified first address to the filtering and verification module.

[0133] e. The web crawler engine's filtering and verification module filters the first address and promptly sends web requests to verify its validity.

[0134] f. The crawler engine's parsing module, while synchronously processing the first address, also asynchronously parses the HTML file to discover more first addresses.

[0135] g. The crawler engine's parsing crawler module completes all tasks and sends completion information to the union module.

[0136] h. Repeat the parallel crawling process, iterating and parsing to obtain new addresses. Repeat the above steps until no more new addresses are found.

[0137] Furthermore, the crawler engine proxy module collects resource information reported by the traversal engine at regular intervals, such as processor and memory information, traversal task information, and reports the address to be traversed when the processor and memory information is sufficient.

[0138] In one embodiment, traversing the at least one first intermediate address to obtain at least one second intermediate address includes:

[0139] Obtain the configuration file, which includes a directory dictionary and / or a filename dictionary;

[0140] Based on the configuration file, the directory portion or filename portion in the at least one first intermediate address is replaced to obtain multiple extended addresses;

[0141] By traversing the plurality of extended addresses, at least one second intermediate address is obtained, wherein the at least one second intermediate address is an address among the plurality of extended addresses that contains a response.

[0142] In this embodiment of the invention, a configuration file is obtained, which includes a directory dictionary and / or a filename dictionary. Based on the configuration file, the directory portion or filename portion in the at least one first intermediate address is replaced to obtain multiple extended addresses. The multiple extended addresses are traversed to obtain the at least one second intermediate address, which is an address among the multiple extended addresses that has a response. In this way, by using the directory dictionary and / or filename dictionary to extend the first intermediate address to obtain the second intermediate address, the amount of crawler data obtained through the second intermediate address can be increased.

[0143] In some embodiments, the directory portion or filename portion in the at least one first intermediate address is replaced based on the configuration file to obtain multiple extended addresses. Alternatively, the directory portion in the first intermediate address may be replaced based on the directories included in the directory dictionary to obtain a second intermediate address.

[0144] In some embodiments, the directory portion or filename portion in the at least one first intermediate address is replaced based on the configuration file to obtain multiple extended addresses. Alternatively, the filename portion in the first intermediate address can be replaced based on the filename included in the filename dictionary to obtain a second intermediate address.

[0145] In some embodiments, the directory portion or filename portion in the at least one first intermediate address is replaced based on the configuration file to obtain multiple extended addresses. This can be achieved by simultaneously using a directory dictionary and a filename dictionary, replacing the directory portion in the first intermediate address with the directories included in the directory dictionary, and replacing the filename portion in the first intermediate address with the filenames included in the filename dictionary to obtain a second intermediate address.

[0146] In one embodiment, before replacing the directory portion or filename portion in the at least one first intermediate address based on the configuration file to obtain multiple extended addresses, the method further includes:

[0147] Obtain preset recursion parameters, which include recursion depth and / or recursion count;

[0148] The process of replacing the directory or filename portion of the at least one first intermediate address based on the configuration file yields multiple extended addresses, including:

[0149] Based on the preset recursive parameters and the configuration file, the directory portion or filename portion in the at least one first intermediate address is replaced to obtain the plurality of extended addresses.

[0150] In this embodiment of the invention, preset recursion parameters are obtained, including recursion depth and / or recursion count. Based on the preset recursion parameters and the configuration file, the directory portion or filename portion in at least one first intermediate address is replaced to obtain the plurality of extended addresses. Thus, by extending using preset recursion parameters, the number of extended second intermediate addresses can be set according to requirements, thereby adjusting the amount of crawler data corresponding to the second addresses obtained by the crawler.

[0151] For example, when it is necessary to improve the efficiency of the crawler and not to obtain too much crawling data, the recursion depth and / or the number of recursions can be set to be small, thus obtaining fewer second intermediate addresses; when it is necessary to obtain too much crawling data, the recursion depth and / or the number of recursions can be set to be large, thus obtaining more second intermediate addresses, and thus the crawler obtains more data.

[0152] In some implementations, the process of traversing the engine to execute detection task requests is as follows: Figure 8 As shown, the specific steps include the following:

[0153] a. The union module of the traversal engine sends the first intermediate address to the core traversal module.

[0154] b. The dictionary management module of the traversal engine loads directory dictionaries and / or file name dictionaries based on the configuration file.

[0155] c. The core traversal module of the traversal engine concatenates the addresses according to the dictionary to obtain the concatenated addresses, and then begins traversing the concatenated addresses to obtain the second intermediate address where the response exists.

[0156] d. The recursion processing module of the traversal engine reads the recursion depth and the number of recursions.

[0157] e. The recursive processing module of the traversal engine repeatedly expands to complete the recursive task.

[0158] f. After the task is completed, send a completion message to the joint module.

[0159] Furthermore, the traversal engine's joint module continuously monitors the crawler engine, reports its basic resource information, and obtains its resource utilization status. This resource information allows the usage of both the traversal and crawler engines to be determined, enabling the centralized scheduling center to decide whether to schedule new tasks.

[0160] Please see Figure 9 , Figure 9 This is a structural diagram of a data crawler device provided in an embodiment of the present invention, as shown below. Figure 9 As shown, the data crawler device 900 includes:

[0161] The receiving module 901 is used to receive a detection task request, wherein the detection task request includes an object to be detected;

[0162] The first crawler module 902 is used to crawl the object to be detected and obtain crawler data corresponding to at least one first address and at least one first intermediate address;

[0163] Traversal module 903 is used to traverse the at least one first intermediate address to obtain at least one second intermediate address, wherein the at least one second intermediate address is an extended address;

[0164] The second crawler module 904 is used to crawl the at least one second intermediate address to obtain crawler data corresponding to the at least one second address.

[0165] The output module 905 is used to output the crawler data corresponding to the at least one first address and the crawler data corresponding to the at least one second address.

[0166] In one embodiment, the first crawler module 902 includes:

[0167] The parsing submodule is used to perform Hypertext Markup Language (HTML) parsing on the object to be detected to obtain at least one first address;

[0168] The splitting submodule is used to split the at least one first address to obtain a main part, a directory part, a file part, and a parameter part;

[0169] A generation submodule is used to generate the at least one first intermediate address based on the main body, the directory, and the file.

[0170] In one embodiment, the first crawler module 902 further includes:

[0171] The filtering submodule is used to filter the at least two first addresses based on preset rules to obtain the filtered first addresses;

[0172] The splitting sub-module includes:

[0173] The splitting unit is used to split the filtered first address to obtain the main part, the directory part, the file part, and the parameter part;

[0174] The preset rules include at least one of the following:

[0175] Filter out duplicate addresses;

[0176] Filter addresses included in the preset list.

[0177] In one embodiment, when the preset rule includes filtering duplicate addresses, the filtering submodule includes:

[0178] A calculation unit is configured to calculate a first distance between the target first address and other first addresses, corresponding to the first distance of the main body portion, a second distance of the directory portion, and a third distance of the file portion, wherein the target first address is one of the at least two first addresses;

[0179] The weighting unit is used to weight the first distance, the second distance and the third distance to obtain the target similarity score between the target first address and the other first addresses;

[0180] The deletion unit is used to delete the target first address from the at least two first addresses when the target similarity score is less than a preset threshold.

[0181] In one embodiment, the computing unit includes at least one of the following:

[0182] The first calculation subunit is configured to set the distance corresponding to the target part to a first preset value when the target first address matches the target part of the other first addresses, wherein the target part is the main part, the directory part, or the file part;

[0183] The second calculation subunit is used to set the distance corresponding to the target portion to a second preset value when the target first address does not match the target portion of the other first addresses.

[0184] In one embodiment, the traversal module 903 includes:

[0185] The first acquisition submodule is used to acquire a configuration file, which includes a directory dictionary and / or a file name dictionary;

[0186] The replacement submodule is used to replace the directory portion or filename portion in the at least one first intermediate address based on the configuration file to obtain multiple extended addresses;

[0187] The traversal submodule is used to traverse the plurality of extended addresses to obtain at least one second intermediate address, wherein the at least one second intermediate address is an address among the plurality of extended addresses that has a corresponding response.

[0188] In one embodiment, prior to the replacement submodule, the traversal module 903 further includes:

[0189] The second acquisition submodule is used to acquire preset recursion parameters, which include recursion depth and / or recursion count;

[0190] The replacement submodule includes:

[0191] The replacement unit is used to replace the directory portion or filename portion in the at least one first intermediate address based on the preset recursive parameters and the configuration file, so as to obtain the plurality of extended addresses.

[0192] The data crawler device provided in this embodiment of the invention can implement each process of the above-described data crawler method, with one-to-one correspondence of technical features and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0193] It should be noted that the data crawler device in the embodiments of the present invention can be a device, or it can be a component, integrated circuit, or chip in an electronic device.

[0194] This invention also provides an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor. When the program is executed by the processor, it implements the various processes of the above-described data crawler method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0195] For details, see Figure 10As shown, this embodiment of the invention also provides an electronic device, including a bus 1001, a transceiver 1002, an antenna 1003, a bus interface 1004, a processor 1005, and a memory 1006.

[0196] The transceiver 1002 is used to receive a detection task request, the detection task request including the object to be detected;

[0197] The processor 1005 is used to crawl the object to be detected and obtain crawler data corresponding to at least one first address and at least one first intermediate address.

[0198] The processor 1005 is configured to traverse the at least one first intermediate address to obtain at least one second intermediate address, wherein the at least one second intermediate address is an extended address;

[0199] The processor 1005 is used to crawl the at least one second intermediate address to obtain crawler data corresponding to the at least one second address.

[0200] The transceiver 1002 is used to output the crawler data corresponding to the at least one first address and the crawler data corresponding to the at least one second address.

[0201] In one embodiment, the step of crawling the object to be detected to obtain crawler data corresponding to at least one first address and at least one first intermediate address includes:

[0202] The object to be detected is parsed using Hypertext Markup Language (HTML) to obtain at least one first address;

[0203] The at least one first address is split to obtain a main part, a directory part, a file part, and a parameter part;

[0204] The at least one first intermediate address is generated based on the main body, the directory, and the file.

[0205] In one embodiment, when the at least one address includes at least two first addresses, after performing Hypertext Markup Language (HTML) parsing on the object to be detected to obtain the at least one first address, the method further includes:

[0206] The at least two first addresses are filtered based on preset rules to obtain the filtered first addresses;

[0207] The step of splitting the at least one first address to obtain a main body, a directory, a file, and a parameter part includes:

[0208] The filtered first address is split to obtain the main body part, the directory part, the file part, and the parameter part;

[0209] The preset rules include at least one of the following:

[0210] Filter out duplicate addresses;

[0211] Filter addresses included in the preset list.

[0212] In one embodiment, when the preset rule includes filtering duplicate addresses, filtering the at least two first addresses based on the preset rule to obtain the filtered first addresses includes:

[0213] Calculate the first distance corresponding to the main body portion, the second distance corresponding to the directory portion, and the third distance corresponding to the file portion between the target first address and other first addresses, wherein the target first address is one of the at least two first addresses;

[0214] The first distance, the second distance, and the third distance are weighted to obtain the target similarity score between the target first address and the other first addresses;

[0215] If the target similarity score is less than a preset threshold, delete the target first address from the at least two first addresses.

[0216] In one embodiment, calculating the first distance corresponding to the main portion, the second distance corresponding to the directory portion, and the third distance corresponding to the file portion between the target first address and other first addresses includes at least one of the following:

[0217] If the target first address matches the target portion of the other first addresses, the distance corresponding to the target portion is set to a first preset value, where the target portion is the main body portion, the directory portion, or the file portion;

[0218] If the target first address does not match the target portion of the other first addresses, the distance corresponding to the target portion is set to a second preset value.

[0219] In one embodiment, traversing the at least one first intermediate address to obtain at least one second intermediate address includes:

[0220] Obtain the configuration file, which includes a directory dictionary and / or a filename dictionary;

[0221] Based on the configuration file, the directory portion or filename portion in the at least one first intermediate address is replaced to obtain multiple extended addresses;

[0222] By traversing the plurality of extended addresses, at least one second intermediate address is obtained, wherein the at least one second intermediate address is an address among the plurality of extended addresses that contains a response.

[0223] In one embodiment, before replacing the directory portion or filename portion in the at least one first intermediate address based on the configuration file to obtain multiple extended addresses, the method further includes:

[0224] Obtain preset recursion parameters, which include recursion depth and / or recursion count;

[0225] The process of replacing the directory or filename portion of the at least one first intermediate address based on the configuration file yields multiple extended addresses, including:

[0226] Based on the preset recursive parameters and the configuration file, the directory portion or filename portion in the at least one first intermediate address is replaced to obtain the multiple extended addresses.

[0227] exist Figure 10 In this document, a bus architecture (represented by bus 1001) is used. Bus 1001 may include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 1005 and memory represented by memory 1006. Bus 1001 may also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 1004 provides an interface between bus 1001 and transceiver 1002. Transceiver 1002 may be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 1005 is transmitted over a wireless medium via antenna 1003, which further receives data and transmits it to processor 1005.

[0228] Processor 1005 is responsible for managing bus 1001 and general processing, and can also provide various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. Memory 1006 can be used to store data used by processor 1005 during operation.

[0229] Optionally, the processor 1005 can be a CPU, ASIC, FPGA, or CPLD.

[0230] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described data crawler method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0231] The present invention also provides a computer program product, including computer instructions that, when executed by a processor, implement the above-described... Figure 1 The various processes of the corresponding data crawling method embodiments can achieve the same technical effect, and will not be described again here to avoid repetition.

[0232] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0233] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0234] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A data crawling method, characterized in that, include: Receive a detection task request, the detection task request including the object to be detected; The object to be detected is crawled to obtain crawler data corresponding to at least one first address and at least one first intermediate address; The at least one first intermediate address is traversed to obtain at least one second intermediate address, wherein the at least one second intermediate address is an extended address; The crawler performs a crawling operation on the at least one second intermediate address to obtain crawler data corresponding to the at least one second address; Output the crawler data corresponding to the at least one first address and the crawler data corresponding to the at least one second address.

2. The method as described in claim 1, characterized in that, The step of crawling the object to be detected to obtain crawler data corresponding to at least one first address and at least one first intermediate address includes: The object to be detected is parsed using Hypertext Markup Language (HTML) to obtain at least one first address; The at least one first address is split to obtain a main part, a directory part, a file part, and a parameter part; The at least one first intermediate address is generated based on the main body, the directory, and the file.

3. The method as described in claim 2, characterized in that, When the at least one address includes at least two first addresses, after performing Hypertext Markup Language (HTML) parsing on the object to be detected to obtain the at least one first address, the method further includes: The at least two first addresses are filtered based on preset rules to obtain the filtered first addresses; The step of splitting the at least one first address to obtain a main body, a directory, a file, and a parameter part includes: The filtered first address is split to obtain the main body part, the directory part, the file part, and the parameter part; The preset rules include at least one of the following: Filter out duplicate addresses; Filter addresses included in the preset list.

4. The method as described in claim 3, characterized in that, When the preset rule includes filtering duplicate addresses, the step of filtering the at least two first addresses based on the preset rule to obtain the filtered first addresses includes: Calculate the first distance corresponding to the main body portion, the second distance corresponding to the directory portion, and the third distance corresponding to the file portion between the target first address and other first addresses, wherein the target first address is one of the at least two first addresses; The first distance, the second distance, and the third distance are weighted to obtain the target similarity score between the target first address and the other first addresses; If the target similarity score is less than a preset threshold, delete the target first address from the at least two first addresses.

5. The method as described in claim 4, characterized in that, The calculation of the first distance corresponding to the main body portion, the second distance corresponding to the directory portion, and the third distance corresponding to the file portion between the target first address and other first addresses includes at least one of the following: If the target first address matches the target portion of the other first addresses, the distance corresponding to the target portion is set to a first preset value, where the target portion is the main body portion, the directory portion, or the file portion; If the target first address does not match the target portion of the other first addresses, the distance corresponding to the target portion is set to a second preset value.

6. The method as described in claim 1, characterized in that, The step of traversing the at least one first intermediate address to obtain at least one second intermediate address includes: Obtain the configuration file, which includes a directory dictionary and / or a filename dictionary; Based on the configuration file, the directory portion or filename portion in the at least one first intermediate address is replaced to obtain multiple extended addresses; By traversing the plurality of extended addresses, at least one second intermediate address is obtained, wherein the at least one second intermediate address is an address among the plurality of extended addresses that contains a response.

7. The method as described in claim 6, characterized in that, Before replacing the directory portion or filename portion in the at least one first intermediate address based on the configuration file to obtain multiple extended addresses, the method further includes: Obtain preset recursion parameters, which include recursion depth and / or recursion count; The process of replacing the directory or filename portion of the at least one first intermediate address based on the configuration file yields multiple extended addresses, including: Based on the preset recursive parameters and the configuration file, the directory portion or filename portion in the at least one first intermediate address is replaced to obtain the multiple extended addresses.

8. A data crawler device, characterized in that, include: The receiving module is used to receive a detection task request, wherein the detection task request includes an object to be detected; The first crawler module is used to crawl the object to be detected and obtain crawler data corresponding to at least one first address and at least one first intermediate address. The traversal module is used to traverse the at least one first intermediate address to obtain at least one second intermediate address, wherein the at least one second intermediate address is an extended address; The second crawler module is used to crawl the at least one second intermediate address to obtain crawler data corresponding to the at least one second address. The output module is used to output the crawler data corresponding to the at least one first address and the crawler data corresponding to the at least one second address.

9. An electronic device, characterized in that, Including transceivers and processors, The transceiver is used to receive a detection task request, which includes the object to be detected. The processor is used to crawl the object to be detected and obtain crawler data corresponding to at least one first address and at least one first intermediate address; The processor is configured to traverse the at least one first intermediate address to obtain at least one second intermediate address, wherein the at least one second intermediate address is an extended address; The processor is configured to crawl the at least one second intermediate address to obtain crawler data corresponding to the at least one second address. The transceiver is used to output the crawler data corresponding to the at least one first address and the crawler data corresponding to the at least one second address.

10. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the data crawler method as described in any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data crawler method as described in any one of claims 1 to 7.

12. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the data crawler method as described in any one of claims 1 to 7.

Images