Network data capture method and device using anti-honeypot technology

Abstract

The present invention discloses a network data crawling method and device with anti-honeypot technology, which is applied to the field of software development technology, including: receiving an initial access link; simulating a browser request through a crawler to access a site through the initial access link; receiving response information returned by the site; determining multiple accessible links and path information; using RPA to open the site page corresponding to the initial access link; using RPA to obtain a handle corresponding to the link through path information for each accessible link, determining the position information of the handle on the site page, and marking the handle; determining whether the link corresponding to the handle is a honeypot based on the handle position information and the marking result; prohibiting access to the link when the link corresponding to the handle is a honeypot; and repeating the above steps with the link as the initial access link when the link corresponding to the handle is not a honeypot, until all links are processed. The present invention can optimize the crawler's ability to crawl network information, avoid honeypot traps, and collect more information.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a network data capture method and device using an anti-honeypot technology. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the invention that are recited in the claims. No statement herein is admitted to be prior art by virtue of its inclusion in this section.

[0003] With the development of the internet, the demand for data collection using crawlers has continued to grow, especially among search engines. Crawlers are often used to capture large amounts of data in a short period of time, which can put a certain amount of traffic pressure on the target website. Therefore, website operators may employ various techniques to prevent crawlers from collecting data, such as using the User-Agent header to prevent crawlers from collecting data, checking or limiting access frequency, and using honeypot technology. Crawlers, in turn, may also employ countermeasures against these techniques, such as setting request headers, setting random access frequencies, and using proxy IP addresses.

[0004] In common honeypot technology, links are intentionally left on web pages that humans cannot see or would never click. Since crawlers retrieve content from source code, they might access such links. In this case, if the website detects an IP address accessing this link, the IP address, User-Agent, Mac address, and other information that could be used to identify the visitor are immediately and permanently blocked. When targeting this honeypot technology, most crawlers employ a targeted crawling approach, accessing only specific portions of the target content and striving to avoid triggering honeypots on web pages where they might be present.

[0005] The aforementioned directional crawler method only crawls content within a specific path. This approach has two drawbacks: First, even if a specific crawler path is determined, there's no guarantee that it will be free of honeypots. Finding a suitable directional path requires a certain amount of trial and error (e.g., IP pools). Second, due to path limitations, directional crawlers can only crawl content within a specific path, resulting in limited information acquisition and reduced efficiency. Summary of the Invention

[0006] An embodiment of the present invention provides a network data crawling method using anti-honeypot technology to optimize the crawler's ability to crawl network information, avoid honeypot traps, and collect more information. The method includes:

[0007] Receive the initial access link to the Site;

[0008] Use crawlers to simulate browser requests and access the site through the initial access link;

[0009] Receive the response information returned by the site;

[0010] Determining a plurality of accessible links and path information of each accessible link according to the response information;

[0011] Use Robotic Process Automation (RPA) to open the site page corresponding to the initial access link;

[0012] For each accessible link, use RPA to obtain the link's corresponding handle based on the path information. Then, operate on the handle to determine its location on the site page, and then mark the handle for display on the site page.

[0013] Determine whether the link corresponding to the handle is a honeypot based on the handle's location information and display marking results;

[0014] When the link corresponding to the handle is a honeypot, access to the link is prohibited;

[0015] When the link corresponding to the handle is not a honeypot, the link is re-used as the initial access link and the above steps are repeated until all accessible links are processed.

[0016] The embodiment of the present invention further provides a network data capture device using anti-honeypot technology to optimize the crawler's ability to capture network information, avoid honeypot traps, and collect more information. The device includes:

[0017] A link receiving module, used for receiving the initial access link of the site;

[0018] The crawler module is used to simulate browser requests through crawlers and access the site through the initial access link;

[0019] Response receiving module, used to receive the response information returned by the site;

[0020] a link determination module, configured to determine a plurality of accessible links and path information of each accessible link according to the response information;

[0021] The RPA module is used to use Robotic Process Automation (RPA) to open the site page corresponding to the initial access link;

[0022] The handle module is used to obtain the handle corresponding to each accessible link through the path information using RPA, determine the location of the handle on the site page by operating on the handle, and display the handle on the site page.

[0023] The honeypot judgment module is used to determine whether the link corresponding to the handle is a honeypot based on the handle's location information and the display marking result;

[0024] The execution module is used to prohibit access to the link when the link corresponding to the handle is a honeypot; when the link corresponding to the handle is not a honeypot, the link is re-used as the initial access link and the above modules are repeatedly triggered until all accessible links are processed.

[0025] An embodiment of the present invention further provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the network data capture method using the anti-honeypot technology described above is implemented.

[0026] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the network data capture method using the anti-honeypot technology is implemented.

[0027] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the network data crawling method of the anti-honeypot technology is implemented.

[0028] In an embodiment of the present invention, an initial access link of a site is received; a crawler simulates a browser request and accesses the site through the initial access link; a response message returned by the site is received; multiple accessible links and path information of each accessible link are determined based on the response information; a robotic process automation (RPA) is used to open the site page corresponding to the initial access link; for each accessible link, the RPA is used to obtain a handle corresponding to the link based on the path information, the handle is operated on to determine the position information of the handle on the site page, and the handle is displayed on the site page; based on the position information of the handle and the display mark result, it is determined whether the link corresponding to the handle is a honeypot; if the link corresponding to the handle is a honeypot, access to the link is prohibited; if the link corresponding to the handle is not a honeypot, the link is re-used as the initial access link, and the above steps are repeated until all accessible links are processed. Compared with the directional crawler solution in the prior art, it can realize the survey of possible honeypots on the web page, avoid possible honeypot traps, no longer be restricted to a specified path, and achieve larger-scale data crawling. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0030] Figure 1Flowchart of a network data capture method using the anti-honeypot technology according to an embodiment of the present invention;

[0031] Figure 2 A diagram showing a specific example of a network data capture method using the anti-honeypot technology according to an embodiment of the present invention;

[0032] Figure 3 This is a structural block diagram of a network data capture device using the anti-honeypot technology according to an embodiment of the present invention;

[0033] Figure 4 This is an interactive diagram of a network data capture device using the anti-honeypot technology in an embodiment of the present invention;

[0034] Figure 5 Schematic diagram of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] To make the purpose, technical solutions and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0036] First, the terms involved in the present invention are explained.

[0037] RPA (Robotic Process Automation) is an agile product that connects systems and reengineers processes. By simulating and enhancing the interaction between humans and computers, it rapidly connects internal and external systems, as well as heterogeneous systems, without disrupting existing IT systems. This allows for business process reengineering and reduces repetitive, low-value labor.

[0038] Crawler: A program or script that automatically crawls World Wide Web information according to certain rules.

[0039] Honeypot: Originated from network attack and defense, it refers to some traps used by website servers to distinguish normal visitors from crawlers. When a visitor accesses a hidden honeypot, the visitor is judged as an abnormal visitor and precautions are taken.

[0040] XPath (XML Path Language): A language used to locate a part of an XML document. Based on the tree structure of XML, it provides the ability to find nodes in the data structure tree.

[0041] Figure 1 The flowchart of the network data crawling method of the anti-honeypot technology in an embodiment of the present invention includes:

[0042] Step 101: receiving an initial access link of a site;

[0043] Step 102: Using a crawler to simulate a browser request, access the site through the initial access link;

[0044] Step 103: Receive response information returned by the site;

[0045] Step 104: determining a plurality of accessible links and path information of each accessible link according to the response information;

[0046] Step 105: Use Robotic Process Automation (RPA) to open the site page corresponding to the initial access link.

[0047] Step 106: For each accessible link, use RPA to obtain the link's corresponding handle based on the path information, operate on the handle to determine the handle's location on the site page, and display the handle on the site page.

[0048] Step 107: Determine whether the link corresponding to the handle is a honeypot based on the handle's location information and the display mark result;

[0049] Step 108: When the link corresponding to the handle is a honeypot, access to the link is prohibited; when the link corresponding to the handle is not a honeypot, the link is re-used as the initial access link, and the above steps are repeated until all accessible links are processed.

[0050] In one embodiment, the initial access link is not a honeypot.

[0051] In one embodiment, a crawler simulates a browser request and accesses a site through an initial access link. A request (request method, request URL, request header, request body) can be composed of code, and then the initial access link sends the request to the site and waits for the site's response.

[0052] In one embodiment, the response information is returned by the site to the requester (client, browser, crawler), and can be divided into three parts: Response Status Code, Response Headers, and Response Body.

[0053] In one embodiment, the response information includes HTML code; determining multiple accessible links and path information of each accessible link based on the response information includes: searching for hyperlinks in the HTML code; determining multiple accessible links and path information of each accessible link based on the found hyperlinks.

[0054] In this embodiment, the identified multiple accessible links can be formed into a link list. The response body in the response message is the textual data of the response message, namely, the HTML code that constitutes the website page. HTML code is a markup language, and one type of keyword is the hyperlink href. These keyword-marked locations are addresses that can jump to other pages within the website page. Simply finding and organizing all the hyperlinks in the HTML code will form a list of accessible links.

[0055] In one embodiment, Robotic Process Automation (RPA) is used to simulate a human opening the site page corresponding to the initial access link. Because RPA technology simulates human operations, these operations are generally permitted by legitimate websites and are less likely to be blocked, making it a suitable auxiliary tool for crawlers. Using RPA technology to identify potential honeypot traps allows crawlers to avoid them, enabling larger-scale data crawling.

[0056] In one embodiment, for each accessible link, RPA is used to obtain the link's corresponding handle through the path information. As mentioned above, HTML code is a markup language consisting of HTML tags and text, similar to this: <h1> This is the title< / h1> , This is the body paragraph , so that the angle brackets (e.g. ) and reverse angle brackets (e.g. ) is called an element in RPA automation tools, and its operation handle can be obtained through the path information XPath.

[0057] In one embodiment, the handle is manipulated to determine the location of the handle on the site page and the handle is displayed on the site page, including: manipulating the handle to determine the coordinates of the handle on the site page and highlighting the handle on the site page. In this embodiment, after obtaining the handle, a series of information about the element corresponding to the handle can be obtained through the handle, such as text content, location, size, tags, etc. In this embodiment, the location information is obtained through the handle and then highlighted.

[0058] In one embodiment, determining whether a link corresponding to the handle is a honeypot based on the handle's location information and display marking results includes: determining that the link corresponding to the handle is a honeypot when the handle's coordinate value is less than or equal to 0, or the handle cannot be highlighted. The handle's location information on the site page includes X and Y coordinate values.

[0059] Figure 2This is a specific example diagram of the network data crawling method of the anti-honeypot technology in an embodiment of the present invention. In this example, the process of the network data crawling method of the anti-honeypot technology may include: (1) The crawler accesses the initial access link. Give the crawler an accessible site link (the initial access link is not a honeypot), and the crawler begins to simulate a browser request to access the site. Under normal circumstances, the HTML code, JSON data, binary data and other content returned by the site will be obtained. (2) Parse and sort out the next accessible link list. In the returned content, the crawler can store information such as data. For the returned HTML code, sort out the next link list that can be requested for access, and give the XPath of each link in the list. (3) Perform RPA test judgment on each accessible link. For the request link list sorted out in step (2), perform RPA testing in the following steps: ① First, use RPA to simulate manually opening the initial access link in step (1); ② For each link in the list, use RPA to obtain the handle of the link through XPath. Operate the acquired handle to obtain its location information (X, Y axis coordinates) on the site page and highlight it (High Line). ③ For handles with coordinate values less than or equal to 0 or that cannot be highlighted, mark their original links as honeypot traps. (4) Arrange the results and loop. Arrange the results of step (3), prohibiting the links marked as honeypots from being accessed in the next step, and repeat the operations of steps (1) to (4) for the others. Repeat the operation until all the required content of the website is crawled.

[0060] The present invention also provides a network data capture device for anti-honeypot technology, as described in the following embodiments. Since the principle of the device to solve the problem is similar to the network data capture method of anti-honeypot technology, the implementation of the device can refer to the implementation of the network data capture method of anti-honeypot technology, and the repeated parts will not be repeated.

[0061] Figure 3 FIG. 1 is a structural block diagram of a network data capture device using the anti-honeypot technology according to an embodiment of the present invention. Figure 3 As shown, the network data capture device 300 of the anti-honeypot technology may include:

[0062] The link receiving module 301 is used to receive the initial access link of the site;

[0063] The crawler module 302 is used to simulate a browser request through a crawler and access the site through the initial access link;

[0064] Response receiving module 303, used to receive response information returned by the site;

[0065] A link determination module 304 is configured to determine a plurality of accessible links and path information of each accessible link according to the response information;

[0066] The RPA module 305 is used to open the site page corresponding to the initial access link using the Robotic Process Automation (RPA) module;

[0067] The handle module 306 is used to use RPA to obtain the handle corresponding to each accessible link based on the path information, determine the position information of the handle on the site page by operating on the handle, and display the handle on the site page.

[0068] Honeypot determination module 307, used to determine whether the link corresponding to the handle is a honeypot based on the handle's location information and the display marking result;

[0069] The execution module 308 is used to prohibit access to the link when the link corresponding to the handle is a honeypot; when the link corresponding to the handle is not a honeypot, the link is re-used as the initial access link and the above modules are repeatedly triggered until all accessible links are processed.

[0070] In one embodiment, the response information includes HTML code. In this embodiment, the link determination module 304 is specifically configured to: search for hyperlinks in the HTML code; and determine multiple accessible links and path information of each accessible link based on the found hyperlinks.

[0071] In one embodiment, the handle module 306 is specifically configured to:

[0072] The coordinates of the handle on the site page are determined by operating the handle, and the handle is highlighted on the site page.

[0073] In one embodiment, the honeypot determination module 307 is specifically configured to:

[0074] When the coordinate value of the handle is less than or equal to 0, or the handle cannot be highlighted, it is determined that the link corresponding to the handle is a honeypot.

[0075] Figure 4 FIG. 1 is an interactive diagram of a network data capture device using the anti-honeypot technology according to an embodiment of the present invention. Figure 4 As shown, the crawler unit receives data in the response information, sorts out the next accessible links, and interacts with the robotic process automation unit as an auxiliary tool for data.

[0076] Based on the above invention concept, Figure 5 As shown, the present invention also proposes a computer device 500, including a memory 501, a processor 503 and a computer program 502 stored in the memory 501 and executable on the processor 503, wherein the processor 503 implements the aforementioned anti-honeypot technology network data capture method when executing the computer program 502.

[0077] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the network data capture method using the anti-honeypot technology is implemented.

[0078] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the network data crawling method of the anti-honeypot technology is implemented.

[0079] In an embodiment of the present invention, an initial access link of a site is received; a crawler simulates a browser request and accesses the site through the initial access link; a response message returned by the site is received; multiple accessible links and path information of each accessible link are determined based on the response information; and a robotic process automation (RPA) is used to open the site page corresponding to the initial access link. Since robotic process automation technology simulates human operations to improve efficiency, this operation is generally allowed by regular websites and is unlikely to be banned. As an auxiliary tool for crawlers, it is very suitable. For each accessible link, RPA is used to obtain the handle corresponding to the link through the path information, and the position information of the handle on the site page is determined by operating the handle, and the handle is displayed on the site page. Based on the handle position information and the display mark result, it is determined whether the link corresponding to the handle is a honeypot; if the link corresponding to the handle is a honeypot, access to the link is prohibited; if the link corresponding to the handle is not a honeypot, the link is re-used as the initial access link, and the above steps are repeated until all accessible links are processed. Compared with the directional crawler solution in the prior art, it can survey the possible honeypots on the web page, avoid the possible honeypot traps, no longer be restricted to the specified path, and achieve larger-scale data crawling.

[0080] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0081] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0082] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0083] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0084] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A network data capture method using anti-honeypot technology, characterized in that: include: Receive the initial access link to the Site; Use crawlers to simulate browser requests and access the site through the initial access link; Receive the response information returned by the site; Determining a plurality of accessible links and path information of each accessible link according to the response information; Use Robotic Process Automation (RPA) to open the site page corresponding to the initial access link; For each accessible link, use RPA to obtain the link's corresponding handle based on the path information. Then, operate on the handle to determine its location on the site page, and then mark the handle for display on the site page. Determine whether the link corresponding to the handle is a honeypot based on the handle's location information and display marking results; When the link corresponding to the handle is a honeypot, access to the link is prohibited; When the link corresponding to the handle is not a honeypot, the link is re-used as the initial access link and the above steps are repeated until all accessible links are processed.

2. The method according to claim 1, wherein The response information includes HTML code; Determining multiple accessible links and path information of each accessible link according to the response information includes: Find hyperlinks in HTML code; According to the found hyperlinks, a plurality of accessible links and path information of each accessible link are determined.

3. The method according to claim 1, wherein The handle location information on the site page is determined by operating the handle, and the handle is displayed on the site page, including: The coordinates of the handle on the site page are determined by operating the handle, and the handle is highlighted on the site page.

4. The method according to claim 3, wherein Based on the handle's location information and display marking results, determine whether the link corresponding to the handle is a honeypot, including: When the coordinate value of the handle is less than or equal to 0, or the handle cannot be highlighted, it is determined that the link corresponding to the handle is a honeypot.

5. A network data capture device using anti-honeypot technology, characterized in that: include: A link receiving module, used for receiving the initial access link of the site; The crawler module is used to simulate browser requests through crawlers and access the site through the initial access link; Response receiving module, used to receive the response information returned by the site; a link determination module, configured to determine a plurality of accessible links and path information of each accessible link according to the response information; The RPA module is used to use Robotic Process Automation (RPA) to open the site page corresponding to the initial access link; The handle module is used to obtain the handle corresponding to each accessible link through the path information using RPA, determine the location of the handle on the site page by operating on the handle, and display the handle on the site page. The honeypot judgment module is used to determine whether the link corresponding to the handle is a honeypot based on the handle's location information and the display marking result; The execution module is used to prohibit access to the link when the link corresponding to the handle is a honeypot; when the link corresponding to the handle is not a honeypot, the link is re-used as the initial access link and the above modules are repeatedly triggered until all accessible links are processed.

6. The device according to claim 5, characterized in that The response information includes HTML code; The link determination module is specifically used for: Find hyperlinks in HTML code; According to the found hyperlinks, a plurality of accessible links and path information of each accessible link are determined.

7. The device according to claim 5, characterized in that The handle module is specifically used for: The coordinates of the handle on the site page are determined by operating the handle, and the handle is highlighted on the site page.

8. The device according to claim 7, wherein The honeypot judgment module is specifically used to: When the coordinate value of the handle is less than or equal to 0, or the handle cannot be highlighted, it is determined that the link corresponding to the handle is a honeypot.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

11. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images