Website dead link dynamic detection method, system, equipment and medium

Abstract

The invention relates to a website dead link dynamic detection method and device and a medium. The method comprises the steps that a to-be-detected website address is put into a to-be-detected page queue; extracting the first website from the to-be-detected page queue; loading a first website page by the WEB operating environment, detecting a page loading state, and if loading fails, putting the first website into a dead link list and returning to select a next website; all network requests and responses in the loading process of the first website page are detected one by one, and if the network requests do not respond or return fails, the URLs of the network requests are added into a dead link list; performing dynamic operation check on the first website page; and putting the first website into the detected page queue, and repeating until all pages in the to-be-detected page queue are detected. Compared with the prior art, the method not only can identify all static links, but also can track dynamically generated links and requests on a page, and automatically detect all links on a specified website.

Description

Technical Field

[0001] The present invention relates to the field of website dead link detection, and in particular to a website dead link dynamic detection method, system, equipment and medium. Background Art

[0002] The Internet has more and more content, and the update speed is accelerating. When old content is deleted or transferred, the corresponding links become dead links. It is necessary to regularly check all links on the website to ensure that the links are valid.

[0003] Currently, most detection of broken links on websites relies on manual detection or using tools to crawl website pages for detection, which has the following defects:

[0004] 1) Manual inspection is to manually check whether all links on the website are correct and valid one by one by clicking on them. This is a labor-intensive and time-consuming process, and is prone to missed inspections or detection errors.

[0005] 2) Tool detection crawls the website page content, analyzes HTML to obtain all resource links in the page, and then visits the links in the page to confirm whether the links are valid. This method is much more efficient than the manual method, but the detection content and scope are limited. Many links and requests on modern websites are dynamically generated by JS scripts in the browser after user operations, and the running results of JS scripts need to be tracked for analysis. Therefore, there are great limitations to simply statically analyzing whether the hyperlinks in HTML are valid, and it can only meet the needs of broken link detection for some small-scale and simple websites.

[0006] Therefore, there is an urgent need to design a website dead link dynamic detection method that can not only identify all static links, but also track dynamically generated links and requests on the page, and automatically detect all links on a specified website. Summary of the invention

[0007] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a website dead link dynamic detection method, system, device and medium, which can not only identify all static links, but also track the links and requests dynamically generated on the page, and automatically detect all links on the specified website.

[0008] The purpose of the present invention can be achieved by the following technical solutions:

[0009] According to a first aspect of the present invention, a method for dynamically detecting broken links on a website is provided, the method comprising:

[0010] S1. Put the website address to be detected into the page queue to be detected, and put the domain to which the website address to be detected belongs into the domain set;

[0011] S2, taking out the first URL from the queue of pages to be detected, and sending an open page instruction to the WEB operating environment;

[0012] S3, after receiving the page opening instruction, the WEB operating environment loads the first URL page and detects the loading status of the first URL page. If the loading fails, the first URL is put into the dead link list, and the process returns to S2 to select the next URL. Otherwise, the process goes to S4;

[0013] S4, detecting all network requests and responses during the loading process of the first URL page one by one, if the network request has no response or returns a failure, the URL of the network request is put into a dead link list, if the resource type returned by the network request is html, the URL of the network request is put into a queue of pages to be detected;

[0014] S5, obtaining all HTML elements of the first URL page and putting them into a queue of elements to be checked, executing a set operation instruction for each element in the queue of elements to be checked, and dynamically checking the first URL page according to the response result;

[0015] S6. Put the first URL into the queue of detected pages, and repeat S2 until all pages in the queue of pages to be detected are detected.

[0016] Preferably, after taking out the first URL from the queue of pages to be detected in S2, if the first URL is already in the dead link list or in the detected queue, then a next URL is selected as the first URL.

[0017] Preferably, after detecting the loading status of the first URL page, if the first URL does not belong to any domain in the domain set, the process goes to S2 to select the next URL.

[0018] Preferably, in said S5, all HTML elements of the first website page are obtained and put into a queue of elements to be checked, and after executing a set operation instruction on each element in the queue of elements to be checked, the first website page is dynamically checked according to the response result, specifically including:

[0019] S51, sending a page element acquisition instruction to the WEB operating environment, acquiring all HTML elements of the first URL page and putting them into a queue of elements to be checked;

[0020] S52, taking out an element from the queue of elements to be checked, and sending an operation instruction to the WEB operating environment;

[0021] S53, after the WEB operating environment executes the operation instruction, if the page jumps, the new page address is put into the queue of pages to be detected, and a back instruction is sent to the WEB operating environment, and the WEB operating environment backs up to the last opened page;

[0022] S54, detecting all network requests and responses during the simulated user operation one by one, if the network request has no response or returns a failure, adding the URL of the network request to the dead link list, if the resource type returned by the network request is html, putting the URL of the network request into the queue of pages to be detected;

[0023] S55, repeat until all elements in the queue of elements to be checked are checked.

[0024] Preferably, the operation instructions in S62 include hovering, clicking and touching instructions.

[0025] Preferably, the method further comprises: monitoring all network requests and responses of the WEB operating environment in real time.

[0026] Preferably, the method further comprises: before the WEB operating environment is started, presetting user data to simulate designated user login detection.

[0027] Preferably, the user data includes a user name, a password and a Token.

[0028] According to a second aspect of the present invention, there is provided an electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, any one of the methods described above is implemented.

[0029] According to a third aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, any one of the methods described above is implemented.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] (1) The present invention creates a WEB operating environment, loads and renders web page content in the WEB operating environment, simulates manual operation to execute action instructions on all elements of the page, and dynamically detects website links based on all network requests and responses during page loading and simulated operation. It can not only identify all static links, but also track dynamically generated links and requests on the page, and automatically detect all links on a specified website. This can effectively solve the problem that traditional static detection methods are difficult to meet the dead link detection needs of large-scale modern websites.

[0032] (2) The present invention presets user data such as user name, password, and token before starting the WEB operating environment, simulates the login detection of a designated user, and solves the problem of being unable to detect the content of a website that requires special permissions. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 The present invention is a flow chart of the method. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0035] Example

[0036] like Figure 1 As shown, this embodiment provides a method for dynamically detecting broken links on a website, the method comprising:

[0037] S1. Put the website address to be checked into the page queue to be checked CHECKING_PAGE_QUEUE, and put the domain to which the website address to be checked belongs into the domain set DOMAIN_SET.

[0038] S2, start the WEB operating environment and initialize it, take out the first URL from the page queue to be checked CHECKING_PAGE_QUEUE, monitor all network requests and responses of the WEB operating environment, and send an open page instruction to the WEB operating environment;

[0039] If the first URL is already in the dead link list DEADLINK_LIST or in the checked page queue CHECKED_PAGE_QUEUE, then the next URL is reselected as the first URL, otherwise, an open page instruction is sent to the WEB operating environment.

[0040] S3. After receiving the open page instruction, the WEB operating environment loads the first URL page and obtains the loading status of the first URL page. If the loading fails, the first URL is put into the dead link list DEADLINK_LIST, and the next URL is selected by returning to S2. If the first URL does not belong to any domain in the domain set DOMAIN_SET of the dead link list DEADLINK_LIST, the process returns to S2.

[0041] S4. Detect all network requests and responses during the loading process of the first URL page one by one. If the network request has no response or returns a failure, add the URL of the network request to the dead link list DEADLINK_LIST. If the resource type returned by the network request is html, put the URL of the network request into the page queue to be detected CHECKING_PAGE_QUEUE.

[0042] S5, obtaining all HTML elements of the first URL page and putting them into the to-be-checked element queue CHECKING_ELEMENT_QUEUE, after executing the set operation instruction for each element in the to-be-checked element queue, dynamically checking the first URL page according to the response result, specifically including the following sub-steps:

[0043] S51, sending a page element acquisition instruction to the WEB operating environment, acquiring all HTML elements of the first URL page and putting them into a check element queue CHECKING_ELEMENT_QUEUE;

[0044] S52, taking an element from the CHECKING_ELEMENT_QUEUE, monitoring all network requests and responses of the WEB runtime environment, and sending an operation instruction to the WEB runtime environment, wherein the operation instruction includes hover, click, and touch instructions;

[0045] S53, after the WEB operating environment executes the operation instruction, if the page jumps, the new page address is put into the to-be-checked page queue CHECKING_PAGE_QUEUE, and a back instruction is sent to the WEB operating environment, and the WEB operating environment backs up to the last opened page;

[0046] S54, check all network requests and responses during the simulated user operation one by one, if the network request has no response or returns a failure, add the URL of the network request to the dead link list DEADLINK_LIST, if the resource type returned by the network request is html, put the URL of the network request into the page queue to be checked CHECKING_PAGE_QUEUE;

[0047] S55. Repeat until all elements in the to-be-checked element queue CHECKING_ELEMENT_QUEUE are checked.

[0048] Taking the click operation as an example, click operation instructions are sent one by one to all elements of the check element queue CHECKING_ELEMENT_QUEUE corresponding to the current page A1, and the page jump and HTTP(S) request of each click are tracked. If the page jumps when element B1 is clicked, the new page address is added to the check page queue CHECKING_PAGE_QUEUE, and a back instruction is sent to the WEB runtime environment. The WEB runtime environment rolls back to the last opened page (A1), and then continues to send click operation instructions to the elements after the current element (B1) until the click action of all elements is triggered.

[0049] S6. Put the first URL into the checked page queue CHECKED_PAGE_QUEUE, and repeat S2 until all pages in the to-be-checked page queue CHECKING_PAGE_QUEUE are checked.

[0050] In summary, the core point of the present invention is to load and render web page content in the created WEB operating environment, simulate manual operation and click all elements of the page, dynamically track and detect all HTTP(S) links in JS operation, and detect whether the link is valid by analyzing network requests and responses. Not only can all static links be identified, but also links and requests dynamically generated on the page can be tracked, and all links on the specified website can be automatically detected.

[0051] The electronic device of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0052] Multiple components in the device are connected to the I / O interface, including: input units, such as keyboards, mice, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunication networks.

[0053] The processing unit performs the various methods and processes described above, such as methods S1 to S6. For example, in some embodiments, methods S1 to S6 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps of methods S1 to S6 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 to S6 in any other appropriate manner (e.g., by means of firmware).

[0054] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip systems (SOCs), load programmable logic devices (CPLDs), and the like.

[0055] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0056] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0057] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for dynamically detecting broken links on a website, characterized in that: The method includes: S1. Put the website address to be detected into the page queue to be detected, and put the domain to which the website address to be detected belongs into the domain set; S2, taking out the first URL from the queue of pages to be detected, and sending an open page instruction to the WEB operating environment; S3, after receiving the page opening instruction, the WEB operating environment loads the first URL page and detects the loading status of the first URL page. If the loading fails, the first URL is put into the dead link list, and the process returns to S2 to select the next URL. Otherwise, the process goes to S4; S4, detecting all network requests and responses during the loading process of the first URL page one by one, if the network request has no response or returns a failure, the URL of the network request is put into a dead link list, if the resource type returned by the network request is html, the URL of the network request is put into a queue of pages to be detected; S5, obtaining all HTML elements of the first URL page and putting them into a queue of elements to be checked, executing a set operation instruction for each element in the queue of elements to be checked, and dynamically checking the first URL page according to the response result; S6. Put the first URL into the queue of detected pages, and repeat S2 until all pages in the queue of pages to be detected are detected.

2. A website broken link dynamic detection method according to claim 1, characterized in that: After the first URL is taken out from the queue of pages to be detected in S2, if the first URL is already in the dead link list or in the queue of detected pages, the next URL is reselected as the first URL.

3. A website broken link dynamic detection method according to claim 1, characterized in that: After detecting the loading status of the first URL page, if the first URL does not belong to any domain in the domain set, the process goes to S2 to select the next URL.

4. A website broken link dynamic detection method according to claim 1, characterized in that: In S5, all HTML elements of the first URL page are obtained and put into a queue of elements to be checked. After executing a set operation instruction for each element in the queue of elements to be checked, the first URL page is dynamically checked according to the response result, specifically including: S51, sending a page element acquisition instruction to the WEB operating environment, acquiring all HTML elements of the first URL page and putting them into a queue of elements to be checked; S52, taking out an element from the queue of elements to be checked, and sending an operation instruction to the WEB operating environment; S53, after the WEB operating environment executes the operation instruction, if the page jumps, the new page address is put into the queue of pages to be detected, and a back instruction is sent to the WEB operating environment, and the WEB operating environment backs up to the last opened page; S54, detecting all network requests and responses during the simulated user operation one by one, if the network request has no response or returns a failure, adding the URL of the network request to the dead link list, if the resource type returned by the network request is html, putting the URL of the network request into the queue of pages to be detected; S55, repeat until all elements in the queue of elements to be checked are checked.

5. A website broken link dynamic detection method according to claim 4, characterized in that: The operation instructions in S62 include hovering, clicking and touching instructions.

6. A website broken link dynamic detection method according to claim 1, characterized in that: The method further comprises: monitoring all network requests and responses of the WEB operating environment in real time.

7. A website broken link dynamic detection method according to claim 1, characterized in that: The method further comprises: before the WEB operating environment is started, presetting user data to simulate a designated user login detection.

8. A website broken link dynamic detection method according to claim 7, characterized in that: The user data includes user name, password and Token.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images