Link validity detection method, apparatus, device, and readable storage medium

Abstract

The application discloses a kind of link validity detection method, device, equipment and readable storage medium.The link validity detection method includes: the link to be detected is attached after platform jump link, and the HTTP status code returned by HTTP request is obtained;When the HTTP status code returned is 302, the location attribute in the response header of the HTTP request is obtained;Based on the location attribute, the validity of the link to be detected is judged.The link validity detection method provided by the application uses script, can quickly and accurately monitor the state of large-scale link, and when detecting link exception, automatic alarm is carried out.Compared with prior art, the method has the advantages of high efficiency and accuracy, rapid batch detection, detailed classification of abnormal link, strong scalability and the like.

Description

Technical Field

[0001] This invention relates to the field of information services and internet operations, and in particular to a method, apparatus, device, and readable storage medium for detecting link validity. Background Technology

[0002] In today's WeChat Official Account operations, especially in the advertising and marketing industry, a large number of links are generated to meet business needs. However, a certain number of links may be blocked by the platform due to malicious reports from competitors or misjudgments by the platform, resulting in unnecessary losses. Existing technical solutions for link validity detection, such as using RPA robots to simulate manual operation, are prone to identification errors and execution anomalies, leading to inaccurate test results. Furthermore, they are relatively inefficient and cannot meet the needs of large-scale batch testing. Methods that generate short links through platform short link application interfaces to obtain link status have access frequency limitations, and the availability of this application interface may change at any time, causing delays and errors during the detection process, and failing to quickly complete the large-scale link testing required. Therefore, to address the above technical problems, it is necessary to provide a link validity detection method, apparatus, device, and readable storage medium.

[0003] The information disclosed in this background section is intended only to enhance the understanding of the overall background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0004] The purpose of this invention is to provide a link validity detection method, apparatus, device, and readable storage medium, which can quickly detect the real-time status of a large number of links, promptly identify abnormal links, improve link management efficiency, and provide more stable and reliable support for information services.

[0005] To achieve the above objectives, the technical solution provided by the present invention is as follows:

[0006] In a first aspect, the present invention provides a link validity detection method, comprising:

[0007] Attach the link to be tested to the platform's redirect link and obtain the returned HTTP status code via an HTTP request;

[0008] When the returned HTTP status code is 302, retrieve the location attribute from the response header of the HTTP request;

[0009] The validity of the link to be detected is determined based on the location attribute.

[0010] In one or more implementations, determining the state of the link to be detected based on the location attribute includes:

[0011] If the location attribute is the link to be detected, then the link to be detected is a valid link;

[0012] If the location attribute is not the link to be detected, then the link to be detected is an abnormal link.

[0013] In one or more embodiments, the method further includes:

[0014] When the link to be detected is an abnormal link, obtain the source code of the page after the abnormal link redirects;

[0015] The text in the redirected page is identified;

[0016] Based on the text, the abnormal links are further classified.

[0017] In one or more embodiments, the secondary classification of the abnormal links based on the text includes:

[0018] If the page contains the words "Access Stopped", the abnormal link will be classified as a broken link.

[0019] If the page contains the words "Continue to visit", then the abnormal link will be classified as a valid link.

[0020] In one or more embodiments, the method further includes:

[0021] When the returned HTTP status code is 200, a second check is performed after a ten-minute interval.

[0022] In one or more embodiments, the method further includes:

[0023] Create the list of items to be detected;

[0024] Identify the links in the database that need to be detected;

[0025] Import the links to be detected into the list of links to be detected in the script.

[0026] In one or more embodiments, the method further includes:

[0027] Use an automated program to open a broken link;

[0028] Get the redirect link returned by the platform.

[0029] Secondly, the present invention provides a link validity detection device, comprising:

[0030] The status module is used to attach the link to be detected to the platform redirect link and obtain the returned HTTP status code through an HTTP request;

[0031] The response header module is used to retrieve the location attribute from the response header of the HTTP request when the returned HTTP status code is 302.

[0032] The judgment module is used to determine the validity of the link to be detected based on the location attribute.

[0033] Thirdly, the present invention provides a computer device comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the link validity detection method by executing the computer instructions.

[0034] Fourthly, the present invention provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores computer instructions for causing a computer to execute the link validity detection method described above.

[0035] Compared with existing technologies, the present invention provides a link validity detection method, apparatus, device, and readable storage medium. The method involves attaching the link to be detected to a platform redirect link, obtaining the returned HTTP status code via an HTTP request, and when the returned HTTP status code is 302, obtaining the location attribute from the response header of the HTTP request. Based on the location attribute, the validity of the link to be detected is determined. This method has the following advantages:

[0036] (1) Fast and accurate: This method can quickly and accurately monitor the status of a large number of links. According to actual tests, it can detect 8 links per second per device. Compared with existing technical solutions, it can not only improve the detection speed, but also reduce the false judgment rate and improve the detection accuracy.

[0037] (2) High degree of automation: This method achieves automated detection through scripts, avoiding errors caused by manual operation. It is also not limited by access frequency and can meet the needs of large-scale batch detection.

[0038] (3) More detailed classification of abnormal links: This method can classify abnormal links in a second way, accurately determine whether the link is blocked or has abnormal access frequency, so as to better guide subsequent processing. Attached Figure Description

[0039] Figure 1This is a schematic diagram illustrating an application scenario of the link validity detection method according to one embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of the link validity detection process in one embodiment of the present invention;

[0041] Figure 3 This is a structural block diagram of a link validity detection device according to an embodiment of the present invention;

[0042] Figure 4 This is a structural block diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0043] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings, but it should be understood that the scope of protection of the present invention is not limited to the specific embodiments.

[0044] Unless otherwise expressly stated, throughout the specification and claims, the term "comprising" or its variations such as "including" or "comprises" shall be understood to include the stated elements or components without excluding other elements or other components.

[0045] To facilitate understanding of the technical solutions of this application, the technical terms that may appear in this invention will be explained in detail below.

[0046] Robotic Process Automation (RPA) systems are business process automation technologies based on software robots and artificial intelligence (AI). An RPA system is an application that automates end-user processes by mimicking manual operations on a computer. RPA tools are technically similar to graphical user interface (GUI) testing tools. These tools automatically interact with a GUI and allow users to demonstrate their processes and then implement them using demonstrative programming. A key difference is that these systems allow data exchange between different applications. For example, receiving an email might involve receiving a payment slip, retrieving its contents, and entering them into a bookkeeping system.

[0047] Python: Designed by Guido van Rossum of the Netherlands Organization for Mathematical and Computer Science in the early 1990s as an alternative to a language called ABC. Python offers efficient high-level data structures and enables simple and effective object-oriented programming. Its syntax, dynamic typing, and interpreted language nature make it a popular programming language for scripting and rapid application development on most platforms. With continuous updates and the addition of new features, it has gradually been used for independent, large-scale project development. The Python interpreter is easily extensible, allowing for the addition of new features and data types using C or C++ (or other languages ​​that can be called from C). Python can also be used as an extension language in customizable software. Python's rich standard library provides source code or machine code for various major system platforms.

[0048] An API (Application Programming Interface) is a set of conventions that connect different components of a software system. Its main purpose is to provide applications and developers with the ability to access a set of routines without needing access to the source code or understanding the details of the internal workings. The software that provides the functionality defined by the API is called the implementation of that API. An API is an interface, and therefore an abstraction. In programming practice, the design of a programming interface must first ensure a reasonable division of responsibilities within the software system. Good interface design can reduce the interdependencies between system parts, increase the cohesion of component units, and reduce the coupling between component units, thereby improving the maintainability and scalability of the system.

[0049] Selenium is a tool for testing web applications. Selenium tests run directly in the browser, just like a real user. Supported browsers include IE (7, 8, 9, 10, 11), Mozilla Firefox, Safari, Google Chrome, Opera, Edge, etc. Its main functions include: testing browser compatibility—testing whether the application works well on different browsers and operating systems; testing system functionality—creating regression tests to verify software functionality and user requirements; and supporting automatic recording of actions and automatic generation of test scripts in different languages ​​such as .NET, Java, and Perl.

[0050] browsermob-proxy (BMP for short) is an HTTP proxy service that allows us to intercept HTTP requests and responses and output performance data as a HAR file.

[0051] URL (Uniform Resource Locator): Available resources on the Internet can be represented by simple strings, which are called "Uniform Resource Locators" (URLs). URLs are a method of specifying the location of information on the World Wide Web service program of the Internet, and have now been compiled into the Internet standard RFC1738 by the World Wide Web Consortium.

[0052] Response Headers: Response headers are HTTP headers that can be used in HTTP responses and are independent of the response message body. Headers like Age, Location, and Server are used to provide more detailed response context. Not all headers appearing in a response are classified as response headers according to the specification; however, all "session-based" headers are generally referred to as response headers in the response message.

[0053] response.content: Data scraped directly from the web without any decoding.

[0054] response.headers: The header content of the HTTP response content is retrieved directly from the network.

[0055] XPath: XPath stands for XML Path Language. It is a language used to locate specific parts of an XML document. Based on the tree structure of XML, XPath provides the ability to find nodes within the data structure tree. Initially, XPath was proposed as a general-purpose syntax model between XPointer and XSL. Later, developers adopted XPath as a small query language.

[0056] Webhook: Webhook is an API concept and one of the usage paradigms of microservice APIs. It is also known as a reverse API, meaning that the front end does not actively send requests, but is entirely pushed by the back end. Simply put, a webhook is a URL that receives HTTP POST (or GET, PUT, DELETE) requests. An API provider that implements webhook will send a message to this configured URL when an event occurs. Unlike request-response, using webhook allows you to receive changes in real time.

[0057] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0058] Please refer to Figure 1The diagram illustrates an application scenario of the link validity detection method provided by this invention. Figure 1 The implementation scenario shown includes a terminal 101 that needs to perform link validity checks and a server 102 with script execution capabilities. Specifically, when the terminal 101 needs to perform link validity checks, it can import the links to be checked into the server 102 with script execution capabilities through the list of links to be checked, and then perform the validity check operation.

[0059] It should be noted that the server 102 with script execution function refers to a server system connected to network 103 that has script execution function and detects the validity of links according to the constructed script. It can include various types of server systems with script execution function, and this embodiment of the present invention does not limit it.

[0060] The link validity detection method of this invention can be applied to the link validity detection device of this invention. This link validity detection device can be configured on a terminal. The terminal may include, but is not limited to, a PC (Personal Computer), a PDA (Tablet PC), a smartphone, a smart wearable device, etc.

[0061] Terminal 101 can establish a communication connection with each server 102 through network 103. This network can be a wireless network or a wired network, and this embodiment of the invention does not limit it.

[0062] The server 102 with script execution function can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0063] Please refer to Figure 2 The diagram shown illustrates a flowchart of link validity detection according to an embodiment of the present invention. The link validity detection method specifically includes the following steps:

[0064] S201: Attach the link to be detected to the platform redirect link and obtain the returned HTTP status code through an HTTP request;

[0065] It should be noted that before performing step S201, it is also necessary to import the link to be detected and obtain the redirection link of the platform.

[0066] In one exemplary embodiment, importing links to be detected and obtaining redirect links from the platform includes: creating the list of links to be detected; identifying links to be detected in the database; importing the links to be detected into the list of links to be detected in the script; opening a broken link using an automated program; and obtaining the redirect links returned by the platform.

[0067] For example, based on existing business needs, a list of links to be detected is created; links that need to be detected in the database are identified and automatically imported into the script's list of links to be detected in real time. Using Selenium in Python, a link that has been blocked by WeChat is used to automatically open the link in a simulated WeChat environment via RPA. A proxy server is started through browsermob-proxy (the browser's proxy), which captures all incoming and outgoing HTTP traffic, converts the captured HAR file into a Python object, extracts the redirect URL, and obtains the jump link used by WeChat before redirecting to weixin110.qq.com (WeChat Security Center).

[0068] Specifically, the returned HTTP status codes include: if the link to be tested redirects normally, the status code is 302; if the link to be tested is accessed too quickly and needs to be accessed again after a period of time, the status code is 200.

[0069] For example, by appending the URL to be checked to the redirect link, a GET request can be made using requests, and the returned HTTP status code can be used for judgment. If the returned status code is 200, it means that the access frequency is too high, and it needs to pause for 10 minutes before continuing the check; if the returned status code is 302, meaning the URL has been redirected, then subsequent check operations can be performed.

[0070] S202: When the returned HTTP status code is 302, retrieve the location attribute from the response header of the HTTP request;

[0071] For example, if the returned status code is 302, meaning the URL has been redirected, you need to use the response.headers method to get the location property in the Response Headers of the current request.

[0072] It should be noted that the original description phrase of status code 302 is "Moved Temporarily," which is a status code in the HTTP protocol. Simply put, it means that the resource originally existed, but its location has been temporarily changed; in other words, the requested resource is temporarily residing at a different URI. Therefore, unless a cache header is specifically specified, this status code cannot be cached.

[0073] S203: Determine the validity of the link to be detected based on the location attribute.

[0074] Specifically, based on the location attribute, the status of the link to be detected is determined, including: if the location attribute is the link to be detected, then the link to be detected is a valid link; if the location attribute is not the link to be detected, then the link to be detected is an abnormal link.

[0075] For example, if using the WeChat platform for detection, if the value of `location` is the link we need to detect, it means the link to be detected is a valid link. If the value of `location` is `https: / / weixin110.qq.com / ...` (WeChat Security Center), it means the link to be detected is in an abnormal state. It should be noted that if the value of `location` is `mp.weixin.qq.com / ...`, it means the detection failed and a second detection is required.

[0076] It should be noted that the link validity detection method also includes a secondary classification of the abnormal links.

[0077] In one exemplary embodiment, the abnormal link is further classified, including: when the link to be detected is an abnormal link, obtaining the source code of the page after the link to be detected redirects; recognizing the text in the page after the redirect; if the page contains the words "stop access", then classifying the abnormal link as an invalid link; if the page contains the words "continue access", then classifying the abnormal link as a valid link.

[0078] For example, in WeChat, abnormal links are represented in two forms: one is completely blocked, with a message indicating that access to the webpage has been stopped; the other requires clicking "continue" to access, indicating that it is not an official WeChat webpage and requires clicking "continue." For the links identified as abnormal, the source code of the redirected page is obtained using the `response.content` method. XPath is then used to locate and identify the text on the page, which is then categorized. If the page contains the phrase "stop access," the abnormal link is classified as an invalid link; if the page contains the phrase "continue access," the abnormal link is classified as a valid link.

[0079] In summary, the link validity detection method provided by this invention involves attaching the link to be detected to a platform redirect link and obtaining the returned HTTP status code via an HTTP request. When the returned HTTP status code is 302, the location attribute in the response header of the HTTP request is obtained. Based on the location attribute, the validity of the link to be detected is determined. This link validity detection method can quickly and accurately monitor the status of a large number of links. Compared with existing technical solutions, it not only improves detection speed but also reduces the false positive rate and improves detection accuracy. Furthermore, the use of a written script enables automated detection, avoiding errors caused by manual operation and not being limited by access frequency, thus meeting the needs of large-scale batch detection. Secondary classification of abnormal links can also accurately identify link anomalies, thereby better guiding subsequent processing.

[0080] Please refer to Figure 3 As shown, based on the same inventive concept as the aforementioned link validity detection method, one embodiment of the present invention provides a link validity detection device 300, which includes a status module 301, a response header module 302, and a judgment module 303.

[0081] It should be noted that the status module 301 is used to attach the link to be detected to the platform redirection link and obtain the returned HTTP status code through an HTTP request; the response header module 302 is used to obtain the location attribute in the response header of the HTTP request when the returned HTTP status code is 302; and the judgment module 303 is used to judge the validity of the link to be detected based on the location attribute.

[0082] Specifically, before the detection begins, the status module 301 needs to create a list of links to be detected, identify the links to be detected in the database, and import the links to be detected into the list of links to be detected in the script. An automated program is used to open a broken link; the redirect link is then obtained. Afterwards, the link to be detected is appended to the redirect link in the platform, and the returned HTTP status code is obtained via an HTTP request.

[0083] When the response header module returns an HTTP status code of 200, it waits for a period of time before continuing the detection; when the returned HTTP status code is 302, it retrieves the location attribute from the response header of the HTTP request.

[0084] The judgment module determines the validity of the link to be detected based on the location attribute, and then performs a secondary classification on the links that have been judged as abnormal. If the location attribute matches the link to be detected, the link status is normal; if the location attribute does not match the link to be detected, the link status is abnormal. For links with abnormal status, the source code of the page after the abnormal link redirects is obtained; the text of the page is recognized; pages containing the words "stop access" are classified as invalid links; pages containing the words "continue access" are classified as valid links.

[0085] Please refer to Figure 4 As shown, embodiments of the present invention also provide an electronic device 400, which includes at least one processor 401, a memory 402 (e.g., non-volatile memory), a main memory 403, and a communication interface 404, wherein the at least one processor 401, the memory 402, the main memory 403, and the communication interface 404 are connected together via a bus 405. The at least one processor 401 is configured to invoke at least one program instruction stored or encoded in the memory 402, causing the at least one processor 401 to perform various operations and functions of the link validity detection method described in the various embodiments of this specification.

[0086] In the embodiments of this specification, electronic device 400 may include, but is not limited to: personal computer, server computer, workstation, desktop computer, laptop computer, notebook computer, mobile electronic device, smartphone, tablet computer, cellular phone, personal digital assistant (PDA), handheld device, messaging device, wearable electronic device, consumer electronic device, etc.

[0087] This invention also provides a computer-readable medium carrying computer-executable instructions, which, when executed by a processor, can be used to implement various operations and functions of the link validity detection method described in the various embodiments of this specification.

[0088] The computer-readable medium in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0089] In this invention, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0090] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0091] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, systems, and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0092] The foregoing description of specific exemplary embodiments of the invention is for illustrative and explanatory purposes. These descriptions are not intended to limit the invention to the precise forms disclosed, and it will be apparent that many changes and variations can be made in accordance with the foregoing teachings. The exemplary embodiments were chosen and described in order to explain the specific principles of the invention and its practical application, thereby enabling those skilled in the art to implement and utilize various different exemplary embodiments of the invention, as well as various different choices and variations. The scope of the invention is intended to be defined by the claims and their equivalents.

Claims

1. A method for detecting link validity, characterized in that, include: Create a list of items to be tested; Identify the links in the database that need to be detected; Import the links to be detected into the list of links to be detected in the script; Use an automated program to open a broken link; Get the redirect link returned by the platform; Attach the link to be tested to the platform's redirect link and obtain the returned HTTP status code via an HTTP request; When the returned HTTP status code is 302, retrieve the location attribute from the response header of the HTTP request; Based on the location attribute, the validity of the link to be detected is determined, specifically including: if the location attribute is the link to be detected, then the link to be detected is a valid link; if the location attribute is not the link to be detected, then the link to be detected is an abnormal link. When the link to be detected is an abnormal link, obtain the source code of the page after the abnormal link redirects; The text in the redirected page is identified; based on the text, the abnormal links are further classified. If the page contains the words "Stop accessing", the abnormal link will be classified as a broken link; if the page contains the words "Continue accessing", the abnormal link will be classified as a valid link.

2. The link validity detection method as described in claim 1, characterized in that, The method further includes: When the returned HTTP status code is 200, a second check is performed after a ten-minute interval.

3. A link validity detection device, applied to the link validity detection method as described in any one of claims 1-2, characterized in that, include: The status module is used to attach the link to be detected to the platform redirect link and obtain the returned HTTP status code through an HTTP request; The response header module is used to retrieve the location attribute from the response header of the HTTP request when the returned HTTP status code is 302. The judgment module is used to determine the validity of the link to be detected based on the location attribute.

4. A computer device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the link validity detection method according to any one of claims 1-2 by executing the computer instructions.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the link validity detection method according to any one of claims 1-2.

Images