A multi-threaded website anomaly detection method and terminal

Abstract

The application provides a multi-thread website exception detection method and terminal, which divides a webpage running process into a background management module and a webpage detection module through system service platform management, wherein the background management module is a core of a whole system hub, mainly responsible for basic information configuration of a website to be detected and webpage detection task construction, specially configured construction frequency and detection depth of a detection task for each website, and then corresponding multi-thread detection and analysis are performed by the webpage detection module, on one hand, whether an exception exists is judged according to the content of a current page, on the other hand, all accessible addresses of the current page are analyzed, and the results are returned to the background management module for data statistics, so that when an exception detection result is received in the background, timely response processing can be performed, and when the final detection is completed, a whole detection result can be seen in the background, and webpage exception detection efficiency is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network anomaly detection, and particularly relates to a multi-threaded website anomaly detection method and a terminal. BACKGROUND

[0002] With the rapid development of the Internet and technological progress, online education in the education field has been applied unprecedentedly. Major education institutions and government units have moved relevant teaching resources online. WEB, as the core software carrier, presents works to users. With the continuous enrichment of application content and functions, the business scenarios and the range of content to be tested become very large. However, as an educational product, the reliability of the resources provided by the application and the accessibility of the website are particularly important, and the release of the application is more cautious than general applications. However, the more good products are, the more frequent the corresponding service release and operation and maintenance are, and the verification time consumed is a considerable investment, but it has to be performed.

[0003] Currently, in addition to manual testing, there are mainly two kinds of automatic solutions on the market, including covering through writing WEB UI automation scripts and detecting by traversing and scanning the links of web pages. The disadvantages are as follows:

[0004] Disadvantage one: if the WEB UI automation method is used to write scripts, it can indeed locate the abnormalities of the relevant pages, but the website has many functions and resources in the later stage, and it is a huge workload to write scripts manually, and the maintenance work investment is uncontrollable; if the web page link is traversed and detected, the coverage is quite limited, and most of the web page jump links are implicit links (addresses jumped by JavaScript internal methods, not by (The address that the href tag redirects to). If the website homepage happens to be all implicit addresses, then this solution will only be able to access the homepage itself and cannot extend to any more webpage addresses.

[0005] The second drawback is that whether you write web automation scripts or use links to traverse web pages, you will encounter an execution efficiency problem. If the website has more than 10,000 sublinks, and each page is parsed, even if the parsing time of a page is 3 seconds, only 1,200 pages can be parsed in an hour, which is very inefficient.

[0006] The third drawback is that the content displayed on the webpage is affected by the stability of the server, and is often probabilistically abnormal. During the traversal and detection process, it may only be a temporary error. However, with the existing conventional methods, it may be possible to detect the abnormal assertion in the process, but it does not know exactly what happened. It may be an abnormality of a dependent server or an abnormality of the local network environment. By the time the report results come out, the situation has been restored, and the results cannot be analyzed in a timely manner.

[0007] Fourthly, the link access method relies more on checking the integrity of page loading. At most, it can only ensure that the content loads normally. Many website errors are based on some preliminary operations, such as clicking an operation to open a window before the pop-up error occurs. There are currently no universal detection methods published in the industry for this type of problem.

[0008] Fifthly, if the page is accessed via a link and contains nested web pages from other external sites, the address obtained by directly concatenating the domain name of the main website will be an invalid URL, resulting in an incorrect request. This will become a source of noise during testing.

[0009] Disadvantage six: Many websites' web pages are actually dynamic content, containing many dynamic links. If a web crawler finds an abnormal page, such as an inaccessible URL, the developers / testers will be at a loss as to where the path came from.

[0010] Disadvantage 7: Whether writing web automation scripts or using links to traverse web pages, there will be an abnormal termination problem. If the program is interrupted due to the instability of the service itself or other factors, the original web page parsing progress will be terminated, and the traversal will need to be started from the beginning, which will consume this part of the repeated parsing time.

[0011] Disadvantage eight, different websites have different access rights for visitors, some visitors cannot browse comprehensively or have limited function operation, but if each webpage access needs login operation, it will undoubtedly increase a process consumption time. SUMMARY

[0012] The technical problem solved by the present application is to provide a multi-thread website anomaly detection method and terminal, which effectively improves the webpage anomaly detection efficiency.

[0013] In order to solve the above technical problems, the technical scheme adopted by the present application is:

[0014] A multi-thread website anomaly detection method, comprising the steps of:

[0015] S1, the background management module acquires the website to be tested and configures the basic information;

[0016] S2, based on the basic information, the webpage detection task is constructed and configured to be traversed by the depth of the detection webpage, and then it is sent to the queue to be consumed;

[0017] S3, the webpage detection module acquires the webpage detection task of the queue to be consumed in real time, and performs webpage anomaly detection and hierarchical webpage address analysis according to the basic information and the depth of the webpage detection task to be traversed through the corresponding number of threads;

[0018] S4, the webpage detection module returns the detection result and the analysis result to the background management module.

[0019] In order to solve the above technical problems, another technical scheme adopted by the present application is:

[0020] A multi-thread website anomaly detection terminal, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to realize the steps of the above multi-thread website anomaly detection method.

[0021] The application has the beneficial effect that a multi-thread website anomaly detection method and terminal are provided, the webpage running process is divided into a background management module and a webpage detection module through system service platform management, the background management module is the core of the whole system hub, mainly responsible for basic information configuration of the website to be detected and webpage detection task construction, a special detection task construction frequency and detection depth are configured for each website, then the webpage detection module performs corresponding multi-thread detection and analysis, on one hand, whether there is an anomaly in the content of the current page is judged, on the other hand, all accessible addresses of the current page are analyzed, and the results are returned to the background management module for data statistics, so that when the background receives the detection result of the anomaly, it can respond in time, and when the final detection is completed, the whole detection result can be seen in the background, and the webpage anomaly detection efficiency is effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 A flowchart of a multi-threaded website anomaly detection method according to an embodiment of the present application;

[0023] Figure 2 A system business diagram of a multi-threaded website anomaly detection method according to an embodiment of the present application;

[0024] Figure 3 A distributed web page analysis diagram of a multi-threaded website anomaly detection method according to an embodiment of the present application;

[0025] Figure 4 A web page analysis flowchart of a multi-threaded website anomaly detection method according to an embodiment of the present application;

[0026] Figure 5 A flowchart of data statistics of a data statistics submodule of a multi-threaded website anomaly detection method according to an embodiment of the present application;

[0027] Figure 6 A structure schematic diagram of a multi-threaded website anomaly detection terminal according to an embodiment of the present application.

[0028] Label explanation:

[0029] 1. A multi-threaded website anomaly detection terminal; 2. A memory; 3. A processor. DETAILED DESCRIPTION

[0030] To explain the technical content, the achieved purposes and effects of the present application in detail, the following will be described in combination with the embodiments and the accompanying drawings.

[0031] Before that, the English terms appearing in this text are explained as follows:

[0032] Playwright: An automated testing framework, compared with the automated testing framework commonly used in the industry, is based on the driving layer, with the characteristics of new technology, strong performance, rich function expansion and fast speed.

[0033] Please refer to Figures 1 to 5 A multi-threaded website anomaly detection method, comprising the steps of:

[0034] S1. The background management module acquires the website to be tested and configures the basic information;

[0035] S2. Based on the basic information, the webpage detection task is constructed and configured to be traversed by the depth of the detection webpage, and then it is issued to the consumer queue;

[0036] S3. The webpage detection module acquires a plurality of webpage detection tasks of the consumer queue in real time, and performs webpage anomaly detection and hierarchical webpage address resolution according to the basic information and the depth of the webpage detection task to be traversed through a corresponding number of threads;

[0037] S4. The webpage detection module returns the detection result and the analysis result to the background management module.

[0038] From the above description, the beneficial effects of the present application are that a multi-threaded website anomaly detection method is provided, which divides the webpage running process into a background management module and a webpage detection module through system service platform management, wherein the background management module is the core of the whole system, mainly responsible for basic information configuration and webpage detection task construction of the website to be tested, and a special detection task construction frequency and detection depth are configured for each website, and then the webpage detection module performs corresponding multi-threaded detection and analysis, on the one hand, judges whether the current page content exists abnormally, and on the other hand, analyzes all accessible addresses of the current page, and returns the results to the background management module for data statistics, so that when the background receives the abnormal detection result, it can respond in time, and when the final detection is completed, the whole detection result can be seen in the background, effectively improving the webpage anomaly detection efficiency.

[0039] Further, the background management module comprises a data management submodule, and the step S1 is specifically:

[0040] S11, the data management submodule acquires the website to be tested and performs basic information configuration, comprising:

[0041] Creating a corresponding website, filling in the corresponding webpage name, and configuring a corresponding website address list, the website address list being at least one address;

[0042] Detection item configuration, including interface anomaly detection configuration, webpage content anomaly detection configuration and console anomaly detection configuration;

[0043] Early warning item configuration, configuring the system to support push WeChat, email and telephone early warning channels, and performing instant notification of the corresponding early warning channel according to whether the early warning notification condition is met when detecting webpage anomaly;

[0044] Custom basic configuration, including basic configuration when the system starts, dynamic loading data mode configuration and filtering text configuration of the website to be tested;

[0045] All configuration items are in JSON data structure;

[0046] S12, the data management submodule acquires all sub-page addresses of the website to be tested, and stores them after corresponding filtering and deduplication.

[0047] From the above description, the configuration of basic information is performed by the data management submodule, all configuration items are in JSON data structure, which can be easily extended and used; meanwhile, the corresponding website name, website address list, detection item, early warning item and custom basic configuration are performed, effectively creating the basic information of the website, facilitating subsequent webpage detection; in addition, all sub-page addresses of the website to be tested need to be deduplicated before storage, so as to support record review while reducing space occupation.

[0048] Further, the system startup basic configuration specifically includes webpage parsing thread number configuration, headless mode enable / disable configuration and page timeout time configuration;

[0049] The dynamic loading data mode configuration of the website to be tested specifically includes page sliding configuration, page turning configuration and load more button configuration;

[0050] The filtering text configuration specifically includes a menu for storing text content that does not need to be clicked.

[0051] From the above description, the system startup basic configuration can effectively ensure system startup and initialization, while increasing the dynamic loading data mode configuration of the website to be tested supporting page sliding, turning and loading more content, ensuring the comprehensiveness of webpage detection, and increasing text filtering to ensure irrelevant text interference with webpage detection, further effectively improving the efficiency of webpage anomaly detection.

[0052] Further, the background management module further includes a construction management submodule, and the step S2 specifically includes:

[0053] S21, the construction management submodule acquires the basic information, constructs a webpage detection task and configures corresponding webpage traversal depth and task construction frequency, while increasing the custom basic configuration, including whether to start headless mode and whether to use coolie login;

[0054] S22, the construction management submodule creates a current webpage detection task based on the time interval of the task construction frequency using the current construction number, and judges whether there is an unfinished task corresponding to the current construction number when creating the task, if yes, the webpage detection task corresponding to the unfinished task is taken as the current webpage detection task, otherwise a new webpage detection task is created through the current construction number as the current webpage detection task;

[0055] S23, the construction management submodule creates a webpage address list to be tested according to the website address list under the current webpage detection task, sets the current path level of the website address to 0 and the total depth to the webpage traversal depth, and adds the current webpage detection task to the to-be-consumed queue as to-be-consumed data.

[0056] As can be known from the above description, the construction management sub-module is responsible for the construction of the webpage detection task. In addition to configuring the webpage traversal depth and the task construction frequency, some custom basic configurations are added to perfect the basic information of the webpage detection task and further improve the efficiency of subsequent webpage anomaly detection. Before the webpage detection task is constructed, the creation of a single-point test task can be tested for the execution of the crawler process to avoid abnormal task creation.

[0057] Further, the step S3 is specifically:

[0058] S31, when the system starts, the webpage detection module obtains the number of webpage parsing threads, starts a corresponding number of threads, and loops to obtain the webpage detection task from the to-be-consumed queue;

[0059] S32, each thread queries the basic information and the webpage traversal depth corresponding to the current webpage detection task, and initializes;

[0060] S33, using the playwright automation test framework, starting the playwright program according to the basic information and the webpage traversal depth, performing detection class related monitoring and webpage link detection related monitoring, obtaining corresponding detection results and parsing results, and recording the detection process through the playwright program, the detection class related monitoring includes interface anomaly detection, webpage content anomaly detection and console anomaly detection, the webpage link detection related monitoring includes new tab page monitoring, page redirection monitoring and page new link address parsing;

[0061] S34, after each thread completes the current webpage detection task, a tracking interface API is generated through the playwright program, the recorded detection process is packaged into a tracking file and uploaded to the cloud through the tracking interface API, and after obtaining the corresponding cloud address, the tracking file stored in the local of the current thread is deleted.

[0062] As can be known from the above description, the webpage detection module subscribes and consumes the webpage detection task in the to-be-consumed queue, can start a corresponding thread according to the basic information configured by the data management submodule to distribute different webpage detection tasks, and each thread performs corresponding detection class related monitoring and webpage link detection related monitoring according to the obtained basic information in the webpage detection task to determine whether there is an anomaly and parse the webpage address, effectively improving the webpage anomaly detection efficiency; at the same time, each thread records the detection and parsing process of the webpage detection task and generates a tracking file stored in the cloud, which can be convenient for subsequent staff to track the anomaly.

[0063] Further, the step S32 of initializing the basic information and the webpage traversal depth further comprises:

[0064] The error details error_content, the sub-page chain list hrefs, the current numerical level deep_now, and the domain name address domain of the current website to be tested are initialized;

[0065] Before the step S34 of uploading the tracking file to the cloud through the tracking interface API, the step further comprises:

[0066] Each thread determines whether the error details error_conten parameter in the current webpage detection task is empty. If not, it indicates that there is an exception. The tracking file corresponding to the current webpage detection task is uploaded to the cloud through the tracking interface API, and the corresponding cloud address is obtained after deleting the tracking file stored locally in the current thread.

[0067] As can be seen from the above description, only the detection process with an exception is tracked to generate a tracking file and store it in the cloud, which facilitates exception tracking while reducing the storage pressure of the cloud and the local storage pressure.

[0068] Further, the step S33 further comprises:

[0069] The page.on("dialog") interface in the playwright program is called to monitor pop-up events. If a pop-up window is detected, the page.content() interface in the playwright program is called to obtain the current webpage content text for exception judgment. After the judgment is completed, ESC is closed.

[0070] Before the playwright program performs an operation click, it is determined whether the current webpage has a scroll bar, a page turning element, and a more loading element according to the page sliding configuration, the page turning configuration, and the more loading button configuration, respectively, and the corresponding traversal click is performed.

[0071] As can be seen from the above description, the webpage is also detected for pop-up windows to avoid blocking other click times due to pop-up windows in the subsequent click operation process. In addition, it is determined whether the webpage to be detected has a scroll bar, a page turning element, and a more loading element according to the page sliding configuration, the page turning configuration, and the more loading button configuration in the data management submodule, so that the traversal click is performed when it is detected, further avoiding blocking the subsequent click behavior.

[0072] Further, the background management module further comprises a data statistics submodule, and the step S4 specifically comprises:

[0073] S41, the webpage detection module returns the detection result corresponding to the current webpage detection task, the parsing result and the cloud address corresponding to the tracking file to the data statistics submodule, and binds the task ID of the current webpage detection task with the corresponding detection result, parsing result and cloud address;

[0074] S42, it is judged whether there is a corresponding website connection under the current webpage detection task, if yes, filtering, otherwise adding the new address in the parsing result to the to-be-consumed queue;

[0075] S43, if the parsing result corresponding to the current webpage detection task is abnormal, the detection result corresponding to the current webpage detection task is read, and the corresponding channel warning is performed according to the pre-warning condition met by the detection result;

[0076] S44, the detection result and the parsing result corresponding to each task ID are visually displayed, and the corresponding cloud address is mapped to the detection result in the form of a jump link.

[0077] Further, the step S4 further comprises:

[0078] S45, periodically summarize the detection result and the parsing result of a plurality of webpage detection tasks completed, and generate a corresponding test summary table, including the total number of detected webpages, the number of passed detected webpages and the number of abnormal webpages;

[0079] The step S4 further comprises a step:

[0080] S5, repeatedly steps S1 to S4 for the detected abnormal webpages at a regular time, and re-performs webpage abnormal detection.

[0081] As can be seen from the above description, the detection result and the parsing result obtained by the webpage detection module performing the webpage detection task are uniformly summarized and managed by the data statistics submodule of the background management module, so that the statistical view of the screening record and the abnormal result is facilitated; at the same time, the abnormal webpages in the detection result are further detected at a regular time, and the detection result is improved.

[0082] Please refer to Figure 6 A multi-threaded website abnormal detection terminal, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the multi-threaded website abnormal detection method.

[0083] From the above description, the beneficial effects of the present application are that based on the same technical concept, a multi-threaded website anomaly detection terminal is provided in cooperation with the above-mentioned multi-threaded website anomaly detection method, the web page running process is divided into two aspects of background management module and web page detection module through system service platform management, wherein the background management module is the core of the whole system hub, mainly responsible for basic information configuration and web page detection task construction of the website to be tested, and the detection task construction frequency and detection depth are configured for each website, then the web page detection module performs corresponding multi-threaded detection and analysis, on one hand, judges whether there is an anomaly in the content of the current page, and on the other hand, analyzes all accessible addresses of the current page, and returns the results to the background management module for data statistics, so that the detection result of the anomaly in the background can be responded in time, and the overall detection result in the background can also be seen when the final detection is completed, thereby effectively improving the web page anomaly detection efficiency.

[0084] The multi-threaded website anomaly detection method and terminal provided by the present application are mainly applied to the scene of detecting the reliability of web page resources provided by a website and the accessibility of a web page, and will be specifically described below in combination with specific embodiments:

[0085] Please refer to Figure 1 , the embodiment one of the present application is:

[0086] A multi-threaded website anomaly detection method, as shown in Figure 1 , comprising the steps of:

[0087] S1, the background management module acquires the website to be tested and configures basic information.

[0088] S2, based on the basic information, the web page detection task is constructed and configured with the depth of the detection web page to be traversed, and then is issued to the to-be-consumed queue.

[0089] S3, the web page detection module acquires a plurality of web page detection tasks of the to-be-consumed queue in real time, and performs web page anomaly detection and hierarchical web page address analysis according to the basic information and the depth of the web page detection task to be traversed through a corresponding number of threads.

[0090] S4, the web page detection module returns the detection result and the analysis result to the background management module.

[0091] That is, in the embodiment, the webpage running process is divided into a background management module and a webpage detection module by system service platform management, wherein the background management module is the core of the whole system and is mainly responsible for basic information configuration and webpage detection task construction of the website to be detected, a special detection task construction frequency and detection depth are configured for each website, then the webpage detection module performs corresponding multi-thread detection and analysis, on one hand, judges whether there is an abnormality in the content of the current page, on the other hand, analyzes all accessible addresses of the current page, and returns the results to the background management module for data statistics, so that the abnormal detection result can be responded in time when the background receives the abnormal detection result, and the overall detection result can be seen in the background when the final detection is completed, and the webpage abnormal detection efficiency is effectively improved.

[0092] Please refer to Figures 2 to 5 , the second embodiment of the present application is:

[0093] A multi-thread website abnormal detection method, on the basis of the above-mentioned first embodiment, in the embodiment, as shown in Figure 2 , the background management module includes three sub-modules, namely a data management submodule Data Management, a construction management submodule Bulid Management and a data statistical submodule Statistical Management, in the embodiment, the data management submodule Data Management supports the creation of website basic information, that is, step S1 is specifically:

[0094] S11, the data management submodule obtains the website to be detected and performs basic information configuration, including:

[0095] Creating a corresponding website, filling in the corresponding webpage name, and configuring a corresponding website address list, the website address list is at least one address, and can also be multiple addresses, to solve the scene that there is no jump relationship between multiple routes of some websites.

[0096] Detection item configuration, including interface abnormality detection configuration, webpage content abnormality detection configuration and console abnormality detection configuration.

[0097] Among them, in this embodiment, the interface detection configuration mainly includes but is not limited to: 1) interface response code http code exception list, such as: [500, 501, 502, 503, 504]; 2) interface request address http url exception list, such as: ["undefined", "null"], but any monitoring and analysis method supported by playwright page.on can be supplemented as needed. Wherein playwright page.on is an initialization monitoring API of playwright automation test framework, and the following response, console, loading, etc. Are parameters of this API, and different parameters can be passed to achieve different monitoring effects;

[0098] The web content exception detection configuration is an exception content list: such as ["web is out of order", "video not found", "course has been offline", "webpage cannot be opened", "cannot access this website"] And the like, used to judge whether the entire web content contains some known abnormal text content or elements;

[0099] The console exception detection configuration is the same as above, which is also an exception content stack keyword list: error_list, used for exception checking when the web detection module detects. Considering that many websites may have some external dependencies that cause exceptions to be thrown, but they do not belong to the real business results, therefore, exclude_error_list is also supported as a whitelist filter to avoid too much noise, wherein exclude_error_list refers to the whitelist stack, for example, a third-party business is dependent on a business, which has an exception thrown, but it has no effect on the business, this exception configuration can be used to not push when it is judged that the exception is in the whitelist of known exceptions.

[0100] Early warning item configuration, the configuration system supports pushing WeChat, email and telephone and other early warning channels, and can perform instant notification of the corresponding early warning channel according to whether the early warning notification condition is met (such as: notify WeChat when 1 exception occurs, notify telephone when 2 exceptions occur in succession) when a web exception is detected.

[0101] Custom basic configuration, including but not limited to basic configuration when the system starts, dynamic loading data configuration of the website to be tested, and filtering text configuration.

[0102] Among them, in this embodiment, the basic configuration when the system starts specifically includes the webpage analysis thread number thread_count configuration (for example, default is 1), headless mode headless enable / disable configuration (for example, default is true) and page timeout time url_timeout configuration (for example, default is 10 seconds);

[0103] The dynamic loading data mode configuration of the website to be tested specifically includes the page sliding scroll value configuration (for example, default is 1, indicating that it is pulled down only once), the page turning next_page (including the need to pass specific positioning element next_page.content and the turning number next_page.count) and the more button more (also including the need to pass specific positioning element more.content and the clicking number more.count);

[0104] The filter text filter_tex configuration is a list, specifically including storing the text content that does not need to be clicked, suitable for some known element filtering configuration that does not need to be clicked.

[0105] Among them, in this embodiment, all configuration items are in JSON data structure, which is convenient for extension and use.

[0106] S12, the data management submodule obtains all sub-page addresses of the website to be tested, and stores them after corresponding filtering and deduplication, so as to support record inquiry while reducing space occupation.

[0107] That is, the basic configuration when the system starts can effectively ensure the initialization after the system starts, while increasing the dynamic loading data mode configuration of the website to be tested supporting page sliding, turning and loading more content, ensuring the comprehensiveness of webpage detection, and increasing text filtering to ensure irrelevant text interference with webpage detection, further effectively improving the efficiency of webpage anomaly detection.

[0108] For example, as shown in Figure 2 In this embodiment, the construction management submodule is a scheduling center responsible for constructing and issuing tasks to the webpage detection module at regular intervals, which can configure the depth of the webpage to be detected and create the frequency of the construction. As shown in Figure 3 The step S2 specifically includes:

[0109] S21, the construction management submodule acquires basic information, constructs a webpage detection task and configures corresponding webpage traversal depth and task construction frequency, simultaneously increases custom basic configuration, thereby perfecting the basic information of the webpage detection task, further improving the efficiency of subsequent webpage anomaly detection, including configuring whether to start the headless mode and whether to use coolie login. For example, by setting storage_state (the path parameter of the transmission parameter) through browser.new_context (an API provided by the playwright automation test framework, used for preloading local cookie information, so that the webpage does not need to perform the login preoperation), the local cookie is directly applied, and the user login operation process is avoided. This step can effectively solve the eighth shortcoming mentioned in the background art.

[0110] Before constructing the webpage detection task, the creation of a single-point test task can also be performed for the observation of the execution crawler process of the webpage, so as to avoid the abnormality of task creation.

[0111] S22, the construction management submodule creates the current webpage detection task based on the time interval of the task construction frequency, and judges whether there is an unfinished task corresponding to the current construction number when creating the task. If yes, the webpage detection task corresponding to the unfinished task is taken as the current webpage detection task, otherwise a new webpage detection task is created as the current webpage detection task through the current construction number.

[0112] S23, after the completion of the task, the construction management submodule creates a list of webpage addresses to be detected under the current webpage detection task according to the website address list configured in the digital management submodule, sets the current path level of the webpage to 0 and the total depth to the webpage traversal depth configured above, and adds the current webpage detection task to the Redis Queue as a to-be-consumed data through redis.list_append (“crawler_task”) (indicating adding a to-be-consumed data to the crawler_task queue of the Redis Queue), as shown in Figure 2 for subsequent webpage detection module Crawler to subscribe and consume and parse.

[0113] Meanwhile, as shown in Figure 2 , the webpage detection module Crawler is the core implementation module of the system scheme, which listens to whether there is a webpage to be detected, parses the corresponding webpage, one is to analyze whether the content of a certain webpage is abnormal, and the other is to analyze the next level of webpage address, and iterates this parsing process. As shown in Figure 4As shown, the Crawler Thread is a web page detection thread, and the Crawler Agent is a server running the web page detection thread. One server can run multiple threads. In this embodiment, step S3 is specifically:

[0114] S31, when the system is started, the web page detection module obtains the number of web page analysis threads thread_count configured by the data management submodule, starts a corresponding number of threads, and obtains web page detection tasks from the to-be-consumed queue Redis Queue through redis.rds.brpop(["crawler_task"])(which means taking one to-be-consumed data from the crawler_task queue of the Redis Queue) in a loop. This method can ensure that each thread consumes a unique web page, avoiding the problem of multiple instances analyzing the same web page address when directly querying the database. At the same time, the multi-threaded listening method of this step can effectively solve the problem of long page analysis time and low efficiency caused by tens of thousands of website sub-links, and can also solve the seven shortcomings mentioned in the background technology.

[0115] S32, after obtaining the web page detection task, each thread queries the basic information corresponding to the current web page detection task (including the detection item configuration, early warning item configuration, and custom basic configuration performed in the background management module, etc.) and the web page traversal depth value, and initializes these configuration variables, including but not limited to initializing the error details error_content, the sub-page link list hrefs, the current numerical level deep_now, and the domain name address domain of the current website to be tested.

[0116] S33, using the playwright automated testing framework, starting the playwright program according to the basic information and the web page traversal depth, performing detection class related listening and web page link detection related listening, obtaining corresponding detection results and analysis results, and recording the detection process through the context.tracing.start(playwright process record trace function enable API) of the playwright program. The detection class related listening includes interface exception detection, web page content exception detection, and console exception detection. The web page link detection related listening includes new tab page listening, page redirection listening, and page new link address analysis.

[0117] In this embodiment, the interface exception detection is specifically interface request exception detection. The interface request is monitored and intercepted through page.on(“response”), which is an API of the playwright framework, and the data of the monitoring callback of the API is processed in this embodiment. It is determined whether the URL of the request contains the exception keywords listed in the configuration, for example: [“undefined”, “null”], and whether the http status code (that is, the interface status code) contains the exception code listed in the configuration, for example: [500, 501, 502, 503, 504]. When an exception is detected, error content description is appended to error_content. The error content description includes: 1) error source: interface request, 2) request address: response.url, 3) http code: response.status, and 4) error text: response.text().

[0118] The web page content exception detection is specifically to obtain the content of the current page through page.content() (an interface provided by playwright to obtain the entire web page content text), and to determine whether the current page contains error content items. When an exception is detected, error content is appended to error_content. The error content description records: 1) error source: page display, and 2) error content: **.

[0119] The console exception detection is specifically to monitor and intercept the console error through page.on(“console”) (a method provided by playwright to monitor the browser console printing), and to determine whether the console_message.type (the error type level printed by the browser console) is error and the console_message.text (the error stack text printed by the browser console) contains the key error in the configuration. When an exception is detected, error content description is appended to error_content. The error content description includes: 1) error source: console, and 2) error stack: console_message.text.

[0120] The new tab page listening specifically detects the opening of a new tab page, and the new tab page is listened to through page.on("popup") (web link address). When the next click operation triggers the opening of a new page tab, the current new tab address is obtained, and the click element information is recorded. The result is then discarded to the data management submodule Statistical Management for database storage judgment of the new sub-page address, that is, whether the corresponding href (referring to the parsed new address from the web address) under the corresponding task task exists. If it exists, it is directly filtered, if it does not exist, it is queried in the database whether the page address exists under the task. If it does not exist, an address record is created in the database, including information: sub-page address, current level deep now+1, parent page address parent_href, element text static_text clicked to jump, and record creation time and state as to be tested (await). At the same time, by calling redis.list_append("crawler_task"), the address to be tested is pushed to the Redis Queue for consumption queue. If deep_now is equal to the system task configuration deep, a new sub-address will not be created, which is used for depth judgment cutoff.

[0121] The page redirection listening specifically detects the redirection of the page, and the page redirection is listened to through page.on("load") (a page redirection listening interface provided by playwright). When the next click operation triggers the page redirection, the current latest page.url (the real-time latest address of the web page) is obtained. As above, the result is discarded to the Statistical Management for database storage judgment of the new sub-page address, and the previous page is rolled back.

[0122] The page new link address parsing specifically detects the href address of the page, and through the page.frames method, all iframes (including the main web page, which is also an iframe, iframe is an HTML element, used to embed another web page in a web page, here referring to the case of nesting other web pages in a web page) on the entire web page are obtained, and frame.locator("a[href]").all() (that is, playwright locates all a elements with href attribute in each frame The ability to link) to obtain the root node address iframe_base_url (that is, the domain name of the embedded webpage) and the corresponding url of each iframe webpage. When the url contains http, then directly proceed to the database judgment of the new sub-page address. If the URL does not contain http, if it is a URL inside the iframe, then href is iframe_base_url+url, if it is the address of the main site, then href is domain+url. After splicing the address, as above, the result is given to the Statistical Management for database judgment of the new sub-page address. This step can effectively solve the fifth shortcoming mentioned in the background art.

[0123] S34, after each thread completes its current webpage detection task, a trace interface API is generated through the context.tracing.stop(playwright custom recording process record, an API that generates a trace playback file) and the recorded detection process is packaged into a trace file (i.e. trace file) and uploaded to the cloud through the trace interface API, and after obtaining the corresponding cloud download address, the trace file stored in the local storage of the current thread is deleted. In this embodiment, in order to improve the efficiency of webpage anomaly detection, only the detection process with anomalies is tracked to generate a trace file and store it in the cloud, which facilitates anomaly tracking while reducing cloud storage pressure and local storage pressure, i.e. before uploading the trace file to the cloud through the trace interface API in step S34, it also includes:

[0124] Each thread determines whether the error_conten parameter in its current webpage detection task is empty. If it is not empty, it indicates that there is an anomaly, the trace file corresponding to the current webpage detection task is uploaded to the cloud through the trace interface API, and after obtaining the corresponding cloud address, the trace file stored in the local storage of the current thread is deleted. At the same time, the specific error content, address, and trace file address are submitted to the data statistical submodule Statistical Management of the background management module, and a problem can be created for follow-up, and the detection status of this webpage address is modified to done, and the process continues to wait for the information of the next webpage to be detected. When there is a webpage to be detected, the above processes are repeated.

[0125] That is, the webpage detection module subscribes to and consumes the webpage detection tasks in the consumption queue, can start corresponding threads according to the basic information configured by the data management submodule to distribute the execution of different webpage detection tasks, and each thread performs corresponding detection class related monitoring and webpage link detection related monitoring according to the basic information obtained in the webpage detection task, determines whether there is an exception and parses the webpage address, effectively improves the webpage exception detection efficiency; at the same time, each thread records the detection and parsing process of the webpage detection task, and generates a tracking file and stores it to the cloud, which can facilitate subsequent staff to track the exception. This step can effectively solve the third and sixth shortcomings mentioned in the background art.

[0126] Meanwhile, in the present embodiment, again Figure 4 As shown, step S33 further includes:

[0127] In order to avoid the pop-up window from blocking other click events during the subsequent click operation process, the page.on("dialog") interface in the playwright program (an API for monitoring web page pop-up windows) is called to monitor pop-up window events. If a pop-up window is detected, the page.content() interface in the playwright program is called to obtain the current web page content text for abnormality judgment. After the judgment is completed, ESC is closed to avoid blocking the interface click operation. This step can effectively solve the fourth shortcoming mentioned in the background art.

[0128] Meanwhile, before the operation click in the playwright program, it is judged according to the page sliding configuration, page turning configuration and load more button configuration whether the current web page has a scroll bar, whether there is a page turning element and whether there is a load more element, and corresponding traversal click is performed.

[0129] When it is judged that the current web page has a scroll bar, it is scrolled to the bottom, and all clickable text elements and image elements on the page are obtained according to page.accessibility.snapshot() (an interface capability of playwright, which provides a list of page clickable elements) to perform traversal click. It is considered that many websites have some common buttons or guides on each page, and some common elements are configured to avoid clicking to improve execution efficiency. It should be noted that UI automation often has some pop-up windows, so this embodiment also supports configuration whether to close the pop-up window by calling ESC to avoid blocking the subsequent click behavior. This step can effectively solve the problem that the website homepage is only implicitly addressed, so it can only access the homepage itself and cannot expand any more web page addresses.

[0130] When it is detected that the current page has a next_page.content page turning element, the corresponding next page element is clicked, and the newly added element content of the page is detected. The operation of monitoring the page redirection is repeated, and the incremental clickable elements are clicked. At this time, it is judged whether next_page.count has been clicked, otherwise it is judged whether there is a next page element. If yes, the operation of monitoring the page redirection and the page new link address analysis is repeated until the next page element does not exist or next_page.count has been clicked.

[0131] When it is detected that there is a more load more element, it is detected whether the current page has a more.load more element. If it exists, the more.load more element is clicked. The process is the same as next_page, and the incremental accessible elements are clicked.

[0132] That is, the pop-up window detection is also performed on the webpage, so as to avoid that the pop-up window in the subsequent clicking operation process blocks other clicking time. In addition, according to the page sliding configuration, the page turning configuration and the load more button configuration performed in the data management submodule, it is judged whether the webpage to be detected exists a scroll bar, a page turning element and a load more element, so that the traversal clicking is performed when the scroll bar, the page turning element and the load more element are detected, and the subsequent clicking behavior is further avoided from being blocked.

[0133] In addition, as shown in the embodiment, step S4 specifically includes: Figure 5

[0134] S41, the webpage detection module returns the detection result corresponding to the current webpage detection task, the analysis result and the cloud address corresponding to the trace file to the data statistics submodule, and binds the task ID of the current webpage detection task with the corresponding detection result, analysis result and cloud address.

[0135] S42, first, in the Crawler analysis stage, the analyzed website data is sent to the data statistics submodule, and after the data statistics submodule receives the analyzed website data, it is judged whether the corresponding website connection under the current webpage detection task exists, if yes, the analyzed website data is filtered, otherwise, the new address in the analysis result is added to the to-be-consumed queue.

[0136] S43, if the analysis result corresponding to the current webpage detection task is abnormal, the corresponding detection result is read, and corresponding channel warning is performed according to the warning notification condition met by the detection result. At the same time, the background can also click the synchronization problem to the corresponding problem recording platform, and set the state to processing. At this time, if the same error is received again, the repeated warning push can be avoided, and the message interference can be reduced.

[0137] Finally, after the detection is completed, the corresponding bug creation synchronization can be supported in the background, and since each webpage detection task has a corresponding task ID, the webpage address and the basic information of each webpage parsed in the local can be viewed under the corresponding task ID, including: address, parent address, level, detection state and detection result.

[0138] S44, the detection result and the analysis result corresponding to each task ID are visually displayed, and when the detection fails, the corresponding cloud address can be mapped to the detection result in the form of a jump link, that is, the details of the error and the corresponding trace record are recorded, the trace file stored in the cloud can be obtained by clicking the cloud address, all performances of the process are watched, the whole test process is completely traced back, and the problem is conveniently and clearly positioned. At the same time, the problem can be synchronously submitted to the corresponding internal problem recording platform.

[0139] In addition, in the embodiment, step S4 further includes: ​

[0140] S45, periodically aggregate the detection results and analysis results of the plurality of completed webpage detection tasks, and generate a corresponding test summary table, including the total number of detected webpages, the number of passed webpages, and the number of abnormal webpages.

[0141] Step S4 further includes the following steps:

[0142] S5, periodically repeating steps S1 to S4 for the detected abnormal webpages to re-perform webpage anomaly detection.

[0143] That is, the detection results and analysis results obtained by the webpage detection module performing the webpage detection task are uniformly aggregated and managed by the data statistics submodule of the background management module, so that the screening records and abnormal result statistics can be viewed; at the same time, the abnormal webpages in the detection results are periodically re-detected for further webpage anomaly detection to improve the detection results.

[0144] In this embodiment, the following specific application scenarios are provided to illustrate the effect of the multi-threaded website anomaly detection method of this embodiment:

[0145] Scenario one: test student A is responsible for testing a WEB online education website. With the continuous promotion of the WEB education website, the resources of the website are increasing, and the resource quantity is in units of ten thousand. At the same time, the teaching types and columns of the education website are also increasing, and during the version iteration process, there have been several times when some resources and courses associated with the columns could not be opened or were abnormally opened due to improper development and release. In addition, there have also been cases of course abnormal opening due to some misoperations of the operation or third-party service failures, and the related abnormalities have been fed back by users several times. In view of this situation, due to the involvement of many cooperative developments and the unclear division of labor of QAs, etc., it is difficult to prevent the occurrence of related problems. At the same time, since there are a large number of webpage subpages involved, the verification method by pure manual means also needs to invest a large amount of manpower cost.

[0146] Through the scheme of this embodiment, A only needs to configure the domain name address of the WEB website, then find the development to collect the common error content text of each module of the website, and configure the text to the error_content field, then set the depth of traversal required, such as 4 levels. Then click Start to execute, which can automatically perform 4-level depth path traversal detection on the website. The number of automatically traversed webpages can be over ten thousand, which is not only fast, but also can automatically filter repeated cross pages. The abnormalities found in the process can be prewarned to A for corresponding promotion processing, and support the final completion of the test of an automatic detection result review.

[0147] Scenario two: based on the background of scenario one, test classmate A just started a device, and a thread was enabled by default. Since the education website operated by A has been in operation for a long time, the resource path is very large, and even if it is checked one by one through automation, it takes 24 hours to detect.

[0148] Through the scheme of this embodiment, A can start the service according to the performance evaluation of the server to start how many threads, and deploy multiple servers when necessary. For example, A configures the number of threads to be 2, and deploys 5 servers. When the first web page to be parsed is received, one of the 10 threads receives the web page to be parsed, parses the home page to 10 sub-page addresses, and then gives the result to StatisticalManagement for storage of new sub-page addresses. Through redis.list_append(“crawler_task”), the web page address after deduplication is given to Redis, and the 10 threads will each take away a sub-page for secondary parsing, and the secondary parsing result is also dropped into the pending consumption queue, and waits for the arrival of the next pending consumption parsing web page information. That is, through the distributed consumption method, the overall parsing efficiency can be improved by multiple times. When the device resources are unlimited, the parsing speed will be quite fast.

[0149] Scenario three: test classmate A is responsible for testing an online education website, which has a recommendation column, and each course dynamically recommends many associated course contents. If A simply scans the link and finds an abnormal web course address, the developer will have no clue to investigate and cannot know where the link comes from, because the link should not exist in the front end, and it is normal to search for it.

[0150] Through the scheme of this embodiment, each retrieved link address will store which link jumps from, and at the same time, it will also store when the element text is clicked. Even if the previous web page is a dynamic web page, it can be known that what content is clicked at that time, which is convenient for developers and QAs to further follow up the problem.

[0151] Scenario four: test classmate A is responsible for the automatic testing of a WEB online education, responsible for the business writing of scripts. In the process of online continuous monitoring regression, especially for some new core columns, although most of the time, the content of the page is the same, and most of the verification is reliability verification, that is, most of the script writing is repeated. The investment for small A is not done, and the cost performance is low. At the same time, the monitoring automation execution time of small A often needs to consume a long time to build execution. A build kit needs to spend an hour or even more time, when pushed to the group, the message is often lagging behind, even if the user feedback is given priority to, the system is automatically restored or developed to repair, and then the push message is received. One early warning is not timely, and two, for some occasional problems, the report cannot intuitively find out the problem.

[0152] Through the scheme of the embodiment, most of the page scanning can be covered, 50* related faults of page calling services can be detected, not only the related early warning can be made in the first time, but also the trace record can be saved and provided to the development for positioning and troubleshooting of the related error process, and the above-mentioned pain points in the scenario are perfectly solved.

[0153] Please refer to Figure 6 , the third embodiment of the present application is:

[0154] A multi-threaded website exception detection terminal 1, comprising a memory 2, a processor 3 and a computer program stored in the memory 2 and executable on the processor 3, the processor 3 executes the computer program to complete the steps in the multi-threaded website exception detection method in the above-mentioned embodiment one or embodiment two.

[0155] In summary, the multi-threaded website exception detection method and terminal provided by the present application have the following beneficial effects:

[0156] 1. The interface related exception detection can be covered, which is a real request means for online webpage service reliability.

[0157] 2. The test personnel can configure exception webpage text according to the demand of the business to perform rapid traversal scanning of the webpage, without additional coding.

[0158] 3. The detection process captures the exception information, which is used to promote the development and processing optimization of potential risk problems.

[0159] 4. Real-time reception of the process exception detection message is performed, without waiting for the entire detection task to end.

[0160] 5、Through tracking records, small probability abnormal problems in the automatic detection process can be perfectly traced back, low probability problems due to service stability are avoided, and the situation that further analysis cannot be carried out at the time due to only screenshots or assertion logs is avoided.

[0161] 6、The execution effect can be doubled through multiple service devices, so that the automatic detection result is delivered in time.

[0162] 7、The multi-threaded execution of multiple web detection tasks makes the remaining threads complete the task by sharing the consumption even if part of the device is abnormal in the process, for example, a thread restarts.

[0163] 8、Through data statistics, the number of web pages covered and the detection pass rate of each detection can be clearly known.

[0164] 9、Through data statistics, the source of the web page can be effectively traced back, and the problem can be quickly located when the problem is investigated.

[0165] The above only describes the embodiments of the present application, and does not limit the patent range of the present application, any equivalent transformation, direct or indirect application in related technical fields by using the content of the specification and drawings of the present application are also included in the patent protection range of the present application.

Claims

1. A multi-threaded website anomaly detection method, characterized in that, Including the following steps: S1. The backend management module obtains the website to be tested and configures its basic information; S2. Based on the aforementioned basic information, periodically construct webpage detection tasks and configure the depth to be traversed for the detected webpages before sending them to the queue to be consumed. S3. The web page detection module obtains multiple web page detection tasks in the queue to be consumed in real time, and performs web page anomaly detection and step-by-step web page address parsing by a corresponding number of threads based on the basic information of each web page detection task and the required traversal depth. S4. The webpage detection module returns the detection results and parsing results to the backend management module; The backend management module also includes a construction management sub-module, and step S2 specifically involves: S21. The construction management submodule obtains the basic information, constructs a webpage detection task and configures the corresponding webpage traversal depth and task construction frequency, and adds custom basic configurations, including whether to enable headless mode and whether to use cookie login. S22. The construction management submodule creates the current web page detection task at regular intervals based on the time interval of the task construction frequency using the current construction number. When creating the task, it determines whether there are any unfinished tasks for the current construction number. If so, the web page detection task corresponding to the unfinished task is used as the current web page detection task. Otherwise, a new web page detection task is created using the current construction number as the current web page detection task. S23. The construction management submodule creates a list of web page addresses to be tested under the current web page detection task based on the website address list, sets the current path level of the URL to 0 and the total depth to the web page traversal depth, and adds the current web page detection task as data to be consumed to the consumption queue. Step S3 specifically involves: S31. When the system starts, the web page detection module obtains the number of web page parsing threads, starts the corresponding number of threads, and cyclically obtains the web page detection tasks from the queue to be consumed. S32. Each thread queries the basic information and webpage traversal depth corresponding to its current webpage detection task, and initializes it. S33. Using the Playwright automated testing framework, start the Playwright program based on the basic information and the webpage traversal depth to perform detection-related listening and webpage link detection-related listening, obtain the corresponding detection results and parsing results, and record the detection process through the Playwright program. The detection-related listening includes interface anomaly detection, webpage content anomaly detection, and console anomaly detection. The webpage link detection-related listening includes new tab page listening, page redirect listening, and page new link address parsing. S34. After each thread completes its current webpage detection task, it generates a tracking interface API through the playwright program, packages the recorded detection process into a tracking file, uploads it to the cloud through the tracking interface API, and deletes the tracking file stored locally by the current thread after obtaining the corresponding cloud address.

2. The multi-threaded website anomaly detection method according to claim 1, characterized in that, The backend management module includes a data management sub-module, and step S1 specifically involves: S11. The data management submodule acquires the website to be tested and configures its basic information, including: Create a corresponding website, fill in the corresponding webpage name, and configure the corresponding website address list, which must contain at least one address; The detection item configuration includes interface anomaly detection configuration, webpage content anomaly detection configuration, and console anomaly detection configuration; The system can be configured to support push notifications via WeChat, email, and telephone when webpage anomalies are detected. The system will then send immediate notifications to the corresponding notification channels based on whether the conditions for notification are met. Customizable basic configuration, including basic configuration at system startup, configuration of dynamic data loading method for the website under test, and configuration of filtered text; All configuration items are in JSON data structure; S12. The data management submodule obtains all sub-webpage addresses of the website to be tested, performs corresponding filtering and deduplication, and then stores them.

3. The multi-threaded website anomaly detection method according to claim 2, characterized in that, The basic configuration at system startup includes configuration for the number of web page parsing threads, configuration for enabling / disabling headless mode, and configuration for page timeout. The dynamic data loading method configuration of the website under test specifically includes page sliding configuration, page turning configuration, and load more button configuration; The filtered text configuration specifically includes configuring a menu for storing text content that does not need to be clicked.

4. The multi-threaded website anomaly detection method according to claim 1, characterized in that, The initialization of the basic information and the webpage traversal depth in step S32 also includes: Initialize the error details (error_content), the subpage chain list (hrefs), the current numerical level (deep_now), and the domain address (domain) of the current website under test; Before uploading the tracking file to the cloud via the tracking interface API in step S34, the method further includes: Each thread checks whether the error_conten parameter in its current webpage detection task is empty. If it is not empty, it indicates that there is an exception. The thread then uploads the trace file corresponding to the current webpage detection task to the cloud through the trace interface API, obtains the corresponding cloud address, and deletes the trace file stored locally in the current thread.

5. The multi-threaded website anomaly detection method according to claim 3, characterized in that, Step S33 further includes: Call the page.on("dialog") interface in the playwright program to listen for pop-up events. If a pop-up is detected, call the page.content() interface in the playwright program to get the current webpage content text for exception handling. After the exception handling is completed, press ESC to close the window. Before clicking in the PlayWright program, it determines whether the current webpage has a scroll bar, a page turning element, and a "load more" element, based on the page swiping configuration, the page turning configuration, and the "load more" button configuration, and then performs the corresponding traversal and clicking.

6. The multi-threaded website anomaly detection method according to claim 2, characterized in that, The backend management module also includes a data statistics sub-module, and step S4 specifically involves: S41. The webpage detection module returns the detection result, the parsing result, and the cloud address corresponding to the tracking file corresponding to the current webpage detection task to the data statistics submodule, and binds the task ID of the current webpage detection task with the corresponding detection result, parsing result, and cloud address; S42. Determine whether a corresponding website link already exists under the current webpage detection task. If so, filter it; otherwise, add the new address in the parsing result to the queue to be consumed. S43. If the parsing result corresponding to the current webpage detection task is abnormal, then read the corresponding detection result and issue the corresponding channel warning according to the warning notification conditions met by the detection result. S44. Visualize the detection results and parsing results corresponding to each task ID, and map the corresponding cloud address to the detection results in the form of a jump link.

7. The multi-threaded website anomaly detection method according to claim 6, characterized in that, Step S4 further includes: S45. Periodically summarize the detection results and parsing results of multiple completed webpage detection tasks, and generate a corresponding test summary table, including the total number of detected webpages, the number of webpages that passed the detection, and the number of abnormal webpages. The step S4 is followed by the following step: S5. Repeat steps S1 to S4 periodically for detected abnormal web pages to re-detect web page anomalies.

8. A multi-threaded website anomaly detection terminal, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the multi-threaded website anomaly detection method according to any one of claims 1 to 7.

Images