Automatic flow monitoring system based on playwright framework

Abstract

The invention discloses an automatic traffic monitoring system based on a playwright framework, which relates to the technical field of network monitoring and comprises an operation script writing module, a playwright controller, a request / response interception module, a traffic acquisition module, an anomaly detection engine, a network simulator, a multi-scene scheduler, a data analysis module and a visual report generator. By deeply integrating the request / response interception capability of Playwright, automatic monitoring integrating flow collection, anomaly detection, multi-scene simulation and performance analysis is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of network monitoring, and more particularly to an automatic flow monitoring system based on a Playwright framework. Background Art

[0002] In the current field of web application performance monitoring, many technologies use Selenium or the Chrome DevTools Protocol (CDP) to simulate page operations and monitor network requests. For example, some tools monitor HTTP requests initiated by browsers, recording information such as the request address, response time, and data size, and outputting them in log format. Typical technologies include using Selenium with browser extensions to capture traffic, or using Node.js to call the CDP API to implement basic monitoring functions.

[0003] Although existing technologies enable browser-driven control of page behavior, request and response data can be obtained through underlying protocol interfaces (such as CDP) for simple recording or analysis, and some tools can be integrated into CI / CD processes, existing monitoring technologies lack in-depth processing capabilities for multi-scenario concurrency, anomaly detection, and performance indicator correlation, resulting in the following drawbacks:

[0004] (1) Unable to accurately intercept all requests and responses: Selenium itself does not support deep interception of request / response content. Although CDP supports it, it requires developers to manually parse a large amount of raw data, resulting in high development costs and easy omission of key information;

[0005] (2) Data collection granularity is coarse and storage is not uniform: Existing tools usually only record URLs and status codes, lacking detailed collection of dimensions such as request frequency, response time, and data volume, and also lack a unified data model for storage and subsequent analysis;

[0006] (3) Lack of real-time anomaly detection mechanism: Traditional practices rely on manual review of logs to determine whether there are anomalies. An automated threshold detection rule system has not been established, making it difficult to detect sudden traffic anomalies in a timely manner.

[0007] (4) Weak network environment adaptability testing capabilities: Existing tools do not have the ability to simulate different network bandwidths and delays, and cannot evaluate the performance of applications under different network conditions;

[0008] (5) Low monitoring accuracy in multi-user concurrent scenarios: Most tools can only run on a single page or a single user session, and cannot effectively distinguish the traffic behavior of multiple users, resulting in distorted monitoring results;

[0009] (6) Disconnection between operation and monitoring: Automated operation scripts and traffic monitoring scripts are usually separated, resulting in an inability to accurately map the causal relationship between operation behavior and traffic changes;

[0010] (7) Lack of correlation analysis of performance indicators: The existing monitoring system fails to correlate network traffic data with page loading performance indicators (such as FP, LCP, CLS, etc.), making it difficult to locate performance bottlenecks;

[0011] (8) Low level of visualization: The reports output by existing tools are mostly text logs, lacking graphical display, which is not conducive to team collaboration and problem reproduction.

[0012] Therefore, how to achieve automated monitoring that integrates traffic collection, anomaly detection, multi-scenario simulation, and performance analysis is an urgent problem that technical personnel in this field need to solve. Summary of the Invention

[0013] In view of this, the present invention provides an automated traffic monitoring system based on the Playwright framework. By deeply integrating Playwright's request / response interception capabilities, it realizes automated monitoring that integrates traffic collection, anomaly detection, multi-scenario simulation, and performance analysis.

[0014] In order to achieve the above object, the present invention adopts the following technical solutions:

[0015] An automated traffic monitoring system based on the Playwright framework, including an operation script writing module, a Playwright controller, a request / response interception module, a traffic collection module, an anomaly detection engine, a network simulator, a multi-scenario scheduler, a data analysis module, and a visual report generator;

[0016] Operation script writing module, writing user operation scripts through playwright;

[0017] Playwright controller, which executes user operation scripts to realize browser automation control;

[0018] The request / response interception module calls the route() function of the playwright controller to intercept each request in the browser, and calls the response() function of the playwright controller to intercept the response details;

[0019] Traffic collection module, which organizes intercepted request and response details into traffic data according to a preset format and stores it in a database or local file;

[0020] The anomaly detection engine retrieves real-time traffic data from a database or local file, compares it with pre-set threshold rules, identifies abnormal events, and triggers real-time warnings.

[0021] Network simulator, call the setNetworkConditions() function of the playwright controller to simulate different network environments;

[0022] The multi-scene scheduler calls the browser.newContext() function of the playwright controller to create an independent context and allocates the context to the user operation script; receives abnormal events, and links the operation script writing module and network simulator based on the abnormal events; calls the PerformanceAPI by calling the page.evaluate() function of the playwright controller to obtain performance indicators;

[0023] The data analysis module retrieves traffic data from the database or local files, and cross-analyzes traffic data and performance indicators to obtain historical traffic and performance trends;

[0024] Visual report generator, generates visual reports based on historical traffic and performance trends, traffic data, abnormal events, threshold rules, and performance indicators.

[0025] Preferably, the user operation script includes clicking a button, filling in a form, jumping to a page, etc., and each user operation script is assigned a unique user identifier or session identifier.

[0026] Preferably, the request includes a request header, and the response details include metadata such as the response header, response body, time consumption, status code, etc.; relevant fields are extracted from the request and response details for organization, and the extracted fields include timestamp, URL, request type, response size, response time, IP location, etc.

[0027] Preferably, the anomaly detection engine is set with several threshold rules, including request failure rate > 10%, response time > 2s, etc.

[0028] Preferably, the parameters of the network environment include bandwidth, delay, packet loss rate, etc. By setting the parameters to simulate different network environments, the performance of the browser page under different conditions is evaluated.

[0029] Preferably, the multi-scene scheduler creates a new page according to the multi-page switching requirements generated by executing the user operation script by calling the context.newPage() function of the playwright controller, calls the page.goto() function to navigate to different URLs, and calls the page.close() function to close the page.

[0030] Preferably, the multi-scene scheduler creates a preset number of contexts by calling the browser.newContext() function of the playwright controller, sorts the user operation scripts to be executed in the playwright controller, and allocates idle contexts to the sorted user operation scripts; each allocated context is automatically associated with the user ID or session ID corresponding to the user operation script, executes the user operation script in the context, generates user operation and traffic data, and automatically associates the corresponding user ID or session ID in the user operation and traffic data.

[0031] Preferably, each context has independent Cookies, LocalStorage, SessionStorage and browsing session status. Multiple contexts can be used to simulate the concurrent behavior of multiple users, and the corresponding user operation scripts can be executed simultaneously in multiple contexts to ensure that the user operations and traffic data generated by each user executing the user operation script can be accurately distinguished, and that the user operations and traffic data are strictly isolated and do not interfere with each other.

[0032] Preferably, the user operation script is adjusted or the network simulator is controlled to switch the network environment according to the abnormal event control operation script writing module; the monitoring frequency parameter in the user operation script is adjusted to increase or decrease the number of monitoring times, or the execution of the user operation script is stopped to end the monitoring.

[0033] Preferably, the performance indicators include first rendering time FP, maximum content rendering time LCP, cumulative layout offset CLS, first input delay FID, etc.

[0034] Preferably, the process of cross-analysis by the data analysis module is:

[0035] Step 1: Clean and preprocess traffic data and performance indicators. Preprocessing includes denoising, filling missing values, and data standardization.

[0036] Step 2: Based on the processed traffic data and performance indicators, calculate the characteristic value that represents the correlation between the traffic data and the performance indicators;

[0037] Step 3: Based on the eigenvalues, use statistical methods or machine learning algorithms to build a correlation model that characterizes the correlation between traffic data and performance indicators to obtain historical traffic and performance trends.

[0038] Preferably, the data analysis module can also compare historical traffic and performance trends according to the preset anomaly detection threshold to achieve anomaly monitoring, and send the anomaly monitoring results to the visual report generator to generate a visual report; the multi-scenario scheduler adjusts or adds threshold rules in the anomaly detection engine according to the anomaly monitoring results.

[0039] Preferably, the content of the visual report includes charts, trend analysis and key performance indicator comparison, etc. The visual report is in the form of PDF, screenshot or dashboard report, and can be exported to HTML or PDF format.

[0040] It can be seen from the above technical solutions that, compared with the prior art, the present invention provides an automated traffic monitoring system based on the Playwright framework, which specifically includes the following beneficial effects:

[0041] (1) Improved traffic monitoring accuracy: The Playwright framework natively supports a deep request / response interception mechanism, which allows for the capture of more granular data, including request headers and response bodies. This solves the problem that traditional tools such as Selenium cannot directly access this information. In addition, the traffic collection module uses a unified format to store collected data, making subsequent analysis more accurate and effective.

[0042] (2) Enhanced anomaly detection capabilities: A threshold rule-based anomaly detection engine is introduced for automated early warning. It can not only automatically identify anomalies based on preset thresholds, but also support dynamic loading of new rules, improving problem discovery efficiency and reducing false alarm rates.

[0043] (3) Improve network adaptability testing capabilities: Using Playwright's built-in setNetworkConditions() method, you can accurately simulate different network environments, so that potential problems of the application under various network conditions can be discovered in advance during the development phase, which is crucial for ensuring user experience.

[0044] (4) Support for concurrent monitoring in multiple scenarios: By creating multiple Playwright contexts, complete isolation between different user sessions is achieved, ensuring the independence and accuracy of monitoring data during concurrent operations by multiple users, and avoiding the problem of result distortion caused by cross-interference.

[0045] (5) Realize the linkage analysis of operation and traffic: The user operation script is closely integrated with the traffic collection process. Through synchronous execution, the traffic changes caused by specific operations and their impact on page performance can be directly tracked, providing a basis for optimization.

[0046] (6) Assisting performance optimization decisions: Combining key performance indicators obtained from the Performance API with traffic data for correlation analysis can help quickly locate performance bottlenecks and guide subsequent optimization work.

[0047] (7) Improve visualization and collaboration efficiency: Generate graphic reports containing detailed charts and key indicators, which enhances the readability and interactivity of the reports and facilitates sharing and discussion of problems among team members and proposing solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0049] Figure 1 This is a schematic diagram of the automated traffic monitoring system based on the Playwright framework provided by the present invention. DETAILED DESCRIPTION

[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0051] The embodiment of the present invention discloses an automated traffic monitoring system based on the Playwright framework, including an operation script writing module, a Playwright controller, a request / response interception module, a traffic collection module, an anomaly detection engine, a network simulator, a multi-scenario scheduler, a data analysis module, and a visual report generator;

[0052] Operation script writing module, writing user operation scripts through playwright;

[0053] Playwright controller, which executes user operation scripts to realize browser automation control;

[0054] The request / response interception module calls the route() function of the playwright controller to intercept each request in the browser, and calls the response() function of the playwright controller to intercept the response details;

[0055] Traffic collection module, which organizes intercepted request and response details into traffic data according to a preset format and stores it in a database or local file;

[0056] The anomaly detection engine retrieves real-time traffic data from the database or local files, compares it with pre-set threshold rules, identifies anomalies, and triggers real-time warnings.

[0057] Network simulator, call the setNetworkConditions() function of the playwright controller to simulate different network environments;

[0058] The multi-scene scheduler calls the browser.newContext() function of the playwright controller to create an independent context and allocates the context to the user operation script; receives abnormal events, and links the operation script writing module and network simulator based on the abnormal events; calls the PerformanceAPI by calling the page.evaluate() function of the playwright controller to obtain performance indicators;

[0059] The data analysis module retrieves traffic data from the database or local files, and cross-analyzes traffic data and performance indicators to obtain historical traffic and performance trends;

[0060] Visual report generator, generates visual reports based on historical traffic and performance trends, traffic data, abnormal events, threshold rules, and performance indicators.

[0061] Furthermore, user operation scripts include clicking buttons, filling out forms, jumping to pages, etc., and each user operation script is assigned a unique user ID or session ID.

[0062] Furthermore, the request includes a request header, and the response details include metadata such as the response header, response body, time consumption, status code, etc.; relevant fields are extracted from the request and response details for organization, and the extracted fields include timestamp, URL, request type, response size, response time, IP location, etc.

[0063] Furthermore, the performance indicators include FP, LCP, CLS, FID, etc.

[0064] Furthermore, visual reports include charts, trend analysis, and key performance indicator comparisons. These reports can be generated as PDFs, screenshots, or dashboards, and exported to HTML or PDF formats. The visual report generator is tightly integrated with the data analysis module. By cross-analyzing collected traffic data and performance indicators, it provides deeper insights, helping developers quickly identify potential issues and optimize the user experience.

[0065] Furthermore, the database may adopt a relational database (MySQL), NoSQL (MongoDB), a time series database (InfluxDB), etc.

[0066] In a specific embodiment, the multi-scene scheduler creates a new page based on the multi-page switching requirements generated by executing the user operation script by calling the context.newPage() function of the playwright controller, calling the page.goto() function to navigate to different URLs, and calling the page.close() function to close the page.

[0067] The multi-scene scheduler creates a preset number of contexts by calling the browser.newContext() function of the playwright controller, sorts the user operation scripts to be executed in the playwright controller, and allocates idle contexts to the sorted user operation scripts; each allocated context is automatically associated with the user ID or session ID corresponding to the user operation script, executes the user operation script in the context, generates user operation and traffic data, and automatically associates the corresponding user ID or session ID in the user operation and traffic data.

[0068] Furthermore, each context has independent Cookies, LocalStorage, SessionStorage and browsing session status. Multiple contexts can be used to simulate the concurrent behavior of multiple users, and the corresponding user operation scripts can be executed in multiple contexts at the same time, ensuring that the user operations and traffic data generated by each user executing the user operation script can be accurately distinguished, and that user operations and traffic data are strictly isolated and do not interfere with each other.

[0069] The multi-scenario scheduler not only supports switching between different pages but also simulates the concurrent behavior of multiple users, ensuring that each user's behavior and traffic can be accurately distinguished. By creating a separate Playwright context, each virtual user or session is provided with a completely isolated environment. This ensures that even when multiple user sessions run within the same browser instance, they do not interfere with each other, thereby improving the accuracy and reliability of monitoring data. Whenever the scheduler needs to start a new user session, it calls browser.newContext() to create a new context. This new context contains all the user's session information, such as cookies and local storage. It also ensures that all activities targeting this user, such as request / response interception, network condition simulation, and performance metric collection, are performed within this independent context. Therefore, even in complex multi-user concurrent scenarios, the system can accurately track each user's actions and their corresponding network activity, greatly improving the flexibility and accuracy of the overall monitoring system. The scheduler ensures that intercepted traffic data in the request / response interception module and traffic collection module is automatically attached with the user / session identifier bound to its source context. Similarly, metrics collected by the performance metric collector are associated with the currently executed page and its context identifier. This strict context isolation and identifier binding mechanism fundamentally prevents crosstalk between traffic and performance data from different users or sessions, ensuring accurate attribution and scenario independence of monitoring data. The scheduler's management of concurrent tasks and resources also ensures the stability and efficiency of the monitoring process.

[0070] In a specific embodiment, the anomaly detection engine is configured with multiple threshold rules, including, for example, request failure rate > 10% or response time > 2s.

[0071] According to the abnormal event control operation script writing module, adjust the user operation script or control the network simulator to switch the network environment; adjust the monitoring frequency parameters in the user operation script to increase or decrease the number of monitoring times, or stop the execution of the user operation script to end the monitoring.

[0072] Furthermore, the parameters of the network environment include bandwidth, delay, packet loss rate, etc., and the performance of the browser page under different conditions is evaluated by simulating different network environments.

[0073] In a specific embodiment, the process of cross-analysis by the data analysis module is as follows:

[0074] S1: Data cleaning and preprocessing of traffic data and performance indicators. Preprocessing includes denoising, filling missing values, and data standardization. Data standardization unifies data from different sources into a unified format for subsequent analysis.

[0075] S2: Based on the processed traffic data and performance indicators, calculate the characteristic values that represent the correlation between the traffic data and performance indicators. Traffic data includes the number of requests and response time, while performance indicators include the first rendering time (FP) and the maximum content rendering time (LCP). The calculated characteristic values include the average response time in each time period and the corresponding LCP change rate.

[0076] S3: Based on the characteristic values, a correlation model is constructed using statistical methods or machine learning algorithms to characterize the correlation between traffic data and performance indicators, and historical traffic and performance trends are obtained.

[0077] Furthermore, the data analysis module can also compare historical traffic and performance trends based on preset anomaly detection thresholds to achieve anomaly monitoring, and send the anomaly monitoring results to the visual report generator to generate a visual report; the multi-scenario scheduler adjusts or adds threshold rules in the anomaly detection engine based on the anomaly monitoring results.

[0078] In one embodiment, the anomaly detection engine can use a machine learning algorithm to identify abnormal events based on traffic data, and the multi-scenario scheduler can adjust model parameters in the machine learning algorithm to adjust the abnormal event identification. Machine learning algorithms include isolation forest algorithms, K-means clustering algorithms, and the like.

[0079] On the other hand, in a specific embodiment, multiple contexts are deployed through virtual machines or containers to implement distributed concurrent behavior simulation of multiple users.

[0080] On the other hand, in a specific embodiment, the visual report generator can use open source libraries such as ECharts or Plotly to generate visual reports; in addition, third-party BI tools such as Grafana and Tableau can also be used as visual report generators to process data.

[0081] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0082] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An automated traffic monitoring system based on the playwright framework, characterized in that: It includes an operation script writing module, a playwright controller, a request / response interception module, a traffic collection module, an anomaly detection engine, a network simulator, a multi-scenario scheduler, a data analysis module, and a visual report generator; Operation script writing module, writing user operation scripts through playwright; Playwright controller, which executes user operation scripts to realize browser automation control; The request / response interception module calls the route() function of the playwright controller to intercept each request in the browser, and calls the response() function of the playwright controller to intercept the response details; Traffic collection module, which organizes intercepted request and response details into traffic data according to a preset format and stores it; The anomaly detection engine retrieves real-time collected traffic data and compares it based on preset threshold rules to identify abnormal events and trigger real-time warnings; Network simulator, call the setNetworkConditions() function of the playwright controller to simulate different network environments; The multi-scenario scheduler calls the browser.newContext() function of the Playwright controller to create an independent context and assigns the context to the user operation script; receives abnormal events, and links the operation script writing module and network simulator based on the abnormal events; and calls the Performance API by calling the page.evaluate() function of the Playwright controller to obtain performance indicators; The data analysis module retrieves traffic data and conducts cross-analysis on traffic data and performance indicators to obtain historical traffic and performance trends; Visual report generator, generates visual reports based on historical traffic and performance trends, traffic data, abnormal events, threshold rules, and performance indicators.

2. The automated traffic monitoring system based on the playwright framework according to claim 1 is characterized in that: User operation scripts include clicking buttons, filling out forms, and jumping to pages. Each user operation script is assigned a unique user ID or session ID.

3. The automated traffic monitoring system based on the playwright framework according to claim 1 is characterized in that: The request includes the request header, and the response details include the response header, response body, time consumption and status code. The relevant fields are extracted from the request and response details for organization. The extracted fields include timestamp, URL, request type, response size, response time and IP location.

4. The automated traffic monitoring system based on the Playwright framework according to claim 1 is characterized in that: The anomaly detection engine is set with several threshold rules.

5. The automated traffic monitoring system based on the playwright framework according to claim 1 is characterized in that: The parameters of the network environment include bandwidth, delay and packet loss rate. Different network environments are simulated by setting parameters.

6. The automated traffic monitoring system based on the Playwright framework according to claim 1 is characterized in that: The multi-scene scheduler creates a new page based on the multi-page switching requirements generated by executing the user operation script by calling the context.newPage() function of the playwright controller, calls the page.goto() function to navigate to different URLs, and calls the page.close() function to close the page.

7. The automated traffic monitoring system based on the Playwright framework according to claim 2 is characterized in that: The multi-scene scheduler creates a preset number of contexts by calling the browser.newContext() function of the playwright controller, sorts the user operation scripts to be executed in the playwright controller, and allocates idle contexts to the sorted user operation scripts; Each allocated context is automatically associated with the user ID or session ID corresponding to the user operation script, the user operation script is executed in the context, user operation and flow data are generated, and the corresponding user ID or session ID is automatically associated with the user operation and flow data.

8. The automated traffic monitoring system based on the Playwright framework according to claim 1, characterized in that: According to the abnormal event control operation script writing module, adjust the user operation script or control the network simulator to switch the network environment; adjust the monitoring frequency parameters in the user operation script, modify the monitoring times, or stop the execution of the user operation script to end the monitoring.

9. The automated traffic monitoring system based on the Playwright framework according to claim 1, characterized in that: The process of cross-analysis in the data analysis module is as follows: Step 1: Clean and preprocess traffic data and performance indicators. Preprocessing includes denoising, filling missing values, and data standardization. Step 2: Based on the processed traffic data and performance indicators, calculate the characteristic value that represents the correlation between the traffic data and the performance indicators; Step 3: Build a correlation model based on the characteristic values to characterize the correlation between traffic data and performance indicators, and obtain historical traffic and performance trends.

10. The automated traffic monitoring system based on the Playwright framework according to claim 1, characterized in that: The data analysis module compares historical traffic and performance trends based on preset anomaly detection thresholds to achieve anomaly monitoring and sends the anomaly monitoring results to the visual report generator to generate a visual report; The multi-scenario scheduler adjusts or adds threshold rules in the anomaly detection engine based on anomaly monitoring results.

Images