Automated traffic monitoring system based on playwright framework

Abstract

The application discloses an automatic flow monitoring system based on a playwright framework, relates to the technical field of network monitoring, and comprises an operation script compiling module, a playwright controller, a request / response interception module, a flow collecting module, an exception detection engine, a network simulator, a multi-scene scheduler, a data analysis module and a visual report generator. Through deep integration of the request / response interception capability of the playwright, the automatic monitoring integrating flow collection, exception detection, multi-scene simulation and performance analysis is realized.

Description

Technical Field

[0001] This invention relates to the field of network monitoring technology, and more specifically to an automated traffic monitoring system based on the Playwright framework. Background Technology

[0002] In the current field of network application performance monitoring, there are already many techniques that use Selenium or Chrome DevTools Protocol (CDP) to simulate page operations and monitor network requests. For example, some tools monitor HTTP requests initiated by the browser, recording information such as request address, response time, and data size, and outputting it in log form. Typical techniques include using Selenium in conjunction with browser extensions for traffic capture, or using Node.js to call the CDP API to implement basic monitoring functions.

[0003] While existing technologies allow browser-driven page behavior control to obtain request and response data through underlying protocol interfaces (such as CDP) for simple recording or analysis, and some tools can be integrated into CI / CD processes, current monitoring technologies lack in-depth processing capabilities for multi-scenario concurrency, anomaly detection, and performance metric correlation, thus exhibiting the following shortcomings:

[0004] (1) Unable to accurately intercept all requests and responses: Selenium itself does not support deep interception of request / response content. Although CDP supports it, it requires developers to manually parse a large amount of raw data, resulting in high development costs and easy omission of key information;

[0005] (2) Coarse data collection granularity and inconsistent storage: Existing tools usually only record URLs and status codes, lacking fine collection of dimensions such as request frequency, response time, and data volume, and also lacking a unified data model for storage and subsequent analysis;

[0006] (3) Lack of real-time anomaly detection mechanism: Traditional practices rely on manual log review to determine whether there are anomalies. There is no automated threshold detection rule system, making it difficult to detect sudden traffic anomalies in a timely manner.

[0007] (4) Weak network environment adaptability testing capability: Existing tools do not have the ability to simulate different network bandwidths and delays, and cannot evaluate the performance of applications under different network conditions;

[0008] (5) Low monitoring accuracy in multi-user concurrent scenarios: Most tools can only run on a single page or in a single user session, which cannot effectively distinguish the traffic behavior of multiple users, resulting in distorted monitoring results;

[0009] (6) Disconnect between operation and monitoring: Automated operation scripts and traffic monitoring scripts are usually separate, which makes it impossible to accurately correspond the causal relationship between operation behavior and traffic changes;

[0010] (7) Lack of performance index correlation analysis: The existing monitoring system fails to perform correlation analysis between network traffic data and page loading performance indicators (such as FP, LCP, CLS, etc.), making it difficult to locate performance bottlenecks;

[0011] (8) Low level of visualization: The reports output by existing tools are mostly text logs, lacking graphical display, which is not conducive to team collaboration and problem reproduction.

[0012] Therefore, how to achieve automated monitoring that integrates traffic collection, anomaly detection, multi-scenario simulation, and performance analysis is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0013] In view of this, the present invention provides an automated traffic monitoring system based on the Playwright framework. By deeply integrating Playwright's request / response interception capabilities, it achieves automated monitoring that integrates traffic collection, anomaly detection, multi-scenario simulation, and performance analysis.

[0014] To achieve the above objectives, the present invention adopts the following technical solution:

[0015] An automated traffic monitoring system based on the Playwright framework includes an operation script writing module, a Playwright controller, a request / response interception module, a traffic acquisition module, an anomaly detection engine, a network simulator, a multi-scenario scheduler, a data analysis module, and a visualization report generator.

[0016] The operation script writing module allows users to write operation scripts using PlayWright.

[0017] The PlayWright controller executes user-defined scripts to automate browser controls.

[0018] The request / response interception module calls the `route()` function of the Playwright controller to intercept each request in the browser and calls the `response()` function of the Playwright controller to intercept response details.

[0019] The traffic acquisition module organizes the intercepted request and response details into traffic data according to a preset format and stores it in a database or local file;

[0020] The anomaly detection engine retrieves real-time traffic data from the database or local files, compares it based on preset threshold rules, identifies abnormal events, and triggers real-time alerts.

[0021] The network simulator calls the setNetworkConditions() function of the PlayWright controller to simulate different network environments.

[0022] The multi-scene scheduler calls the browser.newContext() function of the Playwright controller to create an independent context and assign a context to the user operation script; it receives abnormal events and links the operation script writing module and network simulator according to the abnormal events; it calls the PerformanceAPI to obtain performance metrics by calling the page.evaluate() function of the Playwright controller.

[0023] The data analysis module retrieves traffic data from the database or local files and performs cross-analysis on the traffic data and performance metrics to obtain historical traffic and performance trends.

[0024] A visualization report generator that produces visualization reports based on historical traffic and performance trends, traffic data, abnormal events, threshold rules, and performance metrics.

[0025] Preferably, the user operation script includes clicking a button, filling out a form, or redirecting to a page, and each user operation script is assigned a unique user identifier or session identifier.

[0026] Preferably, the request includes a request header, and the response details include metadata such as response headers, response body, time taken, and status code; relevant fields are extracted from the request and response details and organized, including timestamp, URL, request type, response size, response time, and IP address.

[0027] Preferably, the anomaly detection engine is configured with several threshold rules, including request failure rate >10% and response time >2s.

[0028] Preferably, network environment parameters include bandwidth, latency, packet loss rate, etc. By setting parameters, different network environments can be simulated, and the performance of the browser's page under different conditions can be evaluated.

[0029] Preferably, the multi-scenario scheduler creates a new page by calling the context.newPage() function of the playwright controller, navigates to different URLs by calling the page.goto() function, and closes the page by calling the page.close() function, based on the multi-page switching requirements generated by executing the user operation script.

[0030] Preferably, the multi-scenario scheduler creates a preset number of contexts by calling the browser.newContext() function of the playwright controller, sorts the user operation scripts to be executed in the playwright controller, and allocates idle contexts to the sorted user operation scripts; each allocated context is automatically associated with the user identifier or session identifier corresponding to the user operation script, executes the user operation script in the context, generates user operation and traffic data, and automatically associates the corresponding user identifier or session identifier in the user operation and traffic data.

[0031] Preferably, each context has independent Cookie, LocalStorage, SessionStorage, and browsing session state. Multiple contexts can simulate concurrent behavior of multiple users, and execute corresponding user operation scripts simultaneously in multiple contexts. This ensures that the user operations and traffic data generated by each user executing the user operation script can be accurately distinguished, and that user operations and traffic data are strictly isolated and do not interfere with each other.

[0032] Preferably, the abnormal event control operation script writing module adjusts the user operation script or controls the network simulator to switch network environments; the monitoring frequency parameter in the user operation script is adjusted to increase or decrease the number of monitoring times, or to stop monitoring when the user operation script finishes execution.

[0033] Preferably, performance metrics include First Render Time (FP), Maximum Content Render Time (LCP), Cumulative Layout Offset (CLS), and First Input Delay (FID).

[0034] Preferably, the data analysis module performs cross-analysis as follows:

[0035] Step 1: Perform data cleaning and preprocessing on traffic data and performance metrics. Preprocessing includes noise reduction, filling in missing values, and data standardization.

[0036] Step 2: Based on the processed traffic data and performance metrics, calculate the feature values ​​that characterize the correlation between the traffic data and performance metrics;

[0037] Step 3: Based on the feature values, use statistical methods or machine learning algorithms to construct a correlation model that represents the relationship between traffic data and performance indicators, and obtain historical traffic and performance trends.

[0038] Preferably, the data analysis module can also compare historical traffic and performance trends based on preset anomaly detection thresholds to achieve anomaly monitoring, and send the anomaly monitoring results to the visualization report generator to generate a visualization report; the multi-scenario scheduler adjusts or adds threshold rules in the anomaly detection engine based on the anomaly monitoring results.

[0039] Preferably, the content of the visualization report includes charts, trend analysis, and comparison of key performance indicators. The visualization report can be in the form of PDF, screenshot, or dashboard, and can be exported as HTML or PDF format.

[0040] As can be seen from the above technical solution, compared with the prior art, the present invention discloses an automated traffic monitoring system based on the Playwright framework, which specifically includes the following beneficial effects:

[0041] (1) Improved traffic monitoring accuracy: Through the request / response deep interception mechanism natively supported by the Playwright framework, more granular data, including request headers and response bodies, can be captured, solving the problem that traditional tools such as Selenium cannot directly obtain this information. In addition, the traffic collection module uses a unified format to store the collected data, making subsequent analysis more accurate and effective.

[0042] (2) Enhanced anomaly detection capability: An anomaly detection engine based on threshold rules is introduced for automated early warning. It can not only automatically identify anomalies according to preset thresholds, but also supports dynamic loading of new rules, which improves the efficiency of problem discovery and reduces the false alarm rate.

[0043] (3) Enhance network adaptability testing capabilities: By using the setNetworkConditions() method built into Playwright, different network environments can be accurately simulated, so that potential problems of the application under various network conditions can be discovered in advance during the development stage, which is crucial for ensuring user experience.

[0044] (4) Support for concurrent monitoring in multiple scenarios: By creating multiple Playwright contexts, complete isolation between different user sessions is achieved, ensuring the independence and accuracy of monitoring data during concurrent operations by multiple users, and avoiding the problem of result distortion caused by cross interference.

[0045] (5) Implement operation and traffic linkage analysis: The user operation script is closely integrated with the traffic collection process. Through synchronous execution, the traffic changes caused by specific operations and their impact on page performance can be directly tracked, providing a basis for optimization.

[0046] (6) Assisting performance optimization decision-making: By combining the key performance indicators obtained from the Performance API with traffic data for correlation analysis, it can help to quickly locate performance bottlenecks and guide subsequent optimization work.

[0047] (7) Improve visualization and collaboration efficiency: Generate graphic reports with detailed charts and key indicators, which enhances the readability and interactivity of the reports, making it easier for team members to share and discuss problems and propose solutions. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0049] Figure 1 This is a schematic diagram of an automated traffic monitoring system based on the Playwright framework provided by the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] This invention discloses an automated traffic monitoring system based on the Playwright framework, including an operation script writing module, a Playwright controller, a request / response interception module, a traffic acquisition module, an anomaly detection engine, a network simulator, a multi-scenario scheduler, a data analysis module, and a visualization report generator;

[0052] The operation script writing module allows users to write operation scripts using PlayWright.

[0053] The PlayWright controller executes user-defined scripts to automate browser controls.

[0054] The request / response interception module calls the `route()` function of the Playwright controller to intercept each request in the browser and calls the `response()` function of the Playwright controller to intercept response details.

[0055] The traffic acquisition module organizes the intercepted request and response details into traffic data according to a preset format and stores it in a database or local file;

[0056] The anomaly detection engine retrieves real-time traffic data from the database or local files, compares and identifies abnormal events based on preset threshold rules, and triggers real-time alerts.

[0057] The network simulator calls the setNetworkConditions() function of the PlayWright controller to simulate different network environments.

[0058] The multi-scene scheduler calls the browser.newContext() function of the Playwright controller to create an independent context and assign a context to the user operation script; it receives abnormal events and links the operation script writing module and network simulator according to the abnormal events; it calls the PerformanceAPI to obtain performance metrics by calling the page.evaluate() function of the Playwright controller.

[0059] The data analysis module retrieves traffic data from the database or local files and performs cross-analysis on the traffic data and performance metrics to obtain historical traffic and performance trends.

[0060] A visualization report generator that produces visualization reports based on historical traffic and performance trends, traffic data, abnormal events, threshold rules, and performance metrics.

[0061] Furthermore, user action scripts include clicking buttons, filling out forms, and redirecting to pages, and each user action script is assigned a unique user identifier or session identifier.

[0062] Furthermore, the request includes request headers, and the response details include metadata such as response headers, response body, time taken, and status code; relevant fields are extracted from the request and response details and organized, including timestamp, URL, request type, response size, response time, and IP address.

[0063] Furthermore, performance metrics include FP, LCP, CLS, FID, etc.

[0064] Furthermore, the visualization report includes charts, trend analysis, and key performance indicator comparisons. The report can be in PDF, screenshot, or dashboard format and exported as HTML or PDF. The visualization report generator is tightly integrated with the data analysis module, providing deeper insights through cross-analysis of collected traffic data and performance metrics. This helps developers quickly pinpoint potential issues and optimize the user experience.

[0065] Furthermore, the database can be a relational database (MySQL), a NoSQL database (MongoDB), a time-series database (InfluxDB), etc.

[0066] In one specific embodiment, the multi-scenario scheduler creates a new page by calling the context.newPage() function of the playwright controller, navigates to different URLs by calling the page.goto() function, and closes the page by calling the page.close() function, based on the multi-page switching requirements generated by executing the user operation script.

[0067] The multi-scenario scheduler creates a preset number of contexts by calling the browser.newContext() function of the playwright controller, sorts the user operation scripts to be executed in the playwright controller, and allocates idle contexts to the sorted user operation scripts. Each allocated context is automatically associated with the user identifier or session identifier corresponding to the user operation script. The user operation script is executed in the context, generating user operation and traffic data, and the corresponding user identifier or session identifier is automatically associated in the user operation and traffic data.

[0068] Furthermore, each context has independent Cookie, LocalStorage, SessionStorage, and browsing session state. Multiple contexts can simulate concurrent behavior of multiple users, and execute corresponding user operation scripts simultaneously in multiple contexts. This ensures that the user operations and traffic data generated by each user executing the user operation script can be accurately distinguished, and that user operations and traffic data are strictly isolated and do not interfere with each other.

[0069] The multi-scenario scheduler not only supports switching between different pages but also simulates concurrent behavior of multiple users, ensuring that each user's actions and traffic can be accurately distinguished. By creating independent Playwright contexts, it provides a completely isolated environment for each virtual user or session. This approach ensures that even when multiple user sessions run within the same browser instance, they do not interfere with each other, thereby improving the accuracy and reliability of monitoring data. Whenever the scheduler needs to start a new user session, it calls `browser.newContext()` to create a new context. This new context contains all session information for that user, such as cookies and local storage. It also ensures that all request / response interception, network condition simulation, and performance metric collection activities for this user are performed within this independent context. Therefore, even in complex multi-user concurrent scenarios, the system can accurately track the operations of each user and their corresponding network activities, greatly improving the flexibility and accuracy of the overall monitoring system. The scheduler ensures that the intercepted traffic data in the request / response interception module and traffic collection module is automatically appended with the user / session identifier bound to its source context. Similarly, the metrics acquired by the performance metric collector are also associated with the currently executing page and its context identifier. Through this strict context isolation and identifier binding mechanism, crosstalk between traffic and performance data from different users or sessions is fundamentally avoided, ensuring the accurate attribution and scenario independence of monitoring data. The scheduler's management of concurrent tasks and resources also guarantees the stability and efficiency of the monitoring process.

[0070] In one specific embodiment, the anomaly detection engine is configured with multiple threshold rules, including requests with a failure rate >10% or response time >2 seconds.

[0071] The module for controlling abnormal events can be used to adjust user operation scripts or control the network simulator to switch network environments; the monitoring frequency parameters in the user operation script can be adjusted to increase or decrease the number of monitoring times, or to stop monitoring when the user operation script finishes execution.

[0072] Furthermore, network environment parameters include bandwidth, latency, packet loss rate, etc., and the performance of browser pages under different conditions is evaluated by simulating different network environments.

[0073] In one specific embodiment, the data analysis module performs cross-analysis as follows:

[0074] S1: Perform data cleaning and preprocessing on traffic data and performance metrics. Preprocessing includes noise reduction, missing value imputation, and data standardization. Data standardization unifies the format of data from different sources to facilitate subsequent analysis.

[0075] S2: Based on the processed traffic data and performance metrics, calculate the feature values ​​that characterize the correlation between the traffic data and performance metrics; traffic data includes the number of requests, response time, etc., and performance metrics include first render time (FP), maximum content render time (LCP), etc. The calculated feature values ​​include the average response time and the corresponding LCP change rate in each time period, etc.

[0076] S3: Based on the eigenvalues, use statistical methods or machine learning algorithms to construct a correlation model that represents the relationship between traffic data and performance indicators, and obtain historical traffic and performance trends.

[0077] Furthermore, the data analysis module can compare historical traffic and performance trends based on preset anomaly detection thresholds to achieve anomaly monitoring, and send the anomaly monitoring results to the visualization report generator to generate a visualization report; the multi-scenario scheduler adjusts or adds threshold rules in the anomaly detection engine based on the anomaly monitoring results.

[0078] On the other hand, in one specific embodiment, the anomaly detection engine can use machine learning algorithms to identify abnormal events based on traffic data, and the multi-scenario scheduler can adjust the model parameters in the machine learning algorithm to adjust the anomaly event identification. Machine learning algorithms include the Isolation Forest algorithm, K-means clustering algorithm, etc.

[0079] On the other hand, in one specific embodiment, multiple contexts are deployed through virtual machines or containers to simulate distributed concurrent behavior for multiple users.

[0080] On the other hand, in one specific embodiment, the visualization report generator can use open-source libraries such as ECharts or Plotly to generate visualization reports; alternatively, third-party BI tools such as Grafana and Tableau can be selected as visualization report generators to process data.

[0081] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0082] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An automated traffic monitoring system based on the Playwright framework, characterized in that, It includes an operation script writing module, a playwright controller, a request / response interception module, a traffic collection module, an anomaly detection engine, a network simulator, a multi-scenario scheduler, a data analysis module, and a visualization report generator; The operation script writing module allows users to write user operation scripts using Playwright. User operation scripts include clicking buttons, filling out forms, and redirecting pages. Each user operation script is assigned a unique user identifier or session identifier. The PlayWright controller executes user-defined scripts to automate browser controls. The request / response interception module calls the `route()` function of the Playwright controller to intercept each request in the browser and calls the `response()` function of the Playwright controller to intercept response details. The traffic acquisition module organizes the intercepted request and response details into traffic data according to a preset format and stores it. The anomaly detection engine retrieves real-time traffic data and compares it based on preset threshold rules to identify abnormal events and trigger real-time alerts. The network simulator calls the setNetworkConditions() function of the PlayWright controller to simulate different network environments. The multi-scene scheduler calls the browser.newContext() function of the Playwright controller to create an independent context and assign a context to the user operation script; it receives abnormal events and links the operation script writing module and network simulator according to the abnormal events; it calls the PerformanceAPI to obtain performance metrics by calling the page.evaluate() function of the Playwright controller. The multi-scene scheduler creates a preset number of contexts by calling the browser.newContext() function of the playwright controller, sorts the user operation scripts to be executed in the playwright controller, and allocates idle contexts to the sorted user operation scripts. Each assigned context is automatically associated with the user identifier or session identifier corresponding to the user operation script. The user operation script is executed in the context to generate user operation and traffic data, and the corresponding user identifier or session identifier is automatically associated in the user operation and traffic data. Each context has its own independent Cookie, LocalStorage, SessionStorage, and browsing session state. By simulating concurrent behavior of multiple users through multiple contexts, the corresponding user operation scripts are executed simultaneously in multiple contexts. This ensures that the user operations and traffic data generated by each user's execution of the user operation script are accurately distinguished, and that user operations and traffic data are strictly isolated and do not interfere with each other. Based on the abnormal event control operation script, the module can be used to adjust the user operation script or control the network simulator to switch network environments; The data analysis module retrieves traffic data and performs cross-analysis of traffic data and performance indicators to obtain historical traffic and performance trends. A visualization report generator that produces visualization reports based on historical traffic and performance trends, traffic data, abnormal events, threshold rules, and performance metrics.

2. The automated traffic monitoring system based on the Playwright framework according to claim 1, characterized in that, The request includes a request header, and the response details include a response header, response body, timeout, and status code. Relevant fields are extracted from the request and response details and organized accordingly. The extracted fields include timestamp, URL, request type, response size, response time, and IP address.

3. The automated traffic monitoring system based on the Playwright framework according to claim 1, characterized in that, The anomaly detection engine has several threshold rules set.

4. The automated traffic monitoring system based on the Playwright framework according to claim 1, characterized in that, Network environment parameters include bandwidth, latency, and packet loss rate. Different network environments can be simulated by setting these parameters.

5. The automated traffic monitoring system based on the Playwright framework according to claim 1, characterized in that, The multi-scenario scheduler creates new pages by calling the context.newPage() function of the playwright controller, navigates to different URLs by calling the page.goto() function, and closes the pages by calling the page.close() function, based on the multi-page switching requirements generated by executing user operation scripts.

6. The automated traffic monitoring system based on the Playwright framework according to claim 1, characterized in that, Based on the abnormal event control operation script, the module can adjust the user operation script or control the network simulator to switch network environments; adjust the monitoring frequency parameters in the user operation script, modify the number of monitoring, or stop the monitoring after the execution of the user operation script.

7. The automated traffic monitoring system based on the Playwright framework according to claim 1, characterized in that, The data analysis module performs cross-analysis as follows: Step 1: Perform data cleaning and preprocessing on traffic data and performance metrics. Preprocessing includes noise reduction, missing value filling, and data standardization. Step 2: Based on the processed traffic data and performance metrics, calculate the feature values ​​that characterize the correlation between the traffic data and performance metrics; Step 3: Construct a correlation model based on the feature values ​​to represent the relationship between traffic data and performance indicators, and obtain historical traffic and performance trends.

8. The automated traffic monitoring system based on the Playwright framework according to claim 1, characterized in that, The data analysis module compares historical traffic and performance trends based on preset anomaly detection thresholds to achieve anomaly monitoring, and sends the anomaly monitoring results to the visualization report generator to generate a visualization report; The multi-scenario scheduler adjusts or adds threshold rules in the anomaly detection engine based on the anomaly monitoring results.

Images