Webpage performance monitoring analysis method, system and device and storage medium

Abstract

The invention discloses a webpage performance monitoring analysis method, system and device and a storage medium, relates to the technical field of webpage monitoring, and aims to overcome the limitation of an existing webpage performance monitoring technology. The method comprises the following steps: constructing a high-availability infrastructure to cope with large-scale concurrent access; front-end performance data are collected in real time, and a stream processing technology and a real-time algorithm are adopted for cleaning and aggregation; analyzing data by using a machine learning model, mining performance trends and abnormities, and establishing an intelligent reference; starting a scene intelligent adaptation module, accurately identifying a webpage application scene, and dynamically customizing a monitoring strategy; constructing a monitoring system covering data acquisition, analysis, reporting, alarming and optimization suggestion; and a storage medium is optimized, and data integrity and accessibility are ensured. The method has the advantages that diversified monitoring indexes, intelligent and accurate prediction performance, efficient real-time feedback and dynamic optimization of resource utilization are provided, website performance and user experience are remarkably improved, and a solid guarantee is provided for stable and efficient operation of the website.

Description

Technical Field

[0001] The present invention relates to the technical field of web page monitoring, and in particular to a web page performance monitoring and analysis method, system, device and storage medium. Background Art

[0002] With the rapid advancement of Internet technology, websites have become an important platform for companies to showcase their brands, services and products to the outside world. With the increase in user visits and the complexity of website functions, web performance issues have gradually become a challenge that cannot be ignored. For example, problems such as slow page loading, slow response and operation jams reduce user experience and easily cause user loss and a decline in conversion rate. Therefore, web performance monitoring has become an indispensable part of ensuring the smooth and efficient operation of the website.

[0003] Currently, there are many tools and technical solutions for web performance monitoring on the market, which mainly rely on different technical means to monitor and optimize web performance. The following are several typical existing technologies and their shortcomings: (1) The monitoring indicators are too single: YSlow and Pingdom mainly focus on a few performance parameters such as page loading time and response time, and lack comprehensive consideration of multi-dimensional performance parameters such as user interaction experience, error frequency, and user bounce rate; (2) Insufficient depth of data analysis: Although Google Analytics can obtain massive amounts of user behavior data, it has limitations in further data analysis capabilities and is difficult to reveal the patterns and trends hidden behind the data; (3) The real-time feedback mechanism is not sound: Although GTmetrix provides detailed performance reports, it is insufficient in terms of real-time feedback and cannot immediately reflect the actual operating status of the current web page; (4) Lack of dynamic optimization capabilities: Most existing technologies remain at the performance monitoring and problem diagnosis stage. They lack the ability to automatically adjust and optimize web page performance based on monitoring results and require human intervention.

[0004] Therefore, a more comprehensive, efficient, and intelligent web page performance monitoring and analysis method, system, device, and storage medium are needed. This new technology should be able to monitor and collect comprehensive performance indicator data of web pages in real time at various stages such as loading, rendering, and interaction, use advanced data analysis algorithms and machine learning technologies to conduct in-depth mining and analysis, discover potential performance problems and optimization space, and provide real-time warning and dynamic optimization functions. In this way, website performance can be significantly improved, user experience can be guaranteed, and strong support can be provided for the stable and efficient operation of the website. Summary of the invention

[0005] In view of the deficiencies of the prior art, the present invention provides a web page performance monitoring and analysis method, system, device and storage medium. The core purpose of the present invention is to integrate real-time insight and intelligent prediction engines, accurate anomaly detection and early warning systems, and scene intelligent adaptation and dynamic optimization strategies to create a web page performance monitoring and analysis method, system, device and storage medium. This innovation aims to completely revolutionize the way web page performance is monitored, by capturing performance data in real time, accurately predicting future trends, sensitively detecting and responding to abnormal events in real time, and dynamically adjusting optimization strategies according to user access scenarios, to achieve all-round intelligent management of web page performance.

[0006] Specifically, the present invention improves the immediacy, accuracy, and foresight of monitoring, so that customers can quickly grasp the real-time status of web page performance, predict and prevent potential problems. At the same time, through accurate anomaly detection and automatic early warning mechanisms, the abnormal response time is greatly shortened and potential losses are reduced. It also emphasizes scenario-based intelligent adaptation, dynamically adjusting optimization measures according to different user access scenarios, and providing users with a smoother and more personalized access experience. Finally, through an intuitive and easy-to-use visual interface, it provides customers with a comprehensive and clear performance overview and decision-making support, helping to make accurate decisions and optimize website operation strategies.

[0007] To achieve the above objectives, a web page performance monitoring and analysis method, system, device and storage medium are implemented through the following technical solutions: S1: System deployment and configuration, build a high-availability infrastructure in the cloud server or local data center: deploy high-performance servers, SSD storage arrays, and high-speed network switching equipment to allow hardware resources to fully meet the needs of large-scale concurrent access and data processing; configure the operating system for security reinforcement to ensure the security and stability of the system, deploy the Java runtime environment and database management system (MySQL / PostgreSQL), install necessary middleware services, ApacheHive for data warehouses, and Apache Flume for data stream processing. Then configure the Apache Kafka cluster to process real-time data streams, and deploy Apache Flink to achieve low-latency stream data processing; S2: Data collection and intelligent analysis, through the front-end monitoring tool (SDK), real-time collection of front-end page performance data, including page loading time, first contact time (FCT), time expiration rate (TLP), user interaction time, error rate, bounce rate, throughput, etc. Then use stream processing technology and real-time data analysis algorithms to perform preliminary cleaning and aggregation operations on the collected data, and use machine learning models to automatically analyze the processed data to mine performance trends and potential anomalies. Use historical performance data to train deep learning models, build intelligent benchmark models, integrate statistical methods, cluster analysis, and time series pattern recognition and other anomaly detection algorithm matrices, and characterize the normal fluctuation range of performance indicators; S3: Scenario intelligent adaptation and monitoring strategy generation, start the scenario intelligent adaptation module, use machine learning or rule engine to accurately identify web application scenarios. According to the scenario identification results, dynamically customize performance monitoring strategies and indicators to provide personalized monitoring solutions for different web applications. Continuously optimize monitoring strategies to ensure the accuracy and pertinence of monitoring; S4: Dynamic optimization strategy and resource management, design a dynamic optimization strategy engine, analyze monitoring data in real time and adjust website resources, dynamically adjust the parameters of the optimization strategy engine according to the monitoring results, implement resource loading priority adjustment, image compression optimization and cache management strategy. Ensure that the website maintains optimal performance under different network environments and user behaviors; S5: Performance monitoring system construction. A web page performance monitoring system is constructed based on the above monitoring and analysis methods, covering data collection module, analysis module, report generation module, alarm module and optimization suggestion module. The data collection module is used to obtain web page loading data and user interaction data in real time; the analysis module is used to perform in-depth analysis of data using anomaly detection algorithm matrix and model; the report generation module is used to generate detailed performance reports in a timely manner based on the analysis results; the alarm module is used to set performance thresholds, and automatically trigger alarms or early warning mechanisms when anomalies are detected; the optimization suggestion module is used to provide optimization suggestions based on the analysis results. The system continuously adjusts the monitoring and optimization strategies based on user feedback and optimization effects, forming a closed-loop continuous performance optimization process; S6: Storage media optimization, use reliable storage media to store performance data to ensure data integrity and accessibility. Store real-time performance data on SSDs and store historical data on HDDs or cloud storage to improve overall storage efficiency. Back up key data regularly and perform recovery tests regularly to ensure the validity and recoverability of backup data. And monitor the performance indicators of storage media (read and write speed, latency, throughput, etc.) in real time to identify and resolve potential problems in a timely manner.

[0008] Preferably, in said S1, it also includes: Deploy an Apache Kafka cluster consisting of multiple nodes on a cloud server or local data center. Each node should have sufficient memory, CPU resources, and disk space to support large-scale concurrent access and data processing requirements. Set the replication factor to 2 or higher to ensure data reliability and high availability of the system even in the event of a single point of failure. Next, deploy an Apache Flink cluster, including a master node (JobManager) for coordinating task allocation and state management, and multiple worker nodes (TaskManagers) for performing actual data processing tasks. Adjust the configuration parameters of TaskManager and JobManager, JVM heap size, parallelism, etc. according to the hardware resources to optimize performance. Configure the Kafka producer API to ensure that it can efficiently receive data from the front-end monitoring tool and publish it to the specified Kafka topic. Set up the Flink application as a Kafka consumer, subscribe to the relevant topic, start consuming the data stream from the front-end monitoring tool, and perform preliminary cleaning of the raw data to remove invalid or erroneous records. Through window functions, data within a period of time is aggregated and calculated, and deep learning models are trained based on historical performance data to determine the normal fluctuation range of performance indicators. The trained models are then integrated into Flink tasks to achieve real-time data analysis and prediction.

[0009] Preferably, in said S3, further comprising: The core of this solution is the scenario intelligent adaptation module, which can automatically identify and analyze the application scenarios of web pages, and can tailor performance monitoring strategies and indicators for different web applications to ensure the accuracy and pertinence of monitoring and meet the specific needs of customers. The deep integration of scenario intelligent adaptation and monitoring strategy generation is a static tool or system, and also a process of continuous evolution. Through continuous monitoring, analysis and optimization, it can ensure the continuous improvement of website performance and provide users with a smoother and more satisfactory access experience.

[0010] Preferably, in said S4, further comprising: The dynamic optimization strategy engine is also a core component of this solution. It has the ability to analyze monitoring data in real time and automatically adjust optimization strategies. By continuously monitoring website performance, performance bottlenecks and potential risk points can be quickly identified, and relevant parameter configurations can be adjusted in real time based on the latest monitoring results to ensure that the website can always maintain optimal performance under various network conditions and user behavior patterns, forming a good complementary relationship with the scene intelligent adaptation module. The dynamic optimization strategy engine can receive and process data streams from the performance monitoring system in real time, and flexibly adjust the optimization strategy based on the feedback information of these data. For example, it can rearrange the priority of resource loading to speed up page loading; optimize image compression algorithms to reduce bandwidth consumption; or more finely control cache resources to improve access efficiency. No matter what challenges are faced, the dynamic optimization strategy engine can respond quickly and implement corresponding adjustment measures. This high flexibility and real-time responsiveness ensure that the website always maintains the best operating state even in a complex and changing network environment, providing users with a high-quality access experience.

[0011] Compared with the prior art, the present invention discloses a web page performance monitoring and analysis method, system, device and storage medium. The present invention has the following beneficial effects: 1. Diversified monitoring indicators: The current system is not limited to detecting traditional indicators such as page loading time and response time, but also includes multi-dimensional performance parameters such as first contact time (FCT), time to expiration rate (TLP), user interaction time, error rate, bounce rate, throughput, etc. By comprehensively considering these indicators, it is possible to more comprehensively evaluate web page performance and ensure the accuracy and comprehensiveness of the monitoring results; 2. Intelligence and precision: The system integrates intelligent benchmark models, anomaly detection algorithm matrices, and machine learning models, enabling the system to automatically analyze processed data, mine performance trends and potential anomalies, and achieve accurate predictions. This intelligence not only improves the accuracy and efficiency of anomaly detection, but also effectively filters false positives, reduces missed positives, and provides clear and reliable anomaly signals for the operation and maintenance team; 3. Real-time and high efficiency: By deploying Apache Kafka cluster and Apache Flink, efficient processing of real-time data streams is achieved, ensuring low latency in data processing. The system can respond to performance changes instantly and quickly capture and process performance data. The present invention reduces the average page loading time by 20%, speeds up the response time by 30%, improves user satisfaction by 25%, and the optimized resource management strategy effectively reduces the server burden and reduces bandwidth costs by 30%; 4. Dynamic optimization and efficient resource utilization: A dynamic optimization strategy engine is designed to dynamically adjust website resources based on real-time monitoring data, such as resource loading priority, image compression algorithm, and cache management strategy, to ensure that the website can maintain optimal performance under different network environments and user behaviors and improve resource utilization efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative work: Figure 1 It is a flow chart of system deployment and data collection and analysis of the present invention; Figure 2 It is a flow chart of web page performance monitoring and optimization of the present invention. DETAILED DESCRIPTION

[0013] Step 1: System deployment and configuration S11 builds a high-availability infrastructure in cloud servers or local data centers, including high-performance servers, SSD storage arrays, and high-speed network switching devices. Ensure that all hardware resources support large-scale concurrent access and data processing requirements; S12 installs the Java runtime environment, database management system (MySQL / PostgreSQL), and necessary middleware services (Apache Hive for data warehouse and Apache Flume for data stream processing); S13 configures an Apache Kafka cluster to process real-time data streams and deploys Apache Flink to achieve low-latency stream data processing. Optimize the configuration of TaskManager and JobManager based on system resources to achieve real-time data stream processing.

[0014] Step 2: Data collection and intelligent analysis S21 collects performance data of the front-end pages of e-commerce websites in real time through front-end monitoring tools, including page loading time, first contact time (FCT), time expiration rate (TLP), user interaction time, error rate, bounce rate, and throughput key performance indicators. Ensure that the data collector does not interfere with user access and that the data is accurate and reliable, and send the collected data in JSON format to the specified Topic of the Apache Kafka cluster; S22 configures the Kafka cluster and sets a Topic to receive performance data from the front end; uses the Kafka Producer API to send the collected performance data to the Kafka Topic; uses the Kafka Consumer API to read the data stream in Apache Flink for real-time processing; In Flink's stream processing tasks, S23 uses TensorFlow.js for data cleaning and preprocessing, including removing invalid data and standardizing data formats; building and training machine learning models (LSTM or other deep learning models) to predict web page performance trends; saving the trained models and loading them in Flink for real-time prediction.

[0015] Step 3: Intelligent scene adaptation and monitoring strategy generation S31 develops a scene intelligent recognition module that uses machine learning or rule engines to automatically identify web application scenarios. The recognition results should accurately reflect the user's actual usage scenarios and performance requirements; S32 dynamically generates performance monitoring strategies and indicators based on the scene recognition results. It customizes personalized monitoring solutions for specific web applications to ensure the pertinence and effectiveness of monitoring. S33 continuously collects user feedback and performance data, and iteratively optimizes monitoring strategies. It uses data analysis results to guide performance optimization work and improve the overall performance of the website and user experience.

[0016] Step 4: Dynamic Optimization Strategy and Resource Management S41 intelligently adjusts the resource configuration of the website based on the performance monitoring data collected in real time, and uses Redis or other memory databases for anomaly aggregation and deduplication to ensure the accuracy and effectiveness of the monitoring data; S42 builds a complete set of automated optimization processes, automatically adjusts website resource allocation according to the decisions made by the dynamic optimization strategy engine, establishes a mechanism to continuously monitor the effect of optimization, and makes necessary adjustments to the optimization strategy based on the actual effect, forming a closed-loop optimization process; S43 designs and implements dynamic resource scheduling and load balancing strategies for different network environments and user access patterns. In the case of high concurrent access, the website can still maintain stable and efficient operation. Through intelligent allocation of server resources and reasonable planning of data transmission paths, it effectively avoids service interruptions or performance degradation caused by sudden increases in traffic.

[0017] Step 5: Performance monitoring system construction S51 Build a web page performance monitoring system that includes a data collection module, an analysis module, a report generation module, an alarm module, and an optimization suggestion module; S52 obtains web page loading data and user interaction data in real time, uses anomaly detection algorithm matrix and models to conduct in-depth analysis of the data, supports setting performance thresholds, and immediately triggers an alarm to notify the corresponding personnel once an anomaly is detected.

[0018] Step 6: Optimize storage media S61 uses SSD to store real-time performance data, improves the rapid response of data processing, and stores historical data in the cloud storage system for long-term preservation and data analysis; S62 regularly backs up key data and performs recovery tests to ensure data validity and recoverability, monitors storage media performance indicators in real time, and promptly identifies and resolves potential problems.

Claims

1. A web page performance monitoring and analysis method, system, device and storage medium, characterized in that: The method comprises the following steps: S1: Build a high-availability infrastructure, deploy high-performance servers, SSD storage arrays, and high-speed network switching devices to ensure that hardware resources support large-scale concurrent access and data processing requirements; S2: Collect performance data of front-end pages in real time, including but not limited to page loading time, first contact time (FCT), time to expiration rate (TLP), user interaction time, error rate, bounce rate, throughput, etc., and use stream processing technology and real-time data analysis algorithms for preliminary cleaning and aggregation operations; S3: Use machine learning models to automatically analyze processed data, mine performance trends and potential anomalies, build intelligent benchmark models, and integrate anomaly detection algorithm matrices such as statistical methods, cluster analysis, and time series pattern recognition; S4: Start the scene intelligent adaptation module to accurately identify web application scenarios, dynamically customize performance monitoring strategies and indicators, and provide personalized monitoring solutions for different web applications; S5: Design a dynamic optimization strategy engine to analyze monitoring data in real time and adjust website resources to ensure that the website maintains optimal performance under different network environments and user behaviors; S6: Build a web performance monitoring system, covering data collection, analysis, report generation, alarm and optimization suggestion modules, forming a closed-loop continuous performance optimization process; S7: Optimize storage media, ensure the integrity and accessibility of performance data, improve overall storage efficiency, regularly back up critical data, and perform recovery tests.

2. The method according to claim 1, characterized in that The S1 further comprises: Deploy an Apache Kafka cluster consisting of multiple nodes on a cloud server or local data center; Each node should have sufficient memory, CPU resources, and disk space to support large-scale concurrent access and data processing requirements; Set the replication factor to 2 or higher to ensure data reliability and high system availability even when a single point of failure occurs; Deploy an Apache Flink cluster, including a master node (JobManager) for coordinating task allocation and state management, and multiple worker nodes (TaskManagers) for performing actual data processing tasks; adjust the configuration parameters of TaskManager and JobManager, JVM heap size, parallelism, etc. according to hardware resources to optimize performance; Configure the Kafka producer API to ensure that it can efficiently receive data from the front-end monitoring tool and publish it to the specified Kafka topic; Set the Flink application as a Kafka consumer, subscribe to relevant topics, start consuming data streams from the front-end monitoring tool, perform preliminary cleaning on the raw data, and remove invalid or erroneous records; Through window functions, data within a period of time is aggregated and calculated, and deep learning models are trained based on historical performance data to determine the normal fluctuation range of performance indicators. The trained models are then integrated into Flink tasks to achieve real-time data analysis and prediction.

3. The method according to claim 1, characterized in that The S2 further includes: Develop a scenario intelligent recognition module to automatically identify web application scenarios using machine learning or rule engines; Dynamically generate performance monitoring strategies and indicators based on the identification results.

4. The method according to claim 1, characterized in that: The S3 further includes: Intelligently adjust website resource configuration based on performance monitoring data collected in real time; Establish automated optimization processes and implement dynamic resource scheduling and load balancing strategies.

5. The method according to claim 1, characterized in that The S4 further comprises: By continuously monitoring website performance, performance bottlenecks and potential risk points can be quickly identified; Adjust relevant parameter configurations in real time based on the latest monitoring results; The dynamic optimization strategy engine can receive and process data streams from the performance monitoring system in real time; Flexibly adjust the optimization strategy based on the feedback information from these data.

6. The method according to any one of claims 1 to 4, characterized in that Also includes: Use SSD to store real-time performance data and store historical data in cloud storage; Regularly perform backup and recovery tests on critical data; Monitor storage media performance indicators in real time to identify and resolve potential problems promptly.

Images