Website page access abnormity monitoring method

Abstract

The invention relates to the technical field of network security, and particularly discloses a website page access abnormity monitoring method, which comprises the following steps of: S101, carrying out custom code embedding on a website page; step S102, obtaining page performance indexes and abnormal data by running the codes of the custom embedded codes; step S103, packaging the page performance index and the abnormal data into an event report; step S104, summarizing the event reports to generate an aggregation report; s105, analyzing the page performance index report by using a machine learning algorithm to generate a performance index analysis report; and S106, analyzing the abnormal data report by using an abnormal detection algorithm so as to generate an abnormal data analysis report. Through the method provided by the invention, the stability, the safety and the user experience of the website can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of network security, and in particular to a method and storage medium for monitoring abnormal access to website pages. Background Art

[0002] Large websites serve as external display windows for organizations / enterprises / institutions, providing users with a large amount of relevant information. The user experience when visiting the website will not only be affected by the response performance of the server, but also by the performance of the browser rendering. Therefore, it is no longer possible to fully optimize and improve the user experience based solely on server-side information. In addition, large websites have a large amount of web page information, which is very cumbersome and complicated to maintain. A large number of different regions, network environments, and terminal devices determine that the user experience of visiting the website is very different, so a quantitative method is needed to collect and summarize the runtime performance and abnormal information of each user terminal, so that website maintenance personnel can be informed of the actual problems encountered by users during access, detect website abnormalities in a timely manner, and improve the user experience when visiting the website.

[0003] In view of the above problems, it is necessary to provide a website page access anomaly monitoring method to collect performance indicators and monitor abnormal data of web pages at the user end for use by website maintenance personnel. Summary of the invention

[0004] The object of the present invention is to provide a method for monitoring abnormal website page access, so as to at least solve one or more of the technical problems mentioned above.

[0005] The purpose of the present invention can be achieved through the following technical solutions:

[0006] In a first aspect, the present invention provides a method for monitoring abnormal access to website pages, comprising:

[0007] Step S101: Customize the website page; wherein the custom embedding types include embedding monitoring scripts, custom event embedding, performance data embedding, abnormal data embedding, and resource loading abnormality monitoring;

[0008] Step S102: Obtaining page performance indicators and abnormal data by running the custom embedded code;

[0009] Step S103: Encapsulating the page performance index and abnormal data into an event report; wherein the event report includes a data set consisting of a group of event parameters;

[0010] Step S104: Summarize the event reports to generate an aggregate report; wherein the aggregate report is divided into a page performance indicator report and an abnormal data report;

[0011] Step S105: using a machine learning algorithm to analyze the page performance indicator report to generate a performance indicator analysis report;

[0012] Step S106: using an anomaly detection algorithm to analyze the abnormal data report to generate an abnormal data analysis report;

[0013] The steps of customizing the embedding code include:

[0014] By locating the tags of the HTML elements of the website page, the position of the monitored element in the page structure is determined;

[0015] Embed javascript event listeners into monitoring elements to encapsulate monitoring logic and data collection as DOM elements;

[0016] Based on the DOM element, the performance index and abnormal data of the monitored element are monitored.

[0017] As a further solution of the present invention: using an anomaly detection algorithm to analyze the abnormal data report to generate an abnormal data analysis report includes:

[0018] Extracting and converting features of the abnormal data report;

[0019] Analyzing the abnormal data report by an anomaly detection algorithm to generate an anomaly detection model;

[0020] Use historical data to train the anomaly detection model and evaluate the accuracy and generalization ability of the model through cross-validation method;

[0021] According to the abnormal data report, the results of the abnormal detection are organized into an abnormal data analysis report using the abnormal detection model; wherein the abnormal data analysis report includes statistical data of the abnormal data, trend charts, and detailed information of the abnormal events.

[0022] In a second aspect, the present invention provides a computer-readable storage medium, on which is stored a computer program for abnormal control of website page access. When the computer program is executed by one or more processors, the method described in any one of the first aspects is implemented.

[0023] Through the website page access anomaly monitoring method provided above, the beneficial effects of the embodiment of the present invention are:

[0024] (1) By embedding code in website pages, performance data and abnormal data of the pages can be collected, which is conducive to data analysis in the background of monitoring tools.

[0025] (2) The method provided by the present invention can effectively monitor and analyze website performance, and timely discover and resolve access anomalies, which is of great significance for improving website stability, security and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The present invention will be further described below in conjunction with the accompanying drawings.

[0027] Figure 1 It is a flowchart of a website page access monitoring method according to some embodiments of the present invention. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] Figure 1 1 is a flow chart of a website page access monitoring method 100 according to some embodiments of the present invention.

[0030] like Figure 1 As shown, in method 100, a website page access monitoring method is exemplarily described through an embodiment, and the specific steps include:

[0031] In step S101, custom coding is performed on the website page; wherein the custom coding types include embedded monitoring scripts, custom event coding, performance data coding, abnormal data coding, and resource loading abnormality monitoring.

[0032] The steps of customizing the embedding code include:

[0033] By locating the tags of the HTML elements of the website page, the position of the monitored element in the page structure is determined;

[0034] Embed javascript event listeners into monitoring elements to encapsulate monitoring logic and data collection as DOM elements;

[0035] Based on the DOM element, the performance index and abnormal data of the monitored element are monitored.

[0036] Embedding in web pages is a process of embedding monitoring code in website pages to collect performance data and abnormal data. In order to help technicians in this field understand the exact meaning of embedding, the following is a specific example description showing how to implement page monitoring through embedding:

[0037] In one embodiment, for a page of an e-commerce website, it is necessary to monitor the performance and abnormality of the homepage. The following are the steps and methods of embedding the code:

[0038] First, choose a suitable monitoring tool, such as Google Analytics, New Relic, Raygun, etc. These tools provide SDK or API. Choosing a suitable front-end monitoring tool framework can easily add monitoring code for embedding and integration into the website.

[0039] Second, embed the monitoring script in the page: embed the monitoring script in the HTML template of the website. For example, using the Google Analytics framework, you can add the following HTML code to the tag:

[0040]

[0041]

[0042] The above code will automatically send page visit data to Google Analytics when a user visits the page.

[0043] Again, for specific user interactions or page behaviors, such as button clicks, form submissions, etc., you can add custom events to collect more detailed data. For example, use JavaScript to add a click event listener:

[0044]

[0045] The above code will send an event to Google Analytics when a user clicks the "Buy" button.

[0046] When you need to embed performance data when monitoring page performance indicators, you can use the browser's Performance API to collect performance data such as page loading time and resource loading time. For example, add a Java script:

[0047]

[0048] The above code will send the page load time after the page has finished loading.

[0049] When embedding abnormal data, you can use the window.onerror event listener to capture JavaScript errors and send them to the monitoring system. For example, add a Java script:

[0050]

[0051] The above code will send an error message when a JavaScript error occurs.

[0052] When embedding code to monitor resource loading exceptions, especially for loading exceptions of resources such as images and videos, you can collect data by listening to error events. For example, embedding JavaScript scripts:

[0053]

[0054] The above code will send corresponding events when the image loads successfully or fails.

[0055] Through the above-mentioned description of the embodiment of the embedded code, it is possible to collect the performance data and abnormal data of the page, and then further analyze the collected data through the background of the monitoring tool, so as to optimize the website performance and user experience according to the analysis results.

[0056] In one embodiment, page performance indicators may include: timestamp, first byte time, first screen time), white screen time, first input delay time, interactive time, maximum content drawing, total blocking time, etc.

[0057] After obtaining the timestamp on the web page, you can obtain relevant performance indicators in the following ways, such as:

[0058] TTFB (Time to First Byte): The number of milliseconds from issuing a page request to receiving the first byte of response data; TTFB = responseEnd-requestStart.

[0059] FCP (First Paint Time): Before the page is loaded, the browser has no content to display and displays a white screen. The first paint time is the time when the page content first appears on the browser interface; FCP = First Paint - start Time.

[0060] FP (white screen time): the time when the browser starts to present the page on the screen. The formula is: FP = loadEventEnd-navigationStart.

[0061] FID (First Input Delay): measures the browser's response speed to the user's first interaction; FID = domInteractive - fetchStart.

[0062] TTI (Time to Interactive): The time it takes for the browser to parse the document object model of a web page and resolve it to an interactive state; TTI = domInteractive - navigationStart.

[0063] In addition, you can directly obtain the following indicators by calling the browser's performance getEntries API through Javascript:

[0064] LCP (Largest Contentful Paint): Measures how long it takes for the largest visible element on the page (a block of text or an image) to become visible.

[0065] TBT (Total Blocking Time): The sum of the main thread blocking time of all long tasks when loading a web page.

[0066] In addition to the indicators listed above, some embodiments also detect information such as page path, page site, page region, page title, event generation date, terminal type (PC, mobile device), etc.

[0067] When a Javascript script on a web page runs and an exception occurs, in one embodiment, the exception will be captured by the exception handling code, and then packaged into a GA4 (Google Analytics V4) event and reported to the Google Analytics V4 platform. The GA4 event is a data set consisting of a set of event parameters, among which the event parameters are the exception message content.

[0068] In one embodiment, through the Performance custom event, the event parameters: ttfb, fcp, fp, lcp, fid, tbt, tti, the unit is unified in ms (milliseconds), and ttfb, fcp, fp, lcp, fid, tbt, tti are metrics (Metrics, that is, variables with values, which can be used for aggregation operations such as totaling, calculating averages, and taking maximum / minimum values), and other page parameters are dimensions (Dimensions, that is, labels, used for event classification).

[0069] For the Script_Error custom event, the event parameters include js_error (JS exception content) and / or other page parameters. Script Error reports dimensions.

[0070] When an exception occurs during the execution of a JS script on a web page, the browser's javascript engine will issue an error exception event, which will be received by the event listener. The exception event can contain the error name, error number, description, error message, file where the error occurred, etc. The listener can package the exception event content and the current page path into a GA4 event and report it to Google Analytics.

[0071] In order to further help those skilled in the art understand the technical idea of ​​the present invention, the principle of adding an event listener and the complete process of performing feature analysis, encoding analysis, rendering page events or encapsulating DOM elements are explained below through a specific example.

[0072] In this embodiment, it is necessary to collect keywords input by users and analyze the search behavior of users based on the input events of the search box in the monitoring website.

[0073] First, you need to analyze the characteristics of the search box in order to accurately find and encapsulate this element on the page.

[0074] Element selector: This is the HTML element that identifies the search box. Usually it is an <input> or <input type＝"text"> Label.

[0075] Class name or ID: A search box may have a specific class name (such as .search-input) or an ID (such as searchBox).

[0076] Parent elements: Understand where the search box is located in the page structure so you can select it more accurately.

[0077] Next, you need to write JavaScript code to encapsulate the DOM element and add event listeners.

[0078] <! -- For example, the HTML code of the search box is as follows -->

[0079] <input type＝"text"id＝"searchBox"placeholder＝"Search...">

[0080] The added JavaScript code is:

[0081]

[0082] Next, in your event listener's callback function, do the following:

[0083] Get user input: Get the keywords entered by the user in the search box through event.target.value.

[0084] Feature analysis: Check if the keyword is empty to avoid sending invalid data.

[0085] Coding analysis: Process keywords, such as removing leading and trailing spaces, to ensure data accuracy.

[0086] Rendering page events: Output keywords in the console to facilitate developers' debugging and analysis.

[0087] Encapsulate DOM elements: Send the processed keywords to a monitoring system (such as Google Analytics) to collect and analyze user search behavior.

[0088] Therefore, the monitoring process of the above events can be divided into:

[0089] Page loading: A user visits a website and the page loads.

[0090] Select the search box element: Select the search box element through the getElementById method.

[0091] Add event listener: Add input event listener to the search box element.

[0092] User input: The user enters keywords in the search box.

[0093] Event triggering: The input event is triggered and the callback function is executed.

[0094] Feature analysis: Check if the keyword is empty.

[0095] Coding analysis: Process keywords, such as removing leading and trailing spaces.

[0096] Rendering page event: output keywords in the console.

[0097] Encapsulate DOM elements: send keywords to the monitoring system.

[0098] Through the above complete process example, technicians in this field can effectively monitor the user's input events in the search box, collect keyword data, and perform further analysis and optimization.

[0099] The principle of custom embedding is explained below.

[0100] Custom coding is a process of embedding monitoring code in website pages to collect performance data and abnormal data. The following are the specific steps and methods to achieve this solution:

[0101] First, locate the HTML element in the page through CSS selectors (such as ID, class name, tag name, etc.). For example, if we want to monitor a search box, we can get a reference to this element through document.getElementById('searchBox') or document.querySelector('.search-input').

[0102] Secondly, use the addEventListener method of JavaScript to add an event listener to the monitored element. For example, add an input event listener to the search box:

[0103]

[0104] The above listener will be triggered when the user enters input and the callback function will be executed.

[0105] Next, you need to encapsulate it into a DOM element. Through encapsulation, you can organize the monitoring logic and data collection code together to form a modular code structure. For example, you can create a function to handle the input event of the search box and collect the required data in it:

[0106]

[0107] The collectData function is responsible for sending data to the server or for further processing.

[0108] Then, use the Performance API provided by the browser to collect page performance indicators, such as page loading time, resource loading time, etc. For example, use performance.now() to obtain a high-precision timestamp for calculating the time of event processing:

[0109] var startTime=performance.now();

[0110] / / Perform some operations

[0111] var endTime=performance.now();

[0112] console.log('Operation time:',endTime-startTime,'milliseconds');

[0113] For abnormal data, JavaScript errors can be captured through the window.onerror event and sent to the monitoring system

[0114] In specific implementations, you can also use objects or Maps to store performance data and exception information. For example, create an object to store the performance data of the search box:

[0115]

[0116] When each input event is triggered, the relevant data in this object is updated and sent to the server if necessary.

[0117] Finally, send the collected data to the backend server through AJAX request, Image Beacon or other methods. For example, use navigator.sendBeacon or create an Image object to send data:

[0118]

[0119] Through the above exemplary description, it is possible to effectively monitor specific elements in a website page, collect performance indicators and abnormal data, thereby ensuring network security and optimizing user experience and page performance.

[0120] Further, the process proceeds to step S102, and the page performance index and abnormal data are obtained by running the custom embedded code.

[0121] In one embodiment, it includes:

[0122] Collect performance indicators, including but not limited to: DNS query time, TCP connection time, request response time, content transmission time, and DOM parsing time.

[0123] Collect exception data, including but not limited to: capturing JavaScript errors and resource loading errors, exception throwing errors, and network request failures.

[0124] In step S102, we obtain page performance indicators and abnormal data by running the custom embedded code. The following describes the process of obtaining and processing these data through an embodiment:

[0125] First, obtain the page performance indicators.

[0126] In one embodiment, the browser's Performance API can be used to collect performance data such as page loading time and resource loading time. For example, the timestamps of each stage of page loading are obtained through window.performance.timing, and indicators such as DNS resolution time, TCP connection time, and first screen loading time are calculated.

[0127] For another example, use the window.performance.getEntries() method to obtain data related to resource loading and asynchronous requests, analyze the loading time of each resource in the page, and optimize the loading order and resource size.

[0128] Secondly, the collection of abnormal data.

[0129] In one embodiment, JavaScript error information can be captured through the window.onerror event listener, including the error message, the script URL where the error occurred, the line number, the column number, and the error object.

[0130] For Vue errors, you can use the Vue.config.errorHandler method to capture them.

[0131] For API request errors, you can handle the errors uniformly in the axios response interceptor.

[0132] After collecting page performance indicators and exception data, the collected performance indicators and exception data need to be processed, such as calculating the average response time, error rate, etc.

[0133] In an implementation scenario, data can be reported to the server using AJAX, Image Beacon, or the navigator.sendBeacon() method. The navigator.sendBeacon() method allows data to be sent asynchronously when the page is unloaded, and is suitable for reporting the last data when the page is closed.

[0134] If the navigator.sendBeacon() method is not compatible, you can use the method of creating an Image object as an alternative and send a GET request by setting img.src. This method does not have cross-domain restrictions.

[0135] During the data reporting process, data reporting is required at key moments such as page loading completion, route switching, page visibility changes, etc., to ensure that the collected data can reflect the user's real experience.

[0136] In one embodiment scenario, considering the problem that the asynchronous AJAX request may not be completed when the page is unloaded, navigator.sendBeacon() or Image Beacon can be used to ensure data reporting.

[0137] The reported data structure and encapsulation method can use objects or maps to store performance data and exception information. For example, create an object to store the performance data of the search box, including input time, processing time, number of errors, etc.

[0138] For the reported data, a data structure containing all necessary information, such as event type, timestamp, duration, error information, etc., can be constructed and then sent to the server through the above reporting method.

[0139] Through the description of the above scheme, those skilled in the art can understand the method provided by the present invention, thereby effectively monitoring the performance and abnormal conditions of website pages, collecting key data, and performing analysis and optimization to improve user experience.

[0140] Further, the process proceeds to step S103, where the page performance indicators and abnormal data are packaged into an event report; wherein the event report includes a data set consisting of a group of event parameters.

[0141] In one embodiment, the event report includes at least one object, which is a data structure in JSON format and contains one or more of the following fields: timestamp, page URL, user agent string, performance indicator, exception information

[0142] The following describes the technical implementation steps for encapsulating page performance indicators and abnormal data into event reports.

[0143] During the data collection phase, the browser's Performance API, such as performance.getEntries(), performance.getEntriesByType(), performance.getEntriesByName(), and other methods are used to collect various performance indicators during the page loading process, including but not limited to DNS query time, TCP connection time, request response time, content transmission time, DOM parsing time, etc.

[0144] During the exception capture phase, operations are performed in the following ways: For example, calling window.onerror and window.addEventListener('error') to capture JS errors and resource loading errors.

[0145] You can also use window.addEventListener('unhandledrejection',function(event){}) to capture Promise errors. And rewrite window.XMLHttpRequest and window.fetch to capture network request failures.

[0146] Furthermore, the collected performance data and exception information are encapsulated into an event report object. This object can be a JSON formatted data structure, including fields such as timestamp, page URL, user agent string, performance indicator, exception information, etc.

[0147] For example, an event report object may contain the following structure:

[0148]

[0149]

[0150] The reporting server strategy can refer to the above embodiment description, which will not be repeated here. If the network is unstable, a local cache and delayed reporting strategy can be adopted. For critical information infrastructure event reporting, detailed rules need to be formulated to clarify the reporting time limit requirements, event types, reporting methods and channels, reporting content, etc., to ensure smooth information sharing.

[0151] The beneficial effects of the above scheme are:

[0152] Real-time monitoring: Ability to monitor website performance and abnormalities in real time, and detect and resolve problems in a timely manner.

[0153] Performance optimization: By analyzing performance data, you can optimize the website and improve the user experience.

[0154] Abnormal warning: Quickly discover and respond to abnormal situations to reduce business losses.

[0155] Data-driven decision making: Based on the collected data, more scientific decisions can be made to optimize website architecture and code.

[0156] Compatibility and fault tolerance: By designing a performance SDK and formulating a reporting strategy, we ensure that the SDK runs stably in different businesses and technology stacks and has a fault-tolerant mechanism.

[0157] Therefore, through the above technical solution, page performance indicators and abnormal data can be effectively encapsulated as event reports to provide support for subsequent data analysis and problem solving.

[0158] Further, the process comes to step S104, where the event reports are summarized to generate an aggregate report, wherein the aggregate report is divided into a page performance index report and an abnormal data report.

[0159] In one embodiment, the aggregating the event reports to generate an aggregated report includes: collecting performance data; wherein the performance data at least includes page loading time and resource loading time; collecting exception data; wherein the exception data at least includes exception data captured by an exception capture mechanism. Cleaning the performance data and / or exception data to remove erroneous or invalid data points. Aggregating data using the distributed architecture of the search and data analysis engine to generate aggregated data; and integrating the aggregated data into a central repository using real-time data aggregation technology to generate a page performance indicator report and an exception data report.

[0160] In one embodiment, GA4 receives the reported events from the terminal in real time, and then aggregates them daily. The aggregated data will be delayed for up to 48 hours. For metrics, data aggregation can obtain the interval maximum value, interval minimum value, interval average value, total, etc.

[0161] Aggregate reports are divided into performance indicator reports and JS (javascript) exception reports. Both can take page path, terminal type, event reporting date, etc. as report dimensions.

[0162] Performance indicator reporting indicators may include ttfb, fcp, fp, lcp, fid, tbt, tti and number of events; JS exception reporting indicators may include exception message content and number of events.

[0163] In one embodiment, the GA4 API can be called to generate a JS anomaly aggregation report by selecting event date, page path (landing page), access area, access terminal type, and JS anomaly as dimensions and the number of events (visits) as a metric value.

[0164] Since GA4's data aggregation will be delayed by up to 48 hours, the aggregation report will obtain the aggregation data from 2 days ago on a daily basis.

[0165] The daily aggregate report data is obtained as multiple records by dimension. Each record contains the values ​​of each indicator under the dimension combination and the corresponding number of events. It can be saved in a relational database, such as a MySQL database to save the aggregate report data records.

[0166] By grouping statistics in different dimensions, maintenance personnel can be presented with website performance indicators and JS anomalies from different angles. Since the page path dimension is included, pages that exceed indicator thresholds or generate JS anomalies can be located, and improvements can be made iteratively.

[0167] In one embodiment, each performance indicator is composed of a large number of data samples collected when the user visits. The difference between each indicator sample and the mean of all the indicator samples is verified by the Quantile-Quantile diagram (normal probability and sample probability quantile diagram) to be consistent with the normal distribution with a mean of 0. From the aggregate report, the mean (μ) and the maximum value (max) in the indicator sample can be obtained respectively. According to the cumulative probability of the normal distribution, it can be assumed that the difference between the maximum value and the mean falls at a point of 3 standard deviations, and the standard deviation can be calculated: δ = (max-μ) / 3, thereby obtaining the normal distribution (0, δ) of the difference between the indicator and the mean.

[0168] According to the probability density formula of normal distribution (0,δ):

[0169] f(x)=(1 / (σ√(2π)))·exp(-(x-μ) 2 / (2σ 2 ))

[0170] The cumulative probability of the normal distribution (0,δ) can be obtained:

[0171] F(x)=∫[-∞,x]f(t)dt

[0172] Taking x=2δ, we can obtain: F(x)=0.9545, that is, 95.45% of the index values ​​are less than μ+2δ.

[0173] Therefore, if the threshold of the selected indicator is μ+2δ (i.e., the mean plus 2 times the standard deviation), the 4.55% samples with the highest indicator values ​​can be obtained.

[0174] Through the examples of the above embodiments, the website maintenance personnel can select a time period, count the pages whose indicators exceed the threshold (μ+2δ), and display them according to the indicator sorting.

[0175] In step S104, the collected page performance indicators and abnormal data need to be summarized so as to generate an aggregate report. The aggregate report is usually divided into a page performance indicator report and an abnormal data report. The technical solution is described below by listing a specific embodiment:

[0176] Through the above description of the embodiments of the present invention, those skilled in the art should understand the implementation of the data collection and preprocessing stages. For example, performance data such as page loading time and resource loading time are collected using the Performance API. Exception data is collected through the window.onerror and Promise exception capture mechanisms. The collected data is cleaned to remove invalid or erroneous data points, etc.

[0177] Furthermore, after collecting event data, in the data aggregation stage, the distributed architecture of Elasticsearch can be used for large-scale data aggregation. Elasticsearch's distributed aggregation can process data in parallel and improve aggregation efficiency.

[0178] Subsequently, the aggregated data is used to generate page performance indicator reports and abnormal data reports. For example, Python's Pandas library can be used to process and analyze data so that the aggregated data can be grouped by different dimensions and calculated and analyzed separately.

[0179] The data is then consolidated into a central repository using real-time data aggregation techniques such as ETL (Extract, Transform, Load) or ELT (Extract, Load, Transform) to enable historical data analysis and trend prediction for report generation.

[0180] Finally, users can query and download reports through the provided data access interface, or use data visualization tools such as Tableau to display aggregated data in charts to improve the readability and intuitiveness of the report.

[0181] In the above data aggregation process, the technical difficulties are explained below:

[0182] In order to efficiently aggregate data when processing large-scale distributed data, distributed search engines such as Elasticsearch can be used to aggregate data to take advantage of their distributed architecture and parallel processing capabilities.

[0183] In order to solve the problem of uneven data quality, the collected data may contain errors or inconsistencies, which will affect the accuracy of the aggregation results. This requires data cleaning and verification in the data preprocessing stage to remove invalid or erroneous data points.

[0184] To address the difficulty of quickly generating aggregated reports with high real-time requirements, real-time data aggregation and analysis can be achieved by using real-time data processing technologies such as Apache Kafka and Apache Storm.

[0185] In order to address the difficulty of protecting data security and privacy during data aggregation, data encryption and anonymization can be used to ensure the security of data during transmission and storage.

[0186] Through the description of the above embodiment, step S104 can be effectively implemented to generate a page performance indicator report and an abnormal data report, thereby providing data support for subsequent performance optimization and abnormality processing.

[0187] Further, the process comes to step S105, using a machine learning algorithm to analyze the page performance indicator report to generate a performance indicator analysis report.

[0188] In one embodiment, a machine learning algorithm is used to analyze the page performance indicator report to generate a performance indicator analysis report, including:

[0189] Preset thresholds for each performance indicator.

[0190] Use data analysis tools or application performance monitoring tools to analyze the performance data in the performance indicator report to obtain pages that exceed the preset threshold.

[0191] The performance data is subjected to pattern recognition via machine learning algorithms to predict potential performance bottlenecks.

[0192] By setting anomaly detection rules, you can alert users of increased page loading time or frequent resource loading failures.

[0193] The analysis results are organized into analysis reports, including statistical data of performance indicators, trend charts, and detailed information on abnormal events.

[0194] In step S105, an in-depth analysis of the page performance index report is involved to identify and locate those website pages with poor performance, that is, those that exceed a preset threshold. The following is a specific description through an embodiment:

[0195] When collecting page performance indicators, you can use the Performance API to collect various performance indicators during the page loading process, such as DNS resolution time, TCP connection time, white screen time, DOM rendering completion time, page onload time and other technical solutions:

[0196] Based on business needs and industry standards, you can set preset thresholds for each performance indicator. For example, for an e-commerce website, you may need to set the page load time threshold to less than 3 seconds.

[0197] Use data analysis tools, such as Elasticsearch or professional APM (application performance monitoring) tools, to analyze the collected performance data and identify pages that exceed preset thresholds. Use machine learning algorithms to identify patterns in performance data and predict potential performance bottlenecks.

[0198] During anomaly detection analysis, anomaly detection rules are set to generate alarms for sudden increases in page loading time or frequent resource loading failures.

[0199] Finally, the analysis results are compiled into reports, including statistical data of performance indicators, trend charts, detailed information of abnormal events, etc. And through data visualization tools such as Grafana or Kibana, the performance data and analysis results are intuitively displayed in the form of charts.

[0200] In the analysis results, when the page performance exceeds the preset threshold, an alarm can be automatically notified via email, SMS or instant messaging.

[0201] According to the analysis results, in one embodiment, targeted performance optimization suggestions can also be provided, such as resource compression, code splitting, lazy loading, etc.

[0202] By monitoring the page performance indicators through the above scheme of the present invention, the page loading speed and user experience can be significantly improved by timely discovering and solving performance problems. And performance bottlenecks can be quickly located to reduce user complaints and business losses. Since it is a data-driven decision, that is, a more scientific analysis based on actual performance data, it can provide developers with accurate information so as to optimize the website architecture and code in a targeted manner.

[0203] The solution of the present invention can also continuously monitor performance and establish a continuous performance monitoring mechanism to ensure continuous and stable website performance.

[0204] In terms of resource optimization, performance analysis is used to optimize resource usage, such as reducing unnecessary resource loading and improving the utilization of server and network resources.

[0205] Furthermore, excellent website performance can enhance users’ trust and satisfaction with the brand and strengthen the brand image.

[0206] Through the above-mentioned scheme of the present invention, page performance indicators can be effectively analyzed, website pages exceeding preset thresholds can be located, and corresponding optimization measures can be taken to ensure continuous optimization and improvement of website security and performance.

[0207] Further, the process proceeds to step S106, where an abnormal data analysis report is generated by analyzing the abnormal data report using an abnormality detection algorithm.

[0208] In one embodiment, using an anomaly detection algorithm to analyze the abnormal data report to generate an abnormal data analysis report includes:

[0209] Feature extraction and conversion are performed on the abnormal data report.

[0210] The abnormal data report is analyzed by an anomaly detection algorithm to generate an anomaly detection model.

[0211] The anomaly detection model is trained using historical data and the accuracy and generalization ability of the model are evaluated through cross-validation.

[0212] According to the abnormal data report, the results of the abnormal detection are organized into an abnormal data analysis report using the abnormal detection model; wherein the abnormal data analysis report includes statistical data of the abnormal data, trend charts, and detailed information of the abnormal events.

[0213] The following describes the process of anomaly detection in detail through an example:

[0214] First, in the preprocessing stage, similar to the collection of the above performance indicators, it is also necessary to clean abnormal data and remove invalid or erroneous data points to ensure the accuracy of subsequent analysis.

[0215] Subsequently, the data is analyzed to identify abnormal patterns through various anomaly detection algorithms such as Isolation Forest, Local Outlier Factor (LOF), One-Class SVM, etc. These algorithms are able to handle data sets of different types and distributions and identify anomalies that do not conform to the normal data distribution.

[0216] In order to improve the effectiveness of anomaly detection algorithms, it is necessary to extract and transform data features. For example, PCA can be used for dimensionality reduction to reduce the dimensions of the data while retaining the most important features for anomaly detection.

[0217] Then, the anomaly detection model is trained by using historical data, and the accuracy and generalization ability of the model are evaluated through methods such as cross-validation.

[0218] Finally, the trained anomaly detection model is used to output the anomaly detection results, and the anomaly detection results are organized into a report, including statistics of abnormal data, trend charts, detailed information of abnormal events, etc. Then, data visualization tools such as Grafana or Kibana are used to intuitively display abnormal data and analysis results in the form of charts.

[0219] When abnormal data is detected, relevant personnel are automatically notified via email, SMS or instant messaging.

[0220] In one embodiment, the anomaly detection algorithm improves the accuracy of detection by combining unsupervised learning with supervised learning and using a deep learning model; including:

[0221] Step S901: using the K-means++ algorithm combined with the Elkan algorithm to select the initial cluster center to improve the stability and accuracy of clustering; the steps include:

[0222] 1) Randomly select the first centroid from the dataset;

[0223] 2) For each data point xx, calculate its shortest distance D(x) to all selected centroids.

[0224] 3) Select a new data point as the next centroid, and the probability of selection is proportional to D(x), that is, the probability

[0225] 4) Repeat steps 2) and 3) until K centroids are selected.

[0226] The above selection strategy ensures the dispersion between centroids, thereby improving the clustering effect and reducing the risk of local optimal solutions.

[0227] Step S902: Using the Autoencoder model to learn the representation of normal web page access, or using the deep learning model of the autoencoder to learn the features of the normal access pattern, so as to reconstruct the error to detect anomalies; wherein the convolutional Autoencoder model is used to process image data or structured data; the loss function formula of the Autoencoder is as follows:

[0228]

[0229] Among them, θ is the network parameter, x i is the input data, is the reconstructed data, ε i is the error term;

[0230] Step S903: Use the recurrent neural network with attention mechanism to predict future access patterns. The prediction formula is expressed as:

[0231]

[0232] in, is the predicted value at time point t, y t-1, y t-2, ...,y t-n is the observed value at the previous time point, σ is the model parameter;

[0233] when With y t If the difference exceeds the preset threshold, it is detected as abnormal.

[0234] In some embodiments, targeted performance optimization suggestions, such as code optimization, resource adjustment, etc., can also be provided based on the results of the abnormal analysis. For example, according to the definition of different super-threshold indicators, the corresponding pages can be adjusted, such as: page layout, label setting, page-related asset adjustment, preloading mechanism, etc. Since those skilled in the art can choose to use the above adjustment methods according to actual results, the present invention will not elaborate on them in detail. After the page is improved and republished, it can return to the iterative process of the solution of the present invention to form an iterative closed loop.

[0235] Through the above-mentioned abnormal data detection of the present invention, the accuracy and reliability of data can be improved by identifying and processing abnormal data. Abnormal patterns in data can be discovered in time to prevent possible business risks and security threats. And based on the abnormal data analysis report, data support is provided for business decisions to improve the scientificity and effectiveness of decisions. Through abnormal analysis, system performance bottlenecks can be identified, optimized, and the overall performance of the system can be improved. Further rapid response and processing of abnormal situations can reduce user dissatisfaction and complaints and improve user experience. In addition, through the automated abnormal detection and alarm mechanism of the present invention, manual intervention is reduced and operation and maintenance costs are reduced.

[0236] Through the implementation of the above abnormal data detection, abnormal data can be effectively analyzed and a detailed abnormal data analysis report can be generated to provide data support for business decision-making and system optimization.

[0237] In summary, through the above-provided website page access anomaly monitoring method, by using custom embedded code in the website page, the performance data and anomaly data of the page can be collected, which is conducive to the background data analysis of the monitoring tool. In addition, the method provided by the present invention can effectively monitor and analyze website performance, timely discover and solve access anomalies, and effectively improve website stability, security and user experience.

[0238] This solution uses a method of embedding codes on the user side to collect performance and exception data. It can obtain various indicator data from redirection, caching, DNS resolution, initiating HTTP requests, receiving HTTP responses, and page rendering during the entire page loading cycle. The data obtained is direct data. Compared with the server-side performance monitoring method that monitors the HTTP server request / response time, which can only obtain performance data in the data transmission stage but cannot obtain performance indicators in the page rendering process, it is more comprehensive and has a finer monitoring granularity, which is conducive to web page maintenance personnel to improve the website more targeted.

[0239] Based on the above, the present invention also discloses a computer-readable storage medium, which stores a computer program for monitoring abnormal website page access. When the computer-readable program is executed by one or more processors, the method described in the above multiple embodiments or implementation methods is implemented.

[0240] In some implementation scenarios, the computer-readable storage medium may be any suitable magnetic storage medium or magneto-optical storage medium. Any such computer storage medium may be part of a device or apparatus, or may be accessible or connectable to a device or apparatus. Any application or module described in the present invention may be implemented using computer-readable / executable instructions that may be stored or otherwise maintained by such a computer-readable medium.

[0241] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A method for monitoring abnormal access to website pages, characterized in that: The method comprises the following steps: Step S101: Customize the website page; wherein the custom embedding types include embedding monitoring scripts, custom event embedding, performance data embedding, abnormal data embedding, and resource loading abnormality monitoring; Step S102: Obtaining page performance indicators and abnormal data by running the custom embedded code; Step S103: Encapsulating the page performance index and abnormal data into an event report; wherein the event report includes a data set consisting of a group of event parameters; Step S104: Summarize the event reports to generate an aggregate report; wherein the aggregate report is divided into a page performance indicator report and an abnormal data report; Step S105: using a machine learning algorithm to analyze the page performance indicator report to generate a performance indicator analysis report; Step S106: using an anomaly detection algorithm to analyze the abnormal data report to generate an abnormal data analysis report; The steps of customizing the embedding code include: By locating the tags of the HTML elements of the website page, the position of the monitored element in the page structure is determined; Embed javascript event listeners into monitoring elements to encapsulate monitoring logic and data collection as DOM elements; Based on the DOM element, the performance index and abnormal data of the monitored element are monitored.

2. The method according to claim 1, characterized in that The page performance indicators include: timestamp, first byte time, first screen time, white screen time, first input delay time, interactive time, maximum content drawing, and total blocking time.

3. The method according to claim 1, characterized in that The step of obtaining page performance indicators and abnormal data by running the custom embedded code includes: Collect performance indicators, including but not limited to: DNS query time, TCP connection time, request response time, content transmission time, DOM parsing time; Collect exception data, including but not limited to: capturing JavaScript errors and resource loading errors, exception throwing errors, and network request failures.

4. The method according to claim 1, characterized in that: The event report includes at least one object, which is a data structure in JSON format and contains one or more of the following fields: timestamp, page URL, user agent string, performance indicator, and exception information.

5. The method according to claim 1, characterized in that The event reports are aggregated to generate an aggregate report, including: Collect performance data; wherein the performance data at least includes page loading time and resource loading time; Collecting abnormal data; wherein the abnormal data at least includes abnormal data captured by an abnormal capture mechanism; Cleaning the performance data and / or abnormal data to remove erroneous or invalid data points; Aggregate data using the distributed architecture of search and data analysis engines to generate aggregated data; and The aggregated data is integrated into a central repository using real-time data aggregation technology to generate page performance indicator reports and exception data reports.

6. The method according to claim 1, characterized in that The using of a machine learning algorithm to analyze the page performance indicator report to generate a performance indicator analysis report includes: Preset thresholds for each performance indicator; Use data analysis tools or application performance monitoring tools to analyze the performance data in the performance indicator report to obtain pages that exceed the preset threshold; performing pattern recognition on the performance data through a machine learning algorithm to predict potential performance bottlenecks; By setting anomaly detection rules, alarms can be generated for increases in page loading time or frequent resource loading failures. The analysis results are organized into analysis reports, including statistical data of performance indicators, trend charts, and detailed information on abnormal events.

7. The method according to claim 1, characterized in that The using of anomaly detection algorithm to analyze the abnormal data report to generate an abnormal data analysis report includes: Extracting and converting features of the abnormal data report; Analyzing the abnormal data report by an anomaly detection algorithm to generate an anomaly detection model; Use historical data to train the anomaly detection model and evaluate the accuracy and generalization ability of the model through cross-validation method; According to the abnormal data report, the results of the abnormal detection are organized into an abnormal data analysis report using the abnormal detection model; wherein the abnormal data analysis report includes statistical data of the abnormal data, trend charts, and detailed information of the abnormal events.

8. The method according to claim 6 or 7, characterized in that: Also includes: According to the analysis results, the performance of the method is optimized, including resource compression, code splitting, and lazy loading.

9. The method according to claim 1, characterized in that: The anomaly detection algorithm combines unsupervised learning with supervised learning and uses a deep learning model to improve the accuracy of detection; including: Step S901: using the K-means++ algorithm combined with the Elkan algorithm to select the initial cluster center to improve the stability and accuracy of clustering, the steps include: 1) Randomly select the first centroid from the dataset; 2) For each data point x, calculate its shortest distance D(x) to all selected centroids; 3) Select a new data point as the next centroid, and the probability of selection is proportional to D(x), that is, the probability 4) Repeat steps 2) and 3) until K centroids are selected; Step S902: Using the Autoencoder model to learn the representation of normal web page access, or using the deep learning model of the autoencoder to learn the features of normal access patterns, so as to reconstruct the error to detect anomalies; wherein the convolutional Autoencoder model is used to process image data or structured data; the loss function formula of the model is as follows: Among them, θ is the network parameter, x i is the input data, is the reconstructed data, ε i is the error term; Step S903: Use the recurrent neural network with attention mechanism to predict future access patterns. The prediction formula is expressed as: in, is the predicted value at time point t, y t-1, y t-2, ...,y t-n is the observed value at the previous time point, σ is the model parameter; when With y t If the difference exceeds the preset threshold, it is detected as abnormal.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program for monitoring abnormal website page access. When the computer program is executed by one or more processors, the method according to any one of claims 1 to 9 is implemented.

Images