A web data monitoring method, system, electronic device and storage medium

Abstract

This application relates to a web data monitoring method, system, electronic device, and storage medium. The method includes: embedding a web front-end SDK to obtain user status, front-end application running status, and basic device data, and reporting the data to NIGNX; collecting log data reported by the SDK to NIGNX, parsing and filtering the log data for deduplication according to preset rules, and storing the deduplicated data in a Redis database; periodically extracting the filtered and deduplicated data, performing secondary deduplication, merging similar data, and statistical processing on the extracted data, and entering the processed data into a MySQL database; extracting specified data from the MySQL database according to actual user needs, returning it to the web front-end, performing health monitoring on the web front-end data, and displaying the current running status of the web front-end through a chart component. This solves the problems of difficulty in obtaining website user preferences and weak data processing capabilities in related technologies.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a web data monitoring method, system, electronic device, and storage medium. Background Technology

[0002] With the rapid development of computer technology and the widespread application of network technology, most websites today are trending towards desktop applications, offering increasingly diverse functions. Therefore, website stability and security are becoming increasingly important. However, the drawbacks of web front-end development, such as difficulty in viewing website logs, understanding user preferences, and monitoring website stability, are becoming increasingly apparent.

[0003] To address the aforementioned shortcomings of web front-ends, many solutions have been proposed in related technologies. However, these solutions generally suffer from problems such as weak data processing capabilities, low concurrency, complex data tracking, and difficulty in gaining user approval. Summary of the Invention

[0004] This application provides a web data monitoring method, system, electronic device, and storage medium to at least solve the problems in related technologies, such as difficulty in obtaining website user preferences and weak data processing capabilities.

[0005] In a first aspect, embodiments of this application provide a web data monitoring method, the method comprising:

[0006] The web front-end embedding SDK is used to obtain user status, front-end application running status, and basic device data, and then report the data to nginx.

[0007] Collect log data reported by the SDK to Nginx, parse and filter the log data according to preset rules, and store the deduplicated data in the Redis database;

[0008] Regularly extract the filtered and deduplicated data, perform secondary deduplication, merge similar data, and statistical processing on the extracted data, and then enter the processed data into the MySQL database;

[0009] Based on the user's actual needs, the specified data is extracted from the MySQL database and returned to the web frontend. The web frontend data is then monitored for health, and the current running status of the web frontend is displayed through a chart component.

[0010] In some embodiments, obtaining user status, front-end application running status, and basic device data through the SDK includes:

[0011] Set up user click events, obtain user profiles and information through the user click events, and build a user behavior model;

[0012] Determine the user update frequency by the SDK loading time;

[0013] The SDK reports the number of online users, as well as the real-time data traffic and real-time system resource consumption of the web front-end.

[0014] By listening to browser events, global errors are intercepted, the unhandled rejection error type and the error type are obtained, and then reported to Nignx for data processing.

[0015] The SDK reports webpage performance, HTTP success rate and repetition rate, stress test values, and front-end security vulnerability check data.

[0016] In some embodiments, obtaining user profiles and information through the user click events and constructing user behavior models includes:

[0017] The user's activity status is obtained through the user click event. If no event occurs within a preset time, it is considered that the user has left. The user's dwell time is accumulated from the next event.

[0018] The system also obtains user click count and URL information through the user click events.

[0019] In some embodiments, health monitoring of web front-end data includes:

[0020] Website error monitoring and data display are performed based on the error types in the processed log data.

[0021] A comprehensive health assessment of the web frontend is conducted by using error monitoring, webpage performance, success and repetition rates of HTTP interfaces, stress test values, and frontend security vulnerability check data.

[0022] In some embodiments, the acquisition of webpage performance, HTTP success rate and repetition rate, stress test values, and front-end security vulnerability check data includes:

[0023] Use the `performance` function to obtain various performance metrics of a webpage;

[0024] Use an SDK to intercept and analyze the success rate and repetition rate of HTTP interfaces.

[0025] Use third-party tools to conduct incremental stress tests on the website, and determine the website's stress test value by setting up incremental scoring rules.

[0026] In some embodiments, after collecting the log data reported by the SDK to Nignx, the method further includes:

[0027] If the data reaches the preset outlier limit, an alarm will be triggered via the set contact method to notify relevant personnel for inspection.

[0028] In some embodiments, after performing health monitoring on web front-end data and displaying the current running status of the web front-end through a chart component, the method includes:

[0029] Based on the monitored web front-end health index and operational status, corresponding analysis and detection suggestions are provided and feedback is given to relevant personnel.

[0030] Secondly, embodiments of this application provide a web data monitoring system, the system comprising:

[0031] The acquisition and reporting module is used to embed SDKs in the web front-end, acquire user status, front-end application running status, and basic device data through the SDK, and report the data to nginx.

[0032] The primary processing module is used to collect log data reported by the SDK to Nginx, parse and filter the log data according to preset rules, and store the deduplicated data in the Redis database.

[0033] The secondary processing module is used to periodically extract the filtered and deduplicated data, perform secondary deduplication, merge similar data, and perform statistical processing on the extracted data, and then enter the processed data into the MySQL database.

[0034] The monitoring and display module is used to extract specified data from the MySQL database according to the user's actual needs, return it to the web front-end, perform health monitoring on the web front-end data, and display the current running status of the web front-end through chart components.

[0035] Thirdly, embodiments of this application provide an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the web data monitoring method as described in the first aspect above.

[0036] Fourthly, embodiments of this application provide a storage medium storing a computer program that, when executed by a processor, implements the web data monitoring method as described in the first aspect above.

[0037] Compared to related technologies, the web data monitoring method provided in this application embeds an SDK in the web front-end, obtains user status, front-end application running status, and basic device data through the SDK, and reports the data to NIGNX; collects log data reported by the SDK to NIGNX, parses and filters the log data for deduplication according to preset rules, and stores the deduplicated data in a Redis database; periodically extracts the filtered and deduplicated data, performs secondary deduplication, merges similar data, and performs statistical processing on the extracted data, and enters the processed data into a MySQL database; extracts specified data from the MySQL database according to the user's actual needs, returns it to the web front-end, performs health monitoring on the web front-end data, and displays the current running status of the web front-end through a chart component.

[0038] This application utilizes end-to-end performance and user behavior monitoring to display monitoring data through charts and trend analysis of various metrics. This helps website developers and administrators promptly identify occasional user issues, uncover potential problems, pinpoint user concerns, and improve user engagement and retention. Furthermore, this application provides timely alerts and task allocation for detected anomalies on the website, offering the operations platform rich attribution capabilities, including but not limited to stack trace analysis, scheduling analysis, dimensional analysis, event tracking analysis, and single-point log queries. It addresses the difficulties in obtaining website user preferences and the limited data processing capabilities found in related technologies. Attached Figure Description

[0039] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0040] Figure 1 This is a flowchart of a web data monitoring method according to an embodiment of this application;

[0041] Figure 2 This is a schematic diagram of the data processing flow of the web data monitoring method according to an embodiment of this application;

[0042] Figure 3 This is a structural block diagram of a web data monitoring system according to an embodiment of this application;

[0043] Figure 4 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application. Furthermore, it is understood that although the efforts made in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, modifications to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0045] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0046] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application means two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The terms “first,” “second,” “third,” etc., used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0047] This embodiment provides a web data monitoring method. Figure 1This is a flowchart of a web data monitoring method according to an embodiment of this application, such as... Figure 1 As shown, the process includes the following steps:

[0048] Step S101: Install SDK in the web front-end, obtain user status, front-end application running status and basic device data through SDK, and report the data to nginx.

[0049] Figure 2 This is a schematic diagram of the data processing flow of the web data monitoring method according to an embodiment of this application, such as... Figure 2 As shown, firstly, JavaScript is used to embed SDKs in the web frontend to register global browser events, listeners, and events such as error, popstate, click, and keydown. The SDK also parses and calls functions related to window.performance, window.location, and navigator.userAgent. The SDK then obtains user status, the running status of the frontend application, and basic device data, and reports this data to nginx. Specifically, this reporting method involves accessing nginx images on the web server, using the obtained data as parameters for the images, and recording each access as a log entry.

[0050] Preferably, in this embodiment, obtaining user status, front-end application running status, and basic device data through the SDK specifically includes:

[0051] 1. Every 5 seconds, user profiles and information are obtained through user click events to build a user behavior model. These profiles and information include: user dwell time, number of clicks, application update frequency, user location, device information, and application entry points. Specifically:

[0052] a. User Dwell Time and Clicks. User activity is assessed through click events. If no event occurs within a preset time, the user is considered to have left. Dwell time is accumulated from the next event. Additionally, click events also provide click count data. Click count represents user engagement, while dwell time represents user preferences. Both are linked to specific pages. The statistical data shows the time users spend on a particular page and the number of clicks during each time period, providing a clear picture of user behavior and preferences. This metric helps operators infer users' true needs.

[0053] b. User application update cycle. The user update frequency is determined by the SDK loading time. For example, the time it takes for each SDK load is the user's most recent update cycle;

[0054] c. User Region. When the SDK reports data to the Nginx server, the HTTP request protocol includes the IP address, which can be used to determine the user's region.

[0055] d. User device information. The SDK can obtain user device information, such as browser and operating system information from window.navigator.userAgent, and screen size information from document.documentElement;

[0056] e. Application entry point, i.e., URL information. When an application's URL is opened online, the SDK loads and retrieves the URL information, such as window.location.href. This application entry point is crucial for operators in attracting users and implementing points-based promotions and rebates. It also helps operators understand user behavior habits.

[0057] The data obtained above can be extracted by operations personnel to build user behavior models, thereby analyzing website user preferences and helping website developers and managers to promptly identify occasional problems during user use, uncover potential issues, pinpoint user concerns, and improve user stickiness and retention.

[0058] 2. Every 5 seconds, the SDK reports the number of online users, as well as the real-time website traffic and system resource consumption data from the web frontend to the nginx server. Specifically:

[0059] a. Number of online users. Since each user terminal has a unique UUID (fingerprintjs()), device deduplication can be performed using the open-source library fingerprintjs. Every 5 seconds, a request with the same UUID is retrieved to obtain a unique user. This allows us to obtain the total number of online users during that period.

[0060] b. Website Real-Time Data Traffic. By intercepting XMLHttpRequest and fetching to obtain the Content-Length field value of the HTTP protocol header, and incrementing it every 5 seconds, the number of bytes of data sent / received by the website application per unit time is obtained. By statistically analyzing the time consumed by each request, the average cost per interface is calculated: timeCost = timeStart - Date.now() ms. This determines the website's real-time data traffic.

[0061] c. Real-time system resource consumption. The number of frames per second (FPS) rendered by the front-end page is calculated using the `window.requestAnimationFrame` browser rendering hook function. It should be noted that FPS is a key indicator of page smoothness. The resource consumption of the front-end application in the browser is calculated using `window.performance.memory`. Memory usage is calculated using the formula: `usedJSHeapSize / jsHeapSizeLimit*100%`.

[0062] 3. By listening to browser events, intercept global errors, obtain the unhandledrejection error type and the error type, and use a custom algorithm to determine the error type: JavaScript runtime error, resource loading error, asynchronous interface call error, and report it to Nginx for data processing.

[0063] 4. Report webpage performance, HTTP success rate and repetition rate, stress test values, and front-end security vulnerability check data through the SDK. Specifically:

[0064] a. Obtain various performance metrics of the webpage using the `performance` function. These metrics include: DNS lookup time: `domainLookupEnd-domainLookupStart`, TCP connection time: `connectEnd-connectStart`, SSL secure connection time: `connectEnd-secureConnectionStart`, request time: `responseEnd-responseStart`, DOM tree parsing time: `domComplete-domInteractive`, initial rendering time / white screen time: `responseStart-navigationStart`, DOM ready time: `domContentLoadedEventEnd-navigationStart`, and onload time (total download time): `loadEventEnd-navigationStart`.

[0065] b. Use the SDK to intercept and statistically analyze the success rate and repetition rate of HTTP requests. Specifically, it retrieves HTTP requests; if the HTTP request contains the same [url] + [method] + [parameters] + [body], it's considered a duplicate HTTP call. The SDK intercepts XMLHttpRequest and fetch to obtain the HTTP request parameters, stores the request in a global variable, and uses a system timer to check for duplicate requests every 300 milliseconds. Interfaces without requests after 300 milliseconds are removed from the global variable. Furthermore, the SDK intercepts XMLHttpRequest and fetch to obtain the status value. The status code is used to calculate the HTTP success rate; for example, a status code of 2xx indicates a successful HTTP call.

[0066] c. Conduct incremental stress testing on the website using third-party tools, and determine the website's stress test value based on preset incremental scoring rules. In this implementation, the valid incremental statistical value is: an incremental score (latency rate * success rate) greater than 50 points;

[0067] d. Other front-end security vulnerability checks include: checking whether cookies are HttpOnly; encoding input and URL parameters; HTML encoding of dynamically output JavaScript content to the page; ensuring that locally stored sensitive information such as passwords, phone numbers, and localStorage data does not exceed 1MB; sandbox security attributes; using HTTPS protocol and HSTS (HTTP Strict Transport Security); and using the browser's SRI (Subresource Integrity) function for static CDN resources.

[0068] Step S102: Collect log data reported by SDK to Nginx, parse and filter the log data according to preset rules, and store the deduplicated data in the Redis database.

[0069] Since much of the data acquired in step S101 is real-time data, this embodiment uses a Lua secondary development nginx plugin to achieve 5-second real-time data collection, thereby enabling real-time collection of log data reported by the SDK to nginx. Each set of nginx data is stored as a log on the server or under nginx_log. Each data entry includes the web frontend project ID, error type, error content hash, and various metric monitoring data. The collected data is parsed and aggregated according to custom rules. Specifically, data with the same project and the same hash within the same collection cycle are aggregated into one data entry, and the number of repetitions is recorded in the variable `count` for filtering and deduplication. Data that does not conform to the system protocol is considered invalid and discarded. It should be noted that different types of information use different protocol codes.

[0070] The above steps complete the initial screening and deduplication. The deduplicated data will be stored in a Redis database, such as... Figure 2 As shown, the system awaits further processing from data processing process B. It's important to note that Redis cached data expires periodically to release space.

[0071] Step S103: Periodically extract the filtered and deduplicated data, perform secondary deduplication, merge similar data, and statistical processing on the extracted data, and then enter the processed data into the MySQL database.

[0072] like Figure 2 As shown, data processing process A retrieves initial deduplication data from Redis every hour. The extracted data undergoes secondary deduplication, merging of similar data, and statistical processing. Preferably, data is statistically summarized using daily, monthly, and yearly time dimensions, such as page views (PV), unique visitors (UV), number of users, error rate, and page performance. The processed data—including terminal device UUID, operating system, browser version, and screen size—is stored independently and linked to specific events. These specific events are stored in separate tables. This ensures that data generated by the same user shares a single "basic device information" record, and the same event uses the same database record, significantly saving database space and greatly improving statistical efficiency and query speed.

[0073] The processed data will be entered into the MySQL database for use by subsequent data processing C.

[0074] Step S104: Extract specified data from the MySQL database according to the user's actual needs, return it to the web front-end, perform health monitoring on the web front-end data, and display the current running status of the web front-end through a chart component.

[0075] Based on actual user needs, specified data is extracted from the MySQL database to provide web API services to the front-end page. This handles the constraints in the front-end page's AJAX requests, such as the time period of the event, user, data type, page number, project ID, and user ID. The data is then returned to the web front-end for health monitoring, and the current running status of the web front-end is displayed using chart components. Figure 2 As shown, it can display user preferences, user heatmaps, user growth trends, real-time application status, and real-time error queries. Ultimately, it provides operations personnel with intuitive data analysis, enabling better monitoring of user websites and the provision of better services to users.

[0076] Table 1 is a web front-end data health scoring rule table according to an embodiment of this application. Preferably, the health monitoring of web front-end data includes: monitoring and displaying website errors based on the error types in the processed log data; and conducting a comprehensive health assessment of the web front-end through error monitoring, webpage performance, success rate and repetition rate of HTTP interfaces, stress test values, and front-end security vulnerability check data. Specifically, in conducting the comprehensive health assessment, each indicator is first scored, and then all indicators are weighted according to the custom rules in Table 1 to obtain the comprehensive health assessment, thereby enabling the analysis and monitoring of the health status of the data. For example, the health score for error monitoring is: (number of resource errors / total number of pages * 1.2 + (number of JS errors + number of asynchronous errors) / number of events * 1.35) * 100; the health score for webpage performance is: (DNS + TCP + SSL + request * 1.25 + dom * 1.5 + domready * 1.4 + onload) / 1500 * 100%; the health score for interface success rate and duplication rate is: (1 - failure rate * 0.7 - duplication rate * 0.3) * 100; the health score for stress test values ​​is: (maximum process value / 50 + maximum stress test duration / 10) * 100; the health score for front-end security vulnerability check data is: (each score / 90) * 100.

[0077] Table 1

[0078]

[0079]

[0080] Through steps S101 to S104 above, this embodiment monitors the entire performance and user behavior chain, displaying the monitored data in charts and various indicator trend changes. This helps website developers and managers promptly identify occasional problems during user interaction, uncover potential issues, pinpoint user concerns, and improve user stickiness and retention. Furthermore, this application provides timely alerts and assigns processing for anomalies detected on the website, offering the operation platform rich attribution capabilities, including but not limited to stack trace analysis, scheduling analysis, dimensional analysis, event tracking analysis, and single-point log querying. This solves the problems of difficulty in obtaining website user preferences and weak data processing capabilities in related technologies.

[0081] In some embodiments, after collecting log data reported by the SDK to Nginx, if the data reaches a preset anomaly threshold, an alarm is triggered via a predefined contact method to notify relevant personnel for troubleshooting. Examples of such anomalies include unexpected crashes after online application website upgrades; severe browser incompatibility issues resulting in numerous blank screens; DDoS attacks causing a surge in traffic; or a sudden increase in errors on the online application website.

[0082] In some embodiments, after performing health monitoring on web front-end data and displaying the current operating status of the web front-end through chart components, the system provides operators with corresponding analysis and detection suggestions based on the monitored web front-end health index and operating status, thereby improving maintenance efficiency.

[0083] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0084] This embodiment also provides a web data monitoring system for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0085] Figure 3 This is a structural block diagram of a web data monitoring system according to an embodiment of this application, such as... Figure 3 As shown, the system includes an acquisition and reporting module 31, a primary processing module 32, a secondary processing module 33, and a monitoring and display module 34.

[0086] The acquisition and reporting module 31 is used to embed the SDK in the web front-end, obtain user status, front-end application running status, and basic device data through the SDK, and report the data to nginx; the primary processing module 32 is used to collect log data reported by the SDK to nginx, parse and filter the log data according to preset rules, and store the deduplicated data in the Redis database; the secondary processing module 33 is used to periodically extract the filtered and deduplicated data, perform secondary deduplication, merge similar data, and perform statistical processing on the extracted data, and enter the processed data into the MySQL database; the monitoring and display module 34 is used to extract specified data from the MySQL database according to the actual needs of users, return it to the web front-end, perform health monitoring on the web front-end data, and display the current running status of the web front-end through chart components.

[0087] Through the aforementioned system, this embodiment monitors the entire performance and user behavior chain, displaying the monitored data in charts and trend changes of various indicators. This helps website developers and administrators promptly identify occasional problems during user interaction, uncover potential issues, pinpoint user concerns, and improve user stickiness and retention. Furthermore, this application provides timely alerts and assigns processing for anomalies detected on the website, offering the operation platform rich attribution capabilities, including but not limited to stack trace analysis, scheduling analysis, dimensional analysis, event tracking analysis, and single-point log querying. This solves the problems of difficulty in obtaining website user preferences and weak data processing capabilities found in related technologies.

[0088] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0089] Furthermore, it should be noted that the aforementioned modules can be either functional modules or program modules, and can be implemented through software or hardware. For modules implemented in hardware, these modules can reside in the same processor; alternatively, they can be located in different processors in any combination.

[0090] This embodiment also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0091] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0092] Furthermore, in conjunction with the web data monitoring methods in the above embodiments, this application embodiment can provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the web data monitoring methods in the above embodiments.

[0093] In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, memory, a network interface, a display screen, and an input device connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, it implements a web data monitoring method. The display screen may be a liquid crystal display (LCD) or an e-ink display. The input device may be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0094] In one embodiment, Figure 4 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application, such as... Figure 4 As shown, an electronic device is provided, which can be a server, and its internal structure diagram can be as follows. Figure 4 As shown, the electronic device includes a processor, a network interface, internal memory, and non-volatile memory connected via an internal bus. The non-volatile memory stores an operating system, computer programs, and a database. The processor provides computing and control capabilities, the network interface communicates with external terminals via a network connection, the internal memory provides an environment for the operation of the operating system and computer programs, the computer programs are executed by the processor to implement a web data monitoring method, and the database stores data.

[0095] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0096] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0097] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0098] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A web data monitoring method, characterized in that, The method includes: A web front-end tracking SDK is used to obtain user status, front-end application running status, and basic device data, and this data is reported to NIGNX, including: The SDK reports webpage performance, HTTP success rate and duplication rate, stress test values, and front-end security vulnerability check data, including: obtaining various performance metrics of the webpage through the `performance` function; intercepting and statistically analyzing the success rate and duplication rate of HTTP requests using the SDK, identifying duplicate HTTP requests if they share the same URL, method, parameters, and body; intercepting XMLHttpRequest and fetch requests to obtain HTTP request parameters, storing the requests in global variables, and using a system timer to check for duplicate requests every 300 milliseconds, removing requests without requests after 300 milliseconds from the global variables; intercepting XMLHttpRequest and fetch requests to obtain status values, and using status codes to statistically analyze the HTTP success rate; and conducting incremental stress testing on the website using third-party tools, determining the website's stress test value based on preset incremental scoring rules, with the incremental score calculated as: Incremental Score = Latency Rate. Success rate > 50%; This includes acquiring user profiles and information every 5 seconds through user click events to build a user behavior model. User profiles and information include: user dwell time, user clicks, user application update cycle, user region, user device information, and user application entry point. Specifically, it involves obtaining user device information through the SDK, including browser and operating system information from `window.navigator.userAgent` and screen size information from `document.documentElement`. Front-end security vulnerability checks include: checking if cookies are HttpOnly; encoding input and URL parameters; HTML encoding of dynamically output JavaScript content; local storage of sensitive information including passwords, phone numbers, and localStorage data not exceeding 1MB; sandbox security attributes; use of HTTPS protocol and HSTS; and using the browser's SRI function for static CDN resources. The system collects log data reported by the SDK to Nginx, parses and filters the log data to remove duplicates according to preset rules, and stores the deduplicated data in a Redis database. The parsing and filtering of log data according to preset rules includes: data with the same hash for the same project within the current collection period are grouped into one data set, and the number of repetitions is recorded in the variable count for filtering and deduplication, while data that does not conform to the protocol is discarded. Regularly extract and filter the deduplicated data, perform secondary deduplication, merge similar data, and perform statistical processing on the extracted data, and then enter the processed data into the MySQL database. This includes summarizing various indicators of the data by time dimension. The processed data includes terminal device UUID, device operating system, browser version, and screen size as independent compilation storage, and is used as a reference to specific events. Specific events are stored in separate tables to ensure that data generated by the same user is not excluded. Based on the user's actual needs, specified data is extracted from the MySQL database and returned to the web frontend. Health monitoring of the web frontend data is performed, and the current operating status of the web frontend is displayed through chart components. This health monitoring includes: monitoring and displaying website errors based on the error types in the processed log data; and conducting a comprehensive health assessment of the web frontend based on error monitoring, webpage performance, HTTP success rate and repetition rate, stress test values, and frontend security vulnerability check data.

2. The method according to claim 1, characterized in that, The SDK obtains user status, front-end application running status, and basic device data, including: Determine the user update frequency by the SDK loading time; The SDK reports the number of online users, as well as the real-time data traffic and real-time system resource consumption of the web front-end. Global errors are intercepted by listening to browser events, the unhandledrejection error type and the error type are obtained, and then reported to Nignx for data processing.

3. The method according to claim 2, characterized in that, The process of acquiring user profiles and information through user click events and constructing user behavior models includes: The user's activity status is obtained through the user click event. If no event occurs within a preset time, it is considered that the user has left. The user's dwell time is accumulated from the next event. The system also obtains user click count and URL information through the user click events.

4. The method according to any one of claims 1-3, characterized in that, After collecting the log data reported by the SDK to Nginx, the method further includes: If the data reaches the preset outlier limit, an alarm will be triggered via the set contact method to notify relevant personnel for inspection.

5. The method according to claim 1, characterized in that, After performing health monitoring on the web frontend data and displaying the current running status of the web frontend through a chart component, the method includes: Based on the monitored web front-end health index and operational status, corresponding analysis and detection suggestions are provided and feedback is given to relevant personnel.

6. A web data monitoring system, characterized in that, The system includes: The acquisition and reporting module is used to embed SDKs in the web frontend. These SDKs acquire user status, frontend application running status, and basic device data, and report this data to NGNX. Specifically, it is used to: report webpage performance, HTTP success rate and repetition rate, stress test values, and frontend security vulnerability check data via the SDK. This includes: obtaining various performance metrics of the webpage through the `performance` function; intercepting and statistically analyzing the success rate and repetition rate of HTTP requests via the SDK, acquiring HTTP requests, and determining that HTTP requests are duplicated if they share the same URL, method, parameters, and body; intercepting XMLHttpRequest and fetch requests to acquire HTTP request parameters, storing the requests in global variables, and using a system timer to check for duplicate requests every 300 milliseconds. Interfaces without requests after 300 milliseconds are cleared from the global variables. The SDK also intercepts XMLHttpRequest and fetch requests. Obtain the status value and use the status code to calculate the success rate of the HTTP interface; conduct incremental stress testing on the website using third-party tools, determine the website's stress test value based on preset incremental scoring rules, and the effective incremental statistical value is: Incremental Score = Latency Rate Success rate > 50%; This includes acquiring user profiles and information every 5 seconds through user click events to build a user behavior model. User profiles and information include: user dwell time, user clicks, user application update cycle, user region, user device information, and user application entry point. Specifically, it involves obtaining user device information through the SDK, including browser and operating system information from `window.navigator.userAgent` and screen size information from `document.documentElement`. Front-end security vulnerability checks include: checking if cookies are HttpOnly; encoding input and URL parameters; HTML encoding of dynamically output JavaScript content; local storage of sensitive information including passwords, phone numbers, and localStorage data not exceeding 1MB; sandbox security attributes; use of HTTPS protocol and HSTS; and using the browser's SRI function for static CDN resources. The primary processing module is used to collect log data reported by the SDK to Nginx, parse and filter the log data according to preset rules, and store the deduplicated data in the Redis database. Specifically, it is used to: group data with the same hash of the same project into one data in the current collection period, record the number of repetitions in the variable count, filter and remove duplicates, and discard data that does not conform to the protocol. The secondary processing module is used to periodically extract filtered and deduplicated data, perform secondary deduplication, merge similar data, and perform statistical processing on the extracted data, and then enter the processed data into the MySQL database. Specifically, it is used to perform statistical summarization of various indicators on the data along the time dimension. The processed data includes terminal device UUID, device operating system, browser version, and screen size as independent compilation storage, and is used as a reference to specific events. Specific events are stored in separate tables to keep track of data generated by the same user. The monitoring and display module is used to extract specified data from the MySQL database according to the user's actual needs, return it to the web frontend, perform health monitoring on the web frontend data, and display the current running status of the web frontend through chart components. Specifically, during the health monitoring of the web frontend data, it is used to: monitor and display website errors based on the error types in the processed log data; and conduct a comprehensive health assessment of the web frontend through error monitoring, webpage performance, success rate and repetition rate of HTTP interfaces, stress test values, and frontend security vulnerability check data.

7. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the web data monitoring method according to any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the web data monitoring method according to any one of claims 1 to 5 when running.

Images