An intelligent data collection method, device and medium based on front-end monitoring SDK

Abstract

The present application discloses an intelligent data collection method, device and medium based on a front-end monitoring SDK, and relates to the technical field of front-end monitoring of Web applications. The method includes: when a Web application page is loaded, a front-end monitoring SDK script is introduced through an external file tag, and the data collection logic preset in the front-end monitoring SDK script is loaded; wherein the data collection logic includes: performance data collection logic, user behavior data collection logic, and error log collection logic; when the Web application page is running, intelligent data collection is performed through the idle time data scheduling mechanism preset in the front-end monitoring SDK script; the data to be uploaded is determined through the incremental data upload mechanism preset in the front-end monitoring SDK script, and the data to be uploaded is intelligently compressed; the data upload strategy is dynamically selected based on the real-time network status to upload the compressed data to be uploaded to the data server.

Description

Technical Field

[0001] The present application relates to the technical field of front-end monitoring of Web applications, and in particular to an intelligent data collection method, device and medium based on a front-end monitoring SDK. Background Art

[0002] With the rapid development of Internet technology, Web applications have been widely used in various fields, and their complexity has increased day by day. In this case, front-end performance, user experience and error monitoring have become crucial to ensure the quality and reliability of Web applications.

[0003] Although existing front-end monitoring SDKs can collect performance data (such as page loading time, resource requests, error logs, etc.) and user behavior data, there are many problems. On the one hand, traditional monitoring SDKs usually generate large resource overhead. Frequent data upload operations and complex data processing processes tend to occupy a large amount of browser resources, which will seriously affect the performance of the page. For example, on some low-performance devices, the page loading speed may be significantly slowed down, or even freeze, which greatly reduces the user experience.

[0004] On the other hand, low efficiency in data collection and transmission is another major pain point of existing SDKs. Especially in weak network environments, the traditional real-time data upload mode often causes network congestion, resulting in data loss and failure to guarantee data integrity and accuracy. This makes it difficult for developers to obtain comprehensive and reliable monitoring data, which in turn affects the accurate diagnosis and optimization of Web application problems.

[0005] Therefore, how to achieve efficient and accurate front-end monitoring data collection without affecting user experience has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] The embodiments of the present application provide an intelligent data collection method, device and medium based on a front-end monitoring SDK, which are used to solve the following technical problem: how to achieve efficient and accurate front-end monitoring data collection without affecting the user experience.

[0007] In a first aspect, an embodiment of the present application provides an intelligent data collection method based on a front-end monitoring SDK, the method comprising: when a Web application page is loaded, introducing a front-end monitoring SDK script through an external file tag, and loading the data collection logic preset in the front-end monitoring SDK script; wherein the data collection logic comprises: performance data collection logic, user behavior data collection logic, and error log collection logic; when the Web application page is running, performing intelligent data collection through an idle time data scheduling mechanism preset in the front-end monitoring SDK script; determining the data to be uploaded through an incremental data upload mechanism preset in the front-end monitoring SDK script, and intelligently compressing the data to be uploaded; dynamically selecting a data upload strategy based on real-time network conditions to upload the compressed data to be uploaded to a data server.

[0008] In one implementation of the present application, after introducing the front-end monitoring SDK script through an external file tag, the method also includes: detecting whether the browser supports the key functions to be applied; wherein the key functions to be applied include: requestIdleCallback, Web Worker and network status API; initializing the default configuration; wherein the configuration types of the default configuration include: data collection frequency, upload strategy and scheduling parameters.

[0009] In one implementation of the present application, the performance data collection logic includes: monitoring the user's page interaction behavior through JavaScript to determine performance parameters using the PerformanceAPI; wherein the performance parameters include: page loading time, resource request time, memory usage; based on the data collection frequency, periodically collecting performance parameters; the user behavior data collection logic includes: monitoring user interaction events and recording related event information; wherein user interaction events include: clicks, scrolling, input; and; monitoring the DOM changes of the page through Mutation Observer to collect loading information of dynamic content; the error log collection logic includes: collecting global errors and uncaught Promise exceptions based on page monitoring; and, collecting network request error information and resource loading failure information.

[0010] In one implementation of the present application, performance data collection and user behavior data collection are pre-designated as low-priority tasks, and error log collection is pre-designated as a high-priority task; intelligent data collection is performed through the idle time data scheduling mechanism preset in the front-end monitoring SDK script, specifically including: when the requestIdleCallback function is available, the browser's idle time is detected, and when the browser is busy, the performance data and user behavior data collected by the low-priority tasks are temporarily stored in the memory, and the performance data and user behavior data are read when the browser is idle; when the requestIdleCallback function is not available, based on the real-time rendering load of the page, the delay time for the execution of high-priority tasks or low-priority tasks is dynamically adjusted through the preset setTimeout scheduling scheme.

[0011] In one implementation of the present application, after performing intelligent data collection through the idle time data scheduling mechanism preset in the front-end monitoring SDK script, the method also includes: identifying sensitive data in the collected data; and desensitizing the sensitive data based on preset desensitization rules.

[0012] In one implementation of the present application, the data to be uploaded is determined through the incremental data upload mechanism preset in the front-end monitoring SDK script, and the data to be uploaded is intelligently compressed, specifically including: using a hash algorithm to calculate the hash values ​​of the two collected data of the same type, and when the hash values ​​are different, determining the latter collected data as the data to be uploaded; based on preset preprocessing rules, standardizing the data to be uploaded; determining the real-time network status, and dynamically selecting the corresponding compression algorithm to be applied based on the real-time network status; wherein the real-time network status includes: network type, effective bandwidth, and delay; based on the compression algorithm to be applied, compressing the standardized data to be uploaded.

[0013] In one implementation of the present application, a data upload strategy is dynamically selected based on the real-time network status to upload the compressed data to be uploaded to the data server, specifically including: analyzing whether the real-time network status meets the real-time upload conditions; if the real-time upload conditions are met, uploading the compressed data to be uploaded to the data server in real time; if the real-time upload conditions are not met, dynamically configuring the delayed upload frequency and batch size to upload the compressed data to be uploaded to the data server in batches.

[0014] In one implementation of the present application, the method further includes: utilizing Web Worker to perform asynchronous processing on data collection, data scheduling, data compression, and data uploading.

[0015] In the second aspect, an embodiment of the present application also provides an intelligent data acquisition device based on a front-end monitoring SDK, the device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute an intelligent data acquisition method based on the front-end monitoring SDK such as any one of the above items.

[0016] On the third aspect, an embodiment of the present application also provides a non-volatile computer storage medium for intelligent data collection based on a front-end monitoring SDK, which stores computer executable instructions. When the computer executable instructions are executed, an intelligent data collection method based on a front-end monitoring SDK as described above is implemented.

[0017] The embodiment of the present application provides an intelligent data collection method, device and medium based on a front-end monitoring SDK, which has the following beneficial effects:

[0018] 1. Reduce the consumption of network and system resources through idle time scheduling, incremental upload and compression technology. Collect data using browser idle time to reduce the impact on page loading and user interaction and improve user experience.

[0019] 2. It realizes efficient data collection, transmission and processing, ensuring the accuracy, integrity and security of monitoring data while guaranteeing page performance. It automatically adjusts the upload strategy to ensure efficient data transmission under various network conditions.

[0020] 3. It can intelligently collect and transmit data while minimizing the impact on page performance. It is suitable for front-end monitoring scenarios of various Web applications, especially for use on mobile terminals and in weak network environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0022] Figure 1 A flow chart of an intelligent data collection method based on a front-end monitoring SDK provided in an embodiment of the present application;

[0023] Figure 2 A specific implementation flow chart of intelligent data collection based on front-end monitoring SDK provided by this application;

[0024] Figure 3 A schematic diagram of the internal structure of an intelligent data acquisition device based on a front-end monitoring SDK provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0026] The embodiments of the present application provide an intelligent data collection method, device and medium based on a front-end monitoring SDK, which are used to solve the following technical problem: how to achieve efficient and accurate front-end monitoring data collection without affecting the user experience.

[0027] The technical solution proposed in the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0028] Figure 1 The following is a flow chart of an intelligent data collection method based on a front-end monitoring SDK provided in an embodiment of the present application. Figure 1 As shown, an intelligent data collection method based on a front-end monitoring SDK provided in an embodiment of the present application specifically includes the following steps:

[0029] Step 101: When the Web application page is loaded, the front-end monitoring SDK script is introduced through the external file tag, and the data collection logic preset in the front-end monitoring SDK script is loaded.

[0030] In one embodiment of the present application, in order to realize intelligent data collection based on the front-end monitoring SDK, it is first necessary to load the application page through <script>标签引入监控SDK脚本，确保其在页面加载初期即被执行。

[0031] 进一步的，检测浏览器是否支持待应用关键功能；其中，待应用关键功能包括：requestIdleCallback、Web Worker和网络状态API；初始化默认配置；其中，默认配置的配置类型包括：数据采集频率、上传策略和调度参数。

[0032] 进一步的，加载前端监控SDK脚本中预置的数据采集逻辑。

[0033] 在本申请的一个实施例中，数据采集逻辑包括：性能数据采集逻辑、用户行为数据采集逻辑、错误日志采集逻辑.

[0034] 在一个实施例中，性能数据采集逻辑，包括：通过JavaScript监控用户的页面交互行为，如使用PerformanceAPI采集页面加载时间、资源请求耗时、内存使用情况等性能指标。进一步的将周期性采集页面的FPS（帧率）和内存占用，以监控页面渲染性能。

[0035] 在一个实施例中，用户行为数据采集逻辑，包括：监听用户交互事件并记录相关事件信息；其中，用户交互事件包括：点击、滚动、输入；以及；通过Mutation Observer监控页面的DOM变化，以采集动态内容的加载信息。

[0036] 在一个实施例中，错误日志采集逻辑包括：基于页面监听采集全局错误和未捕获的Promise异常，以捕捉运行时错误。进一步的可以按需采集网络请求错误信息及资源加载失败信息，以便诊断应用性能问题。

[0037] 步骤102、在Web应用页面运行时，通过前端监控SDK脚本中预置的空闲时间数据调度机制，进行智能数据采集。

[0038] 在本申请的一个实施例中，在预先指定性能数据采集与用户行为数据采集为低优先级任务，预先指定错误日志采集为高优先级任务；通过前端监控SDK脚本中预置的空闲时间数据调度机制，进行智能数据采集，具体包括：在requestIdleCallback功能可用的情况下，检测浏览器的空闲时间，并在浏览器器繁忙时，将属于低优先级任务采集的性能数据与用户行为数据暂存在内存中，在浏览器空闲时读取性能数据与用户行为数据；在requestIdleCallback功能不可用的情况下，基于页面的实时渲染负载，通过预设的setTimeout调度方案，动态调整高优先级任务或低优先级任务执行的延迟时间。

[0039] 在一个实施例中，空闲时间数据调度机制可以包括智能任务调度、数据批量处理、自适应空闲时间调度。

[0040] 其中，智能任务调度包括：空闲检测与任务分配、任务优先级管理。

[0041] 具体地，首先利用requestIdleCallback检测浏览器的空闲时间，智能调度数据采集、预处理和上传任务。此机制确保只有在主线程处于空闲状态时才执行数据处理任务，避免阻塞用户交互。进一步的，如果requestIdleCallback不可用，则自动降级为基于setTimeout的调度方案，并根据当前页面的渲染负载动态调整任务执行的延迟时间，确保数据处理不会与页面渲染冲突。进一步的任务按优先级进行分类：高优先级任务（如错误日志、页面崩溃数据）在检测到关键事件时立即采集和上传；低优先级任务（如性能数据、用户行为记录）则在空闲时间批量处理。进一步的，通过调度器动态调整任务的优先级，确保在用户活跃操作期间降低数据处理任务的频率，优先保障页面的响应速度。

[0042] 其中，数据批量处理包括：批量采集与合并、任务节流与防抖。

[0043] 具体地，首先将低优先级的性能数据和用户行为数据临时存储在内存中，待浏览器空闲时统一处理，减少频繁的采集操作对页面性能的影响。进一步的通过内存缓存机制合并采集到的数据包，在浏览器空闲时批量上传，降低网络请求频次，减少服务器压力。进一步的结合节流（throttling）和防抖（debouncing）策略控制数据采集频率。节流机制确保在设定的时间窗口内数据处理和上传任务最多只执行一次，而防抖机制则避免短时间内的高频数据采集，减少资源消耗。进一步的当页面交互频繁或CPU使用率过高时，自动暂停低优先级任务，待系统恢复空闲状态后再继续执行，确保数据采集过程不会影响页面流畅度。

[0044] 其中，自适应空闲时间调度包括：基于用户行为的智能调度、灵魂调整任务时间片。首先通过监控用户的活动状态（如鼠标移动、键盘输入、滚动操作等），在用户活跃时降低数据采集任务的频率；当用户处于非活跃状态（如页面静止超过一定时间）时，自动提升数据采集任务的频率。进一步的通过分析用户行为模式，自适应调整调度策略。例如，在用户切换到后台或页面不可见时（通过document.visibilityState检测），加快数据上传，确保在页面空闲时完成数据同步。进一步的基于requestIdleCallback的deadline参数，动态调整每次任务执行的时间片长度。当deadline.timeRemaining()较长时执行更多的数据处理操作，而在空闲时间不足时优先处理高优先级任务，并将低优先级任务推迟到下一个空闲周期。进一步的通过调节任务的执行时间片，优化系统资源的使用，使数据采集与页面渲染达到平衡。

[0045] 在本申请的一个实施例中，在通过前端监控SDK脚本中预置的空闲时间数据调度机制，进行智能数据采集之后，方法还包括：识别采集数据中的敏感数据；基于预设的脱敏规则对敏感数据进行脱敏处理。

[0046] 步骤103、通过前端监控SDK脚本中预置的增量数据上传机制，确定待上传数据，并对待上传数据进行智能压缩。

[0047] 在本申请的一个实施例中，在通过前端监控SDK脚本中预置的空闲时间数据调度机制，进行智能数据采集之后，通过前端监控SDK脚本中预置的增量数据上传机制，确定待上传数据，并对待上传数据进行智能压缩。

[0048] 具体地，利用哈希算法对同一类型的前后两次采集数据进行哈希值计算，并在哈希值不同的情况下，确定后一次采集数据为待上传数据；基于预设的预处理规则，对待上传数据进行标准化处理；确定实时网络状况，并基于实时网络状况动态选择对应的待应用压缩算法；其中，实时网络状况包括：网络类型、有效带宽、延迟；基于待应用压缩算法，对标准化处理后的待上传数据进行压缩。

[0049] 在一个实施例中，确定待上传数据，并对待上传数据进行智能压缩可以包括：增量数据采集与去重、数据压缩与轻量化传输。

[0050] 其中，增量数据采集与去重包括：数据变动检测与去重、数据分片与批量处理。

[0051] 具体地，首先对采集到的数据使用哈希算法生成唯一标识符（Hash值），每次上传前对比当前数据与上一次上传的数据Hash值，确保仅上传变动的数据，从而减少冗余数据传输。进一步的使用内存缓存存储上次上传的数据快照，并通过数据对比机制筛选出新的数据条目，仅在数据发生变化时触发上传操作，避免重复上传。进一步的将采集到的数据进行分片处理，根据数据类型（如性能指标、用户行为、错误日志）进行分类，每种类型的数据独立生成Hash进行增量检测。进一步的对性能数据和用户行为数据进行合并，以批量上传的方式减少网络请求次数，从而优化带宽使用。

[0052] 其中，数据压缩与轻量化传输包括：智能压缩算法选择、轻量化数据结构。

[0053] 具体地，首先根据当前网络环境动态选择压缩算法。例如，在高带宽环境下使用Gzip进行快速压缩，在低带宽环境下使用Brotli提高压缩比率，减少数据包体积。进一步的通过对数据内容进行预处理（如剔除空白字段、使用短标签替换冗长字段），进一步减少数据体积，从而优化传输效率。进一步的采用轻量化的数据格式（如ProtocolBuffers或MessagePack）替代传统的JSON格式，减少数据序列化和反序列化时的开销，并提高压缩效率。进一步的对日志数据和性能指标进行结构化优化，仅保留关键信息字段，同时删除冗余信息以降低数据负载。

[0054] 步骤104、基于实时网络状况动态选择数据上传策略，以将压缩后的待上传数据上传至数据服务器。

[0055] 在本申请的一个实施例中，在通过前端监控SDK脚本中预置的增量数据上传机制，确定待上传数据，并对待上传数据进行智能压缩之后，基于实时网络状况动态选择数据上传策略，以将压缩后的待上传数据上传至数据服务器。

[0056] 具体地，分析实时网络状况是否符合实时上传条件； 若符合实时上传条件，将压缩后的待上传数据实时上传至数据服务器；若不符合实时上传条件，动态配置延时上传频率和批量大小，以将压缩后的待上传数据分批上传至数据服务器。

[0057] 在一个实施例中，上传数据至数据服务器可以采用如下的自适应上传策略包括：基于网络状态的上传优化、用户行为感知与智能调度、智能上传调度与任务管理、断点续传与重试机制。

[0058] 其中，基于网路状态的上传优化包括：网路状态检测、带宽与延迟感知。

[0059] 具体地，首先通过navigator.connection获取设备当前的网络状态（如网络类型、有效带宽、延迟），根据网络条件调整上传频率和批量大小。在高速网络下（如WiFi和4G），实时上传数据；在低速或高延迟环境下（如3G、Edge），采用批量上传以减少频繁请求。进一步的当检测到用户切换到弱网环境时，自动切换到延迟上传模式，将采集的数据暂存于本地，待网络恢复时再批量上传。

[0060] 其中，用户行为感知与智能调度包括：基于用户的上传优化、页面生命周期时间配合。

[0061] 具体地，首先监测用户的活跃状态（如鼠标、键盘操作、页面滚动等），在用户活跃时减少数据上传频率，优先保障页面的响应速度；在用户不活跃或页面处于后台时（通过document.visibilityState检测），提高上传频率或执行批量数据上传。进一步的当检测到页面切换到后台时（如用户切换标签页），自动触发一次批量上传，以利用空闲时间完成数据同步。进一步的结合页面生命周期事件（如beforeunload和pagehide），在用户关闭页面或刷新时，快速上传关键数据（如错误日志和未上传的性能数据），确保数据不会因页面关闭而丢失。

[0062] 其中，智能上传调度与任务管理包括：多级上传策略、自适应任务时间片调整。

[0063] 具体地，首先实现多级上传模式：高优先级数据（如错误日志、页面崩溃报告）在检测到时立即上传；低优先级数据（如用户行为数据、性能指标）则通过智能调度进行批量上传。进一步的设定动态上传时间窗口，根据页面负载和系统资源情况调整上传频率。例如，当检测到CPU使用率较高时，暂时延迟数据上传，以确保用户体验流畅。进一步的在requestIdleCallback中动态调整上传任务的时间片长度。当检测到浏览器有较多空闲时间时，处理更多的数据包上传；在空闲时间不足时，优先上传高优先级的数据并推迟低优先级的数据处理。

[0064] 其中，断点续传与重试机制包括：数据缓存与断点续传、智能重试与指数退避策略。

[0065] 具体地，首先通过IndexedDB或LocalStorage临时缓存未上传的数据包。在上传失败时，自动保存未完成的数据，待网络恢复后从断点处继续上传，避免数据丢失。进一步的为每个数据包设置唯一的上传标识符，以便在断点续传时快速定位未完成的部分，提高上传效率。进一步的实现上传重试机制，在数据上传失败时，基于指数退避算法（exponentialbackoff）调整重试时间间隔，避免频繁重试导致的网络拥塞。进一步的设定最大重试次数，以防止持续重试消耗过多的系统资源；在重试多次失败后，通知服务器记录异常状态，以便后续分析。

[0066] 在本申请的一个实施例中，方法还包括：利用Web Worker对数据采集、数据调度、数据压缩与数据上传进行异步处理。

[0067] 在一个实施例中，异步处理可以包括：Web Worker的任务分离、异步数据处理与队列管理、异步数据压缩与优化。

[0068] 其中，Web Worker的任务分离包括：任务模块化与并行处理、Web Worker池管理。

[0069] 具体地，首先将数据采集、预处理、压缩和上传等任务划分为多个独立的模块，分别由Web Worker进行异步处理，确保数据采集任务不会阻塞主线程。进一步的对于高优先级的任务（如错误日志、页面崩溃数据）和低优先级的任务（如性能数据、用户行为记录），分配不同的Worker进行并行处理，从而提高数据处理的效率。进一步的实现Web Worker池（WorkerPool）机制，根据任务负载动态分配Worker数量，避免创建过多Worker导致的内存占用和线程争抢。进一步的在高负载情况下，优先分配Worker处理关键任务；在任务高峰过后释放不必要的Worker，以优化内存使用。

[0070] 其中，异步数据处理与队列管理包括：消息队列与批量数据处理、数据缓存与分片处理。

[0071] 具体地，首先通过消息队列在主线程与Web Worker之间进行通信，将采集到的数据按优先级加入队列。Worker在空闲时从队列中批量获取数据进行处理，减少频繁的数据传递开销。进一步的为避免内存溢出，设定队列长度上限。当队列接近满载时，优先处理高优先级任务，并将低优先级任务推迟到下一次空闲时间处理。进一步的在Worker中对大数据包进行分片处理（Chunking），并行处理多个数据片段，加速压缩和上传操作。进一步的通过内存缓存机制临时存储待处理的数据块，确保即使在任务高峰期也能稳定进行数据采集和上传。

[0072] 其中，异步数据压缩与优化包括：非阻塞数据压缩、数据加密与隐私保护。

[0073] 具体地，首先在Web Worker中对采集到的数据进行压缩处理（如Gzip或Brotli），确保压缩过程不会阻塞主线程，从而提高页面渲染速度。进一步的根据数据类型选择不同的压缩策略：对于文本数据使用快速压缩算法，对于性能日志数据则采用高压缩比的算法，以减少上传数据量。进一步的在Worker中进行数据加密（AES加密）操作，确保敏感数据在传输前已加密，提升数据安全性。进一步的所有加密和压缩处理均在Worker内部完成，避免将大量计算任务交由主线程处理，从而降低页面响应时间。

[0074] 图2为本申请提供的一种基于前端监控SDK的智能数据采集具体实施流程图。本申请上述实施例，可以以该流程图具体实施。

[0075] 以上为本申请提出的方法实施例。基于同样的发明构思，本申请实施例还提供了一种基于前端监控SDK的智能数据采集设备，其结构如图3所示。

[0076] 图3为本申请实施例提供的一种基于前端监控SDK的智能数据采集设备内部结构示意图。如图3所示，设备包括：

[0077] 至少一个处理器301；

[0078] 以及，与至少一个处理器通信连接的存储器302；

[0079] 其中，存储器302存储有可被至少一个处理器执行的指令，指令被至少一个处理器301执行，以使至少一个处理器301能够：

[0080] 在Web应用页面加载时，通过外部文件标签引入前端监控SDK脚本，并加载前端监控SDK脚本中预置的数据采集逻辑；其中，数据采集逻辑包括：性能数据采集逻辑、用户行为数据采集逻辑、错误日志采集逻辑；

[0081] 在Web应用页面运行时，通过前端监控SDK脚本中预置的空闲时间数据调度机制，进行智能数据采集；

[0082] 通过前端监控SDK脚本中预置的增量数据上传机制，确定待上传数据，并对待上传数据进行智能压缩；

[0083] 基于实时网络状况动态选择数据上传策略，以将压缩后的待上传数据上传至数据服务器。

[0084] 本申请的一些实施例提供的对应于图1的一种基于前端监控SDK的智能数据采集的非易失性计算机存储介质，存储有计算机可执行指令，计算机可执行指令设置为：

[0085] 在Web应用页面加载时，通过外部文件标签引入前端监控SDK脚本，并加载前端监控SDK脚本中预置的数据采集逻辑；其中，数据采集逻辑包括：性能数据采集逻辑、用户行为数据采集逻辑、错误日志采集逻辑；

[0086] 在Web应用页面运行时，通过前端监控SDK脚本中预置的空闲时间数据调度机制，进行智能数据采集；

[0087] 通过前端监控SDK脚本中预置的增量数据上传机制，确定待上传数据，并对待上传数据进行智能压缩；

[0088] 基于实时网络状况动态选择数据上传策略，以将压缩后的待上传数据上传至数据服务器。

[0089] 本申请中的各个实施例均采用递进的方式描述，各个实施例之间相同相似的部分互相参见即可，每个实施例重点说明的都是与其他实施例的不同之处。尤其，对于物联网设备和介质实施例而言，由于其基本相似于方法实施例，所以描述的比较简单，相关之处参见方法实施例的部分说明即可。

[0090] 本申请实施例提供的系统和介质与方法是一一对应的，因此，系统和介质也具有与其对应的方法类似的有益技术效果，由于上面已经对方法的有益技术效果进行了详细说明，因此，这里不再赘述系统和介质的有益技术效果。

[0091] 本领域内的技术人员应明白，本申请的实施例可提供为方法、系统、或计算机程序产品。因此，本申请可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本申请可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质（包括但不限于磁盘存储器、CD-ROM、光学存储器等）上实施的计算机程序产品的形式。

[0092] 本申请是参照根据本申请实施例的方法、设备（系统）、和计算机程序产品的流程图和／或方框图来描述的。应理解可由计算机程序指令实现流程图和／或方框图中的每一流程和／或方框、以及流程图和／或方框图中的流程和／或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和／或方框图一个方框或多个方框中指定的功能的装置。

[0093] 这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品，该指令装置实现在流程图一个流程或多个流程和／或方框图一个方框或多个方框中指定的功能。

[0094] 这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和／或方框图一个方框或多个方框中指定的功能的步骤。

[0095] 在一个典型的配置中，计算设备包括一个或多个处理器 (CPU)、输入 / 输出接口、网络接口和内存。

[0096] 内存可能包括计算机可读介质中的非永久性存储器，随机存取存储器 (RAM) 和 / 或非易失性内存等形式，如只读存储器 (ROM) 或闪存(flash RAM)。内存是计算机可读介质的示例。

[0097] 计算机可读介质包括永久性和非永久性、可移动和非可移动媒体可以由任何方法或技术来实现信息存储。信息可以是计算机可读指令、数据结构、程序的模块或其他数据。计算机的存储介质的例子包括，但不限于相变内存 (PRAM)、静态随机存取存储器 (SRAM)、动态随机存取存储器 (DRAM)、其他类型的随机存取存储器 (RAM)、只读存储器 (ROM)、电可擦除可编程只读存储器 (EEPROM)、快闪记忆体或其他内存技术、只读光盘只读存储器(CD-ROM)、数字多功能光盘 (DVD) 或其他光学存储、磁盒式磁带，磁带磁磁盘存储或其他磁性存储设备或任何其他非传输介质，可用于存储可以被计算设备访问的信息。按照本文中的界定，计算机可读介质不包括暂存电脑可读媒体 (transitory media)，如调制的数据信号和载波。

[0098] 还需要说明的是，术语"包括”、"包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、商品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、商品或者设备所固有的要素。在没有更多限制的情况下，由语句"包括一个……”限定的要素，并不排除在包括所述要素的过程、方法、商品或者设备中还存在另外的相同要素。

[0099] 以上所述仅为本申请的实施例而已，并不用于限制本申请。对于本领域技术人员来说，本申请可以有各种更改和变化。凡在本申请的精神和原理之内所作的任何修改、等同替换、改进等，均应包含在本申请的权利要求范围之内。< / script>

Claims

1. An intelligent data collection method based on front-end monitoring SDK, characterized in that: The method comprises: When the Web application page is loaded, the front-end monitoring SDK script is introduced through the external file tag, and the data collection logic preset in the front-end monitoring SDK script is loaded; wherein the data collection logic includes: performance data collection logic, user behavior data collection logic, and error log collection logic; When the Web application page is running, intelligent data collection is performed through the idle time data scheduling mechanism preset in the front-end monitoring SDK script; Determine the data to be uploaded through the incremental data upload mechanism preset in the front-end monitoring SDK script, and intelligently compress the data to be uploaded; Dynamically select a data upload strategy based on real-time network conditions to upload the compressed data to be uploaded to a data server; The data to be uploaded is determined through the incremental data upload mechanism preset in the front-end monitoring SDK script, and the data to be uploaded is intelligently compressed, specifically including: The hash algorithm is used to calculate the hash values ​​of the two collected data of the same type, and when the hash values ​​are different, the latter collected data is determined to be the data to be uploaded; Based on preset preprocessing rules, standardize the data to be uploaded; Determine the real-time network status, and dynamically select the corresponding compression algorithm to be applied based on the real-time network status; wherein the real-time network status includes: network type, effective bandwidth, and delay; Based on the compression algorithm to be applied, compressing the data to be uploaded after the standardization processing; Dynamically selecting a data upload strategy based on real-time network conditions to upload the compressed data to be uploaded to the data server specifically includes: Analyzing whether the real-time network status meets the real-time uploading conditions; If the real-time uploading condition is met, the compressed data to be uploaded is uploaded to the data server in real time; If the real-time upload condition is not met, the delayed upload frequency and batch size are dynamically configured to upload the compressed data to be uploaded to the data server in batches.

2. According to claim 1, an intelligent data collection method based on front-end monitoring SDK is characterized in that: After the front-end monitoring SDK script is introduced through the external file tag, the method further includes: Detect whether the browser supports the key functions to be applied; wherein the key functions to be applied include: requestIdleCallback, Web Worker and network status API; Initialize the default configuration; wherein the configuration types of the default configuration include: data collection frequency, upload strategy and scheduling parameters.

3. According to claim 2, an intelligent data collection method based on front-end monitoring SDK is characterized in that: The performance data collection logic includes: Monitor the user's page interaction behavior through JavaScript to determine performance parameters using PerformanceAPI; wherein the performance parameters include: page loading time, resource request time, and memory usage; Based on the data collection frequency, periodically collect the performance parameters; The user behavior data collection logic includes: Monitor user interaction events and record relevant event information; wherein the user interaction events include: click, scroll, input; and; Monitor the DOM changes of the page through Mutation Observer to collect loading information of dynamic content; The error log collection logic includes: Collect global errors and uncaught Promise exceptions based on page monitoring; and, Collect network request error information and resource loading failure information.

4. According to claim 1, the intelligent data collection method based on the front-end monitoring SDK is characterized in that: Pre-designate performance data collection and user behavior data collection as low-priority tasks, and pre-designate error log collection as a high-priority task; Intelligent data collection is performed through the idle time data scheduling mechanism preset in the front-end monitoring SDK script, specifically including: When the requestIdleCallback function is available, the idle time of the browser is detected, and when the browser is busy, the performance data and user behavior data collected by the low-priority task are temporarily stored in the memory, and the performance data and user behavior data are read when the browser is idle; When the requestIdleCallback function is not available, based on the real-time rendering load of the page, the delay time of high-priority tasks or low-priority tasks is dynamically adjusted through the preset setTimeout scheduling scheme.

5. According to claim 4, an intelligent data collection method based on front-end monitoring SDK is characterized in that: After performing intelligent data collection through the idle time data scheduling mechanism preset in the front-end monitoring SDK script, the method further includes: Identify sensitive data in collected data; The sensitive data is desensitized based on preset desensitization rules.

6. The intelligent data collection method based on front-end monitoring SDK according to claim 1 is characterized in that: The method further comprises: Web Worker is used to perform asynchronous processing of data collection, data scheduling, data compression and data upload.

7. An intelligent data acquisition device based on front-end monitoring SDK, characterized in that: The device comprises: at least one processor; and, a memory communicatively coupled to the at least one processor; Wherein, the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute an intelligent data collection method based on a front-end monitoring SDK as described in any one of claims 1-6.

8. A non-volatile computer storage medium for intelligent data collection based on a front-end monitoring SDK, storing computer executable instructions, characterized in that: When the computer executable instructions are executed, an intelligent data collection method based on a front-end monitoring SDK as described in any one of claims 1 to 6 is implemented.

Images