Application behavior data acquisition method and system based on minimum intrusion

Abstract

The invention provides an application behavior data acquisition method and system based on minimum intrusion, and belongs to the technical field of informatization management. Comprising the following steps: automatically injecting an SDK (Software Development Kit) at an application front end by adopting a painless probe injection mode, collecting user behavior data, and reporting the user behavior data in an asynchronous mode; constructing an interceptor mechanism based on the interceptor base class and defining an observer interface; introducing the modified class into a guide area through an observer interface, and injecting a spring area interceptor into an application for back-end data acquisition; a MySQL database and a search engine are adopted to store collected data in a classified mode, and a collaborative mechanism of front-end asynchronous reporting and rear-end real-time preprocessing is established. According to the method, behavior data acquisition of the application can be realized in a minimum intrusion form.

Description

Technical Field

[0001] This invention belongs to the field of information management technology, and in particular relates to a method and system for collecting application behavior data based on minimal intrusion. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] In the field of information management technology, monitoring the operational status, user behavior, and performance indicators of application systems is fundamental for application optimization and decision support. Currently, common monitoring methods mainly rely on network traffic analysis, server audit logs, or monitoring services provided by cloud platforms. These methods typically infer application access and some operational metrics indirectly by parsing network traffic packets and matching application function URLs.

[0004] However, these existing technical solutions have the following obvious technical defects: (1) The technical solutions are highly invasive and have poor adaptability. In order to achieve effective monitoring, existing methods usually require a large amount of monitoring code (tracking) to be inserted into the source code of the application system, or require the application system to make architectural adjustments to adapt to the collection tools. This strong invasiveness not only increases the complexity and maintenance cost of the application, but also makes it difficult for such solutions to be quickly deployed and flexibly expanded in dynamic and heterogeneous modern application architectures such as microservices and cloud native.

[0005] (2) The data collection dimensions are limited and the accuracy is insufficient. Because existing methods rely on indirect inference, it is difficult to accurately capture the fine-grained interactive behaviors of users on the front-end page (such as clicks, jumps, and dwell time) and the complete execution chain of back-end business logic. As a result, the acquired data is too superficial and cannot form an accurate user behavior profile and system performance profile. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, the present invention provides an application behavior data collection method and system based on minimal intrusion, which can realize the collection of application behavior data in a minimally intrusive manner.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: The first aspect of this invention provides a method for collecting application behavior data based on minimizing intrusion.

[0008] A method for collecting application behavior data based on minimal intrusion includes: The application employs a painless probe injection method to automatically inject the SDK (front-end data collection tool) into the application front-end, collect user behavior data, and use an asynchronous reporter based on batch and timeout dual triggering to report the collected data. An extensible interceptor mechanism is built based on a pre-defined interceptor base class, and an observer interface is defined. The modified class is introduced into the bootstrapping area through the observer interface, and the Spring zone interceptor is injected into the application to collect behavioral data of the application's backend business logic. The collected data is stored using a MySQL database and a search engine; the MySQL database stores summary statistics and configuration data, while the search engine stores detailed log data; a periodic cleanup mechanism is also configured to process expired data. Establish a collaborative mechanism between asynchronous front-end reporting and real-time back-end preprocessing to achieve application behavior data collection in a manner that minimizes intrusion.

[0009] Furthermore, the injection process of the SDK (front-end data collection tool) includes: for page applications that are front-end and back-end separated and use nginx (front-end server), the functional modules of nginx (front-end server) are used to perform real-time filtering of HTML content before sending the response content to the browser, locate the head tags in the HTML and insert the SDK (front-end data collection tool) script.

[0010] Furthermore, during the collection of behavioral data of the application's backend business logic, the observer interface communicates with the bootstrap socket (communication channel) and securely injects bytecode modification logic into the target class; subsequently, the monitoring logic is injected into the application using the Spring Boot interceptor mechanism, achieving minimal intrusion into the business code.

[0011] Furthermore, the aforementioned periodic cleanup mechanism automatically cleans up expired data and archives historical data based on time cycles and backup strategies.

[0012] Furthermore, a collaborative mechanism is established between front-end asynchronous reporting and back-end real-time preprocessing, including: the front-end uses an asynchronous reporting device to trigger data reporting based on either a batch size threshold or a time window threshold, and performs final reporting at the end of the page lifecycle; the back-end verifies, cleans, enriches, and standardizes the reported data, and then selects and distributes the processed data to message middleware and storage applications.

[0013] Furthermore, an application behavior data collection method based on minimal intrusion also includes data transformation and business application steps, namely: converting the data collected from the original data layer into basic indicators and computational indicators; and performing statistical analysis on the indicators according to business needs to generate multi-dimensional data dashboards for application access, usage, performance, and anomaly monitoring.

[0014] Furthermore, the application behavior data collection method supports application data collection in both monolithic and microservice architectures.

[0015] A second aspect of the present invention provides an application behavior data acquisition system based on minimal intrusion.

[0016] An application behavior data acquisition system based on minimal intrusion includes: The front-end data acquisition module is configured to automatically inject the SDK (front-end acquisition tool) into the application front-end using a painless probe injection method, collect user behavior data, and report the collected data using an asynchronous reporter based on batch and timeout dual triggering. The backend data collection module is configured to: build an extensible interceptor mechanism based on a preset interceptor base class and define an observer interface; introduce the modified class into the bootstrapping area through the observer interface and inject the Spring zone interceptor into the application to collect behavioral data of the application's backend business logic; The data storage and query module is configured to use a MySQL database and a search engine to categorize and store the collected data. The MySQL database stores summary statistics and configuration data, while the search engine stores detailed log data. A periodic cleanup mechanism is also configured to handle expired data. The data flow closed-loop processing module is configured to establish a collaborative mechanism between front-end asynchronous reporting and back-end real-time preprocessing, thereby achieving the collection of application behavior data in a manner that minimizes intrusion. A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of an application behavior data acquisition method based on minimal intrusion as described in the first aspect of the present invention.

[0017] A fourth aspect of the present invention provides an electronic device including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of an application behavior data acquisition method based on minimal intrusion as described in the first aspect of the present invention.

[0018] The above one or more technical solutions have the following beneficial effects: (1) This invention employs a painless probe injection method, automatically injecting an SDK (front-end data collection tool) into the application front-end to collect user behavior data and report it asynchronously; and monitoring business logic in the back-end through a scalable interceptor mechanism. This series of technical means eliminates the need to modify the application system's source code or recompile or modify the business system at the architectural level during data collection. This greatly reduces the intrusion into existing application systems, simplifies deployment and maintenance processes, and enables the solution to seamlessly support rapid deployment and flexible expansion from traditional monolithic architectures to modern heterogeneous application architectures such as microservices and cloud-native.

[0019] (2) This invention establishes a collaborative mechanism between asynchronous front-end reporting and real-time back-end preprocessing to collect application behavior data in a minimally invasive manner. This allows for the accurate capture of complete behavior data, from user front-end interface interactions to back-end service calls, interface performance, database operations, and even distributed tracing. This collection method, covering the entire user operation lifecycle, breaks through the limitations of indirect inference methods in existing technologies, significantly enriching the acquired data in terms of dimension and granularity. Based on this high-quality first-hand data, accurate user behavior profiles and system performance profiles can be constructed, providing a solid data foundation for transparent operation, refined optimization, and value assessment of application systems.

[0020] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0021] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0022] Figure 1 This is a flowchart of an application behavior data collection method based on minimal intrusion in Embodiment 1 of the present invention. Detailed Implementation

[0023] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0024] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0025] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0026] Example 1 This embodiment discloses a method for collecting application behavior data based on minimizing intrusion.

[0027] like Figure 1 As shown, a method for collecting application behavior data based on minimal intrusion includes: Step S1: Using a painless probe injection method, automatically inject the SDK (front-end collection tool) into the application front end to collect user behavior data, and use an asynchronous reporter based on batch and timeout dual triggering to report the collected data. Step S2: Build an extensible interceptor mechanism based on the preset interceptor base class and define the observer interface; introduce the modified class into the bootstrapping area through the observer interface, and inject the Spring zone interceptor into the application to collect behavioral data of the application's backend business logic; Step S3: Use a MySQL database and a search engine to categorize and store the collected data; the MySQL database stores summary statistics and configuration data, and the search engine stores log details; at the same time, configure a regular cleanup mechanism to process expired data; Step S4: Establish a collaborative mechanism between front-end asynchronous reporting and back-end real-time preprocessing to achieve application behavior data collection in a manner that minimizes intrusion.

[0028] Based on the above process, this invention can achieve application behavior data collection in a minimally invasive (low-invasive or zero-invasive) manner. To facilitate understanding of the technical solution of this invention, the specific implementation methods of this invention will be further explained and described below.

[0029] In step S1, a painless probe injection method is used to automatically inject the SDK (front-end collection tool) into the application front end to collect user behavior data, and an asynchronous reporter based on batch and timeout dual triggering is used to report the collected data.

[0030] The front-end data collection process mainly includes three parts: front-end probe injection, data collection, and data reporting. Probe injection is implemented based on nginx (the front-end server) automatically importing images and using an automatic code replacement tool. Data collection includes automatic loading and validation of the SDK (front-end data collection tool), collecting information on page redirection, in-page clicks, page communication, and page anomalies. It can also collect specific types of information based on the tracking switch. Data reporting uses a standard data format extended from the zipkin format and then calls a unified asynchronous data reporting tool. Specifically, this can be implemented using the following methods: 1) Nginx (front-end server) automatically imports the image.

[0031] For web applications with a front-end / back-end separation architecture and using nginx (the front-end server), this solution automatically injects SDK (front-end data collection tool) content using dedicated string replacement directives within nginx (the front-end server), without modifying any front-end source code. The underlying technology involves nginx filtering the HTML content in real-time before sending the response to the browser and inserting the SDK script at the top of the page code. This process is entirely handled by the nginx layer and does not affect the back-end application logic. For virtual machine deployments, a pre-configured, lightweight nginx installation package is provided, using a version compatible with mainstream Linux distributions to ensure seamless integration with the system environment. For container deployments, a streamlined Linux operating system is provided, with the SDK path and injection location configured via environment variables to avoid conflicts with the original container environment. Deployment uses a separate configuration file, which is activated by nginx configuration directives, ensuring a smooth transition without affecting existing service configurations. Service interruption time is controlled to the second level, guaranteeing system stability and continuous data collection.

[0032] 2) The working mechanism of the automatic code replacement.

[0033] The core of achieving automatic front-end code injection in a standalone Java application lies in the intelligent working mechanism of the "code auto-replacer." This mechanism employs a precise replacement strategy based on tag matching for front-end file types such as HTML and JSP: it inserts the SDK script only at the top of the page code, strictly limiting the replacement scope to specified files under the configured publishing path to avoid affecting other code structures. The replacement is triggered by scheduled scanning or real-time file modification detection to ensure timely and accurate data collection. To ensure normal front-end functionality, the application system automatically checks the syntax compliance of the code after replacement, ensuring the correct display of page information.

[0034] 3) Automatic loading and verification of SDK (front-end data collection tool).

[0035] SDK loading is triggered by page initialization. The browser parses the HTML to... <script>标签时，会立即（阻塞）或异步发起网络请求获取并执行脚本。用户操作（如点击）可通过动态创建script元素的方式按需加载。

[0036] SDK的校验内容包括完整性校验和版本兼容性校验。其中，完整性校验主要是通过对比传输前后的前端采集工具脚本内容，以确保脚本未被篡改。版本兼容性可以通过读取浏览器的类型和版本号，来判断当前浏览器是否支持前端采集工具脚本所需的现代API或语法。

[0037] 在校验过程中，遵循如下容错机制：如果校验失败，则会阻止SDK（前端采集工具）脚本的继续执行，并触发一个error事件。由此，可以确保被采集系统功能不受影响。

[0038] 在步骤S2中，基于预设的拦截器基类构建可扩展的拦截器机制，并定义观察者接口；通过观察者接口将修改后的类引入引导区，并将spring区域拦截器注入应用，对应用后端业务逻辑的行为数据进行采集。

[0039] 后端监控端（javaagent端）采用纯JDK编写，不采用第三方控件，以免与被监控程序产生版本冲突。在此基础上，后端数据的采集主要包括：1）定义拦截器基类，描述拦截器修改位置、修改方法等信息；基于拦截器基类定义若干拦截器对应拦截不同的信息。2）定义观察者接口，并关联至引导区socket（通信通道）；实现若干观察者接口对应修改不同的类功能，并通过观察者将修改类引入引导区。3）通过springboot拦截器，将spring区域拦截器注入系统。4）通过包引入器，将spring扩展程序加入被监测程序。具体可以通过以下方法来实现：第一步，定义拦截器基类 (Interceptor Base Class) 。

[0040] 在后端采集中，拦截器基类是所有具体拦截器（如对 Spring MVC、HttpClient、JDBC 等的拦截）的抽象模板和统一契约。

[0041] 拦截器基类的核心属性，包括：1）methodName、className：用于标识被拦截的目标方法及其类。

[0042] 2）enhancedInstance：用于对被拦截目标对象实例的引用。探针会通过字节码增强技术将一些上下文信息注入到此对象中，以便在方法调用前后传递上下文。

[0043] 拦截器基类的核心方法 (生命周期钩子)，包括：1）beforeMethod(...)：在目标方法执行前被调用。此处通常用于获取方法参数、创建并启动一个本地Span、记录开始时间、将上下文信息存入 enhancedInstance（即内置缓存区中的临时数据库）等。

[0044] 2）afterMethod(...)：在目标方法执行后（包括正常返回和异常抛出）被调用。此处用于捕获返回值或异常、结束Span、上报数据到后端等。

[0045] 3）handleMethodException(...)：专用于处理目标方法执行过程中抛出的异常，确保异常信息能被捕获并记录到Span（即记录用户操作事件的最小单元，比如发起一次页面请求，查询一次数据库等）中。

[0046] 拦截器机制具有高度的可扩展性，其核心是 "基于约定的插件化” 架构，具体的：新增拦截器的接入，包括：首先，实现具体拦截器：开发者需要创建一个新的Java类，继承自抽象的拦截器基类。随后，重写生命周期方法：在新类中按需重写beforeMethod（即在用户操作被系统执行前，预设被执行的业务逻辑）、afterMethod（即在用户操作被系统执行后，预设被执行的业务逻辑）和handleMethodException（即在系统执行中出现异常后，预设被执行的业务逻辑）方法，填入针对特定组件（如一个新的RPC框架或数据库驱动）的埋点逻辑。最后，声明拦截点：在一个独立的配置文件中声明目标类、方法与新建拦截器的匹配关系。探针在启动时会加载此配置，并通过字节码增强技术自动将拦截器织入到目标方法中。

[0047] 在此基础上，完全不需要修改基类代码，开闭原则在此得到了完美体现，即对扩展开放，对修改关闭。其中，基类提供了稳定的、通用的接口和默认实现；要扩展对新库的支持，只需添加新的具体拦截器类和配置文件，而无需修改基类或探针核心代码。

[0048] 第二步，将观察者接口关联引导区socket（通信通道）。

[0049] 后端主要监测HTTP socket（通信通道），请求的核心原理是通过字节码增强技术拦截关键类库的网络操作。具体针对HTTP请求，探针会拦截JDK的核心类（如连接建立、请求发送的方法），在方法执行前后插入埋点逻辑。当应用程序发起HTTP调用时，探针会捕获请求信息（如URL、方法类型）、开始时间戳，并创建分布式追踪的上下文（Trace Context），通过请求头将追踪信息传递给下游服务，从而实现跨服务的链路追踪。

[0050] 为实现对这些核心类的拦截，探针采用"观察者与引导区交互”机制：探针通过HTTP协议建立通信通道：观察者（探针的拦截逻辑）识别到需要增强的类后，将类信息和原始字节码通过HTTP请求发送给引导区的服务端；服务端执行字节码修改（如插入埋点代码），并将修改后的字节码返回；最终重新定义类。

[0051] 关联后实现"修改类引入引导区”的本质是职责分离，即：探针仅负责识别和调度，实际修改操作在隔离的引导环境中完成。这种方式既规避了类加载器隔离问题，又确保了JVM核心类的稳定性和安全性，使得HTTP请求的监控对应用完全透明无侵入。

[0052] 第三步，将SpringBoot拦截器注入应用系统。

[0053] 探针对Spring Boot应用的监控主要通过字节码增强技术拦截Web请求和处理流程。其注入时机主要在应用启动阶段，当Spring Boot容器初始化时，探针在类加载前拦截关键的Spring Bean（如`DispatcherServlet`、Controller类及`RestTemplate`等组件），并植入埋点逻辑。这种启动时注入确保了监控代码在业务逻辑执行前就已就位，避免了运行时修改可能引发的稳定性问题。

[0054] 在兼容性处理上，采用高度透明的集成策略。探针严格遵循Spring Boot的编程模型，仅通过非侵入方式增强原有类的方法（如在请求处理前后插入切面），而非直接修改业务代码或Spring配置。它自动识别Spring Boot的版本和内置容器（如Tomcat），并适配其上下文传递机制，确保Trace信息在异步线程、过滤器链和控制器间正确传递，同时与原生的Spring AOP、Spring Cloud组件无缝协作，以避免双切面冲突或上下文丢失问题。

[0055] 为最小化性能损耗，采集器采用轻量级异步上报机制，即：埋点代码仅收集关键指标和上下文（如HTTP方法、URL和状态码），所有数据先存入内存队列，后由独立线程异步上报至OAP服务，以避免阻塞业务线程。同时，探针通过采样率控制和精简字节码增强范围（仅监控核心链路）减少开销，以保障生产环境的可用性。

[0056] 在步骤S3中，采用MySQL数据库与搜索引擎分类存储所采集的数据；其中，MySQL数据库存储汇总统计数据与配置数据，搜索引擎存储日志明细数据；同时，配置定期清理机制对过期数据进行处理。

[0057] 后台日志服务端采用微服务微应用建构，基于国网技术路线，通过平台提供的弹性扩展能力提高日志收集的并发性能。数据存储采用MySQL数据与ES分类存储，由MySQL端存储汇总统计数据与配置数据，ES端存储日志明细数据，并设计定期清理功能，以保证存储空间有效利用。

[0058] 在MySQL与搜索引擎分类存储的架构中，数据划分的核心标准是数据的查询模式、频率、量级及对事务性的要求，旨在将合适的存储引擎用于合适的场景，以实现系统性能、成本与功能的最优平衡。换言之，分类的核心依据是查询频率与数据量级。MySQL擅长处理低到中量级、需要事务和强一致性的"热”数据与"温”数据；而Elasticsearch则专为海量、高吞吐、弱事务要求的"温”到"冷”数据检索设计。此外，数据的生命周期也是重要依据：MySQL常存储在线业务所需的当前状态数据，而大量历史明细日志在Elasticsearch中存储一定周期后，通常会归档至更廉价的对象存储或数据仓库中进行离线分析，从而实现成本与效率的多级优化。具体划分标准如下：1）汇总统计数据存储在MySQL中。这类数据主要包括业务实体的核心状态、结果数据和低频修改的配置信息，例如用户pv、uv的最终状态、系统访问次数、系统参数表等；其中，pv表示用户访问量，uv表示访客数。其特点是数据量相对可控、结构规整、需要严格的ACID事务保证和复杂的关联查询（如多表JOIN）。MySQL的关系型模型和强大的事务能力非常适合此类场景。

[0059] 2）配置数据同样优先存入MySQL。它们通常是键值对或小型列表（如省市地区编码、业务类型枚举、开关配置），特点是数据量小、读多写极少、强一致性要求高，利用MySQL的主键查询效率极高，且易于管理。

[0060] 3）日志明细数据作为使用搜索引擎（如Elasticsearch）的核心场景。这类数据包括用户行为流水（点击、搜索、浏览）、API调用日志、系统监控事件等。其特点是数据量巨大（TB / PB级）、写入吞吐量极高、几乎无更新操作、查询模式复杂：需要按多个维度（如时间范围、用户ID、操作类型）进行快速过滤、聚合与分析。MySQL难以应对此类数据的海量存储和高效检索，而Elasticsearch的倒排索引和分布式架构天然支持高速、灵活的全文检索与聚合分析。

[0061] 后端的定期清理功能采用了一套兼顾数据可靠性、存储成本与查询连续性的自动化策略。其清理任务由触发条件启动：基于时间周期的定时任务，通常设置为每日凌晨一点低峰期执行；清理的范围聚焦于过期数据而非冗余数据。具体执行逻辑如下：应用系统直接清除九个月前的历史索引数据，但对于六个月前至九个月前这一时间段的数据，则采取更为审慎的处理流程。首先，应用系统会比对该时间段内所有已备份的快照（Snapshot）与ES中现存的实际索引，若发现ES中存在但快照中缺失的索引，则先为其创建快照进行备份，确保数据可追溯。待备份成功后，再执行删除操作，最终保证ES中仅保留最近六个月的在线热数据，而六个月至九个月的数据则归档至快照中，实现成本与可用性的平衡。

[0062] 在整个清理过程中，应用系统通过以下机制保障数据查询不中断：快照创建基于Elasticsearch的快照API实现，其过程为增量式且非阻塞，创建时索引仍可正常读写，对查询透明。删除操作则采用分批次策略，每次仅删除10个索引，避免一次性大量删除引发ES集群资源（如线程池、IO）竞争，从而防止删除操作对实时查询请求造成延迟或中断。这一精细化流程既实现了存储资源的有效回收，又确保了运维操作对查询服务的无感知。

[0063] 在步骤S4中，建立前端异步上报与后端实时预处理的协同机制，以最小化侵入的形式实现应用的行为数据采集。

[0064] 建立的前端异步上报与后端实时预处理的协同机制，包括：前端采用异步上报器，根据批量大小阈值及时间窗口阈值中的任意一种触发数据上报，并在页面生命周期结束时进行最终上报；后端对上报数据进行校验、清洗、丰富和标准化处理，并将处理后的数据选择分发至消息中间件和存储应用。该协同机制的实现可以用以下策略来实现：1、前端数据向后台传递策略。

[0065] 数据传输采用HTTP作为传输载体，其内容遵循定制的扩展标准数据格式（基于JSON）。这种协议与格式的适配关系是：HTTP协议提供了通用的、防火墙友好的传输通道，而内嵌的标准JSON数据格式则确保了数据的自描述性和可扩展性，使得后端OAP服务能够无损地解析出前端性能指标（如页面加载时序、AJAX调用耗时、用户行为事件、错误信息等）和分布式追踪上下文。

[0066] 异步上报器的工作机制是保障性能的关键。它作为一个独立的生产者-消费者系统运行：1）触发上报的时机主要由两个条件控制，以满足实时性与批量处理的平衡：一是批量大小阈值，当内存队列中累积的采集信息段数达到设定值（例如10条）时立即触发一次批量上报；二是时间窗口阈值，若在特定时间间隔（如5秒）内未达到批量条数，则超时后也会自动触发上报，防止数据长时间滞留。此外，在页面生命周期结束（如用户关闭页签或导航离开）时，会优先调用浏览器自带的数据发送接口，该接口能确保在页面卸载场景下可靠地发出HTTP请求，从而确保关键数据可以发送成功，且该接口不阻塞页面切换过程。

[0067] 2）数据缓存策略则侧重于鲁棒性与资源控制：采集线程产生的新数据首先被快速写入一个内存队列，该队列通常设有容量上限（如100条），以防止内存过度消耗。上报线程从队列中消费数据，进行批量合并与轻量压缩（如gzip）后发出。

[0068] 2、后台数据预处理策略。

[0069] 在后端服务接收上报的数据后，启动一个高效、多层次的实时预处理流水线。首先，进行数据校验与清洗：通过校验规则确保数据基本合法性，包括验证数据格式（JSONSchema）、检查必填字段（如`traceID`、`spanID`、`timestamp`）是否存在与类型正确、解析用户代理（UA）生成设备与浏览器信息、以及IP地址解析。随后，进入数据丰富与标准化阶段，对有效数据打上环境标签（如来源应用、版本号）、进行数据格式统一（如时间戳转换、数值归一化）、并衍生新维度字段。

[0070] 对于异常数据，应用系统采用多层过滤机制：除基础校验丢弃的无效数据外，还会识别并处理脏数据（如异常数值、乱码字段）。预处理后的洁净且增强的数据流会被发布到消息中间件datahub进行缓冲与解耦。

[0071] 最终，根据数据类型和用途进行分配：聚合性指标数据（如每日活跃用户数、事件PV / UV）会通过流处理引擎（Flink）进行实时窗口聚合后，批量写入MySQL等关系型数据库，供报表系统查询；而需要明细查询和深度分析的原始事件数据（如用户行为流水日志）则并行写入搜索引擎（Elasticsearch），以支持灵活的多维分析与快速检索。整个流程保障了数据的高可用、低延迟处理与高效存储。

[0072] 进一步地，一种基于最小化侵入的应用行为数据采集方法，还包括数据转换与业务应用步骤，该步骤主要由埋点数据模型来完成。该数据模型是处理前后端数据的完整流程，由埋点采集工具采集的前后端原始数据经过该流程处理后，由可读性差，无业务价值的原始数据（如时间戳、请求状态码等），变为易懂，能反映系统运行情况的业务指标（如系统访问量、功能活跃率、平均响应时长等）。埋点数据模型总共分为三层，第一层是原始数据层，通过前后端采集器采集基础数据与用户补充的数据。第二层是数据转换层，原始数据进入数据转换层后，部分经初步转换后成为基础指标，部分按指标计算公式计算后成为计算类指标。第三层是业务层，根据业务需求，将各类指标进行统计分析，形成数据看板，看板涉及系统访问情况、使用情况、性能情况3大方面，以及功能、页面、服务、接口、数据库、调用拓扑6个维度，全面展现系统应用运行数据。

[0073] 基于以上方法，本发明运用数据埋点、APM工具等多种技术，采用无痛探针引入的方式实现应用系统行为数据获取，除各类APP外，无需其他业务系统进行源代码、研发级别改动和重新编译发布，仅需技术人员调整原应用系统部署方式即可；且，同时支持传统单体架构和微服务微应用架构系统的数据采集，支持云上和云外不同部署方式的数据采集。

[0074] 实施例二本实施例公开了一种基于最小化侵入的应用行为数据采集系统。

[0075] 一种基于最小化侵入的应用行为数据采集系统，包括：前端数据采集模块，被配置为：采用无痛探针注入方式，在应用前端自动注入SDK（前端采集工具），采集用户行为数据，并采用基于批量与超时双重触发的异步上报器上报采集数据；后端数据采集模块，被配置为：基于预设的拦截器基类构建可扩展的拦截器机制，并定义观察者接口；通过观察者接口将修改后的类引入引导区，并将spring区域拦截器注入应用，对应用后端业务逻辑的行为数据进行采集；数据存储与查询模块，被配置为：采用MySQL数据库与搜索引擎分类存储所采集的数据；其中，MySQL数据库存储汇总统计数据与配置数据，搜索引擎存储日志明细数据；同时，配置定期清理机制对过期数据进行处理；数据流闭环处理模块，被配置为：建立前端异步上报与后端实时预处理的协同机制，以最小化侵入的形式实现应用的行为数据采集。实施例三本实施例的目的是提供计算机可读存储介质。

[0076] 计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现如本公开实施例一所述的一种基于最小化侵入的应用行为数据采集方法中的步骤。

[0077] 实施例四本实施例的目的是提供电子设备。

[0078] 电子设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的程序，所述处理器执行所述程序时实现如本公开实施例一所述的一种基于最小化侵入的应用行为数据采集方法中的步骤。

[0079] 以上实施例二、三和四的装置中涉及的各步骤与方法实施例一相对应，具体实施方式可参见实施例一的相关说明部分。术语"计算机可读存储介质”应该理解为包括一个或多个指令集的单个介质或多个介质；还应当被理解为包括任何介质，所述任何介质能够存储、编码或承载用于由处理器执行的指令集并使处理器执行本发明中的任一方法。

[0080] 本领域技术人员应该明白，上述本发明的各模块或各步骤可以用通用的计算机装置来实现，可选地，它们可以用计算装置可执行的程序代码来实现，从而，可以将它们存储在存储装置中由计算装置来执行，或者将它们分别制作成各个集成电路模块，或者将它们中的多个模块或步骤制作成单个集成电路模块来实现。本发明不限制于任何特定的硬件和软件的结合。

[0081] 上述虽然结合附图对本发明的具体实施方式进行了描述，但并非对本发明保护范围的限制，所属领域技术人员应该明白，在本发明的技术方案的基础上，本领域技术人员不需要付出创造性劳动即可做出的各种修改或变形仍在本发明的保护范围以内。< / script>

Claims

1. A method for collecting application behavior data based on minimal intrusion, characterized in that, include: The application employs a painless probe injection method to automatically inject the SDK into the application front end, collect user behavior data, and use an asynchronous reporter based on batch and timeout dual triggering to report the collected data. An extensible interceptor mechanism is built based on a pre-defined interceptor base class, and an observer interface is defined. The modified class is introduced into the bootstrapping area through the observer interface, and the Spring zone interceptor is injected into the application to collect behavioral data of the application's backend business logic. The collected data is stored using a MySQL database and a search engine; the MySQL database stores summary statistics and configuration data, while the search engine stores detailed log data; a periodic cleanup mechanism is also configured to process expired data. Establish a collaborative mechanism between asynchronous front-end reporting and real-time back-end preprocessing to achieve application behavior data collection in a manner that minimizes intrusion.

2. The application behavior data collection method based on minimal intrusion as described in claim 1, characterized in that, The SDK injection process includes: for front-end and back-end separated page applications using nginx, using nginx's functional modules to perform real-time filtering of HTML content before sending the response content to the browser, locating the head tags in the HTML and inserting the SDK script.

3. The application behavior data collection method based on minimal intrusion as described in claim 1, characterized in that, During the process of collecting behavioral data of the application's backend business logic, the observer interface communicates with the bootstrap socket and securely injects bytecode modification logic into the target class; subsequently, the monitoring logic is injected into the application using the Spring Boot interceptor mechanism, achieving minimal intrusion into the business code.

4. The application behavior data collection method based on minimal intrusion as described in claim 1, characterized in that, The aforementioned periodic cleanup mechanism automatically cleans up expired data and archives historical data based on time cycles and backup strategies.

5. The application behavior data collection method based on minimal intrusion as described in claim 1, characterized in that, Establish a collaborative mechanism between front-end asynchronous reporting and back-end real-time preprocessing, including: the front-end uses an asynchronous reporting device to trigger data reporting based on either a batch size threshold or a time window threshold, and performs a final reporting at the end of the page lifecycle; the back-end performs verification, cleaning, enrichment and standardization processing on the reported data, and selects to distribute the processed data to message middleware and storage applications.

6. The application behavior data collection method based on minimal intrusion as described in claim 1, characterized in that, It also includes data transformation and business application steps, namely: converting the data collected from the raw data layer into basic indicators and calculation indicators; and performing statistical analysis on the indicators according to business needs to generate multi-dimensional data dashboards for application access, usage, performance, and anomaly monitoring.

7. The application behavior data collection method based on minimal intrusion as described in claim 1, characterized in that, The application behavior data collection method supports application data collection in both monolithic and microservice architectures.

8. An application behavior data acquisition system based on minimal intrusion, characterized in that, include: The front-end data acquisition module is configured to: automatically inject the SDK into the application front-end using a painless probe injection method, collect user behavior data, and report the collected data using an asynchronous reporter based on batch and timeout dual triggering. The backend data collection module is configured to: build an extensible interceptor mechanism based on a preset interceptor base class and define an observer interface; introduce the modified class into the bootstrapping area through the observer interface and inject the Spring zone interceptor into the application to collect behavioral data of the application's backend business logic; The data storage and query module is configured to use a MySQL database and a search engine to categorize and store the collected data. The MySQL database stores summary statistics and configuration data, while the search engine stores detailed log data. A periodic cleanup mechanism is also configured to handle expired data. The data flow closed-loop processing module is configured to establish a collaborative mechanism between front-end asynchronous reporting and back-end real-time preprocessing, thereby achieving the collection of application behavior data in a manner that minimizes intrusion.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the application behavior data acquisition method based on minimal intrusion as described in any one of claims 1-7.

10. An electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the application behavior data acquisition method based on minimal intrusion as described in any one of claims 1-7.

Images