Log collection optimization method and system

Abstract

The present invention belongs to the field of data collection technology and discloses a log collection optimization method and system. The method includes: based on an established log collection probe component, monitoring the business call process in a SpringBoot application in real time, calculating the current sampling rate to dynamically adjust the current sampling rate, intercepting the business call layer corresponding to the current task according to customized interception rules to collect log collection objects and related parameters in the current task, forming a call location chain corresponding to the current task when stored in a database, and determining the corresponding call location chain to track the behavior information of a specified application or track risk data. The present invention can effectively capture detailed call data, reducing the complexity and cost of monitoring and debugging.

Description

Technical Field

[0001] The present invention relates to the field of data collection technology, and in particular to a log collection optimization method and system. Background Art

[0002] In modern software development, Spring Boot has become a mainstream framework due to its automatic configuration and rapid deployment. However, call process monitoring in a microservices architecture presents challenges: Traditional logging leads to data explosion (log volume increases exponentially under high concurrency). While APM tools (such as SkyWalking and Zipkin) support distributed tracing, they are resource-intensive (requiring independent servers and memory usage exceeding 1GB) and complex to configure (requiring application code modifications or relying on specific SDKs).

[0003] In addition, existing lightweight probes (such as Spring Boot Actuator) only provide basic health checks and lack detailed information about the call link (unable to capture parameters and return values); existing real-time monitoring tools are expensive and highly invasive.

[0004] Existing technologies also have the following issues: Logging limitations. In Spring Boot applications, developers typically add log statements to track method calls. However, this approach results in large and difficult-to-manage log files, especially in highly concurrent environments. Logging is often asynchronous and cannot reflect the current application state in real time. Breakpoint debugging can help developers understand the code execution flow, but it interrupts normal program execution, resulting in performance degradation and making it unsuitable for production environments. While APM systems provide powerful monitoring and analysis capabilities, they require complex configuration and maintenance, increasing development and operations costs.

[0005] Therefore, it is necessary to provide a new log collection optimization method and system. Summary of the Invention

[0006] To address the technical problems that existing lightweight probes (such as Spring Boot Actuator) only provide basic health checks and lack detailed information about the call chain, resulting in an inability to find problematic data and identify the problem link in a timely manner when problems occur, and existing real-time monitoring tools are expensive and highly invasive. The technical problems to be solved by the present invention are achieved through the following technical solutions.

[0007] The first aspect of the present invention proposes a log collection optimization method, which monitors the business call process in the SpringBoot application in real time based on the established log collection probe component, and specifically establishes a log collection probe component based on the Java agent function; calculates the current sampling rate to dynamically adjust the current sampling rate; based on the dynamically adjusted sampling rate, collects the log collection objects and related parameters in the current task according to the current task triggered by the user operation. During the collection process, the business call layer corresponding to the current task is intercepted according to the customized interception rules to collect the log collection objects and related parameters in the current task, so as to form a call location chain corresponding to the current task when storing in the database; when a log collection request is monitored, the log collection probe component is triggered to perform relevant log collection, and the corresponding call location chain is determined to track the behavior information of the specified application or track risk data.

[0008] The second aspect of the present invention proposes a log collection optimization system, which executes the log collection optimization method described in the first aspect of the present invention. The log collection optimization system includes: a monitoring module, which is used to monitor the business call process in the SpringBoot application in real time based on the established log collection probe component, and specifically establishes a log collection probe component based on the Java agent function; a calculation and determination module, which is used to calculate the current sampling rate to dynamically adjust the current sampling rate; a collection module, which is used to collect the log collection objects and related parameters in the current task based on the dynamically adjusted sampling rate and the current task triggered by the user operation. During the collection process, the business call layer corresponding to the current task is intercepted according to the customized interception rules to collect the log collection objects and related parameters in the current task, so as to form a call location chain corresponding to the current task when storing in the database; a tracking and processing module, which triggers the log collection probe component to perform relevant log collection when a log collection request is monitored, and determines the corresponding call location chain to track the behavior information of the specified application or track risk data.

[0009] The third aspect of the present invention provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the log collection optimization method described in the first aspect of the present invention.

[0010] A fourth aspect of the present invention provides a computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the log collection optimization method described in the first aspect of the present invention is implemented.

[0011] The embodiments of the present invention include the following advantages:

[0012] Compared with the prior art, the present invention is based on the established log collection probe component, which monitors the business call process in the SpringBoot application in real time. Specifically, the log collection probe component is established based on the Java agent function, and the current sampling rate is calculated to dynamically adjust the current sampling rate; based on the dynamically adjusted sampling rate, the log collection object and its related parameters in the current task are collected according to the current task triggered by the user operation. During the collection process, the business call layer corresponding to the current task is intercepted according to the customized interception rule to collect the log collection object and its related parameters in the current task, so as to form a call location chain corresponding to the current task when the database is stored. When a log collection request is monitored, the log collection probe component is triggered to perform relevant log collection, and the corresponding call location chain is determined to track the behavior information of the specified application or track risk data. The present invention can realize real-time monitoring of the SpringBoot application call process without relying on heavy logging or breakpoint debugging that interrupts program execution. By providing a flexible probe component that can be dynamically inserted into key interception points of the application as needed to capture detailed call data, including caller, callee, parameter list, return value, and execution time, it can improve the stability and maintainability of the application while reducing the complexity and cost of monitoring and debugging. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a flow chart of an example of the log collection optimization method of the present invention;

[0014] Figure 2 This is a partial flow diagram of starting the log collection probe component in the log collection optimization method of the present invention;

[0015] Figure 3 It is a structural block diagram of the log collection optimization system of the present invention;

[0016] Figure 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention;

[0017] Figure 5 is a schematic structural diagram of an embodiment of a computer-readable medium according to the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and beneficial effects of the present invention more clearly understood, the method and device provided by the present invention are described in detail below in conjunction with specific processes and examples. It should be understood that the following content is only a preferred embodiment of the present invention, and any modifications or substitutions made within the spirit and essential principles of the present invention shall be included in the scope of protection of the present invention.

[0019] The present invention provides a log collection optimization method. The method monitors the business call process in the SpringBoot application in real time based on the established log collection probe component. Specifically, the log collection probe component is established based on the Java proxy function. The current sampling rate is calculated to dynamically adjust the current sampling rate. Based on the dynamically adjusted sampling rate, the log collection object and its related parameters in the current task are collected according to the current task triggered by the user operation. During the collection process, the business call layer corresponding to the current task is intercepted according to the customized interception rule to collect the log collection object and its related parameters in the current task, so as to form a call location chain corresponding to the current task when the database is stored. When a log collection request is monitored, the log collection probe component is triggered to perform relevant log collection, and the corresponding call location chain is determined to track the behavior information of the specified application or track risk data. The present invention can realize real-time monitoring of the SpringBoot application call process without relying on heavy logging or breakpoint debugging that interrupts program execution. By providing a flexible probe component that can be dynamically inserted into key interception points of the application as needed to capture detailed call data, including caller, callee, parameter list, return value, and execution time, it can improve the stability and maintainability of the application while reducing the complexity and cost of monitoring and debugging.

[0020] It should be noted that the invention of the present invention has a wide range of applications, and is particularly suitable for traffic management systems, which are used to manage business data related to traffic services such as international traffic, domestic traffic, and directional traffic.

[0021] Example 1

[0022] Refer to the following Figure 1 、 Figure 2 , the contents of the present invention will be described in detail.

[0023] Figure 1 This is a flowchart of an example of the log collection optimization method of the present invention. Figure 2 It is a partial flow diagram of starting the log collection probe component in the log collection optimization method of the present invention.

[0024] Reference Figure 1 and Figure 2 In step S101, based on the established log collection probe component, the business call process in the SpringBoot application is monitored in real time. Specifically, the log collection probe component is established based on the Java agent function.

[0025] Specifically, the system collects business data related to traffic services such as backward traffic, international traffic, domestic traffic, and directional traffic in SpringBoot applications, and monitors the business call processes related to these traffic services. These business call processes include processes related to user requests, the corresponding response information, the called call layer, and the called business methods. For example, business methods include methods corresponding to different classes such as Filter, Controller, Service, and DAO. For example, non-business methods include Utilities.

[0026] More specifically, log collection targets include business method logs and non-business method logs related to traffic services such as backward traffic, international traffic, domestic traffic, and directional traffic. Business method logs are logs generated during the access of business methods in classes such as Controller, Service, Filter, and DAO that participate in the business interaction process (at different call layers, with each class corresponding to at least one call layer). Non-business method logs are logs generated by methods that assist in implementing specific business operations within business methods.

[0027] In a specific implementation, a log collection probe component is established based on the Java agent function, and specifically ByteBuddy is used as the Java library.

[0028] Use the log collection probe component to obtain request and response information from Spring Boot applications. For business method logs, this includes the method's input and output parameters, header information related to user requests and responses, body information, and method execution duration. For non-business method logs, this includes the call's input and output parameters, and the call time.

[0029] By providing a flexible probe component, namely the log collection probe component, it can be dynamically inserted into the key interception points of the application as needed to capture detailed call data.

[0030] Specifically, the "key interception point" refers to the core anchor position in the application execution process where the monitoring logic can be injected, including: dynamically identifying business-sensitive nodes (such as business-sensitive nodes related to backward traffic) by parsing configuration files based on the real-time configuration loading capability of the probe component at runtime.

[0031] The "key interception points" also include bytecode-level instrumentation, which specifically utilizes the bytecode weaving function to dynamically modify the bytecode during the class loading phase and insert collection logic before and after the execution of the called method.

[0032] Establish customized interception strategies for different business types, including those related to backward traffic (e.g., anchoring the payment callback handler), international traffic, and directional traffic. For backward traffic, for example, use the processing method of the backward callback controller (BackwardCallbackController.process()) to anchor the payment callback handler. For international traffic, at the order validation service layer, for example, use the external channel API gateway entry, specifically OrderValidationService.checkQuota(), to receive a unified gateway entry class and method for API requests from external channels (e.g., short video or shopping service platforms).

[0033] For directional traffic services, for example, using external channel API network management entrances (such as DouyinOrderGateway.

[0034] receiveRequest()), to receive external API requests.

[0035] The entrance of each traffic service is monitored through multi-dimensional universal monitoring trigger points, including HTTP request controller (control layer method entrance), service layer method boundary (service layer method execution point), database access entrance (database operation interception point), etc.

[0036] It should be noted that ByteBuddy is a powerful Java library for dynamically generating and modifying bytecode at runtime. By using ByteBuddy, custom logic can be injected into the target class without modifying the existing code, thereby enabling monitoring of the SpringBoot application call process. In addition, the real-time monitoring of the business call process in the SpringBoot application by the log collection probe component also relies on message middleware (such as a kafka database), a Java programming auxiliary library (such as Lombok), and a configuration file reading module (such as SNakeYaml, used to read yml or yaml configuration files). The above is only explained as an optional example and should not be understood as a limitation of the present invention.

[0037] Different log collection strategies are used for different log collection objects, and the log collection strategies include real-time calculated sampling rates. The sampling calculation process will be described in detail below in conjunction with step S102.

[0038] Next, in step S102 , the current sampling rate is calculated to dynamically adjust the current sampling rate.

[0039] Based on the configuration file loaded when the log collection probe component is started, the key interception points related to the business call process in the SpringBoot application are identified. The dynamic injection process of the log collection probe component includes bytecode-level instrumentation, which specifically utilizes the bytecode weaving function to dynamically modify the bytecode during the class loading phase and insert collection logic before and after the execution of the called method.

[0040] Furthermore, the dynamic injection process includes configuration-driven hot updates. Based on configuration file rules read during startup and runtime, interceptor instances matching the current business scenario are instantly generated and injected, without requiring an application restart. Next, interception points are dynamically bound. Using class loader isolation, the call path to be intercepted is located within the method execution stack, dynamically anchoring the collection logic to control layer entry points, service layer method boundaries, and database operation interfaces.

[0041] When fluctuations in CPU usage or changes in historical error rates are detected, the collection density is adjusted in real time by reloading the lightweight interceptor instance to ensure that the monitoring granularity is dynamically adapted to the system load. Specifically, when the CPU usage is monitored to exceed the preset baseline (for example, greater than or equal to 75%), when the historical risk score is greater than or equal to the preset threshold (for example, 0.8, 0.85 or 0.9), or when the business scenario type is switched, the current collection rate is calculated in real time to adjust the current collection rate so that the current task is dynamically adapted to the resource load. The dynamic injection process ensures that the collection logic is decoupled from the business code, and realizes zero-intrusive log collection through dynamic weaving at the bytecode instruction set level. Therefore, by coordinating the sampling rate linkage adjustment, log collection for various business scenarios can be achieved more efficiently.

[0042] For business scenario type switching, for example, when a context change event is detected in which the business scenario type switches between a backward traffic scenario, an international traffic scenario, and a directional traffic scenario, the current sampling rate is automatically triggered to be calculated.

[0043] When the log collection probe component monitors a new request entering the business call process, or when the system operation status indicator reaches a preset threshold condition (for example, the CPU usage exceeds the preset baseline), the current sampling rate is triggered to be calculated through the request entry or threshold detection port.

[0044] In a specific embodiment, when the CPU usage of the traffic management system exceeds a preset baseline (e.g., greater than or equal to 75%), the current sampling rate is calculated in real time. Specifically, the following expression is used to calculate the current sampling rate:

[0045] F s =F×log 10 (L cpu )×(1+S fr );

[0046] Among them, F s Indicates the current sampling rate calculated based on the CPU usage and historical error rate of the current Java virtual machine; F indicates the basic sampling rate; L cpu Indicates the CPU usage of the current Java virtual machine, log 10 (L cpu ) represents the logarithmic smoothing value of the CPU usage of the current Java virtual machine with a base of 10, which is used to compress the numerical fluctuation of the CPU usage; S fr A historical risk score representing historical abnormal signals, used to quantify the health of the traffic management system, calculated based on the historical error rate within a time window.

[0047] By calculating the current sampling rate in real time and dynamically adjusting it, we can effectively balance the amount of collected data with the diagnostic data requirements. Adaptive sampling rate adjustment is achieved by dynamically coupling system status with historical abnormal signals.

[0048] By CPU usage L cpu Real-time JVM load sensing automatically reduces the sampling rate during high load conditions to alleviate system pressure and balance resource usage. High load refers to CPU usage exceeding 75% during peak monthly subscription periods. By coordinating sampling rate adjustments, during periods of high traffic management system load (such as peak monthly subscription periods), the sampling rate is lowered to reduce the number of messages in the message queue, alleviating log storage processing pressure and enabling more efficient log collection for various business scenarios.

[0049] It should be noted that in this example, the core technical role of the Java Virtual Machine (JVM) lies in its role as the runtime for the log collection probe component and the central control hub for dynamic monitoring. The JVM exposes real-time system operational status metrics, including current CPU usage, memory stack status, and thread activity data, through built-in management interfaces. These system operational status metrics form the primary input source for dynamic sampling rate calculation. Specifically, the JVM's bytecode instrumentation mechanism provides fundamental support for bytecode enhancement. By intercepting target business classes (such as the payment controller) during the class loading phase and inserting collection logic at method execution boundaries, this achieves non-intrusive monitoring. This bytecode instrumentation mechanism also supports runtime redefinition of the log collection probe component. Furthermore, the JVM's resource awareness plays a crucial role in sampling rate regulation. When CPU usage exceeds a preset threshold (e.g., ≥75%), a downsampling mechanism is automatically triggered. By reducing the log collection density of non-core business methods, the load on the collection message consumption system is reduced, allowing the sampling rate to dynamically adapt to business scenario requirements. By tightly coupling JVM resource status with the sampling rate adjustment logic, a dynamic closed-loop control system is formed, reducing monitoring costs while ensuring visibility of critical business links.

[0050] For example, when abnormal signals in data collection occur frequently, the sampling rate can be increased to capture more diagnostic data.

[0051] The basic sampling rate F refers to the default sampling rate in the system steady state. It needs to be set in detail based on the business scenario. For details, please refer to Table 1 below:

[0052] Table 1

[0053]

[0054] Table 1 is an example table of basic sampling rate settings for different business scenario types.

[0055] For setting principles, refer to the Nyquist criterion. For example, the value of F should be at least twice the call frequency of specific business methods. The specific business methods include those related to specific businesses such as purchasing international traffic and exchanging domestic traffic.

[0056] In addition, the upper limit is inferred based on the throughput capacity of downstream storage (specifically the database storing data, such as the Kafka cluster) to constrain storage costs. For example, when the throughput of a single node is 10MB / s and a single log is 1KB, then F max Less than 10,000 records per second.

[0057] For CPU usage, for example, the ManagementFactory#getPlatformMXBean method in the JVM can be used to directly obtain the specified application object, and then the getSystemCpuLoad method can be used to obtain the CPU usage of the specified application object (value range: [0-1]).

[0058] In the case of multiple sampling, instantaneous jitter is avoided by averaging (for example, taking 3 times in 1 second).

[0059] It should be noted that in this example, the JVM process CPU and the host CPU need to be distinguished in the container environment.

[0060] The difference between using logarithmic scale and linear scale in the above expressions is shown in Table 2 below.

[0061] Table 2

[0062]

[0063] Table 2 is an example table of the difference parameters and effects of using logarithmic scale and linear scale in the above expressions.

[0064] By using a logarithmic function to represent CPU usage and calculate the sampling rate, we can effectively compress the high-value range of the sampling rate, making the gradient slower in the high CPU range (such as 0.8-1.0), preventing a cliff drop in the sampling rate. We can also enhance low-value sensitivity, making the gradient steeper in the high and low CPU ranges (such as 0.1-0.3), allowing the sampling rate to be quickly increased when resources are abundant.

[0065] For the historical risk score S fr (also known as the historical error rate), is calculated based on the historical error rate within the time window using the following expression:

[0066] ;

[0067] Among them, S fr The historical risk score of historical abnormal signals is obtained based on the historical error rate calculated as the ratio of abnormal business requests to the total number of business requests within the time window. It is used to quantify the health of the traffic management system. It is calculated based on the historical error rate within a configurable sliding time window (for example, within 10 minutes). Specifically, it refers to the ratio of failed business requests to the total number of business requests. recent Indicates the number of abnormal requests in the most recent time window (such as 5 minutes); E base The value represents the average number of abnormal requests in a baseline window (e.g., one hour); δ represents the standard deviation of the number of abnormal requests, reflecting volatility; α represents the first weighting factor, and β represents the second weighting factor. For example, we use a rolling time window (default 10 minutes) to continuously collect statistics and refresh data samples every 30 seconds to calculate the historical error rate.

[0068] Optionally, α=0.7, β=0.3.

[0069] It should be noted that business request failures are classified as HTTP-layer failures, where a server-side error with a response status code greater than or equal to 500 (e.g., an interface call timeout) is identified. System control-layer failures are those where a method fails to return normally, triggering a method exception (e.g., a payment order does not exist, resulting in a data update failure and a business exception being thrown). These failures also include failures in underlying components and runtime exceptions (e.g., database connection timeouts and message queue delivery failures).

[0070] Specifically, when the short video channel ordering error rate (corresponding to the historical error rate) of the targeted traffic business is monitored to increase, the sampling rate will be automatically increased to the preset baseline to ensure that the risk positioning call chain is fully captured.

[0071] It should be noted that the above is merely provided as an optional example and should not be construed as a limitation to the present invention.

[0072] Next, in step S103, based on the dynamically adjusted sampling rate, according to the current task triggered by the user operation, the log collection objects and related parameters in the current task are collected. During the collection process, the business call layer corresponding to the current task is intercepted according to the customized interception rules to collect the log collection objects and related parameters in the current task, so as to form a call location chain corresponding to the current task when storing in the database.

[0073] Specifically, according to the sampling rate dynamically adjusted in step S102, the log collection objects and related parameters in the current task are collected according to the current task triggered by the user operation. The current task is, for example, a directional traffic collection task.

[0074] To start collecting business data, you need to start the log collection probe component.

[0075] For starting the log collection probe component, there are two ways to start the Java Web service: basic mode startup and container mode startup.

[0076] For example, you can use the java command to directly start a web service for projects such as SpringBoot and SpringCloud. When starting the log collection probe component, you must add the -javaagent parameter before the JAR file to be started. For example, run the command java -javaagent: / path / to / agent.jar -jar / path / to / service.jar.

[0077] For example, when starting a container service, the container has a built-in method for starting a web service, which requires additional parameter configuration for use in projects such as SpringMVC and JBoss. For example, you can specify probe startup parameters in the configuration file of a container service such as Tomcat or WebLogic.

[0078] For Tomcat container services, modify catalina.sh (Linux environment) or catalina.bat (Windows environment) and add "-javaagent: / path / to / agent.jar" to the JAVA_OPTS environment variable.

[0079] For WebLogic container services, for example, modify startWeblogic.sh (Linux environment) or startWeblogic.bat (Windows environment) and add "-javaagent: / path / to / agent.jar" to the beginning of the JAVA_OPTIONS environment variable.

[0080] The starting of the log collection probe component includes the following steps:

[0081] Step S201: Use the premian method in the Agent class as the probe entry.

[0082] Step S202: Use maven-shaded-plugin in pom.xml of packaging tool Maven to perform packaging operation.

[0083] Specifically, you need to define manifest-related information and specify the startup method to the premain method in the Agent class.

[0084] Step S203: After the log collection probe component is started, the configuration file path set by the environment variable is used to read the configuration file content to initialize the interceptor and the message queue production end according to the configuration information content.

[0085] Specifically, the configuration file content includes probe interception rule configuration information and message queue configuration information.

[0086] During the collection process, the service call layer corresponding to the current task is intercepted according to customized interception rules to collect the log collection objects and their related parameters. The collection objects are the business data generated by the specified user group and their purchase of targeted traffic (for example, traffic usage for a video playback app). The related parameters include the call layer invoked, the input and output parameters of the call, etc.

[0087] The customized interception rules include initializing customized interceptors for each call layer according to the contents of the configuration file. In one embodiment, the first interceptor for the first call layer (e.g., the Controller interceptor) is initialized, the second interceptor for the second call layer (e.g., the Service interceptor) is initialized, the third interceptor for the third call layer (e.g., the DAO interceptor) is initialized, and the fourth interceptor for the fourth call layer (e.g., the Default interceptor) is initialized.

[0088] Specifically, the first interceptor is used to intercept Controller classes in the Spring architecture. The second interceptor is used to intercept Service classes in the Spring architecture. The third interceptor is used to intercept DAO classes in the Spring architecture. The fourth interceptor is used to intercept other utility classes outside the Spring architecture. Extensible interceptors are used for static class interfaces and can be extended to implement other interception services.

[0089] Furthermore, a call location chain corresponding to the current task is formed when the database is stored.

[0090] It should be noted that in the present invention, the log collection probe component can be started or shut down as needed, without burdening the production environment. Furthermore, the data collected by the log collection probe component can be directly used to generate a visual call chain diagram to help developers quickly locate problems. Furthermore, the log collection probe component can be integrated with other systems, such as distributed tracing systems, to provide more in-depth performance analysis and fault diagnosis capabilities. The above is provided only as an optional example and should not be construed as a limitation of the present invention.

[0091] In one specific implementation, for example, to collect business data for user a in a specified user group, the collected log data includes calls to business method AA at the first call layer, followed by calls to business method BD at the second call layer, business method CC at the third call layer, and data from cluster G3 in the database. Thus, the following call location chain can be imagined: User a - Controller business method AA - Service business method BD - DAO business method CC - G3.

[0092] It should be noted that the above is merely provided as an optional example and should not be construed as a limitation to the present invention.

[0093] Next, in step S104, when a log collection request is detected, the log collection probe component is triggered to collect relevant logs and determine the corresponding call location chain to track the behavior information of the specified application or track risk data.

[0094] When a log collection request is detected, the log collection probe component is triggered to collect relevant logs and determine the corresponding call location chain to track the behavior information of the specified application or track risk data.

[0095] For example, when a log collection request is detected, the gateway service receives the log collection request, calls the corresponding business port, and generates a log tracking identifier (i.e., TraceID).

[0096] For the log collection of the gateway service, a TraceID is generated at the gateway service entrance. The collection example is as follows:

[0097] {

[0098] "timestamp": "2025-03-12T14:30:22.123Z",

[0099] "service": "gateway-service",

[0100] "class-method": "com.gateway.Router#route",

[0101] "trace_id": "7a3b5c8d9e0f1a2b", / / Newly generated TraceID

[0102] "parent_span_id": null, / / The root node has no parent Span

[0103] "span_id": "7a3b5c8d9e0f1a2b", / / Root SpanID=TraceID

[0104] "input_parameters": {"path": " / order / create", "method": "POST"},

[0105] "output_parameters": {"status": "ROUTED"},

[0106] "request": {"headers": {"X-User-Id": "U12345"}},

[0107] "response": {"target_service": "order-service"},

[0108] "status_code": 200,

[0109] "duration": 12,

[0110] "event_type": "ENTRY_CALL" / / Mark entry call

[0111] }.

[0112] For example, the log collection example of the order service is as follows: Receive TraceID and generate sub-Span

[0113] {

[0114] "timestamp": "2025-03-12T14:30:22.456Z",

[0115] "service": "order-service",

[0116] "class-method": "com.order.OrderService#createOrder",

[0117] "trace_id": "7a3b5c8d9e0f1a2b", / / Inherit the TraceID of the gateway

[0118] "parent_span_id": "7a3b5c8d9e0f1a2b", / / Parent Span = Gateway's SpanID

[0119] "span_id": "d4e5f6a7b8c9d0e1", / / Generate a new sub-SpanID

[0120] "input_parameters": {"userId": "U12345", "sku": "1001", "amount": 2},

[0121] "output_parameters": {"orderId": "ORD-20250605-001"},

[0122] "request": {"body": "encrypted ciphertext"},

[0123] "response": {"result": "SUCCESS"},

[0124] "status_code": 200,

[0125] "duration": 145,

[0126] "event_type": "BIZ_PROCESS" / / Marks the core business logic

[0127] }.

[0128] For another example, the log collection example for the Kuhu service (a downstream service of the Order service) is as follows:

[0129] {

[0130] "timestamp": "2025-03-12T14:30:22.789Z",

[0131] "service": "stock-service",

[0132] "class-method": "com.stock.StockService#deduct",

[0133] "trace_id": "7a3b5c8d9e0f1a2b", / / Keep the global TraceID

[0134] "parent_span_id": "d4e5f6a7b8c9d0e1", / / Parent Span = SpanID of order service

[0135] "span_id": "3f4g5h6i7j8k9l0", / / Generate a new sub-SpanID

[0136] "input_parameters": {"sku": "1001", "quantity": 2},

[0137] "output_parameters": {"remaining": 98},

[0138] "request": {"source": "order-service"},

[0139] "response": {"code": "STOCK-200", "message": "OK"},

[0140] "status_code": 200,

[0141] "duration": 42,

[0142] "event_type": "DB_UPDATE" / / Mark database operation

[0143] }.

[0144] For the different log collection objects and their related parameters, a standardized and structured JSON format collection log is formed, which specifically includes the following fixed fields: collection time point (for example, represented by timestamp), service name (for example, represented by service), collection object class and method (for example, represented by class-method and separated by "#"), log tracking identifier (for example, represented by trace_id), parent log ID (for example, represented by parent_span_id), log ID (for example, represented by span_id), input parameter list (for example, represented by input_parameters), meal output list (for example, represented by output_parameters), request object (for example, represented by request), response object (for example, represented by response), response status code (for example, represented by status_code), method execution time (for example, represented by duration), and log type (for example, represented by event_type).

[0145] The transfer field rules of the call location chain include a trace ID (TraceID) transfer mechanism and a sub-service ID (SpanID) generation rule.

[0146] The specific mechanism for delivering a trace ID (TraceID) is to generate it from the entry service and pass it downstream via a request header (such as X-B3-TraceId). Bytecode instrumentation automatically injects the request header into calls between microservices, eliminating the need for manual coding. Subservice ID (SpanID) generation rules include the root service and subservices. Subservice IDs correspond to trace IDs, generating independent subservice IDs (SpanIDs). These are then linked to the upstream subservice ID (SpanID) through the parent subservice ID (parent_span_id).

[0147] In addition, for log information, it is necessary to automatically mask fields such as ID cards and mobile phone numbers in input and output parameters and request and response objects to desensitize sensitive data.

[0148] It should be noted that the above is merely provided as an optional example and should not be construed as a limitation to the present invention.

[0149] Compared with the prior art, the present invention is based on the established log collection probe component, which monitors the business call process in the SpringBoot application in real time. Specifically, the log collection probe component is established based on the Java agent function. By calculating the current sampling rate, different business scenario types are determined to dynamically adjust the current sampling rate; based on the dynamically adjusted sampling rate, according to the current task triggered by the user operation, the log collection object and its related parameters in the current task are collected. During the collection process, the business call layer corresponding to the current task is intercepted according to the customized interception rule to collect the log collection object and its related parameters in the current task, so as to form a call location chain corresponding to the current task when the database is stored. When a log collection request is monitored, the log collection probe component is triggered to perform relevant log collection, and the corresponding call location chain is determined to track the behavior information of the specified application or track risk data. The present invention can realize real-time monitoring of the SpringBoot application call process without relying on heavy logging or breakpoint debugging that interrupts program execution. By providing a flexible probe component that can be dynamically inserted into key interception points of the application as needed to capture detailed call data, including caller, callee, parameter list, return value, and execution time, it can improve the stability and maintainability of the application while reducing the complexity and cost of monitoring and debugging.

[0150] Example 2

[0151] The following are system embodiments of the present invention, which can be used to implement the method embodiments of the present invention. For details not disclosed in the system embodiments of the present invention, please refer to the method embodiments of the present invention.

[0152] Figure 3 This is a schematic diagram of an example of a log collection optimization system according to the present invention. Figure 3 , a log collection optimization system 300 is described, and the log collection optimization system 300 executes the log collection optimization method described in Example 1 of the present invention.

[0153] The log collection optimization system 300 includes a monitoring module 310 , a calculation and determination module 320 , a collection module 330 , and a tracking and processing module 340 .

[0154] In a specific embodiment, the monitoring module 310 is used to monitor the business call process in the SpringBoot application in real time based on the established log collection probe component, and specifically establishes the log collection probe component based on the Java agent function. The calculation and determination module 320 is used to calculate the current sampling rate to dynamically adjust the current sampling rate. The collection module 330 is used to collect the log collection objects and related parameters in the current task based on the current task triggered by the user operation based on the dynamically adjusted sampling rate. During the collection process, the business call layer corresponding to the current task is intercepted according to the customized interception rules to collect the log collection objects and related parameters in the current task, so as to form a call location chain corresponding to the current task when storing in the database. When the tracking processing module 340 monitors the log collection request, it triggers the log collection probe component to perform relevant log collection and determines the corresponding call location chain to track the behavior information of the specified application or track risk data.

[0155] According to an optional implementation, starting the log collection probe component includes the following steps:

[0156] Step S201: Use the premian method in the Agent class as the probe entry;

[0157] Step S202: Use maven-shaded-plugin in pom.xml of packaging tool Maven to perform packaging operation;

[0158] Step S203: After the log collection probe component is started, the configuration file path set by the environment variable is used to read the configuration file content to initialize the interceptor and the message queue production end according to the configuration information content.

[0159] Specifically, the following expression is used to calculate the current sampling rate:

[0160] F s =F×log 10 (L cpu )×(1+S fr );

[0161] Among them, F s Indicates the current sampling rate calculated based on the CPU usage and historical error rate of the current Java virtual machine; F indicates the basic sampling rate; L cpu Indicates the CPU usage of the current Java virtual machine, log 10 (L cpu ) represents the logarithmic smoothing value of the CPU usage of the current Java virtual machine with a base of 10, which is used to compress the numerical fluctuation of the CPU usage; S frA historical risk score representing historical abnormal signals, used to quantify the health of the traffic management system, calculated based on the historical error rate within a time window.

[0162] For the historical risk score S fr , based on the error rate within the time window, is calculated using the following expression:

[0163] ;

[0164] Among them, S fr The historical risk score of historical abnormal signals, that is, the historical error rate obtained based on the ratio of abnormal business requests to the total number of business requests within the time window, is used to quantify the health of the traffic management system; E recent Indicates the number of abnormal requests in the time window closest to the current time window; E base represents the average number of abnormal requests in the baseline window; δ represents the standard deviation of the number of abnormal requests, which is used to reflect volatility; α represents the first weight coefficient, and β represents the second weight coefficient.

[0165] According to an optional implementation, when the CPU usage is monitored to exceed a preset baseline, when the historical risk score is greater than or equal to a preset threshold, or when the business scenario type is switched, the current acquisition rate is calculated in real time to adjust the current acquisition rate so that the current task is dynamically adapted to the resource load.

[0166] According to an optional implementation method, based on the configuration file loaded when the log collection probe component is started, the key interception points related to the business call process are identified, and the dynamic injection process of the log collection probe component includes bytecode-level instrumentation, which specifically utilizes the bytecode weaving function to dynamically modify the bytecode during the class loading phase, and insert the collection logic before and after the execution of the called method.

[0167] According to an optional implementation, when the log collection probe component monitors a new request entering the business call process, or when the system operation status indicator reaches a preset threshold condition, the current sampling rate is triggered to be calculated through the request entry or threshold detection port.

[0168] It should be noted that due to Figure 3 The log collection optimization method implemented by the log collection optimization system is Figure 1 The log collection optimization methods in the examples are roughly the same, so the descriptions of the common parts are omitted.

[0169] Compared with the prior art, the present invention is based on the established log collection probe component, which monitors the business call process in the SpringBoot application in real time. Specifically, the log collection probe component is established based on the Java agent function. By calculating the current sampling rate, different business scenario types are determined to dynamically adjust the current sampling rate; based on the dynamically adjusted sampling rate, according to the current task triggered by the user operation, the log collection object and its related parameters in the current task are collected. During the collection process, the business call layer corresponding to the current task is intercepted according to the customized interception rule to collect the log collection object and its related parameters in the current task, so as to form a call location chain corresponding to the current task when the database is stored. When a log collection request is monitored, the log collection probe component is triggered to perform relevant log collection, and the corresponding call location chain is determined to track the behavior information of the specified application or track risk data. The present invention can realize real-time monitoring of the SpringBoot application call process without relying on heavy logging or breakpoint debugging that interrupts program execution. By providing a flexible probe component that can be dynamically inserted into key interception points of the application as needed to capture detailed call data, including caller, callee, parameter list, return value, and execution time, it can improve the stability and maintainability of the application while reducing the complexity and cost of monitoring and debugging.

[0170] Figure 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

[0171] like Figure 4 As shown, the electronic device is implemented as a general-purpose computing device. The processor may be one or multiple processors working in concert. The present invention also does not exclude distributed processing, meaning that the processors may be dispersed across different physical devices. The electronic device of the present invention is not limited to a single entity but may also be the sum of multiple physical devices.

[0172] The memory stores a computer executable program, typically a machine-readable code, which can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least some of the steps in the method.

[0173] The memory includes a volatile memory, such as a random access memory unit (RAM) and / or a cache memory unit, and may also be a non-volatile memory, such as a read-only memory unit (ROM).

[0174] Optionally, in this embodiment, the electronic device further includes an I / O interface for exchanging data with an external device. The I / O interface may represent one or more of several types of bus structures, including a storage unit bus or storage unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus structures.

[0175] It should be understood that Figure 4 The electronic device shown is merely an example of the present invention. The electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as screens, and some electronic devices also include human-computer interaction elements such as buttons and keyboards. As long as the electronic device can execute a computer-readable program stored in its memory to implement the method of the present invention or at least some of the steps of the method, it is considered an electronic device covered by the present invention.

[0176] Through the above description of the embodiments, it is easy for those skilled in the art to understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Figure 5 As shown, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of commands to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the embodiment of the present invention.

[0177] The software product may utilize any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0178] The computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. The data signal propagated may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, device, or component. The program code contained on the readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0179] Program code for performing the operations of the present invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0180] The computer-readable medium carries one or more programs. When the one or more programs are executed by a device, the computer-readable medium implements the data interaction method of the present disclosure.

[0181] Those skilled in the art will appreciate that the modules described above can be distributed in the device according to the description of the embodiment, or can be modified accordingly to be used in one or more devices that are different from the embodiment. The modules of the above embodiment can be combined into one module or further divided into multiple submodules.

[0182] From the above description of the embodiments, those skilled in the art will readily appreciate that the exemplary embodiments described herein can be implemented via software or via a combination of software and necessary hardware. Therefore, the technical solutions according to the embodiments of the present invention can be embodied in the form of a software product, which can be stored on a non-volatile storage medium (such as a CD-ROM, USB flash drive, or mobile hard drive) or on a network and includes commands that cause a computing device (such as a personal computer, server, mobile terminal, or network device) to execute the methods according to the embodiments of the present invention.

[0183] It should be noted that the above detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the art to which this application belongs.

[0184] In the above detailed description, reference is made to the accompanying drawings, which form a part hereof. In the drawings, similar symbols typically identify similar components, unless the context dictates otherwise. The illustrated embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be used, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein.

[0185] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A log collection optimization method, characterized in that: include: Based on the established log collection probe component, the business call process in the SpringBoot application is monitored in real time. Specifically, the log collection probe component is established based on the Java agent function; Calculate the current sampling rate to dynamically adjust the current sampling rate; specifically, use the following expression to calculate the current sampling rate: F s =F×log 10 (L cpu )×(1+S fr ); Among them, F s Indicates the current sampling rate calculated based on the CPU usage and historical error rate of the current Java virtual machine; F indicates the basic sampling rate; L cpu Indicates the CPU usage of the current Java virtual machine, log 10 (L cpu ) represents the logarithmic smoothing value of the CPU usage of the current Java virtual machine with a base of 10, which is used to compress the numerical fluctuation of the CPU usage; S fr The historical risk score of historical abnormal signals is used to quantify the health of the traffic management system, which is calculated based on the historical error rate within the time window; S fr The historical risk score is calculated based on the error rate within the time window using the following expression: ; Among them, S fr The historical risk score of historical abnormal signals, that is, the historical error rate obtained based on the ratio of abnormal business requests to the total number of business requests within the time window, is used to quantify the health of the traffic management system; E recent Indicates the number of abnormal requests in the time window closest to the current time window; E base represents the average number of abnormal requests in the baseline window; δ represents the standard deviation of the number of abnormal requests, which is used to reflect volatility; α represents the first weight coefficient, and β represents the second weight coefficient; Based on the dynamically adjusted sampling rate, according to the current task triggered by the user operation, the log collection objects and related parameters in the current task are collected. During the collection process, the business call layer corresponding to the current task is intercepted according to the customized interception rules to collect the log collection objects and related parameters in the current task, so as to form a call location chain corresponding to the current task when storing in the database; When a log collection request is detected, the log collection probe component is triggered to collect relevant logs and determine the corresponding call location chain to track the behavior information of the specified application or track risk data.

2. The log collection optimization method according to claim 1, characterized in that: Further including: Starting the log collection probe component includes the following steps: Step S201: Use the premian method in the Agent class as the probe entry; Step S202: Use maven-shaded-plugin in pom.xml of packaging tool Maven to perform packaging operation; Step S203: After the log collection probe component is started, the configuration file path set by the environment variable is used to read the configuration file content to initialize the interceptor and the message queue production end according to the configuration information content.

3. The log collection optimization method according to claim 1, characterized in that: Further including: When the CPU usage is monitored to exceed the preset baseline, when the historical risk score is greater than or equal to the preset threshold, or when the business scenario type is switched, the current acquisition rate is calculated in real time to adjust the current acquisition rate so that the current task is dynamically adapted to the resource load.

4. The log collection optimization method according to claim 1, characterized in that: Further including: Based on the configuration file loaded when the log collection probe component is started, the key interception points related to the business call process are identified. The dynamic injection process of the log collection probe component includes bytecode-level instrumentation, which specifically utilizes the bytecode weaving function to dynamically modify the bytecode during the class loading phase and insert the collection logic before and after the execution of the called method.

5. The log collection optimization method according to claim 1, characterized in that: When the log collection probe component monitors a new request entering the business call process, or when the system operation status indicator reaches a preset threshold condition, it triggers the calculation of the current sampling rate through the request entry or threshold detection port.

6. A log collection optimization system, characterized in that: It is used to execute the log collection optimization method according to any one of claims 1 to 5, and the log collection optimization system includes: The monitoring module is used to monitor the business call process in the SpringBoot application in real time based on the established log collection probe component. Specifically, the log collection probe component is established based on the Java agent function; A calculation and determination module is used to calculate the current sampling rate to dynamically adjust the current sampling rate; The collection module is used to collect log collection objects and related parameters in the current task based on the dynamically adjusted sampling rate and the current task triggered by the user operation. During the collection process, the business call layer corresponding to the current task is intercepted according to the customized interception rules to collect the log collection objects and related parameters in the current task, so as to form a call location chain corresponding to the current task when storing in the database; The tracking processing module, when monitoring the log collection request, triggers the log collection probe component to collect relevant logs and determine the corresponding call location chain to track the behavior information of the specified application or track risk data.

7. The log collection optimization system according to claim 6, characterized in that: Starting the log collection probe component includes the following steps: Step S201: Use the premian method in the Agent class as the probe entry; Step S202: Use maven-shaded-plugin in pom.xml of packaging tool Maven to perform packaging operation; Step S203: After the log collection probe component is started, the configuration file path set by the environment variable is used to read the configuration file content to initialize the interceptor and the message queue production end according to the configuration information content.

8. The log collection optimization system according to claim 6, characterized in that: Specifically, the following expression is used to calculate the current sampling rate: F s =F×log 10 (L cpu )×(1+S fr ); Among them, F s Indicates the current sampling rate calculated based on the CPU usage and historical error rate of the current Java virtual machine; F indicates the basic sampling rate; L cpu Indicates the CPU usage of the current Java virtual machine, log 10 (L cpu ) represents the logarithmic smoothing value of the CPU usage of the current Java virtual machine with a base of 10, which is used to compress the numerical fluctuation of the CPU usage; S fr A historical risk score representing historical abnormal signals, used to quantify the health of the traffic management system, calculated based on the historical error rate within a time window.

Images