Log collection optimization method and system

Abstract

The invention belongs to the technical field of data acquisition, and discloses a log acquisition optimization method and system. The method comprises the steps of monitoring a service calling process in a SpringBoot application in real time based on an established log collection probe assembly, calculating a current sampling rate so as to dynamically adjust the current sampling rate, intercepting a service calling layer corresponding to a current task according to a customized interception rule so as to collect a log collection object in the current task and related parameters of the log collection object, and determining the current task to form a call positioning chain corresponding to the current task when a database stores the current task, and determining the corresponding call positioning chain to track behavior information or risk data of a specified application program. According to the method, detailed calling data can be effectively captured, and the complexity and cost of monitoring and debugging are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of data collection, and in particular to a method and system for optimizing log collection. Background Art

[0002] In modern software development, Spring Boot has become the mainstream framework due to its automatic configuration and fast deployment characteristics. However, monitoring the call process under the microservices architecture faces challenges: traditional logging leads to data explosion (the log volume grows exponentially under high concurrency), and although APM tools (such as SkyWalking, Zipkin) support distributed tracing, they consume high resources (require an independent server, with memory occupancy greater than 1GB) and have complex configurations (require modifying the application code or depending on a specific SDK).

[0003] In addition, existing lightweight probes (such as Spring Boot Actuator) only provide basic health checks and lack detailed call chain information (unable to capture parameters and return values); existing real-time monitoring tools are costly and highly invasive.

[0004] In addition, the following problems also exist in the prior art: limitations of logging. In Spring Boot applications, developers usually add log statements to track method calls. However, this method results in large and difficult-to-manage log files. Especially in a high-concurrency environment, logging is usually asynchronous and cannot reflect the current state of the application in real time; using breakpoint debugging can help developers understand the code execution process, but this method interrupts the normal execution of the program, causes performance degradation, and is not applicable to the production environment. Although APM systems provide powerful monitoring and analysis functions, they require complex configuration and maintenance, increasing the development and operation costs.

[0005] Therefore, it is necessary to provide a new method and system for optimizing log collection. Summary of the Invention

[0006] To solve the technical problems that existing lightweight probes (such as Spring Boot Actuator) only provide basic health checks and lack detailed call chain information, resulting in the inability to timely find problem data and determine the problem points when problems occur, and existing real-time monitoring tools are costly and highly invasive, etc., the technical problems to be solved by the present invention are realized through the following technical solutions.

[0007] In the first aspect of the present invention, a method for optimizing log collection is proposed. Based on the established log collection probe component, it monitors the business call process in the Spring Boot application in real time. Specifically, the log collection probe component is established on the basis of the Java proxy function; calculates the current sampling rate to dynamically adjust the current sampling rate; based on the dynamically adjusted sampling rate, according to the current task triggered by the user operation, collects the log collection object and its related parameters in the current task. During the collection process, intercepts the business call layer corresponding to the current task according to the customized interception rule to collect the log collection object and its related parameters in the current task, so as to form a call positioning chain corresponding to the current task when storing in the database; when a log collection request is detected, triggers the log collection probe component to perform relevant log collection, determines the corresponding call positioning chain, so as to track the behavior information of the specified application program or track risk data.

[0008] In the second aspect of the present invention, a log collection optimization system is proposed. It executes the log collection optimization method described in the first aspect of the present invention. The log collection optimization system includes: a monitoring module, which is used to monitor the business call process in the Spring Boot application in real time based on the established log collection probe component. Specifically, the log collection probe component is established on the basis of the Java proxy function; a calculation and determination module, which is used to calculate the current sampling rate to dynamically adjust the current sampling rate; a collection module, which is used to collect the log collection object and its related parameters in the current task according to the current task triggered by the user operation based on the dynamically adjusted sampling rate. During the collection process, intercepts the business call layer corresponding to the current task according to the customized interception rule to collect the log collection object and its related parameters in the current task, so as to form a call positioning chain corresponding to the current task when storing in the database; a tracking and processing module, when a log collection request is detected, triggers the log collection probe component to perform relevant log collection, determines the corresponding call positioning chain, so as to track the behavior information of the specified application program or track risk data.

[0009] In the third aspect of the present invention, an electronic device is provided, including: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the log collection optimization method described in the first aspect of the present invention.

[0010] In the fourth aspect of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it implements the log collection optimization method described in the first aspect of the present invention.

[0011] The embodiments of the present invention include the following advantages: Compared with the prior art, based on the established log collection probe component, the present invention monitors the business call process in the Spring Boot application in real time. Specifically, a log collection probe component is established on the basis of the Java proxy function, and the current sampling rate is calculated to dynamically adjust the current sampling rate. Based on the dynamically adjusted sampling rate, according to the current task triggered by the user operation, the log collection object and its related parameters in the current task are collected. During the collection process, according to the customized interception rules, the business call layer corresponding to the current task is intercepted to collect the log collection object and its related parameters in the current task, so as to form a call positioning chain corresponding to the current task when stored in the database. When a log collection request is detected, the log collection probe component is triggered to perform relevant log collection, and the corresponding call positioning chain is determined to track the behavior information of the specified application program or track risk data. The present invention can realize the real-time monitoring of the Spring Boot application call process without relying on heavy logging or breakpoint debugging that interrupts program execution. By providing a flexible probe component, it can be dynamically inserted into the key interception points of the application as needed to capture detailed call data, including the caller, the callee, the parameter list, the return value, and the execution time, etc., which can improve the stability and maintainability of the application, and at the same time reduce the complexity and cost of monitoring and debugging. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a flowchart of an example of the log collection optimization method of the present invention; Figure 2 is a partial flowchart diagram of starting the log collection probe component in the log collection optimization method of the present invention; Figure 3 is a structural block diagram of the log collection optimization system of the present invention; Figure 4 is a structural schematic diagram of an electronic device embodiment according to the present invention; Figure 5 is a structural schematic diagram of a computer-readable medium embodiment according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0013] In order to make the objectives, technical solutions, and beneficial effects of the present invention clearer and more understandable, the methods and devices provided by the present invention are described in detail below in combination with specific processes and embodiments. It should be understood that the following content is only the preferred embodiment of the present invention, and any modifications or substitutions made within the spirit and essential principles of the present invention should be included within the protection scope of the present invention.

[0014] The present invention provides a method for optimizing log collection. Based on the established log collection probe component, this method monitors the business call process in the SpringBoot application in real time. Specifically, a log collection probe component is established on the basis of the Java agent function, and the current sampling rate is calculated to dynamically adjust the current sampling rate. Based on the dynamically adjusted sampling rate, according to the current task triggered by the user operation, the log collection object and its related parameters in the current task are collected. During the collection process, according to the customized interception rules, the business call layer corresponding to the current task is intercepted to collect the log collection object and its related parameters in the current task, so as to form a call positioning chain corresponding to the current task when storing in the database. When a log collection request is detected, the log collection probe component is triggered to perform relevant log collection, and the corresponding call positioning chain is determined to track the behavior information of the specified application program or track risk data. The present invention can realize the real-time monitoring of the call process of the SpringBoot application without relying on heavy logging or breakpoint debugging that interrupts program execution. By providing a flexible probe component, it can be dynamically inserted into the key interception points of the application as needed to capture detailed call data, including the caller, the callee, the parameter list, the return value, and the execution time, etc., which can improve the stability and maintainability of the application, and at the same time reduce the complexity and cost of monitoring and debugging.

[0015] It should be noted that the invention of the present application has a wide range of applications and is particularly suitable for a traffic management system, which is used to manage business data related to traffic services such as international traffic, domestic traffic, and targeted traffic.

[0016] Embodiment 1 The following will refer to Figure 1 、 Figure 2 and will elaborate on the content of the present invention in detail.

[0017] Figure 1 is a step flowchart of an example of the log collection optimization method of the present invention. Figure 2 is a partial process schematic diagram of starting the log collection probe component in the log collection optimization method of the present invention.

[0018] Refer to Figure 1 and Figure 2 In step S101, based on the established log collection probe component, the business call process in the SpringBoot application is monitored in real time. Specifically, a log collection probe component is established on the basis of the Java agent function.

[0019] Specifically, collect business data related to traffic services such as backward traffic, international traffic, domestic traffic, and targeted traffic in the Spring Boot application, and monitor the business call processes related to traffic services such as backward traffic, international traffic, domestic traffic, and targeted traffic. The business call processes include processes related to user requests, response information corresponding to user requests, the invoked call layer, the invoked business methods, etc. For example. The business methods include methods corresponding to different classes such as Filter, Controller, Service, and DAO. For example, the non-business methods include Utilities, etc.

[0020] More specifically, the log collection objects include business method logs and non-business method logs related to traffic services such as backward traffic, international traffic, domestic traffic, and targeted traffic. Among them, the business method logs are logs generated during the access of business methods in classes such as Controller, Service, Filter, and DAO that participate in the business interaction process (for each class, at least one call layer is corresponding at different call layers). The non-business method logs are logs generated by methods that assist in the specific business implementation in the business methods.

[0021] In a specific embodiment, a log collection probe component is established based on the Java agent function, and ByteBuddy is specifically used as the Java library.

[0022] Use the log collection probe component to obtain relevant information about requests (Request) and responses (Response) from the Spring Boot application. For the collection of business method logs, it specifically includes the input parameters, output parameters, Header information, Body information related to user requests and response information, and the execution duration of the method. For the collection of non-business method logs, it specifically includes the input parameters, output parameters, and call time during the call.

[0023] By providing a flexible probe component, namely the log collection probe component, it can be dynamically inserted into the key interception points of the application as needed to capture detailed call data.

[0024] Specifically, the "key interception point" refers to the core anchoring position in the application execution process where monitoring logic can be injected, specifically including: based on the real-time configuration loading ability of the probe component at runtime, dynamically identify business-sensitive nodes by parsing the configuration file (such as business-sensitive nodes related to backward traffic).

[0025] The "key interception point" also includes bytecode-level instrumentation, specifically using the bytecode weaving function to dynamically modify the bytecode during the class loading phase, and inserting collection logic before and after the execution of the invoked method.

[0026] Customized interception strategies are established for different business types, specifically including customized interception strategies related to backward traffic services (such as anchoring payment callback processors), customized interception strategies related to international traffic services, and customized interception strategies related to directed traffic services. Among them, for backward traffic services, such as the processing method using the reverse callback controller (BackwardCallbackController.process()), an anchoring payment callback processor is used. For international traffic services, at the order verification service layer, such as using the external channel API gateway entry, specifically OrderValidationService.checkQuota(), to receive the unified gateway entry class and method for API requests from external channels (such as service platforms for short videos or shopping, etc.).

[0027] For directed traffic services, such as using the external channel API network management entry (such as DouyinOrderGateway. receiveRequest()), to receive external API requests.

[0028] The entry points of each traffic service are monitored through multi-dimensional general monitoring trigger points, specifically including the HTTP request controller (the entry point of the control layer method), the service layer method boundary (the execution point of the service layer method), the database access entry (the database operation interception point), etc.

[0029] It should be noted that ByteBuddy is a powerful Java library used to dynamically generate and modify bytecode at runtime. By using ByteBuddy, custom logic can be injected into the target class without modifying the existing code, thereby realizing the monitoring of the Spring Boot application call process. In addition, the real-time listening of the business call process in the Spring Boot application by the log collection probe component also depends on the message middleware (such as the kafka database), the Java programming assistance library (such as Lombok), and the configuration file reading module (such as SnakeYaml, used to read yml or yaml configuration files). The above are only illustrative as optional examples and should not be construed as limitations to the present invention.

[0030] For different log collection objects, different log collection strategies are adopted, and the log collection strategies include the sampling rate of real-time calculation. The calculation process of sampling will be specifically described below in combination with step S102.

[0031] Next, in step S102, the current sampling rate is calculated to dynamically adjust the current sampling rate.

[0032] Based on the configuration file loaded when the log collection probe component starts, identify the key interception points related to the business call process in the SpringBoot application. The dynamic injection process of the log collection probe component includes bytecode-level instrumentation, specifically using the bytecode weaving function to dynamically modify the bytecode at the class loading stage and insert the collection logic before and after the execution of the called method.

[0033] Furthermore, the dynamic injection process includes configuration-driven hot update. Specifically, according to the configuration file rules read during startup and runtime, interceptors instances that match the current business scenario are generated and injected immediately without restarting the application. Then, the interception points are dynamically bound. Through the class loader isolation mechanism, the call paths to be intercepted are located in the method execution stack, and the collection logic is dynamically anchored to the control layer entry, the service layer method boundary, and the database operation interface.

[0034] When detecting fluctuations in CPU usage or changes in historical error rates, the collection density is adjusted in real time by reloading lightweight interceptor instances to ensure that the monitoring granularity is dynamically adapted to the system load. Specifically, when it is monitored that the CPU usage exceeds the preset baseline (for example, greater than or equal to 75%), when the historical risk score is greater than or equal to the preset threshold (for example, 0.8, 0.85, or 0.9), or when the business scenario type switches, the current collection rate is calculated in real time to adjust the current collection rate so that the current task is dynamically adapted to the resource load. The dynamic injection process ensures the decoupling of the collection logic from the business code and realizes zero-invasive log collection through dynamic weaving at the bytecode instruction set level. Thus, through the coordinated adjustment of the sampling rate linkage, the log collection of each business scenario can be achieved more efficiently.

[0035] For the switching of business scenario types, for example, when detecting context change events such as the switching of business scenario types among the backward traffic scenario, international traffic scenario, and directed traffic scenario, the current sampling rate is automatically triggered for calculation.

[0036] When the log collection probe component hears a new request entering the business call process, or when the system operation status indicators reach the preset threshold conditions (for example, the CPU usage exceeds the preset baseline), the current sampling rate is triggered for calculation through the request entry or the threshold detection port.

[0037] In a specific embodiment, when the CPU usage of the traffic management system exceeds the preset baseline (for example, greater than or equal to 75%), the current collection rate is calculated in real time. Specifically, the following expression is used to calculate the current sampling rate: F , s ,

[0037] , cpu , , fr , s , 10 =F×log 10 (L cpu )×(1+S fr ); Among them, F srepresents the current sampling rate calculated based on the CPU usage rate and historical error rate of the current Java virtual machine; F represents the base sampling rate; L cpu represents the CPU usage rate of the current Java virtual machine, log 10 (L cpu ) represents the logarithmically smoothed value of the CPU usage rate of the current Java virtual machine with base 10, which is used to compress the numerical fluctuations of the CPU usage rate; S fr represents the historical risk score of historical exception signals, which is used to quantify the health of the traffic management system and is calculated based on the historical error rate within a time window.

[0038] By calculating the current sampling rate in real time as described above to dynamically adjust the current sampling rate, it is possible to effectively balance the amount of collected data and the diagnostic data requirements. By dynamically coupling the system state with historical exception signals, an adaptive adjustment of the sampling rate is achieved.

[0039] Through the CPU usage rate L cpu real-time perception of the JVM load, automatically reducing the sampling rate when the load is high to relieve the system pressure and balance resource usage. The high load includes that the CPU usage is greater than or equal to 75% corresponding to the peak period of continuous monthly subscription. By cooperating with the linked adjustment of the sampling rate, when the traffic management system is under high load (such as during the peak period of continuous monthly subscription), reducing the sampling rate to reduce the number of messages in the message queue and relieve the pressure of log storage processing, it is possible to more efficiently implement the log collection for each business scenario.

[0040] It should be noted that in this example, the core technical role of the Java virtual machine (JVM) is reflected as the operating carrier of the log collection probe component and the control center of dynamic monitoring. The JVM exposes the system operation status indicators in real time through the built-in management interface, including the current CPU usage rate, memory stack status, and thread activity data. These system operation status indicators constitute the original input source for the dynamic calculation of the sampling rate. Specifically, the bytecode instrumentation mechanism of the JVM provides the basic support for bytecode enhancement. By intercepting the target business class (such as the payment controller) during the class loading phase and inserting the collection logic at the method execution boundary, zero-intrusive monitoring is achieved. This bytecode instrumentation mechanism also supports the runtime redefinition of the log collection probe component. In addition, the resource perception ability of the JVM plays a decisive role in the sampling rate regulation. When it is detected that the CPU usage rate exceeds the preset threshold (such as ≥75%), the downsampling mechanism is automatically triggered, and by reducing the log collection density of non-core business methods, the load of the collection message consumption system is reduced, driving the dynamic adaptation of the sampling rate to the business scenario requirements. By tightly coupling the JVM resource status with the sampling rate adjustment logic, a dynamic closed-loop control system is formed, ensuring the visibility of critical business links while reducing the monitoring cost.

[0041] For example, when abnormal signals in data collection occur frequently, the sampling rate is increased to capture more diagnostic data.

[0042] For the basic sampling rate F, it refers to the default sampling rate under the steady state of the system, which needs to be refined according to the business scenario. For specific values, please refer to Table 1 below: Table 1

[0043] Table 1 is an example table for setting the basic sampling rates of different business scenario types.

[0044] For the setting principle, the Nyquist criterion can be referred to. For example, the value of F should be at least twice the call frequency of a specific business method. The specific business methods include methods related to specific services such as purchasing international volume and exchanging domestic traffic.

[0045] In addition, the upper limit is inversely deduced based on the throughput capacity of downstream storage (specifically corresponding to the database for storing data, such as a Kafka cluster) for storage cost constraints. For example, when the throughput of a single node is 10MB / s and the size of a single log is 1KB, then F max is less than 10,000 records per second.

[0046] For CPU usage rate, for example, the specified application object can be directly obtained through the ManagementFactory#getPlatformMXBean method in the JVM, and then the CPU usage rate (value range: [0 - 1]) of the specified application object can be obtained through the getSystemCpuLoad method.

[0047] For the case of multiple samplings, the instantaneous jitter is avoided by taking the average (e.g., taking 3 samples within 1 second).

[0048] It should be noted that in this example, in the container environment, it is necessary to distinguish between the CPU of the JVM process and the CPU of the host.

[0049] For the differences between using logarithmic scale and linear scale in the above expressions, please refer to Table 2 below.

[0050] Table 2

[0051] Table 2 is an example table for the difference parameters and effects between using logarithmic scale and linear scale in the above expressions.

[0052] By using a logarithmic function to characterize the CPU usage rate to calculate the sampling rate, the high - value region of the sampling rate can be effectively compressed, making the gradient in the high - CPU range (such as 0.8 - 1.0) gentler and preventing a cliff - like drop in the sampling rate. It can also enhance the sensitivity to low values, making the gradient in the low - and high - CPU range (such as 0.1 - 0.3) steeper, and quickly increasing the sampling rate when resources are abundant.

[0053] For the historical risk score S fr (i.e., the historical error rate), based on the historical error rate within the time window, it is calculated using the following expression: ;

[0054] where S fr represents the historical risk score of the historical anomaly signal, which is the historical error rate obtained based on the ratio of the number of abnormal requests to the total number of business requests within the time window, and is used to quantify the health of the traffic management system. It is calculated based on the historical error rate within a configurable sliding time window (e.g., within 10 minutes), specifically referring to the ratio of the number of failed business requests to the total number of business requests; E recent represents the number of abnormal requests in the most recent time window (e.g., 5 minutes); E base represents the average number of abnormal requests in the baseline window (e.g., 1 hour); δ represents the standard deviation of the number of abnormal requests, which is used to reflect the volatility; α represents the first weight coefficient, and β represents the second weight coefficient. For example, continuous statistics are performed using a rolling time window (default 10 minutes), and the data samples are refreshed every 30 seconds to calculate the historical error rate.

[0055] Optionally, α = 0.7 and β = 0.3.

[0056] It should be noted that for the failure judgment of business requests, failures belonging to the HTTP layer are server errors with a response status code greater than or equal to 500 (e.g., interface call timeout). Failures belonging to the system control layer occur when, in the system control layer, the method in the system control layer fails to return normally and triggers a method exception (such as a payment order not existing, resulting in failure to complete data update and thus a business exception being thrown), that is, corresponding to the failure of the business request. The failure of business requests also includes basic component failures. Runtime exceptions occur (such as database connection timeout exceptions, message queue delivery failure exceptions, etc.).

[0057] Specifically, when it is detected that the subscription error rate of the short video channel for the targeted traffic service (corresponding to the historical error rate) increases, the sampling rate is automatically increased to the preset baseline to ensure complete capture of the risk location call chain.

[0058] It should be noted that the above is only for illustrative purposes as an optional example and should not be construed as a limitation of the present invention.

[0059] Next, in step S103, based on the dynamically adjusted sampling rate, according to the current task triggered by the user operation, collect the log collection object and its related parameters in the current task. During the collection process, intercept the business call layer corresponding to the current task according to the customized interception rules to collect the log collection object and its related parameters in the current task, so as to form a call positioning chain corresponding to the current task when storing in the database.

[0060] Specifically, according to the dynamically adjusted sampling rate in step S102, according to the current task triggered by the user operation, collect the log collection object and its related parameters in the current task. The current task is, for example, a targeted traffic collection task.

[0061] When starting to collect business data, it is necessary to start the log collection probe component.

[0062] For the startup of the log collection probe component, there are two startup methods for Java's Web services: startup in basic mode and startup in container mode.

[0063] For example, use the java command to directly start the Web service, which is used for projects related to, for example, SpringBoot and SpringCloud. When starting the log collection probe component, an additional -javaagent parameter needs to be added, and this parameter is set before the jar package to be started. For example, java -javaagent: / path / to / agent.jar -jar / path / to / service.jar.

[0064] For example, when starting a container service, there is a built-in method in the container to start the Web service, and additional parameter configuration is required, which is used for projects related to, for example, SpringMVC and JBoss. For example, the probe startup parameters can be specified in the configuration file of container services such as Tomcat or WebLogic.

[0065] For the Tomcat container service, for example, modify catalina.sh (Linux environment) or catalina.bat (Windows environment), and add "-javaagent: / path / to / agent.jar" to the environment variable JAVA_OPTS.

[0066] For the WebLogic container service, for example, modify startWeblogic.sh (Linux environment) or startWeblogic.bat (Windows environment), and add "-javaagent: / path / to / agent.jar" at the very front of the environment variable JAVA_OPTIONS.

[0067] The described startup log collection probe component includes the following steps: Step S201: Use the premian method in the Agent class as the probe entry point.

[0068] Step S202: Use the maven-shaded-plugin in the pom.xml of the packaging tool maven for the packaging operation.

[0069] Specifically, it is necessary to define the manifest-related information and specify the startup method to the premain method in the Agent class.

[0070] Step S203: After the log collection probe component starts, read the content of the configuration file through the configuration file path set by the environment variable to initialize the interceptor and the message queue producer side according to the content of the configuration information.

[0071] Specifically, the content of the configuration file includes probe interception rule configuration information and message queue configuration information.

[0072] During the collection process, intercept the business call layer corresponding to the current task according to the customized interception rules to collect the log collection object and its related parameters in the current task. The collection object is a specified user group and the business data generated by the specified user group's purchase of directed traffic (such as the usage traffic of a certain video playback APP). The related parameters include the called call layer, the input and output parameters during the call, etc.

[0073] The customized interception rules include initializing interceptors for each customized call layer according to the content of the configuration file. In a specific embodiment, initialize the first interceptor of the first call layer (such as the Controller interceptor), the second interceptor of the second call layer (such as the Service interceptor), the third interceptor of the third call layer (such as the DAO interceptor), and the fourth interceptor of the fourth call layer (such as the Default interceptor).

[0074] Specifically, the first interceptor is used to intercept the Controller class in the Spring architecture. The second interceptor is used to intercept the Service class in the Spring architecture. The third interceptor is used to intercept the DAO class in the Spring architecture. The fourth interceptor is used to intercept other tool classes that are not in the Spring architecture. The extensible interceptor is used for the interfaces of static classes, specifically for the basic extensible implementation of other interception services.

[0075] Furthermore, a call positioning chain corresponding to the current task is formed during storage in the database.

[0076] It should be noted that in the present invention, the log collection probe component can be started or shut down as needed, without imposing a burden on the production environment. Meanwhile, the data collected by the log collection probe component can be directly used to generate a visualized call chain diagram to help developers quickly locate the problem. In addition, the log collection probe component can also be integrated with other systems, such as a distributed tracing system, to provide more in-depth performance analysis and fault diagnosis capabilities. The above is only described as an optional example and should not be construed as a limitation to the present invention.

[0077] In a specific embodiment, for example, business data of user a in a specified user group is collected. The collected log data includes calling business method A-A of the first call layer, then calling business method B-D of the second call layer, calling business method C-C of the third call layer, and data of cluster G3 in the database. Thus, the following call positioning chain is formed: user a - Controller business method A-A - Service business method B-D - DAO business method C-C - G3.

[0078] It should be noted that the above is only described as an optional example and should not be construed as a limitation to the present invention.

[0079] Next, in step S104, when a log collection request is monitored, the log collection probe component is triggered to perform relevant log collection, and the corresponding call positioning chain is determined to track the behavior information of a specified application program or track risk data.

[0080] When a log collection request is monitored, the log collection probe component is triggered to perform relevant log collection, and the corresponding call positioning chain is determined to track the behavior information of a specified application program or track risk data.

[0081] For example, when a log collection request is monitored, the gateway service receives the log collection request, calls the corresponding service port, and generates a log trace identifier (i.e., TraceID).

[0082] For the log collection of the gateway service, a TraceID is specifically generated at the gateway service entry. The collection example is as follows: { "timestamp": "2025-03-12T14:30:22.123Z", "service": "gateway-service", "class-method": "com.gateway.Router#route", "trace_id": "7a3b5c8d9e0f1a2b", / / Newly generated TraceID "parent_span_id": null, / / The root node has no parent Span "span_id": "7a3b5c8d9e0f1a2b", / / The root Span ID = TraceID "input_parameters": {"path": " / order / create", "method": "POST"}, "output_parameters": {"status": "ROUTED"}, "request": {"headers": {"X-User-Id": "U12345"}}, "response": {"target_service": "order-service"}, "status_code": 200, "duration": 12, "event_type": "ENTRY_CALL" / / Mark the entry call }.

[0083] For example, the log collection example of the order service is as follows: Receive the TraceID and generate a child Span { "timestamp": "2025-03-12T14:30:22.456Z", "service": "order-service", "class-method": "com.order.OrderService#createOrder", "trace_id": "7a3b5c8d9e0f1a2b", / / Inherit the TraceID of the gateway "parent_span_id": "7a3b5c8d9e0f1a2b", / / The parent Span = the Span ID of the gateway "span_id": "d4e5f6a7b8c9d0e1", / / Generate a new child Span ID "input_parameters": {"userId": "U12345", "sku": "1001", "amount": 2}, "output_parameters": {"orderId": "ORD-20250605-001"}, "request": {"body": "Encrypted ciphertext"}, "response": {"result": "SUCCESS"}, "status_code": 200, "duration": 145, "event_type": "BIZ_PROCESS" / / Mark the core business logic }。

[0084] For another example, the log collection example for the inventory service (as a downstream service of the order service) is as follows: { "timestamp": "2025-03-12T14:30:22.789Z", "service": "stock-service", "class-method": "com.stock.StockService#deduct", "trace_id": "7a3b5c8d9e0f1a2b", / / Keep the global TraceID "parent_span_id": "d4e5f6a7b8c9d0e1", / / Parent Span = SpanID of the order service "span_id": "3f4g5h6i7j8k9l0", / / Generate a new child SpanID "input_parameters": {"sku": "1001", "quantity": 2}, "output_parameters": {"remaining": 98}, "request": {"source": "order-service"}, "response": {"code": "STOCK-200", "message": "OK"}, "status_code ": 200, "duration": 42, "event_type": "DB_UPDATE" / / Mark database operations }。

[0085] For the different log collection objects and their related parameters collected, standardized and structured JSON-format collection logs are formed, which specifically include the following fixed fields: collection time point (e.g., represented by timestamp), service name (e.g., represented by service), collection object class and method (e.g., represented by class-method and separated by "#"), log trace identifier (e.g., represented by trace_id), parent log ID (e.g., represented by parent_span_id), log ID (e.g., represented by span_id), input parameter list (e.g., represented by input_parameters), output parameter list (e.g., represented by output_parameters), request object (e.g., represented by request), response object (e.g., represented by response), response status code (e.g., represented by status_code), method execution time (e.g., represented by duration), log type (e.g., represented by event_type).

[0086] The transfer field rules of the call location chain include the TraceID transfer mechanism and the SpanID generation rule for sub-services.

[0087] The TraceID transfer mechanism is specifically generated from the entry service and passed to the downstream through the request header (such as X-B3-TraceId). The call between microservices automatically injects the request header through bytecode instrumentation without manual coding. The SpanID generation rule for sub-services includes the root service and sub-services. Among them, the sub-service ID corresponds to the TraceID, an independent sub-service ID (SpanID) is generated, and the sub-service ID (SpanID) on the upstream side is associated through the parent sub-service ID (parent_span_id).

[0088] In addition, for log information, fields such as ID cards and mobile phone numbers need to be automatically masked in the input and output parameters and the request and response objects to achieve desensitization of sensitive data.

[0089] It should be noted that the above is only for illustrative purposes as an optional example and should not be construed as a limitation to the present invention.

[0090] Compared with the prior art, based on the established log collection probe component, the present invention monitors the business call process in the Spring Boot application in real time. Specifically, a log collection probe component is established on the basis of the Java proxy function. By calculating the current sampling rate, different business scenario types are determined to dynamically adjust the current sampling rate. Based on the dynamically adjusted sampling rate, according to the current task triggered by the user operation, the log collection object and its related parameters in the current task are collected. During the collection process, the business call layer corresponding to the current task is intercepted according to the customized interception rules to collect the log collection object and its related parameters in the current task, so as to form a call positioning chain corresponding to the current task when stored in the database. When a log collection request is detected, the log collection probe component is triggered to perform relevant log collection and determine the corresponding call positioning chain to track the behavior information of the specified application or track risk data. The present invention can realize the real-time monitoring of the call process of the Spring Boot application without relying on heavy logging or breakpoint debugging that interrupts program execution. By providing a flexible probe component, it can be dynamically inserted into the key interception points of the application as needed to capture detailed call data, including the caller, the callee, the parameter list, the return value, and the execution time, etc., which can improve the stability and maintainability of the application, and at the same time reduce the complexity and cost of monitoring and debugging.

[0091] Embodiment 2 The following is an embodiment of the system of the present invention, which can be used to execute the embodiment of the method of the present invention. For the details not disclosed in the embodiment of the system of the present invention, please refer to the embodiment of the method of the present invention.

[0092] Figure 3 is a schematic structural diagram of an example of the log collection optimization system according to the present invention. The following will refer to Figure 3 , to describe the log collection optimization system 300. The log collection optimization system 300 executes the log collection optimization method described in Embodiment 1 of the present invention.

[0093] The log collection optimization system 300 includes a monitoring module 310, a calculation and determination module 320, a collection module 330, and a tracking and processing module 340.

[0094] In a specific embodiment, the monitoring module 310 is used to monitor the business call process in the Spring Boot application in real time based on the established log collection probe component, specifically, the log collection probe component is established on the basis of the Java agent function. The calculation and determination module 320 is used to calculate the current sampling rate to dynamically adjust the current sampling rate. The collection module 330 is used to collect the log collection objects and their related parameters in the current task based on the dynamically adjusted sampling rate according to the current task triggered by the user operation. During the collection process, the business call layer corresponding to the current task is intercepted according to the customized interception rules to collect the log collection objects and their related parameters in the current task, so as to form a call positioning chain corresponding to the current task when stored in the database. When the tracking and processing module 340 detects a log collection request, it triggers the log collection probe component to perform relevant log collection, determines the corresponding call positioning chain, so as to track the behavior information of the specified application program or track risk data.

[0095] According to an alternative embodiment, starting the log collection probe component includes the following steps: Step S201: Use the premian method in the Agent class as the probe entry; Step S202: Use the maven-shaded-plugin in the pom.xml of the packaging tool maven to perform the packaging operation; Step S203: After the log collection probe component is started, read the content of the configuration file through the configuration file path set by the environment variable, so as to initialize the interceptor and the message queue producer end according to the content of the configuration information.

[0096] Specifically, the following expression is used to calculate the current sampling rate: F s =F×log 10 (L cpu )×(1+S fr ); Where, F s represents the current sampling rate calculated according to the CPU usage rate and historical error rate of the current Java virtual machine; F represents the basic sampling rate; L cpu represents the CPU usage rate of the current Java virtual machine, and log 10 (L cpu ) represents the logarithm smoothing value of the CPU usage rate of the current Java virtual machine with base 10, which is used to compress the numerical fluctuation of the CPU usage rate; S fr represents the historical risk score of the historical exception signal, which is used to quantify the health of the traffic management system and is calculated based on the historical error rate within the time window.

[0097] For the historical risk score S fr, which is calculated using the following expression based on the error rate within a time window: ;

[0098] where S fr represents the historical risk score of historical anomaly signals, that is, the historical error rate obtained based on the proportion of the number of abnormal requests in business requests within a time window to the total number of business requests, which is used to quantify the health of the traffic management system; E recent represents the number of abnormal requests in the time window closest to the current time window; E base represents the average number of abnormal requests in the baseline window; δ represents the standard deviation of the number of abnormal requests, which is used to reflect volatility; α represents the first weight coefficient, and β represents the second weight coefficient.

[0099] According to an alternative implementation, when it is monitored that the CPU usage rate exceeds the preset baseline, when the historical risk score is greater than or equal to the preset threshold, or when the business scenario type switches, the current sampling rate is calculated in real time to adjust the current sampling rate so that the current task is dynamically adapted to the resource load.

[0100] According to an alternative implementation, based on the configuration file loaded when the log collection probe component is started, key interception points related to the business call process are identified. The dynamic injection process of the log collection probe component includes bytecode-level instrumentation, specifically using the bytecode weaving function to dynamically modify the bytecode at the class loading stage and insert collection logic before and after the execution of the called method.

[0101] According to an alternative implementation, when the log collection probe component monitors the entry of a new request in the business call process, or when the system operation status indicator reaches the preset threshold condition, the current sampling rate is triggered through the request entry or the threshold detection port.

[0102] It should be noted that since Figure 3 the log collection optimization method executed by the log collection optimization system of Figure 1 is substantially the same as the log collection optimization method in the example of

[0103] Compared with the prior art, based on the established log collection probe component, the present invention monitors the business call process in the SpringBoot application in real time. Specifically, a log collection probe component is established on the basis of the Java agent function. By calculating the current sampling rate, different business scenario types are determined to dynamically adjust the current sampling rate. Based on the dynamically adjusted sampling rate, according to the current task triggered by the user operation, the log collection objects and their related parameters in the current task are collected. During the collection process, the business call layer corresponding to the current task is intercepted according to the customized interception rules to collect the log collection objects and their related parameters in the current task, so as to form a call positioning chain corresponding to the current task when stored in the database. When a log collection request is detected, the log collection probe component is triggered to perform relevant log collection and determine the corresponding call positioning chain to track the behavior information of the specified application or track risk data. The present invention can realize the real-time monitoring of the SpringBoot application call process without relying on heavy logging or breakpoint debugging that interrupts program execution. By providing a flexible probe component, it can be dynamically inserted into the key interception points of the application as needed to capture detailed call data, including the caller, the callee, the parameter list, the return value, and the execution time, etc., which can improve the stability and maintainability of the application, and at the same time reduce the complexity and cost of monitoring and debugging.

[0104] Figure 4 It is a schematic structural diagram of an embodiment of an electronic device according to the present invention.

[0105] As Figure 4 shown, the electronic device is presented in the form of a general-purpose computing device. The processor can be one or multiple and work cooperatively. The present invention does not exclude distributed processing, that is, the processors can be dispersed in different physical devices. The electronic device of the present invention is not limited to a single entity, but can also be the sum of multiple physical devices.

[0106] The memory stores computer-executable programs, usually machine-readable code. The computer-readable program can be executed by the processor so that the electronic device can execute the method of the present invention or at least part of the steps in the method.

[0107] The memory includes volatile memory, such as a random access storage unit (RAM) and / or a cache storage unit, and can also be non-volatile memory, such as a read-only storage unit (ROM).

[0108] Optionally, in this embodiment, the electronic device further includes an I / O interface, which is used for the electronic device to exchange data with external devices. The I / O interface may represent one or more of several types of bus structures, including a memory unit bus or a memory unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any bus structure in a variety of bus structures.

[0109] It should be understood that Figure 4 The electronic device shown is only an example of the present invention. The electronic device of the present invention may further include elements or components not shown in the above example. For example, some electronic devices further include a display unit such as a display screen, and some electronic devices further include human-computer interaction elements such as buttons and keyboards. As long as the electronic device can execute the computer-readable program in the memory to implement at least part of the steps of the method of the present invention, it can be considered as the electronic device covered by the present invention.

[0110] Through the description of the above embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by a combination of software and necessary hardware. Therefore, as Figure 5 shown, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several commands to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the embodiment of the present invention.

[0111] The software product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0112] The computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, in which the readable program code is carried. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The readable storage medium may also be any readable medium other than the readable storage medium, which may send, propagate, or transmit a program for use by or in connection with a command execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0113] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's device, executed as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., by connecting through the Internet using an Internet service provider).

[0114] The above computer-readable medium carries one or more programs, which when executed by such a device, cause the computer-readable medium to implement the data interaction method of the present disclosure.

[0115] Those skilled in the art can understand that the above-mentioned modules may be distributed in the device according to the description of the embodiments, or may be correspondingly changed and distributed in one or more devices that are uniquely different from the present embodiment. The modules of the above embodiments may be combined into one module, or may be further split into multiple sub-modules.

[0116] Through the description of the above embodiments, those skilled in the art can easily understand that the exemplary embodiments described herein can be implemented by software or by a combination of software and necessary hardware. Therefore, the technical solution according to the embodiments of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which may be a CD-ROM, USB flash drive, mobile hard disk, etc.) or on a network, including several commands to cause a computing device (which may be a personal computer, server, mobile terminal, or network device, etc.) to execute the method according to the embodiments of the present invention.

[0117] It should be noted that the above detailed description is exemplary and is intended to provide further illustration of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs.

[0118] In the above detailed description, reference has been made to the accompanying drawings, which form a part hereof. In the drawings, like symbols typically identify like components, unless the context indicates otherwise. The illustrated embodiments described in the detailed description, the drawings, and the claims are not meant to be limiting. Other embodiments may be used and other changes may be made without departing from the spirit or scope of the subject matter presented herein.

[0119] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. An optimized method for log collection, characterized in that, including: Based on the established log collection probe component, it listens in real time to the business call process in the SpringBoot application and establishes a log collection probe component based on the Java agent function; Calculate the current sampling rate to dynamically adjust the current sampling rate; Based on the dynamically adjusted sampling rate, according to the current task triggered by the user operation, collect the log collection object and its related parameters in the current task. During the collection process, intercept the business call layer corresponding to the current task according to the customized interception rules to collect the log collection object and its related parameters in the current task, so as to form a call positioning chain corresponding to the current task when storing in the database; When a log collection request is detected, trigger the log collection probe component to perform relevant log collection, determine the corresponding call positioning chain, so as to track the behavior information of the specified application or track risk data.

2. The log collection optimization method according to claim 1, wherein Further including: Start the log collection probe component, including the following steps: Step S201: Use the premian method in the Agent class as the probe entry; Step S202: Use the maven-shaded-plugin in the pom.xml of the packaging tool maven to perform the packaging operation; Step S203: After the log collection probe component is started, read the content of the configuration file through the configuration file path set by the environment variable, so as to initialize the interceptor and the message queue producer end according to the content of the configuration information.

3. The log collection optimization method according to claim 1, wherein Specifically, use the following expression to calculate the current sampling rate: F s = F × log 10 (L cpu ) × (1 + S fr ); Among them, F s represents the current sampling rate calculated based on the CPU usage rate and historical error rate of the current Java virtual machine; F represents the base sampling rate; L cpu represents the CPU usage rate of the current Java virtual machine, log 10 (L cpu ) represents the logarithmically smoothed value of the CPU usage rate of the current Java virtual machine with base 10, which is used to compress the numerical fluctuations of the CPU usage rate; S fr represents the historical risk score of historical anomaly signals, which is used to quantify the health of the traffic management system and is calculated based on the historical error rate within a time window.

4. The log collection optimization method according to claim 3, wherein Further including: For the historical risk score S fr , it is calculated using the following expression based on the error rate within the time window: ; Among them, S fr represents the historical risk score of historical anomaly signals, that is, the historical error rate obtained based on the proportion of the number of abnormal requests in business requests within the time window to the total number of business requests, which is used to quantify the health of the traffic management system; E recent represents the number of abnormal requests in the time window closest to the current time window; E base represents the average number of abnormal requests in the baseline window; δ represents the standard deviation of the number of abnormal requests, which is used to reflect the volatility; α represents the first weight coefficient, and β represents the second weight coefficient.

5. The log collection optimization method according to claim 4, wherein Further including: When it is monitored that the CPU usage rate exceeds the preset baseline, when the historical risk score is greater than or equal to the preset threshold, or when the business scenario type switches, calculate the current collection rate in real time to adjust the current collection rate so that the current task is dynamically adapted to the resource load.

6. The log collection optimization method according to claim 1, wherein Further including: Based on the configuration file loaded when the log collection probe component is started, identify the key interception points related to the business call process. The dynamic injection process of the log collection probe component includes bytecode-level instrumentation. Specifically, use the bytecode weaving function to dynamically modify the bytecode during the class loading stage and insert the collection logic before and after the execution of the called method.

7. The log collection optimization method according to claim 1, characterized in that When the log collection probe component monitors the entry of a new request in the business call process, or when the system operation status indicator reaches the preset threshold condition, trigger the calculation of the current sampling rate through the request entry or the threshold detection port.

8. An optimized system for log collection, characterized in that, It is used to execute the log collection optimization method according to any one of claims 1 to 7. The log collection optimization system includes: A monitoring module, which is used to listen in real time to the business call process in the SpringBoot application based on the established log collection probe component, and establish a log collection probe component based on the Java agent function; A calculation and determination module, which is used to calculate the current sampling rate to dynamically adjust the current sampling rate; The acquisition module is used to acquire the log acquisition object and its related parameters in the current task based on the dynamically adjusted sampling rate according to the current task triggered by the user operation. During the acquisition process, the business call layer corresponding to the current task is intercepted according to the customized interception rules to acquire the log acquisition object and its related parameters in the current task, so as to form a call positioning chain corresponding to the current task when stored in the database; The tracking processing module triggers the log acquisition probe component to perform relevant log acquisition when a log acquisition request is monitored, determines the corresponding call positioning chain, so as to track the behavior information of the specified application program or track risk data.

9. The log acquisition optimization system according to claim 8, wherein Starting the log acquisition probe component includes the following steps: Step S201: Use the premian method in the Agent class as the probe entry; Step S202: Use the maven-shaded-plugin in the pom.xml of the packaging tool maven to perform the packaging operation; Step S203: After the log acquisition probe component is started, read the content of the configuration file through the configuration file path set by the environment variable, so as to initialize the interceptor and the message queue producer end according to the content of the configuration information.

10. The log collection optimization system according to claim 8, wherein Specifically, the following expression is used to calculate the current sampling rate: F s = F × log 10 (L cpu ) × (1 + S fr ); Among them, F s represents the current sampling rate calculated based on the CPU usage rate and historical error rate of the current Java virtual machine; F represents the base sampling rate; L cpu represents the CPU usage rate of the current Java virtual machine, and log 10 (L cpu ) represents the logarithmically smoothed value of the CPU usage rate of the current Java virtual machine with base 10, which is used to compress the numerical fluctuations of the CPU usage rate; S fr represents the historical risk score of historical exception signals, which is used to quantify the health of the traffic management system and is calculated based on the historical error rate within a time window.

Images