Log data management method and device, equipment and storage medium

Abstract

The invention relates to the technical field of data processing, and particularly provides a log data management method and device, equipment and a storage medium, and the method comprises the steps: starting an agent class of a target application when it is detected that the target application is started; a log collection instruction is inserted into the bytecode file of the target application through the proxy class, a modified bytecode file is obtained, and the log collection instruction is the bytecode file; in the process that the virtual machine runs the modified byte code file, it is monitored that a log collection instruction is triggered, and log data are obtained through the triggered log collection instruction. Due to the fact that in the byte code file loading process of the application, the log collection instruction used for collecting the log data is dynamically inserted into the byte code file through the agent class of the application, log collection is achieved without modifying service codes, non-intrusive log collection is achieved, independence of the service codes is kept, and the code coupling degree is reduced.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a log data management method, device, equipment and storage medium. Background Art

[0002] In large-scale distributed systems, it is very important to collect log data of Java applications. Currently, the common way to collect logs of Java applications is to explicitly add logging code in the business code.

[0003] The above log collection method requires embedding a large amount of logging code in the business code, which significantly increases the coupling degree of the code. Moreover, the logging function is tightly bound to the business function, which destroys the independence of the code and is not conducive to code reuse and modular development. Summary of the invention

[0004] In order to solve the above technical problems, the present application provides a log data management method, device, equipment and storage medium, which realizes non-intrusive log data collection, does not modify the source code, maintains the independence of the business code, reduces code coupling, and facilitates maintenance and upgrading.

[0005] In a first aspect, the present application provides a log data management method, the method comprising: when detecting that a target application is started, starting a proxy class of the target application; inserting a log collection instruction into a bytecode file of the target application through the proxy class to obtain a modified bytecode file, the log collection instruction being a bytecode file; in the process of a virtual machine running the modified bytecode file, monitoring that the log collection instruction is triggered, and obtaining log data using the triggered log collection instruction.

[0006] In the second aspect, the present application provides a log data management device, which includes: a proxy class startup module, which is used to start the proxy class of the target application when it is detected that the target application is started; a file modification module, which is used to insert log collection instructions into the bytecode file of the target application through the proxy class to obtain a modified bytecode file, and the log collection instructions are bytecode files; a log acquisition module, which is used to monitor the triggering of the log collection instruction during the process of the virtual machine running the modified bytecode file, and obtain log data using the triggered log collection instruction.

[0007] In a third aspect, the present application provides an electronic device, which includes a log data management device, and the device includes: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by one or more processors, the one or more processors implement the log data management method as described in the first aspect above.

[0008] In a fourth aspect, the present application provides a storage medium, which may be a computer-readable storage medium, storing a computer program thereon, which, when executed by a processor, implements the log data management method in the first aspect described above.

[0009] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program or instructions, and when the computer program or instructions are executed by a processor, implements any log data management method as described in the first aspect above.

[0010] Compared with the prior art, the technical solution provided by the embodiments of the present application has the following advantages:

[0011] An embodiment of the present application provides a log data management method, apparatus, device and storage medium, the method comprising: when detecting that a target application is started, starting a proxy class of the target application; inserting a log collection instruction into a bytecode file of the target application through the proxy class to obtain a modified bytecode file, the log collection instruction being the bytecode file; during the process of a virtual machine running the modified bytecode file, monitoring that the log collection instruction is triggered, and obtaining log data using the triggered log collection instruction.

[0012] During the loading process of the bytecode file of the application, the log collection instructions for collecting log data are dynamically inserted into the bytecode file through the application proxy class. There is no need to modify the business code to implement logging, thus achieving non-intrusive log collection, maintaining the independence of the business code, reducing code coupling, and facilitating maintenance and upgrades. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0014] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0015] Figure 1 A flow chart of a log data management method provided in an embodiment of the present application;

[0016] Figure 2 A flowchart of another log data management method provided in an embodiment of the present application;

[0017] Figure 3 A schematic diagram of the structure of a log data management device provided in an embodiment of the present application;

[0018] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0020] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0021] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0022] It should be noted that the concepts such as "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0023] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0024] The log data management method provided by the present application is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0025] Figure 1 This is a flow chart of a log data management method in an embodiment of the present application. This embodiment can be applied to collecting and managing application logs. The method can be executed by a log data management device. The log data management device can be implemented in software and / or hardware. The log data management device can be configured in an electronic device.

[0026] like Figure 1 As shown, the log data management method provided in the embodiment of the present application mainly includes steps S101-S103.

[0027] S101. When it is detected that a target application is started, a proxy class of the target application is started.

[0028] An application may also be referred to as an application program, which is an independent program written in the Java programming language. For example, the above-mentioned application may include at least one or more of the following: command line application, desktop application, web application, mobile application, enterprise application, embedded system application, game application, cloud computing service, etc.

[0029] The target application can be understood as the application whose log data needs to be collected during the application operation. The target application startup can be understood as the entire process from the target application starting to run to entering the normal working state.

[0030] In Java, a proxy class generally refers to a class created through a dynamic proxy mechanism. The proxy class provides a way to control object access, that is, to indirectly access another object through a "proxy" object to add additional behavior or control. In the embodiment of the present application, the proxy class is taken as an example for explanation. The Agent class refers to a class used to implement JavaAgent. The Agent class is mainly used to modify the bytecode when the JVM is started or during operation, thereby changing the behavior of the application without modifying the source code.

[0031] When the target application is started, a Java Virtual Machine (JVM) needs to be started first. The JVM is an environment for running Java programs and is mainly used to interpret or compile Java bytecodes and interact with underlying hardware and operating systems. Detecting the start of the target application may include detecting the start of the JVM of the target application.

[0032] Starting the agent class of the target application can be understood as loading the Agent class of the target application. In other words, when the JVM startup of the target application is detected, the Agent class of the target application is loaded. The Agent class can modify the bytecode file when the JVM is started or during operation.

[0033] Furthermore, the Agent class can be loaded by calling the executable file of the Agent class or calling an API.

[0034] Optionally, the Agent class is injected into the target application through the premain method. The premain method is a special entry point method used to implement the Agent. It executes custom initialization logic before the JVM is started but before the bytecode file is executed.

[0035] S102. Inserting a log collection instruction into a bytecode file of a target application through a proxy class to obtain a modified bytecode file, where the log collection instruction is a bytecode file.

[0036] The bytecode file of the target application is an intermediate file generated by compiling the business code file (.java) of the Java target application using a Java compiler. It has a .class extension. The bytecode file contains the instruction set that the JVM can execute and is an intermediate representation that is independent of the platform. Each .class file corresponds to a Java class or interface and contains information about all methods, fields, and properties of the class or interface.

[0037] The above bytecode files refer to the class files required by the target application that the JVM uses the class loader to load. The class loader will load classes in a certain order, first the bootstrap classes, then the extension classes, and finally the application classes.

[0038] Log collection instructions refer to specific operations or method calls embedded in bytecode files to capture log data during application execution. Log collection instructions can be understood as code logic for capturing log data, which has a .class extension.

[0039] Inserting log collection instructions into the bytecode file includes one of the following methods: Compile-time instrumentation, modifying the bytecode file when the Java compiler compiles the source code file into the bytecode file, using tools to enhance the bytecode after compilation but before packaging, adding log collection instructions, and obtaining the modified bytecode file. Load-time instrumentation: Using Java Agent technology, dynamically modify the bytecode through the java.lang.instrument.ClassFileTransformer interface during the JVM loading the bytecode file. Implementing functions such as adding logging at runtime without changing the original code. Run-time instrumentation, intercepting method calls at runtime and inserting logging logic through reflection or proxy mode.

[0040] After the log collection instruction is inserted into the bytecode file of the target application, a modified bytecode file is obtained. The modified bytecode file includes not only the bytecode file of the target application but also the inserted log collection instruction.

[0041] In the embodiment of the present application, the start of the target application mainly involves the process of starting the JVM and loading the edited bytecode file into the JVM. When the start of the JVM is detected, the Agent class of the target application is loaded.

[0042] JVM does not load all the bytecode files that may be used into memory at once when it starts. It will only load a bytecode file when the target application really needs to use it. Therefore, each time the JVM loads a bytecode file, the log collection instructions can be inserted into the bytecode file through the Agent class to obtain the modified bytecode file.

[0043] Furthermore, the location to insert the log collection instruction is determined based on the specific behavior and needs of monitoring. Specifically, the log collection instruction is inserted at the method entrance and exit. When a method is called, the log collection instruction inserted at the method entrance can be used to record the method entry time, incoming parameters and other information. When a method is about to return or throw an exception, the log collection instruction inserted at the method exit can be used to record the method exit time, return value or exception stack. Insert the log collection instruction at the location of the exception handling block to capture and record exception information. For example, the log collection instruction can be inserted at the end of the try block and in the catch block.

[0044] Insert log collection instructions before and after the read and write operations of the field to record the access of the field. Insert log collection instructions in the static initialization block (i.e., static code block) to record information during class loading.

[0045] When inserting log collection instructions, you need to correctly process the instruction stream and operand stack of the bytecode file to ensure the structural integrity and correctness of the bytecode file and avoid the application from failing to run normally due to bytecode modification.

[0046] In one possible implementation, inserting log collection instructions into the bytecode file of the target application through a proxy class includes: identifying the bytecode file of the target application through the proxy class to obtain at least one key position, where the key position is a position in the bytecode file of the target application where the probability of an exception occurring exceeds a set value; and inserting the log collection instruction at the key position of the bytecode file of the target application.

[0047] Use open source bytecode manipulation libraries (such as ASM, Byte Buddy, etc.) to scan the bytecode files of the target application and identify code locations that may cause exceptions. Open source bytecode manipulation libraries can parse class files and analyze their control flow graphs to determine which code paths may cause exceptions. Some locations where exceptions may occur include: method entry and exit, conditional branches (such as if, switch statements), loop structures (such as for, while loops), and exception handling blocks (such as try-catch-finally). Static analysis can locate code locations with complex logic or potential risks.

[0048] Run the target application in the development environment and record the exception rate of each location where an exception may occur as analyzed by the bytecode operation library. If the exception rate of a location where an exception may occur exceeds the set threshold, the location where the exception may occur is regarded as a key location. Insert log collection instructions at the key location in the bytecode file.

[0049] The above threshold value can be set according to actual conditions, for example, the threshold value is set to 5%.

[0050] S103: During the process of the virtual machine running the modified bytecode file, it is detected that a log collection instruction is triggered, and log data is obtained using the triggered log collection instruction.

[0051] The modified bytecode file and its dependencies are deployed to the virtual machine, the virtual machine starts to execute the modified bytecode file, and the target application enters a normal running state. During the process of the virtual machine running the modified bytecode file, the insertion log collection instruction inserted in the bytecode file continuously monitors the execution flow of the application.

[0052] If the preset conditions are met, the log collection instruction is triggered, and detailed log data is generated according to the pre-defined log collection rules. Furthermore, the log data is formatted into an easy-to-parse form, such as a JSON string, to facilitate subsequent transmission and processing.

[0053] If a situation that meets a preset condition is monitored, the log collection instruction is monitored to be triggered, including: if an exception is detected to be thrown, the log collection instruction is triggered.

[0054] In a possible implementation, the log data includes at least one of the following: exception information of the log collection instruction insertion position, current thread information, and context information of the exception information; the current thread refers to the thread that throws the exception information.

[0055] The exception information of the log collection instruction insertion position refers to relevant detail information captured when an exception occurs at the code position where the log collection instruction is inserted.

[0056] Abnormal information mainly includes one or more of the following:

[0057] Exception stack: The exception stack is a list of information generated by the JVM when an exception is thrown, which contains the call chain when the exception occurs. Each line represents a method call, starting from the innermost method to the outermost method. The exception stack can help locate the exact location of the exception, understand the exception propagation path, and find possible causes.

[0058] Exception type: The exception type refers to the specific type of exception thrown, such as NullPointerException, IllegalArgumentException, etc. The exception type helps to identify the category of the exception, thereby narrowing down the scope of the problem more quickly.

[0059] Exception message: The exception message is a descriptive message carried by the exception object, mainly used to explain what went wrong. The exception message can provide additional context to help understand the cause behind the exception.

[0060] The current thread information can be understood as various information related to the execution thread when the exception occurs. The current thread information mainly includes one or more of the following: thread ID, thread group, thread priority and thread state.

[0061] The thread ID is used to uniquely identify a thread. It is mainly used to identify which thread has an exception when multiple threads are executed in parallel. The thread group can provide information about the hierarchy to which the thread belongs, which helps to locate and manage threads. The thread priority can indicate the execution environment of the thread. The thread state is used to indicate the behavior mode of the thread when an exception occurs, such as whether it is blocked or waiting for a resource.

[0062] The context information of exception information refers to the additional information about the background and environment of the exception when the exception occurs. Context information helps to more fully understand the conditions, causes and scope of the exception, which can greatly simplify the troubleshooting process and provide clues for fixing the problem. The context information of exception information mainly includes one or more of the following: method parameters, local variables, configuration settings, external dependency status, etc.

[0063] If the exception occurs inside a method, the method parameters are used to confirm whether the input data is reasonable or whether there are boundary conditions that are not handled correctly. Local variables are used to reproduce the problem scenario and conduct in-depth analysis of the specific cause of the exception. Configuration settings are used to confirm whether the configuration is correct and whether there are any problems caused by the configuration. The external dependency status mainly records the status information of the interaction with external systems (such as databases, remote services), such as response time, error code, etc. The external dependency status can determine whether the exception is caused by an external dependency and the health of the external system.

[0064] The embodiment of the present application provides a log data management method, which includes: when detecting that the target application is started, starting the proxy class of the target application; inserting a log collection instruction in the bytecode file of the target application through the proxy class to obtain a modified bytecode file, the log collection instruction is a bytecode file; in the process of the virtual machine running the modified bytecode file, monitoring that the log collection instruction is triggered, and obtaining log data using the triggered log collection instruction. Since the log collection instruction for collecting log data is dynamically inserted into the bytecode file through the proxy class of the application during the loading process of the bytecode file of the application, there is no need to modify the source code to implement log recording, thus realizing non-intrusive log collection, maintaining the independence of the business code, reducing the code coupling, and facilitating maintenance and upgrading.

[0065] On the basis of the above-mentioned embodiments, the embodiments of the present application further optimize the log data management method. In the embodiments of the present application, the log data management method is mainly described by taking the deployment of the log data management method in the k8s environment as an example.

[0066] The K8s environment refers to a cluster environment where the Kubernetes platform is deployed, which is mainly used to automate the deployment, expansion, and management of containerized applications.

[0067] Specifically, the target application is deployed in the application container, the proxy class is deployed in the sidecar container, and the sidecar container and the application container are deployed in the same Pod.

[0068] In the K8s environment, Pod is the smallest deployable unit. A Pod can contain one or more containers. Application containers and sidecar containers refer to containers of different roles running in the same Pod.

[0069] An application container is a container that actually runs the target application. For example, if the target application is a web application, then the application container is a container that runs the web server, processes HTTP requests, and returns responses.

[0070] The sidecar container is a design pattern in the K8s environment. In the sidecar mode, additional log collection functions are added to an auxiliary container in the same Pod instead of being directly integrated into the application container. The auxiliary container can be called a sidecar container. The sidecar container itself is not part of the main logic of the application. The sidecar container can collect log data from the application container and forward it to the logging system.

[0071] Furthermore, in the k8s environment, the sidecar container can communicate with the application container by configuring appropriate network policies and shared storage.

[0072] In the k8s environment, network policy is a security feature that limits the traffic that can enter or leave the Pod. For the communication between the sidecar container and the application container, since they are located in the same Pod, the sidecar container and the application container can access each other through localhost.

[0073] Furthermore, you can use the NetworkPolicy resource to specify network policies, such as allowing only traffic from a specific namespace or Pods matching a label selector to enter or leave.

[0074] By configuring persistent volumes and persistent volume claims, or using a temporary empty directory as shared storage, data can be shared between the sidecar container and the application container, providing the ability to exchange data between containers.

[0075] In the embodiment of the present application, the log collection function and the target application are deployed in different containers, achieving complete physical isolation, improving the scalability and flexibility of the system, and facilitating deployment and management in a distributed environment.

[0076] Based on the above embodiment, it is monitored that a log collection instruction is triggered, and log data is obtained by using the triggered log collection instruction, including: using the sidecar container to monitor the log collection instruction by setting a network protocol, where the set network protocol is a network protocol between the sidecar container and the application container; when the sidecar container monitors that the log collection instruction is triggered, the log data obtained by the triggered log collection instruction is received by setting the network protocol.

[0077] Setting the network protocol refers to the communication protocol between the sidecar container and the application container. The setting network protocol may include Hypertext Transfer Protocol (HTTP), gRPC, etc.

[0078] The sidecar container is responsible for receiving the log data collected by the bytecode-enhanced application. The sidecar container listens to the log data from the application through a specific network protocol (such as HTTP, gRPC, etc.). After receiving the log data, the sidecar container performs preliminary processing on the log data, such as data format verification, data compression (if necessary), etc., and then sends the processed log data to the log system.

[0079] In the above embodiment, the method for collecting log data is mainly described. In the embodiment of the present application, the abnormal information in the collected log data is mainly processed. Figure 2 As shown, the process of processing abnormal information provided by the embodiment of the present application mainly includes:

[0080] S201. Determine the exception type corresponding to the exception information.

[0081] In the embodiment of the present application, an inheritance hierarchy of exception classes is pre-constructed, and exceptions are divided into runtime exceptions and check exceptions, and then further subdivided into specific exception types.

[0082] Checked exceptions are exception types inherited from the java.lang.Exception class, excluding RuntimeException and its subclasses. Checked exceptions are exception types that the compiler must handle in order to make staff aware of possible problems and prepare countermeasures in advance.

[0083] Runtime exceptions are also called unchecked exceptions, which are inherited from the java.lang.RuntimeException class or its subclasses, or directly inherited from the Error class. The Error class is used to represent problems encountered by the JVM itself, such as OutOfMemoryError. Runtime exceptions are exceptions that the compiler does not require to be handled. In other words, the program can compile normally even if no handling code is written for possible runtime exceptions.

[0084] Correctly distinguishing and using runtime exceptions and checked exceptions can build more robust and maintainable applications.

[0085] The subtypes of checked exceptions include one or more of the following: input and output exception (IOException), SQL exception (SQLException), class not found exception (ClassNotFoundException), interrupt exception (ClassNotFoundException), method not found exception (NoSuchMethodException), parser configuration exception (ParserConfigurationException), conversion exception (TransformerException), unsupported encoding exception (UnsupportedEncodingException), reflective operation exception (ReflectiveOperationException), cloning not supported exception (CloneNotSupportedException), AWT exception (AWTException).

[0086] The subtypes of runtime exceptions include one or more of the following: ArithmeticException, ArrayIndexOutOfBoundsException, ClassCastException, IllegalArgumentException, IllegalStateException, NullPointerException, NumberFormatException, SecurityException, UnsupportedOperationException, InputMismatchException, StringIndexOutOfBoundsException, NoSuchElementException, and NegativeArraySizeException.

[0087] In a possible implementation, determining the exception type corresponding to the exception information includes: reading an inheritance tree of the exception information; searching the inheritance tree for a parent node of the exception information; and using the exception class corresponding to the parent node as the exception type of the exception information.

[0088] In the embodiment of the present application, the inheritance tree of the exception information is checked. If the exception information is indirectly inherited from Exception, it is directly inherited from the exception class corresponding to the parent node. And the exception class corresponding to the parent node is also directly inherited from Exception and is not a subclass of RuntimeException. Therefore, the exception class corresponding to the parent node is the exception type of the exception information.

[0089] Take the exception information FileNotFoundException as an example to explain and view the inheritance tree of FileNotFoundException.

[0090] The inheritance hierarchy of FileNotFoundException is as follows: Object->Throwable->Exception->IOException->FileNotFoundException.

[0091] From the inheritance tree above, we can see that FileNotFoundException is ultimately inherited indirectly from Exception and directly from IOException. Since IOException is also directly inherited from Exception and is not a subclass of RuntimeException, IOException is the exception type corresponding to FileNotFoundException. In other words, FileNotFoundException is a subtype of the checked exception IOException.

[0092] S202: In the predetermined correspondence between the exception type and the exception level, query the exception level corresponding to the exception information.

[0093] According to the predefined configuration rules, different exception levels are set for different exception types, and different processing methods are set for different exception levels.

[0094] In the embodiment of the present application, two levels of exceptions are set, namely, level 1 and level 2, and the severity of level 1 is higher than that of level 2. In other words, level 1 exceptions refer to exceptions that must be handled immediately, such as exceptions that cause application crashes. Level 2 exceptions refer to minor exceptions that the system may automatically recover from, such as recoverable network connection exceptions.

[0095] The predefined correspondence between the exception type and the exception level, after determining the exception type of the exception information in S201, the exception type and the exception level correspondence are searched based on the exception type of the exception information to obtain the exception level of the exception information.

[0096] S203: If the abnormal level corresponding to the abnormal information is the first level, the abnormal information is sent to the log system.

[0097] If the exception level corresponding to the exception information is the first level, it means that the exception indicated by the exception information is an exception that must be handled immediately. Therefore, it needs to be sent to the log system immediately so that the log system can trigger the alarm mechanism so that the staff can notice the exception in a short time.

[0098] S204: If the abnormal level corresponding to the abnormal information is the second level, the acquired abnormal information is periodically sent to the log system.

[0099] If the exception level corresponding to the exception information is the second level, it means that the exception indicated by the exception information is a relatively minor exception and the system may automatically respond to it. Therefore, a certain amount of exception information can be collected within a certain period and then sent in batches to reduce the pressure on the log system.

[0100] In this embodiment, dynamic processing is performed according to the exception type and level, thereby optimizing the efficiency and accuracy of log sending.

[0101] In one possible implementation, the system load of the log system is obtained; when the system load is greater than a first set threshold, the period for sending the exception information is extended; when the system load is less than a second set threshold, the period for sending the exception information is shortened; the first set threshold is greater than the second set threshold.

[0102] System load refers to the current workload of the computer or server running the log system. System load reflects the usage of system resources (such as CPU, memory, disk I / O, and network I / O). The first set threshold and the second set threshold are two critical points used to characterize the system load level.

[0103] When the system load is greater than the first set threshold, it indicates that the log system is in a high-load state. The sending cycle of abnormal information is extended, the log recording frequency is reduced, etc., to reduce the load pressure of the log system.

[0104] When the system load is lower than the second set threshold, it indicates that the log system has recovered to a relatively idle state, and the sending cycle of abnormal information can be shortened, the log recording frequency can be increased, and the like.

[0105] Taking the CPU load average as an example, the first threshold is set to 3.0 and the second threshold is set to 2.0. When the CPU's 1-minute load average exceeds 3.0, the abnormal information sending cycle will be extended, such as from once every minute to once every five minutes. When the CPU's 1-minute load average drops below 2.0, it will return to the normal shorter sending cycle, such as once every minute.

[0106] The second set threshold is set lower than the first set threshold to avoid frequent state switching due to fluctuation of system load near the threshold, resulting in unnecessary performance overhead or inconsistent log records.

[0107] Figure 3 Schematic diagram of the structure of a log data management device in an embodiment of the present application. Figure 3 As shown, the log data management device 30 provided in the embodiment of the present application mainly includes: an agent class startup module 31, a file modification module 32 and a log acquisition module 33.

[0108] Among them, the proxy class startup module 31 is used to start the proxy class of the target application when it is detected that the target application is started; the file modification module 32 is used to insert log collection instructions into the bytecode file of the target application through the proxy class to obtain a modified bytecode file, and the log collection instructions are bytecode files; the log acquisition module 33 is used to monitor the triggering of the log collection instruction during the process of the virtual machine running the modified bytecode file, and obtain log data using the triggered log collection instruction.

[0109] An embodiment of the present application provides a log data management device, which is mainly used to execute the following process: when it is detected that a target application is started, the proxy class of the target application is started; a log collection instruction is inserted into the bytecode file of the target application through the proxy class to obtain a modified bytecode file, and the log collection instruction is the bytecode file; during the process of the virtual machine running the modified bytecode file, it is monitored that the log collection instruction is triggered, and log data is obtained using the triggered log collection instruction.

[0110] During the loading process of the bytecode file of the application, the log collection instructions for collecting log data are dynamically inserted into the bytecode file through the application proxy class. There is no need to modify the business code to implement logging, thus achieving non-intrusive log collection, maintaining the independence of the business code, reducing code coupling, and facilitating maintenance and upgrades.

[0111] In one possible implementation, the file modification module 32 is specifically used to identify the bytecode file of the target application through a proxy class to obtain at least one key position, where the key position is a code position in the bytecode file of the target application where the probability of an exception occurring exceeds a set value; and insert a log collection instruction at the key position of the bytecode file of the target application.

[0112] In a possible implementation, the log data includes at least one of the following: exception information of the log collection instruction insertion position, current thread information, and context information of the exception information; the current thread refers to the thread that throws the exception information.

[0113] In one possible implementation, the target application is deployed in an application container, the proxy class is deployed in a sidecar container, and the sidecar container and the application container are deployed in the same Pod.

[0114] In one possible implementation, the log acquisition module 33 is specifically used to use the sidecar container to monitor the log collection instruction by setting the network protocol, where the set network protocol is the network protocol between the sidecar container and the application container; when the sidecar container detects that the log collection instruction is triggered, the log data obtained by the triggered log collection instruction is received by setting the network protocol.

[0115] In a possible implementation, it also includes: an exception information processing module, which is used to determine the exception type corresponding to the exception information; in the predetermined correspondence between the exception type and the exception level, query the exception level corresponding to the exception information; if the exception level corresponding to the exception information is the first level, send the exception information to the log system; if the exception level corresponding to the exception information is the second level, periodically send the acquired exception information to the log system.

[0116] In a possible implementation, the exception information processing module is specifically used to read the inheritance tree of the exception information; query the parent node of the exception information in the inheritance tree; and use the exception class corresponding to the parent node as the exception type corresponding to the exception information.

[0117] In a possible implementation, it also includes: a sending period adjustment module, which is used to obtain the system load of the log system; when the system load is greater than a first set threshold, the sending period of the exception information is extended; when the system load is less than a second set threshold, the sending period of the exception information is shortened; the first set threshold is greater than the second set threshold.

[0118] The log data management device provided in the embodiment of the present application can execute the log data management method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0119] Figure 4 is a schematic diagram of the structure of an electronic device provided in this embodiment. The electronic device may include a log data management device, such as Figure 4 As shown, the electronic device 400 includes a processor 410, a memory 420, an input device 430, and an output device 440; the number of the processor 410 in the electronic device can be one or more. Figure 4 A processor 410 is taken as an example; the processor 410, the memory 420, the input device 430 and the output device 440 in the electronic device can be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0120] The memory 420 is a computer-readable storage medium that can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the log data management method in the embodiment of the present invention. The processor 410 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the memory 420, that is, the log data management method provided in the embodiment of the present invention is implemented.

[0121] The memory 420 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 420 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 420 may further include a memory remotely arranged relative to the processor 410, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0122] The input device 430 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the electronic device, and may include a keyboard, a mouse, etc. The output device 440 may include a display device such as a display screen.

[0123] This embodiment also provides a storage medium containing computer executable instructions, and when the computer executable instructions are executed by a computer processor, they are used to implement the log data management method provided by the embodiment of the present invention.

[0124] Of course, the computer executable instructions of a storage medium including computer executable instructions provided by an embodiment of the present invention are not limited to the operations of the method described above, and can also execute related operations in the log data management method provided by any embodiment of the present invention.

[0125] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0126] It is worth noting that in the embodiment of the above-mentioned log data management device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0127] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0128] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A log data management method, characterized in that: The method comprises: When detecting that a target application is started, starting a proxy class of the target application; Inserting a log collection instruction into the bytecode file of the target application through the proxy class to obtain a modified bytecode file, wherein the log collection instruction is a bytecode file; During the process of the virtual machine running the modified bytecode file, it is monitored that the log collection instruction is triggered, and the log data is obtained using the triggered log collection instruction.

2. The method according to claim 1, characterized in that The inserting log collection instructions into the bytecode file of the target application through the proxy class includes: Identifying the bytecode file of the target application by the proxy class to obtain at least one key position, where the key position is a code position in the bytecode file of the target application where the probability of an exception occurring exceeds a set value; Insert log collection instructions at key locations of the bytecode file of the target application.

3. The method according to claim 2, characterized in that The log data includes at least one of the following: exception information of the position where the log collection instruction is inserted, current thread information, and context information of the exception information; the current thread refers to the thread that throws the exception information.

4. The method according to claim 1 or 2, characterized in that: The target application is deployed in an application container, the proxy class is deployed in a sidecar container, and the sidecar container and the application container are deployed in the same Pod.

5. The method according to claim 4, characterized in that The monitoring that the log collection instruction is triggered, and obtaining log data using the triggered log collection instruction, includes: Using the sidecar container to monitor the log collection instruction by setting a network protocol, where the set network protocol is a network protocol between the sidecar container and the application container; The sidecar container detects that the log collection instruction is triggered, and receives log data obtained by the triggered log collection instruction through the set network protocol.

6. The method according to claim 3, characterized in that Also includes: Determine the exception type corresponding to the exception information; In the predetermined correspondence between the exception type and the exception level, query the exception level corresponding to the exception information; If the abnormal level corresponding to the abnormal information is the first level, sending the abnormal information to the log system; If the abnormal level corresponding to the abnormal information is the second level, the acquired abnormal information is periodically sent to the log system.

7. The method according to claim 6, characterized in that The determining the exception type corresponding to the exception information includes: Read the inheritance tree of the exception information; Query the inheritance tree for a parent node of the abnormal information; The exception class corresponding to the parent node is used as the exception type corresponding to the exception information.

8. The method according to claim 6, characterized in that Also includes: Obtaining the system load of the log system; When the system load is greater than a first set threshold, extending a sending period of the abnormal information; When the system load is less than a second set threshold, the sending period of the abnormal information is shortened; and the first set threshold is greater than the second set threshold.

9. A log data management device, characterized in that: The device comprises: The proxy class startup module is used to start the proxy class of the target application when detecting that the target application is started; A file modification module, used for inserting a log collection instruction into the bytecode file of the target application through the proxy class to obtain a modified bytecode file, wherein the log collection instruction is a bytecode file; The log acquisition module is used to monitor that the log collection instruction is triggered during the process of the virtual machine running the modified bytecode file, and obtain log data using the triggered log collection instruction.

10. An electronic device, characterized in that: The device comprises: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the log data management method according to any one of claims 1 to 8.

11. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the log data management method according to any one of claims 1 to 8 is implemented.

Images