Application log collection method and apparatus, device, and storage medium

Abstract

Embodiments of the present application disclose an application log collection method, device, equipment and storage medium. The method comprises the following steps: in response to starting of a target application, loading a log collection agent file associated with the target application; adjusting original log collection statements in an original bytecode file of the target application through bytecode adjustment statements in the log collection agent file, to obtain target log collection statements capable of collecting target log data in a running process of the target application; and obtaining logs in the running process of the application in accordance with business requirements by adjusting the bytecode of the application. The method does not need to modify source code of the application, solves the technical problem that the source code is not reusable in the prior art, does not need to depend on or inherit a log class of the application, solves the problems that the prior art is highly invasive and has high complexity, and the method can collect required logs by loading the log collection agent file, and has low performance loss.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to an application log collection method, apparatus, device, and storage medium. Background Technology

[0002] Currently, the development of an application project is usually carried out collaboratively by multiple people. During the application development phase, different developers insert different logging code into the application source code. This may result in the application outputting logs that do not meet business requirements after development is completed. For example, it may output a large number of useless logs, or fail to output the logs required by the business, thus wasting log storage space and reducing the efficiency of viewing and locating application problems.

[0003] To address the issue of application-output runtime logs not meeting business requirements, existing technologies typically employ hard-coding, relying on or inheriting the logging class provided by the application to modify the application's source code; alternatively, aspect-oriented programming is used to modify the application's source code so that the application can output runtime logs that meet business requirements.

[0004] In the process of developing this invention, the inventors discovered at least the following technical problems in the prior art: Hard-coding requires dependence on or inheritance of the application's log class, resulting in strong intrusion into the system and high implementation complexity. Furthermore, hard-coding requires modification of the application's source code, making the application source code unusable. Aspect-oriented programming (AOP) is runtime-based, incurring significant performance overhead, and also requires modification of the application's source code. Summary of the Invention

[0005] This invention provides an application log collection method, apparatus, device, and storage medium.

[0006] According to one aspect of the present invention, an application log collection method is provided, the method comprising:

[0007] In response to the startup of the target application, a log collection agent file associated with the target application is loaded, wherein the log collection agent file contains bytecode adjustment statements;

[0008] Based on the bytecode adjustment statements, the original bytecode file of the target application is accessed, and the original log collection statements in the original bytecode file are adjusted to obtain the target bytecode file.

[0009] Based on the target log collection statement in the target bytecode file, target log data corresponding to the target log collection statement is collected during the operation of the target application.

[0010] According to another aspect of the present invention, an application log collection device is provided, the device comprising:

[0011] The proxy file loading module is used to load a log collection proxy file associated with the target application in response to the startup of the target application, wherein the log collection proxy file contains bytecode adjustment statements;

[0012] The bytecode adjustment module is used to access the original bytecode file of the target application based on the bytecode adjustment statement, and adjust the original log collection statement in the original bytecode file to obtain the target bytecode file.

[0013] The log collection module is used to collect target log data corresponding to the target log collection statement in the target bytecode file during the operation of the target application.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0015] At least one processor; and

[0016] A memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the application log collection method according to any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the application log collection method described in any embodiment of the present invention.

[0019] One embodiment of the above invention has the following advantages or beneficial effects:

[0020] By responding to the startup of the target application, a log collection proxy file associated with the target application is loaded simultaneously. The original log collection statements in the target application's original bytecode file are adjusted using bytecode adjustment statements in the log collection proxy file, resulting in target log collection statements capable of collecting target log data during the application's runtime. This method obtains logs that meet business requirements during application runtime by adjusting the application's bytecode. It does not require modification of the application's source code, solving the technical problem of non-reusable source code in existing technologies. Furthermore, it does not require dependency on or inheritance of the application's log class, addressing the issues of high invasiveness and complexity in existing technologies. Additionally, this method achieves the required log collection simply by loading the log collection proxy file, resulting in minimal performance overhead.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A flowchart illustrating an application log collection method according to an embodiment of the present invention;

[0024] Figure 2 A flowchart illustrating another application log collection method provided in one embodiment of the present invention;

[0025] Figure 3A A flowchart illustrating another application log collection method provided in one embodiment of the present invention;

[0026] Figure 3B This is a schematic diagram of a log collection process provided in one embodiment of the present invention;

[0027] Figure 3C This is a schematic diagram of an application log processing procedure provided in one embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of the structure of an application log collection device according to an embodiment of the present invention;

[0029] Figure 5 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] Figure 1 This is a flowchart illustrating an application log collection method according to an embodiment of the present invention. This embodiment is applicable to applications that have already been developed. To obtain logs that meet business requirements during application operation, a pre-configured log collection agent file is loaded when the application starts, thereby adjusting the application's bytecode file to obtain logs that meet business requirements during application operation. This method can be executed by an application log collection device, which can be implemented in hardware and / or software and can be configured in electronic devices such as computers, smartphones, or smart tablets. Figure 1 As shown, the method includes:

[0033] S110. In response to the startup of the target application, load the log collection agent file associated with the target application, wherein the log collection agent file contains bytecode adjustment statements.

[0034] The target application can be a pre-developed application. For example, the target application can be a monolithic application or a distributed application. The log collection agent file associated with the target application can be a pre-generated file attached to the target application, which will be loaded when the target application starts.

[0035] For example, the source code files of the target application and the log collection agent files associated with the target application can be pre-configured on the same server, and a loading command for loading the log collection agent files when the target application starts can be set. According to the loading command, the log collection agent files are loaded when the target application is detected to start.

[0036] In this embodiment, the log collection agent file can be used to adjust the original bytecode file of the target application. For example, the target collection agent file can be a JAR file. Specifically, the log collection agent file contains bytecode adjustment statements. After loading the target collection agent file, the original bytecode file of the target application can be adjusted using the bytecode adjustment statements in the target collection agent file. These bytecode adjustment statements can be used to delete, modify, or insert information in the original bytecode file of the target application.

[0037] It should be noted that, in this embodiment, the purpose of loading the log collection proxy file for adjusting the original bytecode file of the target application in response to the startup of the target application is to adjust the original bytecode file of the target application through the bytecode adjustment statements in the log collection proxy file, thereby adjusting the log code collection logic in the target application, so that the target application can output logs that meet business requirements, thus avoiding the need to adjust the source code of the target application.

[0038] In this embodiment, corresponding bytecode adjustment statements can be pre-configured based on the source code of the target application, and then log collection agent files can be generated based on the bytecode adjustment statements.

[0039] In one specific implementation, the method provided in this embodiment further includes: identifying key statements in the source code of the target application for connecting middleware corresponding to the target application; and identifying bytecode adjustment statements based on the key statements and the source code.

[0040] The middleware corresponding to the target application can be a relational database, a distributed database (such as Hadoop Database), a key-value database, or an index database (such as Elasticsearch). Key statements can be statements in the source code used to connect to the middleware, such as the critical paths of the middleware. Specifically, key statements can be located in the target application's source code, and further, bytecode adjustments can be made based on these key statements and the source code.

[0041] For example, determining bytecode adjustment statements based on key statements and source code can be as follows: determine the original log collection statements in the source code used for log collection, determine the required log collection statements based on the key statements, and determine the bytecode adjustment statements based on the original log collection statements and the required log collection statements.

[0042] The requirement log collection statement can be a statement used to collect connection information between the target application and middleware during runtime. Each key statement used to connect to the middleware can correspond to a requirement log collection statement, used to collect the connection information between the target application and each middleware, that is, the input parameter information and output parameter information of the target application.

[0043] Specifically, determining bytecode adjustment statements based on original log collection statements and demand log collection statements can be achieved by: identifying repeated statements in the original log collection statements and other statements besides demand log collection statements as log collection statements to be deleted, and generating bytecode adjustment statements based on the log collection statements to be deleted; and / or, identifying other statements in the demand log collection statements besides original log collection statements as log collection statements to be added, and generating bytecode adjustment statements based on the log collection statements to be added.

[0044] The advantage of locating key statements and determining bytecode adjustment statements based on these key statements and the source code is that the connection information between the application and the middleware is usually important. By locating the key statements in the source code that connect to the middleware, and generating bytecode adjustment statements from these key statements, the target application can output key log information about the connection to the middleware. This enables the collection of information about the connection between the target application and the middleware, such as the input or output parameters of the target application, which facilitates subsequent troubleshooting and location of problems in the target application.

[0045] In this embodiment, in addition to operation statements used to perform deletion, modification or addition operations on the bytecode file, the bytecode adjustment statements may also include basic statements that support editing the bytecode file, or statements used to call other bytecode editing libraries.

[0046] In one specific implementation, determining bytecode adjustment statements based on key statements and source code includes: determining bytecode adjustment statements based on key statements, source code, and call statements for bytecode editing libraries.

[0047] Specifically, operation statements can be generated based on key statements and source code. These operation statements include at least one of deletion, modification, and addition statements. Furthermore, bytecode adjustment statements are determined based on the operation statements and the call statements for the bytecode editing library.

[0048] Operation statements are used to perform deletion, modification, or addition operations on bytecode files, while call statements are used to invoke bytecode editing libraries to support the execution of operation statements through files within the bytecode editing libraries. For example, a call statement could be a statement that calls a Log interface method and needs to be inserted at the location where the main information is collected.

[0049] In this implementation, bytecode adjustment statements are generated through key statements, source code, and call statements for the bytecode editing library. This enables the bytecode editing function of the bytecode editing library to be invoked through the call statements, thereby realizing bytecode adjustment based on the bytecode editing library. There is no need to configure basic statements that support bytecode file editing separately in the log collection agent file; the existing bytecode editing library can be called directly, which improves the generation efficiency of the log collection agent file.

[0050] For example, bytecode editing libraries include language programming assistant libraries, such as the Javassist library. Of course, bytecode editing libraries are not limited to language programming assistant libraries; they can also be libraries within bytecode manipulation frameworks (such as the Java bytecode manipulation framework).

[0051] It should be noted that the advantages of calling the language programming assistant library are: compared with the bytecode manipulation framework, the language programming assistant library is more lightweight. By calling the language programming assistant library, the loading resources of the log collection agent file can be reduced, and the efficiency of adjusting the raw bytecode file of the target application can be improved.

[0052] S120. Based on the bytecode adjustment statements, access the original bytecode file of the target application and adjust the original log collection statements in the original bytecode file to obtain the target bytecode file.

[0053] Specifically, after loading the log collection agent file, the bytecode adjustment statements in the log collection agent file can be used to access the target application's original bytecode file and modify, delete, or insert the original log collection statements in the original bytecode file.

[0054] The raw bytecode file can be an intermediate code file compiled from the source code of the target application. For example, the raw bytecode file can be a .class file in Java, composed of bytecode compiled from the source code. The raw log collection statements in the raw bytecode file can be code composed of individual bytecode segments.

[0055] In one specific implementation, adjusting the original log collection statements in the original bytecode file includes at least one of the following: deleting some of the original log collection statements in the original bytecode file based on the deletion statements in the bytecode adjustment statements; modifying some or all of the original log collection statements in the original bytecode file based on the modification statements in the bytecode adjustment statements; determining the log collection statements to be added based on the addition statements in the bytecode adjustment statements, and inserting the log collection statements to be added into the original bytecode file.

[0056] The delete statement can be used to remove parts of the original log collection statements from the original bytecode file, such as deleting duplicate original log collection statements. The modify statement can be used to update parts of the original log collection statements in the original bytecode file, such as changing the log storage path. The add statement can insert the log collection statements to be added into the original bytecode file.

[0057] Specifically, after adjusting the original log collection statements in the original bytecode file, the adjusted original bytecode file can be used as the target bytecode file.

[0058] In the above implementation, the deletion statement in the bytecode adjustment statement can be used to delete duplicate or useless log collection statements, thereby avoiding the collection of a large amount of useless or duplicate log data and solving the technical problem of editing useless or duplicate log collection statements in the source code during the application development phase. The modification statement in the bytecode adjustment statement can be used to modify erroneous log collection statements, thereby avoiding the inability to collect log data due to errors in the log collection statements and solving the technical problem of editing erroneous log collection statements in the source code during the application development phase. The addition statement in the bytecode adjustment statement can be used to add required log collection statements, thereby enabling the collection of business requirement logs and solving the technical problem of not editing useful log collection statements in the source code during the application development phase.

[0059] S130. Based on the target log collection statement in the target bytecode file, collect the target log data corresponding to the target log collection statement during the operation of the target application.

[0060] Specifically, after adjusting the original bytecode file to obtain the target bytecode file, target log data during the execution of the target application can be collected based on the target log collection statements in the target bytecode file. The target log collection statements can be code within the target bytecode file used for log collection.

[0061] For example, after obtaining the target bytecode file, the target code file corresponding to the target application can be determined based on the target bytecode file. Furthermore, by running the target code file, the target application can automatically output target log data corresponding to the target log collection statements.

[0062] Specifically, a virtual machine can be used to interpret the target bytecode file into a target code file. For example, the target code file can be a machine code file. For instance, when the virtual machine starts, it can adjust the original bytecode file based on the log collection agent file and interpret the target bytecode file into an executable machine code file.

[0063] It should be noted that the method provided in this embodiment adjusts the original bytecode file of the target application without affecting the source code file of the target application, thus solving the technical problem in the prior art where modifying the source code leads to the non-reusability of the source code. In the method provided in this embodiment, if the target application needs to output logs for other business requirements, the log collection proxy file can be reconfigured according to the business requirements, and the target application can be restarted to obtain the original bytecode file of the target application again. Then, the original bytecode file can be adjusted using the method provided in this embodiment to obtain log data that meets the business requirements, making application log collection flexible. Furthermore, if there are security risks, the prior art requires checking all directly and indirectly referenced dependency packages, while the method provided in this embodiment can directly locate the problem through the collected application runtime logs without checking the referenced dependency packages.

[0064] The technical solution of this embodiment loads a log collection proxy file associated with the target application in response to the application's startup. By adjusting the bytecode in the log collection proxy file, the original log collection statements in the target application's original bytecode file are modified to obtain target log collection statements capable of collecting target log data during the application's operation. This method obtains logs that meet business requirements during application operation by adjusting the application's bytecode. It does not require modification of the application's source code, solving the technical problem of non-reusable source code in existing technologies. Furthermore, it does not require dependence on or inheritance of the application's log class, solving the problems of high invasiveness and complexity in existing technologies. In addition, this method achieves the collection of required logs simply by loading the log collection proxy file, resulting in minimal performance overhead.

[0065] Figure 2 This is a flowchart illustrating another application log collection method provided in an embodiment of the present invention. Based on the above embodiment, this embodiment adds a process for collecting logs of the process of adjusting the original bytecode file based on bytecode adjustment statements, which can be based on proxy log collection statements in the log collection proxy file. For example... Figure 2 As shown, the method includes:

[0066] S210. In response to the startup of the target application, load the log collection agent file associated with the target application, wherein the log collection agent file contains bytecode adjustment statements and proxy log collection statements.

[0067] S220. Based on the bytecode adjustment statements, access the original bytecode file of the target application and adjust the original log collection statements in the original bytecode file to obtain the target bytecode file.

[0068] S230. Based on the target log collection statement in the target bytecode file, collect the target log data corresponding to the target log collection statement during the operation of the target application.

[0069] S240. Based on the proxy log collection statement, logs are collected on the process of adjusting the original bytecode file based on the bytecode adjustment statement to obtain the proxy log data corresponding to the log collection proxy file.

[0070] Among them, the proxy log collection statement can be a statement in the log collection proxy file used to collect log data after the log collection proxy file is loaded.

[0071] Specifically, after the log collection agent file is loaded, the log collection agent program written in the file is executed. The log collection agent program can adjust the original bytecode file of the target application according to the bytecode adjustment statements, so that the target application performs log collection based on the adjusted target bytecode file. In this embodiment, the proxy log collection statements can be used to collect the runtime logs of the log collection agent program, that is, to collect logs of the process of adjusting the original bytecode file based on the bytecode adjustment statements, and obtain proxy log data.

[0072] In this embodiment, corresponding bytecode adjustment statements can be pre-configured based on the source code of the target application, and statements for collecting log data during the bytecode adjustment process can also be configured.

[0073] In one specific implementation, the method provided in this embodiment further includes: obtaining a first statement for defining the proxy log storage path and a second statement for defining the proxy log level; and determining the proxy log collection statement based on the first statement and the second statement.

[0074] In other words, the configuration process for the proxy log collection statement can be divided into the following steps: First, define the storage path of the proxy log data to obtain the first statement; then, define the log level of the proxy log data to obtain the second statement; finally, construct the proxy log collection statement based on the first statement and the second statement.

[0075] In this context, the log level of the proxy log data represents the output priority of the logs. For example, the proxy log level can be: severe (normal information), warning (warning), info (info), configuration (config), fine (friendly information), finer (more friendly information), and finest (most friendly information); higher-level log data can be output first.

[0076] In the above implementation, the first and second statements can be generated by default, or they can be obtained in response to user operations on the terminal device, with the user editing the first and / or second statements. By obtaining the first and second statements and determining the proxy log collection statements based on them, log collection of the bytecode adjustment process is achieved, facilitating subsequent troubleshooting and location of problems.

[0077] In the above implementation, in addition to the first and second statements, the proxy log collection statement may also include basic statements that support log collection, or it may also include statements for calling the log framework.

[0078] In one specific implementation, the proxy log collection statement is determined based on the first statement and the second statement, including: determining the proxy log collection statement based on the first statement, the second statement, and the call statement for the log framework.

[0079] Specifically, the statement that invokes the logging framework can call the logging implementation class within the logging framework. In particular, the statement that invokes the framework, the first statement, and the second statement can be combined to obtain the statement for proxy log collection.

[0080] The advantage of determining the proxy log collection statement based on the call statement for the logging framework is that it enables log collection based on the existing logging framework without having to configure basic statements supporting log implementation separately in the log collection proxy file, thus improving the generation efficiency of the log collection proxy file.

[0081] For example, the logging framework includes the system's native logging framework. For instance, the system's native logging framework could be the JUL (Java Util Logging) framework. The advantages of using a native logging framework are: log collection can be implemented using the built-in class library without referencing other redundant class libraries, improving the efficiency of generating log collection proxy files while reducing the resources required to load them.

[0082] Of course, in addition to the first and second statements, a third statement for defining the agent log size limit and a fourth statement for defining the agent log output format can also be obtained; furthermore, agent log collection statements can be constructed based on the first, second, third, and fourth statements.

[0083] It should also be noted that in this embodiment, the execution order of S230 and S240 is not important. It can be that during the process of adjusting the original bytecode file based on the bytecode adjustment statement, the proxy log data is collected based on the proxy log collection statement, and after adjusting the original bytecode file based on the bytecode adjustment statement, the target log data is collected based on the adjusted target bytecode file.

[0084] The technical solution of this embodiment adjusts the original bytecode file of the target application based on the bytecode adjustment statements in the log collection proxy file so that the target application outputs target log data. At the same time, based on the proxy log collection statements in the log collection proxy file, logs are collected for the adjustment process of the original bytecode file, and proxy log data related to the log collection proxy file is obtained. This realizes log collection for the log collection proxy file, which facilitates subsequent troubleshooting and location.

[0085] Figure 3A This is a flowchart illustrating another application log collection method provided in an embodiment of the present invention. Based on the above embodiment, this embodiment adds a process of associating and storing target log data and / or proxy log data according to log transmission statements in a log collection proxy file. For example... Figure 3A As shown, the method includes:

[0086] S310. In response to the startup of the target application, load the log collection agent file associated with the target application, wherein the log collection agent file contains bytecode adjustment statements, proxy log collection statements and log transmission statements.

[0087] S320. Based on the bytecode adjustment statements, access the original bytecode file of the target application and adjust the original log collection statements in the original bytecode file to obtain the target bytecode file.

[0088] S330. Based on the target log collection statement in the target bytecode file, collect the target log data corresponding to the target log collection statement during the operation of the target application.

[0089] S340. Based on the proxy log collection statement, logs are collected on the process of adjusting the original bytecode file based on the bytecode adjustment statement to obtain the proxy log data corresponding to the log collection proxy file.

[0090] S350. Based on the log transmission statement, generate a log identifier corresponding to the current log data to be transmitted, and send the current log data to be transmitted and the log identifier to the index database for associated storage; wherein, the current log data to be transmitted includes target log data and / or proxy log data.

[0091] In this embodiment, after collecting target log data and proxy log data, the target log data and / or proxy log data can be sent to an index database for storage according to the log transmission statements in the log collection proxy file. The index database can be an Elasticsearch database.

[0092] Specifically, based on the log transmission statement, a log identifier corresponding to the log data to be transmitted can be generated, and then the log data to be transmitted and the log identifier can be associated and stored in the index database.

[0093] For example, generating a log identifier corresponding to the log data to be transmitted can be done by: generating a log identifier corresponding to the log data to be transmitted based on the data source of the log data to be transmitted, so that log data with the same data source have the same log identifier, and log data with different data sources have different log identifiers.

[0094] In this context, "same data source" can mean both the same target application and the same client requesting the target application. For example, a thread-local variable can be used to determine the client identifier corresponding to the current client, and then this client identifier can be used as the log identifier corresponding to the target log data generated by the current client.

[0095] See Figure 3B The diagram illustrates a log collection process. Taking a single target application as an example, when multiple clients concurrently request access to the target application, corresponding target log data (i.e., traffic, which can be request or response data) can be collected for each client. Furthermore, a log identifier can be generated for each client's corresponding target log data. The target log data and its corresponding log identifier are then associated and stored in an index database to ensure traffic consistency for the same client, ensuring that the traffic identifier for the same client is always the same, and preventing the inability to distinguish the traffic of different clients.

[0096] Considering that when the amount of log data to be transmitted is large, high-concurrency write operations to the index database may put too much pressure on the index database, a message queue can be used to buffer high-concurrency write operations to the index database.

[0097] In one specific implementation, sending the current log data to be transmitted and the log identifier to the index database for associated storage includes: if it is detected that the amount of log data to be transmitted exceeds a preset threshold, then sending the current log data to be transmitted and the log identifier to the index database read area in the preset message queue corresponding to the index database, so that the index database can obtain the current log data to be transmitted and the log identifier from the index database read area.

[0098] The preset message queue can be understood as middleware between the target application and the index database, or as middleware between the log collection agent file and the index database. For example, the preset message queue could be Kafka middleware, RabbitMessage Queue, ZeroMessage Queue, or MeteMessage Queue.

[0099] Specifically, when the amount of log data exceeds a preset threshold, the current log data to be transmitted and the log identifier can be sent to a preset message queue, so that the preset message queue can cache the current log data to be transmitted and the log identifier in the index database reading area. Furthermore, the index database can read the current log data to be transmitted and the log identifier from the index database reading area.

[0100] By setting a preset message queue to cache log data, the inability to handle high-concurrency write operations to the index database is avoided, ensuring that log data can be written to the index database.

[0101] After the target log data and / or proxy log data are written to the index database, the front end can retrieve and display the log data stored in the index database by sending a log data viewing request.

[0102] For example, such as Figure 3C The diagram illustrates an application log processing procedure. It shows how target log data is collected from each target application by loading log collection agent files associated with that application. The target log data is then sent to a pre-defined message queue, which stores the log data in an index database. Furthermore, the front-end can request the target log data from the index database via an interface provided by the index database and display the obtained log data.

[0103] The technical solution of this embodiment generates a corresponding log identifier for the log data to be transmitted by means of log transmission statements in the log collection agent file. Then, the log data to be transmitted and the log identifier are sent to the index database for associated storage. This makes it easy to distinguish the source of each log data in the index database according to the identifier. At the same time, it can also ensure that the identifiers of log data with the same source in the index database are consistent, which helps in troubleshooting and locating problems.

[0104] Figure 4 This is a schematic diagram of an application log collection device provided in one embodiment of the present invention. Figure 4 As shown, the device includes a proxy file loading module 410, a bytecode adjustment module 420, and a log collection module 430.

[0105] The proxy file loading module 410 is used to load the log collection proxy file associated with the target application in response to the startup of the target application. The log collection proxy file contains bytecode adjustment statements.

[0106] The bytecode adjustment module 420 is used to access the original bytecode file of the target application based on the bytecode adjustment statement, and adjust the original log collection statement in the original bytecode file to obtain the target bytecode file.

[0107] The log collection module 430 is used to collect target log data corresponding to the target log collection statement in the target bytecode file during the operation of the target application.

[0108] The technical solution of this embodiment, through a proxy file loading module, loads a log collection proxy file associated with the target application in response to the target application's startup. Through a bytecode adjustment module, based on the bytecode adjustment statements in the log collection proxy file, the original log collection statements in the target application's original bytecode file are adjusted to obtain target log collection statements capable of collecting target log data during the target application's operation. Then, through a log collection module, log collection is performed based on the target bytecode file. By adjusting the application's bytecode, logs that meet business requirements during application operation can be obtained. This method does not require modification of the application's source code, solving the technical problem of non-reusable source code in existing technologies. Furthermore, it does not require dependence on or inheritance of the application's log class, solving the problems of high invasiveness and complexity in existing technologies. In addition, this method can achieve the required log collection simply by loading the log collection proxy file, resulting in minimal performance overhead.

[0109] Based on the above embodiments, the application log collection device further includes a proxy collection module; the proxy collection module is used to collect logs of the process of adjusting the original bytecode file based on the bytecode adjustment statement, based on the proxy log collection statement, to obtain proxy log data corresponding to the log collection proxy file.

[0110] Based on the above embodiments, the application log collection device further includes a log transmission module, which is used to generate a log identifier corresponding to the currently transmitted log data based on the log transmission statement, and send the currently transmitted log data and the log identifier to an index database for associated storage; wherein, the currently transmitted log data includes the target log data and / or the proxy log data.

[0111] Based on the above embodiments, the log transmission module is further configured to, if it is detected that the amount of log data to be transmitted exceeds a preset threshold, send the log data to be transmitted and the log identifier to the index database reading area in a preset message queue corresponding to the index database, so that the index database can obtain the log data to be transmitted and the log identifier from the index database reading area.

[0112] Based on the above embodiments, the application log collection device further includes an adjustment statement generation module; the adjustment statement generation module is used to determine key statements in the source code of the target application for connecting middleware corresponding to the target application, and to determine the bytecode adjustment statement based on the key statements and the source code.

[0113] Based on the above embodiments, the application log collection device further includes a proxy statement generation module; the proxy statement generation module is used to obtain a first statement for defining the proxy log storage path and a second statement for defining the proxy log level; and to determine the proxy log collection statement based on the first statement and the second statement.

[0114] Based on the above embodiments, the adjustment statement generation module is further used to determine the bytecode adjustment statement based on the key statement, the source code, and the call statement for the bytecode editing library.

[0115] Based on the above embodiments, the proxy statement generation module is further configured to determine the proxy log collection statement based on the first statement, the second statement, and the call statement for the log framework.

[0116] Based on the above embodiments, the bytecode adjustment module 420 is also configured to perform at least one of the following operations:

[0117] Based on the deletion statement in the bytecode adjustment statement, some original log collection statements in the original bytecode file are deleted; based on the modification statement in the bytecode adjustment statement, some or all of the original log collection statements in the original bytecode file are modified; based on the addition statement in the bytecode adjustment statement, log collection statements to be added are determined and inserted into the original bytecode file.

[0118] Based on the above embodiments, the bytecode editing library includes a language programming assistant library.

[0119] Based on the above embodiments, the logging framework includes the system's native logging framework.

[0120] The application log collection device provided in this embodiment of the invention can execute the application log collection method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0121] Figure 5 This is a schematic diagram of an electronic device provided according to one embodiment of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0122] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0123] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0124] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as application log collection methods.

[0125] In some embodiments, the application log collection method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the application log collection method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the application log collection method by any other suitable means (e.g., by means of firmware).

[0126] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0127] Computer programs used to implement the application log collection method of embodiments of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on the machine, partially on the machine, or as a standalone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0128] This invention also provides a computer-readable storage medium storing computer instructions for causing a processor to execute an application log collection method, the method comprising:

[0129] In response to the startup of the target application, a log collection agent file associated with the target application is loaded, wherein the log collection agent file contains bytecode adjustment statements;

[0130] Based on the bytecode adjustment statements, the original bytecode file of the target application is accessed, and the original log collection statements in the original bytecode file are adjusted to obtain the target bytecode file.

[0131] Based on the target log collection statement in the target bytecode file, target log data corresponding to the target log collection statement is collected during the operation of the target application.

[0132] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0133] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0134] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0135] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0136] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0137] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An application log collection method characterized by comprising: The method comprises: loading a log collection agent file associated with the target application in response to the target application being started, wherein the log collection agent file contains bytecode adjustment statements and agent log collection statements; accessing an original bytecode file of the target application based on the bytecode adjustment statements, and adjusting original log collection statements in the original bytecode file to obtain a target bytecode file; collecting target log data corresponding to the target log collection statements in the target bytecode file during running of the target application based on the target log collection statements in the target bytecode file; collecting a process of adjusting the original bytecode file based on the bytecode adjustment statements based on the agent log collection statements to obtain agent log data corresponding to the log collection agent file; The method further comprises: obtaining a first statement for defining an agent log storage path and a second statement for defining an agent log level, wherein the agent log level represents the output priority of the log; determining the agent log collection statement based on the first statement and the second statement.

2. The method of claim 1, wherein, The log collection agent file further contains a log transmission statement, and the method further comprises: generating a log identifier corresponding to current to-be-transmitted log data based on the log transmission statement, and sending the current to-be-transmitted log data and the log identifier to an index database for associated storage; wherein the current to-be-transmitted log data comprises the target log data and / or the agent log data.

3. The method of claim 2, wherein, The sending of the current to-be-transmitted log data and the log identifier to the index database for associated storage comprises: if it is detected that the log data amount of the current to-be-transmitted log data exceeds a preset threshold, sending the current to-be-transmitted log data and the log identifier to an index library reading area in a preset message queue corresponding to the index database, so that the index database obtains the current to-be-transmitted log data and the log identifier from the index library reading area.

4. The method of claim 1, wherein, The method further comprises: determining a key statement for connecting middleware corresponding to the target application in source code of the target application; determining the bytecode adjustment statement based on the key statement and the source code.

5. The method of claim 4, wherein, The determining of the bytecode adjustment statement based on the key statement and the source code comprises: determining the bytecode adjustment statement based on the key statement, the source code, and a call statement for a bytecode editing library.

6. The method of claim 1, wherein, The determining of the agent log collection statement based on the first statement and the second statement comprises: determining the agent log collection statement based on the first statement, the second statement, and a call statement for a log framework.

7. The method of claim 1, wherein, The adjusting of the original log collection statements in the original bytecode file comprises at least one of the following: performing deletion processing on part of the original log collection statements in the original bytecode file based on a deletion statement in the bytecode adjustment statement; modifying part or all of original log collection statements in the original bytecode file based on the modification statements in the bytecode adjustment statements; determining to-be-added log collection statements based on the addition statements in the bytecode adjustment statements, and inserting the to-be-added log collection statements into the original bytecode file.

8. An application log collecting apparatus characterized by comprising: Comprise: The agent file loading module is configured to load a log collection agent file associated with the target application in response to the target application being started, wherein the log collection agent file contains bytecode adjustment statements and agent log collection statements; The bytecode adjustment module is configured to access an original bytecode file of the target application based on the bytecode adjustment statements, and adjust original log collection statements in the original bytecode file to obtain a target bytecode file; The log collection module is configured to collect target log data corresponding to target log collection statements in the target bytecode file during running of the target application based on the target log collection statements; The agent collection module is configured to collect agent log data corresponding to the log collection agent file based on the agent log collection statements based on a process of adjusting the original bytecode file based on the bytecode adjustment statements. The device further comprises: The agent statement generation module is configured to obtain a first statement for defining an agent log storage path, and a second statement for defining an agent log level; wherein the agent log level represents an output priority of a log; and the agent log collection statement is determined based on the first statement and the second statement.

9. An electronic device, comprising: The electronic device comprises: At least one processor; and The memory is in communication connection with the at least one processor; wherein The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the application log collection method in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for enabling the processor to execute the application log collection method in any one of claims 1-7 when executed.

Images