Log aggregation method and system based on MDC, electronic equipment and storage medium

Abstract

The invention provides an MDC-based log aggregation method and system, electronic equipment and a storage medium. The method comprises the steps of obtaining a user operation request; the user operation request is used for indicating a user to operate the system; tracking and aggregating logs involved in the user operation request based on the MDC according to the operation object type involved in the user operation request; the operation object type is at least one of single-thread operation, cross-thread operation and cross-service operation; the log tracking comprises at least one of thread internal log tracking, cross-thread log tracking and cross-service log tracking; that is to say, thread internal log tracking, cross-thread log aggregation and cross-service log aggregation can be realized; operation and maintenance are more convenient, and problem positioning is more convenient.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and more specifically, particularly relates to a log aggregation method and system based on MDC, an electronic device, and a storage medium. Background Art

[0002] In modern software development, logging is an integral part that helps developers monitor the status of applications, diagnose problems, and optimize performance. Currently, many systems use the log4j component as their logging solution. log4j is a widely used Java logging tool that provides flexible configuration options and powerful log management capabilities.

[0003] The log4j component allows developers to record log information at different levels (such as DEBUG, INFO, WARN, ERROR, etc.) by integrating it in the background service. This integration makes log generation and maintenance relatively simple. However, although log4j provides basic logging functions, it has some limitations in log management in multi-threaded and microservice architectures.

[0004] However, there are challenges in dealing with log aggregation in a multi-threaded and multi-service environment, which makes it impossible to effectively collect and analyze log data from different threads, that is, it is impossible to aggregate logs. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a log aggregation method and system based on MDC, an electronic device, and a storage medium for realizing thread internal log tracking, cross-thread log aggregation, and cross-service log aggregation; making operation and maintenance more convenient and locating problems more convenient.

[0006] The first aspect of the present application discloses a log aggregation method based on MDC, including:

[0007] Obtaining a user operation request; the user operation request is used to indicate a user's request to operate the system;

[0008] According to the type of operation object involved in the user operation request, the logs involved in the user operation request are tracked and aggregated based on MDC; the operation object type is at least one of single-thread operation, cross-thread operation and cross-service operation; log tracking includes: at least one of thread internal log tracking, cross-thread log tracking and cross-service log tracking.

[0009] Optionally, when the operation object type includes a single-thread operation, the log tracking includes: thread internal log tracking;

[0010] When the operation object type includes a cross-thread operation, the log tracking includes: cross-thread log tracking;

[0011] When the operation object type includes a cross-service operation, the log tracking includes: cross-service log tracking.

[0012] Optionally, when the log tracking is thread internal log tracking, the tracking and aggregating of the logs involved in the user operation request based on the MDC includes:

[0013] Intercept the service request corresponding to the user operation request through AOP;

[0014] Creating an identifier corresponding to the service request; the identifier is used to indicate a unique identifier of the service request; different service requests have different corresponding identifiers;

[0015] The identifier is saved to the aggregation field of MDC so that all methods in the current thread can access the aggregation field, and when each method writes a log, the log framework automatically prints the aggregation field and aggregates all logs written by the current service for the user operation request.

[0016] Optionally, the identifier is a random number; different service requests correspond to different random numbers.

[0017] Optionally, when the log tracing is cross-thread log tracing, the tracing and aggregating the logs involved in the user operation request based on the MDC includes:

[0018] The main thread creates the MDC aggregation field according to the thread internal log tracking;

[0019] When the main thread creates a child thread, the data of the aggregation field of the MDC in the main thread is copied to the aggregation field of the MDC of the child thread, so that all methods in the child thread can access the aggregation field of the child thread, and when each method writes a log, the log framework automatically prints the aggregation field, and aggregates all logs written by the main thread and the child thread for the user operation request.

[0020] Optionally, when the log tracking is cross-service log tracking, the tracking and aggregating of the logs involved in the user operation request based on the MDC includes:

[0021] When the user accesses the first service, the first service generates an aggregate field according to the thread internal log tracking;

[0022] When the first service accesses the second service, the data of the aggregation field is obtained from the MDC of the first service, and the data of the aggregation field is set in the request header of the http request, and the http request is issued;

[0023] After the second service receives the http request from the first service, the second service parses the http request, obtains the data of the aggregation field of the first service, and saves the data of the aggregation field to the aggregation field of the MDC of the second service, so that all methods in the first service and the second service can access their respective corresponding aggregation fields, and the data of the aggregation fields are the same. The log framework automatically prints the aggregation field and aggregates all logs written by the first service and the second service for the user operation request.

[0024] The second aspect of the present application discloses a log aggregation system based on MDC, including:

[0025] An acquisition module, used to acquire a user operation request; the user operation request is used to indicate a user's request to operate the system;

[0026] A tracing module is used to track and aggregate logs involved in the user operation request based on the MDC according to the type of operation object involved in the user operation request; the operation object type is at least one of single-thread operation, cross-thread operation and cross-service operation; log tracing includes: at least one of thread internal log tracing, cross-thread log tracing and cross-service log tracing.

[0027] Optionally, the tracking module is used to track and aggregate the logs involved in the user operation request based on the MDC according to the type of the operation object involved in the user operation request, specifically to:

[0028] When the log tracking is internal log tracking of a thread: intercept the service request corresponding to the user operation request through AOP; create an identifier corresponding to the service request; the identifier is used to indicate a service request; different service requests have different corresponding identifiers; save the identifier to the aggregation field of MDC so that all methods in the current thread can access the aggregation field, and when each method writes a log, the log framework automatically prints the aggregation field, and aggregates all logs written by the current service for the user operation request;

[0029] When the log tracking is cross-thread log tracking: the main thread creates an aggregate field of MDC according to the internal log tracking of the thread; when the main thread creates a child thread, the data of the aggregate field of MDC in the main thread is copied to the aggregate field of MDC in the child thread, so that all methods in the child thread can access the aggregate field of the child thread, and when each method writes a log, the log framework automatically prints the aggregate field, and aggregates all logs written by the main thread and the child thread for the user operation request;

[0030] When the log tracking is cross-service log tracking: when the user accesses the first service, the first service generates an aggregate field according to the internal log tracking of the thread; when the first service accesses the second service, the data of the aggregate field is obtained from the MDC of the first service, and the data of the aggregate field is set in the request header of the http request, and an http request is issued; after the second service receives the http request from the first service, the second service parses the http request, obtains the data of the aggregate field of the first service, and saves the data of the aggregate field to the aggregate field of the MDC of the second service, so that all methods in the first service and the second service can access their respective corresponding aggregate fields, and the data of the aggregate fields are the same, the log framework automatically prints the aggregate field, and aggregates all logs written by the first service and the second service for the user operation request.

[0031] The third aspect of the present application discloses an electronic device, comprising:

[0032] one or more processors;

[0033] a storage device having one or more programs stored thereon;

[0034] When one or more programs are executed by one or more processors, the one or more processors implement the MDC-based log aggregation method as described in any one of the first aspects of the present application.

[0035] A fourth aspect of the present application discloses a storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the MDC-based log aggregation method as described in any one of the first aspects of the present application is implemented.

[0036] It can be seen from the above technical scheme that the present invention provides a log aggregation method based on MDC, including: obtaining user operation requests; user operation requests are used to indicate user requests for operating the system; based on the type of operation objects involved in the user operation requests, tracking and aggregating logs involved in the user operation requests based on MDC; the operation object type is at least one of single-thread operation, cross-thread operation and cross-service operation; log tracking includes: at least one of thread internal log tracking, cross-thread log tracking and cross-service log tracking; that is, the present application can realize thread internal log tracking, cross-thread log aggregation, and cross-service log aggregation; making operation and maintenance more convenient and locating problems more convenient. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 is a flow chart of a log aggregation method based on MDC provided by an embodiment of the present invention;

[0039] Figure 2-Figure 5 is a flow chart of another log aggregation method based on MDC provided in an embodiment of the present invention;

[0040] Figure 6-Figure 9 It is a verification schematic diagram involved in a log aggregation method based on MDC provided in an embodiment of the present invention;

[0041] Fig.10 is a flow chart of another log aggregation method based on MDC provided in an embodiment of the present invention;

[0042] Figure 11-13 It is a verification schematic diagram involved in a log aggregation method based on MDC provided in an embodiment of the present invention;

[0043] Fig.14 is a schematic diagram of a log aggregation system based on MDC provided in an embodiment of the present invention;

[0044] Fig.15 It is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] In the present application, the term "comprise", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements defined by the sentence "comprise one ..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements. In addition, the terms "first", "second", "third", "fourth", etc. (if present) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable in appropriate circumstances, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here, for example.

[0047] Terminology explanation:

[0048] tomcat: backend application server.

[0049] ThreadLocal: Thread variable, which means that the variables filled in ThreadLocal belong to the current thread.

[0050] http request: service request that complies with the http protocol.

[0051] HTTP header: Part of the HTTP protocol, used to pass context information between the client and the server.

[0052] MDC: MDC (Mapped Diagnostic Context), a thread-level log context storage mechanism that allows context information to be stored and passed in the same thread.

[0053] TraceId: Link tracking ID, usually a 32-bit random number, used for log aggregation.

[0054] AOP: Spring AOP (Aspect-Oriented Programming) is an aspect-oriented programming technology provided by the Spring framework.

[0055] TTL: TTL scheme (transmittable-thread-local), a technology for transmitting aggregate field information between threads.

[0056] httpclient: a java-based http service access component.

[0057] Okhttp: A Java-based http service access component.

[0058] Swagger codegen: Swagger is an API interface management component; Swagger codegen automatically generates components based on Swagger's client SDK.

[0059] mq: Message queue, message middleware.

[0060] skywalking: Distributed link tracing component.

[0061] The embodiment of the present application discloses a log aggregation method based on MDC, which is used to solve the challenges in the prior art when processing log aggregation in a multi-threaded and multi-service environment, which makes it impossible to effectively collect and analyze log data from different threads, that is, it is impossible to perform log aggregation.

[0062] See also Figure 1 , the log aggregation method based on MDC includes:

[0063] S101: Obtain a user operation request.

[0064] User operation requests are used to indicate user requests to operate the system.

[0065] In the process of user interaction with application programs or operating systems, user operation requests are instructions issued by users to the system in some way (such as clicking, inputting, dragging, etc.). These requests need to be correctly identified and processed by the system.

[0066] The operation request may include at least one of: login, registration, query, add, delete, modify, submit form, trigger event; of course, it may also include other types, which will not be described here one by one, and are all within the protection scope of this application.

[0067] S102: According to the type of operation object involved in the user operation request, the logs involved in the user operation request are tracked and aggregated based on the MDC.

[0068] The operation object type is at least one of a single-thread operation, a cross-thread operation, and a cross-service operation.

[0069] It should be noted that services usually refer to background threads running in the operating system, providing functions or data to other programs, users, or the operating system itself. Services usually run for a long time, even if no user directly interacts with them. For example: print services, network services, database services, etc. in the Windows operating system.

[0070] A thread is the smallest unit of program execution and the smallest unit that the operating system can schedule operations on. A thread is contained in a thread and is the actual running unit in a thread. For example, a web server may use multiple threads to process requests from multiple clients at the same time.

[0071] Services can contain multiple threads to support concurrent operations and improve efficiency. Threads are the basis for services to implement concurrency and multitasking.

[0072] Single-threaded operation: This operation object type refers to all user request processing performed in a single thread. In single-threaded mode, all tasks are executed sequentially without parallel processing. This method is simple, easy to implement and debug, but may not be suitable for situations that require high concurrency processing. Single-threaded operation is suitable for scenarios with simple tasks and low concurrency requirements.

[0073] Cross-thread operations: Cross-thread operations involve multiple independent threads. Each thread may run in a different memory space and have its own resources. When a user operation request needs to share data or coordinate tasks between different threads, a cross-thread operation is required. This operation may require an inter-thread communication (IPC) mechanism such as pipes, message queues, shared memory, or sockets. Cross-thread operations are suitable for scenarios that require thread isolation, high security, or clear resource allocation.

[0074] Cross-service operations: Services usually refer to background programs that run independently in an operating system and can provide specific functions or resources. Cross-service operations refer to user requests that require multiple services to interact or work together. For example, in a microservice architecture, a user request may need to trigger a series of operations on multiple services to complete. Cross-service operations may require network communication and may involve complex mechanisms such as service discovery and load balancing. Cross-service operations are suitable for complex applications built on distributed systems that require high scalability and high availability.

[0075] Log tracing includes at least one of: intra-thread log tracing, cross-thread log tracing, and cross-service log tracing.

[0076] Log tracing is a technology used in software development to monitor and debug applications. It records detailed information about the program during operation, including state changes, user behavior, system events, etc. According to the scope and context of log tracing, it can be divided into thread-internal log tracing, cross-thread log tracing, and cross-service log tracing.

[0077] Intra-thread log tracing: Occurs in a single thread and records a series of operations or events for a specific service or component. All log records are executed sequentially in the same thread, without concurrency issues. It is easy to understand and analyze because the order of log records is consistent with the order of code execution.

[0078] Cross-thread log tracing: When the application uses multiple threads, cross-thread log tracing records operations or events in different threads. Log records may come from different threads, and the thread identity needs to be specially marked to distinguish the log source; synchronization mechanisms may be required to avoid race conditions when logging. It is suitable for multi-threaded applications that need to monitor and debug concurrent operations in different threads.

[0079] Cross-service log tracing: In a distributed system, cross-service log tracing records the interactions and events of different services or components. Log records may be distributed across different service instances or servers and need to be collected and consolidated over the network. A distributed tracing system may be required to correlate related log records from different services. It is suitable for microservice architectures or distributed systems where services need to work together to complete complex business logic.

[0080] MDC (Mapped Diagnostic Context) is a tool used in multithreaded environments that allows developers to associate information (called context) with execution threads. This context information can be any data, but is often used to store relevant information in log records, such as user ID, session ID, transaction ID, etc. MDC is a feature of Apache Log4j and some other logging frameworks, but its concept can be applied to any logging system that requires context information.

[0081] When a user operation starts, relevant context information, such as a user identifier (such as a user ID), a request identifier (such as a request ID), etc., is placed in the MDC.

[0082] When user operations generate logs during execution, these logs automatically contain the context information in the MDC. This means that all logs recorded in the same MDC context will contain the same context information.

[0083] Passing MDC across threads: If your operation involves multiple threads, make sure to pass the MDC context when creating a thread or dispatching a task. This can be achieved by copying the MDC content to the new thread before starting the thread.

[0084] Cross-service MDC transfer: In a distributed system, if user operations need to be performed across services, the MDC context can be transferred to other services through http headers, message queue attributes, or other mechanisms.

[0085] Log aggregation tools or services can aggregate logs from different threads or services based on the context information in the MDC. In this way, even if the logs are distributed in multiple locations, they can be associated based on the context information.

[0086] The present application can be applied to imaging platforms, and of course other platforms, which will not be described in detail here, and are all within the scope of protection of the present application.

[0087] Optionally, when the operation object type includes single-threaded operation, the log tracing includes: thread internal log tracing.

[0088] When the operation object type includes a cross-thread operation, the log tracing includes: cross-thread log tracing.

[0089] When the operation object type includes a cross-service operation, log tracing includes: cross-service log tracing.

[0090] It should be noted that cross-thread log tracing and cross-service log tracing also involve thread internal log tracing; cross-service tracing may also involve cross-thread log tracing.

[0091] In this embodiment, a user operation request is obtained; the user operation request is used to indicate a user's request to operate the system; based on the type of operation object involved in the user operation request, the logs involved in the user operation request are tracked and aggregated based on the MDC; the operation object type is at least one of a single-thread operation, a cross-thread operation, and a cross-service operation; the log tracking includes: at least one of thread internal log tracking, cross-thread log tracking, and cross-service log tracking; that is, the present application can implement thread internal log tracking, cross-thread log aggregation, and cross-service log aggregation; making operation and maintenance more convenient and locating problems more convenient.

[0092] Different operation requests correspond to different log tracing. The following describes thread internal log tracing, cross-thread log tracing, and cross-service log tracing respectively:

[0093] (1) When log tracking is thread internal log tracking, the logs involved in the user operation request are tracked and aggregated based on MDC, including:

[0094] First, intercept the service request corresponding to the user operation request through AOP; second, create an identifier corresponding to the service request; the identifier is used to indicate the unique identifier of the service request; different service requests have different corresponding identifiers; finally, save the identifier to the aggregation field of MDC so that all methods in the current thread can access the aggregation field, and when each method writes a log, the log framework automatically prints the aggregation field and aggregates all logs written by the current service for the user operation request.

[0095] Optionally, the identifier is a random number; different service requests correspond to different random numbers; the random number is a set of digital sequences arranged according to the principle of randomness, and each number has the same probability of appearing in the sequence. Of course, the identifier can also be a character, which is not specifically limited here and can be determined according to the actual situation, all of which are within the scope of protection of this application.

[0096] It should be noted that operation requests usually refer to specific behavior requests issued by users or systems, such as login, registration, query, add, delete, modify, submit form, trigger event, etc. These operation requests are the direct reflection of the interaction between users and systems, and each operation corresponds to a specific task that the user wants to perform.

[0097] Service requests are more like interactions between background services. In a microservices architecture, a service may call another service to complete certain functions. This cross-service call is a service request. Service requests usually involve more complex business logic and data processing.

[0098] User operations trigger service requests; user operation requests (such as submitting a form) may trigger a series of processing by the background service, which in turn generates service requests (such as calling user services to verify form data).

[0099] Service requests implement operation requests; backend services complete operation requests issued by users by processing service requests. For example, when a user requests to register a new account, the backend may call multiple services such as user services and email services to complete the registration process.

[0100] The operation request is the trigger point of the service request; in many cases, the user's operation request is the starting point of the background service request. Each user operation may trigger a series of background service calls and processing.

[0101] When querying logs, you can easily analyze the logs of a background service through log aggregation. This aggregation field is implemented through MDC.

[0102] Specifically, first use AOP to generate a random number at the entrance of all background services (such as the Controller layer) and save it in the MDC. The various methods of subsequent threads can obtain the aggregate fields (such as the traceId field) from the MDC and print them to the log. More specifically, first, create an aspect class to define pointcuts and advice. Secondly, use the @Pointcut annotation to define a pointcut, which is usually selected on the entry method of the Controller layer. Then, in the before advice (BeforeAdvice) of the pointcut, generate a random number and store it in the MDC using the put method of the MDC. Finally, get and print the aggregate field from the MDC: In the logging method, get the previously stored aggregate field from the MDC and print it to the log.

[0103] like Figure 2 As shown in the figure, when the client initiates a request to service A, the backend system intercepts the request through AOP technology. Before service A starts processing, the system automatically generates a unique random number and stores this value as an aggregate field in the MDC. The aggregate field in the MDC acts as a context identifier for the log record, so that all methods involved in processing the request in service A can access this unified identifier.

[0104] Regardless of how method 1, method 2, or method 3 is called internally by service A, these methods are all running in the same service thread, so they can all access the aggregate field in the MDC. When these methods perform logging operations, the logging framework (whether Log4j or other frameworks) will automatically include the aggregate field in the log output. In this way, all log entries related to service A will carry the same aggregate field, making it possible to track and associate all log information generated by service A when processing a specific request.

[0105] That is to say, set the MDC aggregation field at the entrance of all services through AOP; in the log configuration, use this field as the aggregation field of the aggregate log; set the log4j configuration file, and log4j will automatically print the MDC aggregation field into the log file.

[0106] In this way, the log aggregation function is realized, which greatly facilitates problem diagnosis and system monitoring and improves the availability of log data.

[0107] It should be noted that the generated random number needs to be guaranteed to be unique throughout the entire life cycle of service A to avoid log confusion between different requests.

[0108] Each method is contained in the background service A, and executes the business logic of service A.

[0109] In most business processes, the main thread of a single background service is sufficient to handle requests. However, in special scenarios such as imaging platforms, background services may need to start child threads to perform additional tasks, such as asynchronous file transfer. In order to maintain consistency and convenience in problem location, even in child threads, it is necessary to ensure that logs can be effectively aggregated with logs in the main thread. This requires that when a child thread is started, the MDC aggregation field in the main thread can be passed to the child thread to ensure that the log entries of the two can be associated through the same context identifier.

[0110] (2) When log tracing is cross-thread log tracing, the logs involved in the user operation request are tracked and aggregated based on MDC, including:

[0111] First, the main thread creates the aggregation field of MDC according to the internal log tracking of the thread; that is, the MDC data of the main thread is the identifier previously set by AOP; then, when the main thread creates a child thread, the data of the aggregation field of MDC in the main thread is copied to the aggregation field of MDC in the child thread, so that all methods in the child thread can access the aggregation field of the child thread, and when each method writes a log, the log framework automatically prints the aggregation field, and aggregates all logs written by the main thread and the child thread for user operation requests.

[0112] That is to say, before the child thread is started, the current aggregate field is obtained from the MDC; the obtained aggregate field is passed to the child thread; in the child thread, the same aggregate field is used to update its MDC to ensure the consistency of log output.

[0113] Therefore, no matter it is the main thread or the child thread, the log output of these threads will carry the same aggregation field, so that the log framework can realize cross-thread log aggregation based on the aggregation field. This not only improves the readability and traceability of the log, but also provides strong support for the stable operation of the system and problem diagnosis.

[0114] There are many scenarios for creating child threads, mainly for some time-consuming operations, such as saving files or accessing remote network services. By creating child threads, the processing efficiency of the main thread can be improved and the main thread can be prevented from getting stuck.

[0115] In the operation of the image platform, we often encounter tasks that require asynchronous processing, especially in the immediate upload mode. Once the image index creation service completes its core responsibility, that is, saving the image index, it will trigger an asynchronous thread to continue the upload task. Although this upload action is performed independently in the asynchronous thread, it is closely related to the image index creation service and is an important part of the entire service process.

[0116] In order to maintain the integrity of the entire service process and ensure that the source of the problem can be quickly located during problem diagnosis, log aggregation between the main thread and the asynchronous thread is required. This means that no matter whether the operation occurs in the main thread or the asynchronous thread, the related log records should carry the same context information so that they can be effectively associated.

[0117] like Figure 3 As shown in the figure, at the entrance of the main thread, a globally unique identifier is generated through AOP technology and stored in MDC. This identifier will be used as the aggregation field of log aggregation; when the main thread creates a child thread, a copy of the main thread's MDC data is copied to the child thread, so that the child thread has the aggregation field in the main thread's MDC; with this aggregation field, each method of the child thread (such as Figure 3 The method E) shown can access this aggregation field; the log framework can connect the logs of the main thread and the child thread in series based on this aggregation field, and finally realize the function of service log aggregation.

[0118] That is to say, if the child thread can access this aggregate field, then when the log is printed in the child thread later, the log framework can automatically obtain the aggregate field in the child thread and print it, ultimately realizing the log aggregation function.

[0119] In the technical architecture of the AIP-Cloud platform, the above cross-thread synchronization aggregation field mechanism can be adopted to create and manage asynchronous threads. It can automatically realize log aggregation with the main thread while the asynchronous thread is generated. This means that no matter where this mechanism is applied, the asynchronous thread will carry the context information of the main thread to ensure the consistency of log records.

[0120] Of course, there are other solutions, such as the TTL (transmittable-thread-local) mechanism, which can realize the transmission of aggregated fields between threads, but this solution often requires a custom MDC adapter or a custom thread pool, which is relatively cumbersome. In contrast, the adopted mechanism stands out for its simplicity and ease of use, providing a more elegant and efficient solution for log aggregation in a multi-threaded environment.

[0121] It should be noted that in many scenarios, cross-service response to user operation requests is required, and therefore, when crossing services, log tracking and aggregation are still required. For example, in a triggered upload scenario, the ViewTrans (asynchronous triggered upload service) / SyncViewTrans (synchronous triggered upload service) service calls the transferFileToDms (file triggered upload service) service published by the branch cache to trigger the upload; the branch cache calls the upload service published by the head office cache to upload the file; these are a complete process, and the steps to trigger the upload need to be aggregated. For example, in the cross-network image copy service, the services involved in AIP-Cloud include: the user end accesses the entry / recordCopy of the cross-network copy master service; the master service accesses the source index information service / getMappingAllDetail to obtain the information to be copied; the master service accesses the target index information saving service / saveMappingAllDetail to save the target index information, so this application proposes cross-service log tracking.

[0122] (3) When log tracing is cross-service log tracing, the logs involved in the user operation request are tracked and aggregated based on the MDC, including:

[0123] First, when the user accesses the first service, the first service generates an aggregate field according to the thread internal log tracking; second, when the first service accesses the second service, it obtains the data of the aggregate field from the MDC of the first service, sets the data of the aggregate field to the request header of the http request, and issues an http request; finally, after the second service receives the http request from the first service, the second service parses the http request, obtains the data of the aggregate field of the first service, and saves the data of the aggregate field to the aggregate field of the MDC of the second service, so that all methods in the first service and the second service can access their respective corresponding aggregate fields, and the data of the aggregate field is the same, the log framework automatically prints the aggregate field, and aggregates all logs written by the first service and the second service for user operation requests.

[0124] That is to say, the traceId in the client thread MDC is set to the http header and passed to the server; a filter is set on the server to receive the request from the client and parse the traceId in the http header. If traceId is found, it is saved in the MDC as an aggregate field.

[0125] More specifically, Figure 4 As shown:

[0126] 1) Log aggregation initialization of service A: When a user first accesses service A, the system automatically creates a unique aggregation field, namely traceId. This traceId is then saved in the MDC as the aggregation basis for all log records of service A. Therefore, when each method in service A (such as method A) prints a log, it will automatically include this traceId, thereby achieving log aggregation and tracking.

[0127] 2) Cross-service transmission of traceId: Before service A needs to access service B, the system will extract the traceId from the MDC and pass it to service B as a request header of the http request.

[0128] 3) Service B receives traceId: After receiving the http request from service A, service B parses the request header through a preset filter and extracts the traceId from it.

[0129] 4) Log aggregation settings for service B: Once the traceId is resolved, service B uses it as its own aggregation field and sets it in the MDC, thus taking over the log context from service A.

[0130] 5) Log aggregation of service B: At this time, the logs printed by each method executed in service B (such as method E) will automatically include this traceId, ensuring that even in service B, log records can be aggregated by traceId.

[0131] Implementation of cross-service node log tracing: In this way, traceId is transmitted between different service nodes, successfully implementing cross-service node log tracing. This not only strengthens the consistency of log data, but also provides strong support for system problem diagnosis and performance optimization.

[0132] The specific application scenarios are described below:

[0133] (1) Figure 5 As shown in the figure, taking the trigger upload service of the imaging platform AIP-Cloud accessing DMS-Cloud through httpclient as an example, it explains how to implement cross-service log aggregation:

[0134] SyncViewTrans service overall timing Figure 5First, the browser accesses AIP-Cloud and initiates a SyncViewTrans service request; second, AIP-Cloud generates a traceId; then, AIP-Cloud calls the / transFileToDms service of the branch cache dms-cloud and passes the traceId to the branch cache through the http header; finally, the branch cache dms-cloud calls the / upload service of the head office cache dms-cloud to upload the file and passes the traceId to the head office cache through the http header.

[0135] The code implementation can be: AIP-Cloud sets traceId through httpclient interceptor; so when AIP-Cloud accesses branch cache dms-cloud, it can pass traceId to branch cache dms-cloud. After receiving the request, branch cache dms-cloud obtains traceId from AIP-Cloud through filter; branch cache dms-cloud sets traceId through httpclient interceptor; so when branch cache DMS-Cloud accesses head office cache, it can pass traceId to head office cache dms-cloud. Head office cache dms-cloud obtains traceId from branch cache dms-cloud through filter.

[0136] After completing the above steps, the aggregate field traceId can be transferred between AIP-Cloud / branch cache / total cache.

[0137] Before using this method, the aggregation fields of AIP-Cloud and DMS-Cloud are managed separately. Figure 6 As shown in the figure, the aggregation field of AIP-Cloud is: 3013e604059a4bfe8762a6c7182e0e96; Figure 7 As shown in the figure, the aggregate field of DMS-Cloud is: c9e9340ce96949c4b1ae2a75c610e5ee. That is, the image background service and the image transmission service print different aggregate fields.

[0138] After using this method, if Figure 8 As shown in the figure, the aggregation fields of the AIP-Cloud background service syncviewtrans are all: 4847fbe87c5a4f1bb0f8b313469a2ac5; Fig. 9As shown in the figure, the aggregate field of the branch cache dms-cloud is also: 4847fbe87c5a4f1bb0f8b313469a2ac5; that is, the aggregate fields printed by the image background service and the image transmission service are consistent, achieving the goal of log aggregation.

[0139] (2) Fig.10 As shown in the figure, taking cross-network copy as an example, it explains how to implement cross-service log aggregation:

[0140] First, the user accesses the AIP-Cloud platform through a browser and initiates a cross-network copy service request pointing to / recordCopy. Secondly, after receiving the cross-network copy request, AIP-Cloud first generates a globally unique tracking identifier traceId to maintain the consistency and traceability of the log throughout the copy process. Then, AIP-Cloud subsequently calls the / getMappingAIIDetail service on the AIP-Cloud instance where the original image index is located to obtain the source index information. In this process, traceId is passed to the AIP-Cloud node where the original image index is located through the http request header, ensuring the consistency of the log information. Finally, after obtaining the source index information, AIP-Cloud continues to call the / saveMappingAIIDetail service on the AIP-Cloud instance where the target image is located to save the target index directory. Similarly, traceId is also passed to the AIP-Cloud node where the target image is located through the http request header, thereby ensuring the log aggregation and consistency of operations across service nodes.

[0141] During the code deployment phase, you need to define an Okhttp interceptor OkhttpTraceIdInterceptor in AIP-Cloud. The reason why the Okhttp interceptor is used here instead of reusing the original httpClient interceptor httpClientTraceIdInterceptor is that in cross-network access, AIP-Cloud calls the jar package of swagger codegen to access the peer AIP-Cloud service, and the underlying layer of swagger codegen is Okhttp. You also need to register the Okhttp interceptor OkhttpTraceIdInterceptor. Specifically, register the Okhttp interceptor in GetMapping and SaveMapping respectively. You also need to configure a filter in AIP-Cloud. Specifically, AIP-Cloud configures the filter: LogTraceFilter. It is used to parse the traceId field in the http request header and to aggregate fields (if there is a traceId field in the http requestheader, create a random number as the traceId). At the same time, the MDC aggregation field originally set in LogRecordAspect.java should be commented out.

[0142] The code can be deployed to a test environment and verified.

[0143] Specifically, Fig.11 As shown, adjust the cross-network data; source APP: TESTP target APP: TEST; and call the test case and view the log. Fig.12 As shown in the figure, the aggregated fields of the record service and the getmapping service are: 15bbcbb875714b6e954e5898f480c5c8. It should be noted that the address of the record service entry and the getmapping service address for obtaining the source index are both 125. Fig.13 As shown in the figure, the aggregate field of the savemapipng service is also: 15bbcbb875714b6e954e5898f480c5c8. It should be noted that the address of the savemapping service that saves the target index is 158.

[0144] Another embodiment of the present application provides a log aggregation system based on MDC.

[0145] See also Fig.14 , a log aggregation system based on MDC, including:

[0146] The acquisition module 101 is used to acquire a user operation request; the user operation request is used to indicate a user's request to operate the system.

[0147] The tracking module 102 is used to track and aggregate the logs involved in the user operation request based on the MDC according to the type of operation object involved in the user operation request; the operation object type is at least one of single-thread operation, cross-thread operation and cross-service operation; the log tracking includes: at least one of thread internal log tracking, cross-thread log tracking and cross-service log tracking.

[0148] Optionally, the tracking module 102 is used to track and aggregate the logs involved in the user operation request based on the MDC according to the type of the operation object involved in the user operation request, specifically to:

[0149] When the log tracking is internal thread log tracking: intercept the service request corresponding to the user operation request through AOP; create an identifier corresponding to the service request; the identifier is used to indicate a service request; different service requests have different corresponding identifiers; save the identifier to the aggregation field of MDC so that all methods in the current thread can access the aggregation field, and when each method writes a log, the log framework automatically prints the aggregation field, and aggregates all logs written by the current service for the user operation request;

[0150] When the log tracking is cross-thread log tracking: the main thread creates the MDC aggregation field according to the internal log tracking of the thread; when the main thread creates a child thread, the data of the MDC aggregation field in the main thread is copied to the MDC aggregation field of the child thread, so that all methods in the child thread can access the aggregation field of the child thread, and when each method writes a log, the log framework automatically prints the aggregation field, and aggregates all logs written by the main thread and the child thread for user operation requests.

[0151] When the log tracking is cross-service log tracking: when the user accesses the first service, the first service generates an aggregate field according to the thread internal log tracking; when the first service accesses the second service, the data of the aggregate field is obtained from the MDC of the first service, and the data of the aggregate field is set to the request header of the http request, and an http request is issued; after the second service receives the http request from the first service, the second service parses the http request, obtains the data of the aggregate field of the first service, and saves the data of the aggregate field to the aggregate field of the MDC of the second service, so that all methods in the first service and the second service can access their respective corresponding aggregate fields, and the data of the aggregate field is the same, the log framework automatically prints the aggregate field, and aggregates all logs written by the first service and the second service for user operation requests.

[0152] For details of the specific working process and principle of each of the above modules, please refer to the MDC-based log aggregation method provided in the above embodiment, which will not be described here one by one. It depends on the actual situation and is within the protection scope of this application.

[0153] In this embodiment, the acquisition module 101 acquires a user operation request; the user operation request is used to indicate a user's request for an operation on the system; the tracking module 102 tracks and aggregates the logs involved in the user operation request based on the MDC according to the type of operation object involved in the user operation request; the operation object type is at least one of a single-thread operation, a cross-thread operation, and a cross-service operation; the log tracking includes: at least one of thread internal log tracking, cross-thread log tracking, and cross-service log tracking; that is, the present application can implement thread internal log tracking, cross-thread log aggregation, and cross-service log aggregation; making operation and maintenance more convenient and locating problems more convenient.

[0154] Another embodiment of the present application provides a storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the MDC-based log aggregation method as described in any one of the above embodiments is implemented.

[0155] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0156] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0157] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0158] Another embodiment of the present invention provides an electronic device, such as Fig.15 As shown, including:

[0159] One or more processors 201.

[0160] The storage device 202 stores one or more programs.

[0161] When one or more programs are executed by one or more processors 201 , the one or more processors 201 implement the MDC-based log aggregation method as described in any one of the above embodiments.

[0162] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart.

[0163] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

[0164] Although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0165] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0166] The features recorded in the various embodiments in this specification can be replaced or combined with each other, and the same and similar parts between the various embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0167] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0168] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A log aggregation method based on MDC, characterized in that: include: Get user operation request; The user operation request is used to indicate a user's request to operate the system; According to the type of operation object involved in the user operation request, based on the MDC, the logs involved in the user operation request are tracked and aggregated; The operation object type is at least one of a single-thread operation, a cross-thread operation, and a cross-service operation; the log tracing includes at least one of a thread internal log tracing, a cross-thread log tracing, and a cross-service log tracing.

2. The log aggregation method based on MDC according to claim 1, characterized in that: When the operation object type includes single-thread operation, the log tracking includes: thread internal log tracking; When the operation object type includes a cross-thread operation, the log tracking includes: cross-thread log tracking; When the operation object type includes a cross-service operation, the log tracking includes: cross-service log tracking.

3. The log aggregation method based on MDC according to claim 1, characterized in that: When the log tracking is thread internal log tracking, the tracking and aggregating of the logs involved in the user operation request based on the MDC includes: Intercept the service request corresponding to the user operation request through AOP; Creating an identifier corresponding to the service request; the identifier is used to indicate a unique identifier of the service request; different service requests have different corresponding identifiers; The identifier is saved to the aggregation field of MDC so that all methods in the current thread can access the aggregation field, and when each method writes a log, the log framework automatically prints the aggregation field and aggregates all logs written by the current service for the user operation request.

4. The log aggregation method based on MDC according to claim 3 is characterized in that: The identifier is a random number; different service requests correspond to different random numbers.

5. The log aggregation method based on MDC according to claim 1, characterized in that: When the log tracking is cross-thread log tracking, the tracking and aggregating of the logs involved in the user operation request based on the MDC includes: The main thread creates the MDC aggregation field according to the thread internal log tracking; When the main thread creates a child thread, the data of the aggregation field of the MDC in the main thread is copied to the aggregation field of the MDC of the child thread, so that all methods in the child thread can access the aggregation field of the child thread, and when each method writes a log, the log framework automatically prints the aggregation field, and aggregates all logs written by the main thread and the child thread for the user operation request.

6. The log aggregation method based on MDC according to claim 1, characterized in that: When the log tracking is cross-service log tracking, the tracking and aggregating of the logs involved in the user operation request based on the MDC includes: When the user accesses the first service, the first service generates an aggregate field according to the thread internal log tracking; When the first service accesses the second service, the data of the aggregation field is obtained from the MDC of the first service, and the data of the aggregation field is set in the request header of the http request, and the http request is issued; After the second service receives the http request from the first service, the second service parses the http request, obtains the data of the aggregation field of the first service, and saves the data of the aggregation field to the aggregation field of the MDC of the second service, so that all methods in the first service and the second service can access their respective corresponding aggregation fields, and the data of the aggregation fields are the same. The log framework automatically prints the aggregation field and aggregates all logs written by the first service and the second service for the user operation request.

7. A log aggregation system based on MDC, characterized in that: include: An acquisition module is used to obtain user operation requests; The user operation request is used to indicate a user's request to operate the system; A tracking module, used for tracking and aggregating logs involved in the user operation request based on the MDC according to the type of operation object involved in the user operation request; The operation object type is at least one of a single-thread operation, a cross-thread operation, and a cross-service operation; the log tracing includes at least one of a thread internal log tracing, a cross-thread log tracing, and a cross-service log tracing.

8. The MDC-based log aggregation system according to claim 7, characterized in that: The tracking module is used to track and aggregate the logs involved in the user operation request based on the MDC according to the type of the operation object involved in the user operation request, specifically for: When the log tracking is thread internal log tracking: intercept the service request corresponding to the user operation request through AOP; create an identifier corresponding to the service request; the identifier is used to indicate a service request; different service requests have different corresponding identifiers; The identifier is saved to the aggregation field of the MDC so that all methods in the current thread can access the aggregation field, and when each method writes a log, the log framework automatically prints the aggregation field and aggregates all logs written by the current service for the user operation request; When the log tracking is cross-thread log tracking: the main thread creates an MDC aggregation field according to the thread internal log tracking; When the main thread creates a child thread, the data of the aggregation field of the MDC in the main thread is copied to the aggregation field of the MDC of the child thread, so that all methods in the child thread can access the aggregation field of the child thread, and when each method writes a log, the log framework automatically prints the aggregation field, and aggregates all logs written by the main thread and the child thread for the user operation request; When the log tracking is cross-service log tracking: when the user accesses the first service, the first service generates an aggregate field according to the internal log tracking of the thread; when the first service accesses the second service, the data of the aggregate field is obtained from the MDC of the first service, and the data of the aggregate field is set in the request header of the http request, and an http request is issued; after the second service receives the http request from the first service, the second service parses the http request, obtains the data of the aggregate field of the first service, and saves the data of the aggregate field to the aggregate field of the MDC of the second service, so that all methods in the first service and the second service can access their respective corresponding aggregate fields, and the data of the aggregate fields are the same, the log framework automatically prints the aggregate field, and aggregates all logs written by the first service and the second service for the user operation request.

9. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon; When one or more programs are executed by one or more processors, the one or more processors implement the MDC-based log aggregation method as described in any one of claims 1 to 6.

10. A storage medium, characterized in that: A computer program is stored thereon, wherein when the computer program is executed by a processor, the MDC-based log aggregation method as described in any one of claims 1 to 6 is implemented.

Images