Business log processing method and device, equipment, medium and product

Abstract

The invention relates to the technical field of data processing, and discloses a business log processing method and device, equipment, a medium and a product, the method comprises the following steps: in response to a primary business interface calling request, starting a thread, and obtaining log basic information from the request; obtaining log configuration information for the service interface, wherein the log configuration information comprises logic isolation dimension configuration for a service log; according to the logic isolation dimension configuration, obtaining a value corresponding to the logic isolation dimension from the request as log context information; and generating a service log according to the log basic information and the log context information, and writing the service log into a global shared cache according to the identification code of the thread. According to the method, on the basis of distinguishing business scenes, business-oriented log fine management is realized through two-level isolation of threads and self-defined isolation dimensions, and the problems of log continuity and reliability in high-concurrency business scenes are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and particularly relates to a business log processing method, device, equipment, medium and product. Background Art

[0002] The statements in this part only provide background technical information related to the present invention, and do not necessarily constitute prior art.

[0003] Traditional log systems usually rely on centralized log servers, where all log data is centrally stored and processed. As the data volume increases, the write performance of the log server becomes a bottleneck, especially in high-concurrency scenarios. Centralized storage also makes data retrieval and analysis slow, and it is difficult to quickly locate problems.

[0004] In the prior art, through caching technology, the write performance pressure is alleviated to a certain extent. By temporarily storing logs in the cache and writing them to the storage system in batches after reaching a certain quantity, the I / O operations are reduced. However, in a multi-threaded environment, if multiple threads simultaneously read and write to the cache without an appropriate synchronization mechanism, it may lead to data competition (Race Condition), which in turn may cause problems such as data inconsistency, program crashes, or unpredictable results. For large business systems, due to the numerous sub-businesses involved, when multiple threads write logs to a common cache space, it will result in data confusion and disorder between different transactions, which is not conducive to data management and problem tracing.

[0005] The prior art proposes to solve data isolation between threads based on ThreadLocal. However, even if the thread data is isolated, the problem of data discontinuity cannot be completely solved. Specifically, when a process processes a request, if the request contains batch tasks, such as operating 10 documents simultaneously, the operations for these batch tasks will be written to a single log. When tracing one of the documents, there will be a problem of data discontinuity. In addition, for complex business systems, due to the large number of business function modules in the system and the huge amount of logs generated, the problem of log analysis and retrieval efficiency is more prominent in high-concurrency business scenarios. Summary of the Invention

[0006] To overcome the deficiencies of the above prior art, the present invention provides a business log processing method, device, equipment, medium and product. It can achieve logical isolation at multiple levels from the dimensions of business, threads, and custom for business scenarios, and solve the problems of log continuity and reliability in high-concurrency business scenarios.

[0007] To achieve the above object, the first aspect of the present invention provides a business log processing method, including the following steps:

[0008] In response to a business interface call request, start a thread and obtain basic log information from the request;

[0009] Obtain log configuration information for the business interface, where the log configuration information includes a logical isolation dimension configuration for business logs;

[0010] According to the logical isolation dimension configuration, obtain the value corresponding to the logical isolation dimension from the request as log context information;

[0011] Generate a business log according to the basic log information and the log context information, and write it into the global shared cache according to the identification code of the thread.

[0012] In some embodiments, the log configuration information further includes log level configuration information.

[0013] In some embodiments, the log configuration information further includes cache expiration time configuration and / or cache maximum capacity configuration.

[0014] In some embodiments, the log configuration information further includes a log compression depth configuration; after generating the business log, compress the business log according to the log compression depth configuration and then write it into the global shared cache.

[0015] In some embodiments, the log compression depth configuration includes at least one of a fixed word compression configuration, a fixed string compression configuration, a prefix timestamp compression configuration, and a stack path compression configuration.

[0016] In some embodiments, the fixed word compression configuration includes setting a unique code for each fixed word; the fixed string compression configuration includes setting a unique code for each fixed string; the prefix timestamp compression configuration is to record the time of the first log write as the cache start time for each cache space, and calculate the difference between the time of each log write into the cache and the start time as the prefix timestamp of the subsequent log; the stack path compression configuration is to retain the complete stack path related to the system for the own stack path and briefly identify the native stack path.

[0017] A second aspect of the present invention provides a business log processing device, including:

[0018] A business thread trigger module, configured to start a thread in response to a business interface call request and obtain basic log information from the request;

[0019] An isolation dimension determination module, configured to obtain log configuration information for the service interface, where the log configuration information includes a logical isolation dimension configuration for service logs; configured to obtain, according to the logical isolation dimension configuration, a value corresponding to the logical isolation dimension from the request as log context information;

[0020] A service log generation module, generates service logs according to the log basic information and the log context information, and writes them into the global shared cache according to the identification code of the thread.

[0021] A third aspect of the present invention provides an electronic device, including a processor and a memory, where a computer instruction is stored on the memory, and when the computer instruction is executed by the processor, the electronic device executes the method described above.

[0022] A fourth aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented.

[0023] A fifth aspect of the present invention provides a computer program product, where the computer program product includes a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0024] Based on the above one or more technical solutions, through logical isolation at two levels of threads and custom isolation dimensions on the basis of differentiating business scenarios, and by dividing cache areas in these dimensions, fine-grained management of service-oriented logs is achieved, solving the problems of log continuity and reliability in high-concurrency business scenarios. Description of the Drawings

[0025] The specification drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention.

[0026] Figure 1 It is a flowchart of a service log processing method in one or more embodiments of the present invention;

[0027] Figure 2 It is a schematic diagram of a log cache framework in one or more embodiments of the present invention. Detailed Embodiments

[0028] Embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not used to limit the protection scope of the present application.

[0029] In the description of the embodiments of the present application, the term "including" and its similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "at least partially based on".

[0030] As described in the background art, in the existing cache technology in a multi-threaded scenario, since all threads share a cache space, it leads to chaotic and disorderly data between services and complex problem positioning. Even if thread isolation is performed, it is difficult to distinguish between services. In the face of a complex business system, due to the huge amount of logs generated, it has a great impact on the performance of log retrieval and analysis. Based on this, one or more embodiments of the present invention provide a business log processing method that isolates both from the business dimension and the thread dimension at the stage of log generation. At the same time, for the logs generated by the same thread, a custom isolation dimension is also provided, and users can flexibly configure the isolation dimension according to the characteristics of their own business data, achieving refined log management that meets user needs and solving problems such as low retrieval efficiency and difficult log analysis caused by discontinuous data.

[0031] Specifically, the business log processing method includes the following steps:

[0032] S1: In response to a business interface call request, start a thread and obtain basic log information from the request.

[0033] S2: Obtain log configuration information for the business interface, where the log configuration information includes a logical isolation dimension configuration for business logs.

[0034] S3: According to the logical isolation dimension configuration, obtain the value corresponding to the logical isolation dimension from the request as log context information.

[0035] S4: Generate business logs according to the basic log information and the log context information, and write them into the global shared cache according to the identification code of the thread.

[0036] The global shared cache is the global MAP. The key of the global MAP is the identification code of the thread. Each thread corresponds to a unique identification code, which is stored in the thread local variable (ThreadLocal). The key of the global MAP is the business logs generated by each thread.

[0037] In the above method, first, the interfaces are divided according to the business, avoiding the confusion of log data between different businesses. Secondly, every time a business interface call request is responded to, a thread is started to generate and process business logs. The logs in the global shared cache are associated through the thread identification code, achieving thread-level logical isolation, which helps to reduce the confusion of logs between different transactions of the same business and improve the readability and traceability of the logs. Further, by adding an isolation dimension to such business logs, deeper-dimensional logical isolation for the business is achieved. The log system can manage log data more precisely, improving the queryability, analyzability, and usability of the logs. That is, two levels of logical isolation are achieved. While the logs of different threads can be effectively distinguished, the logs within each thread can be further distinguished based on the custom isolation dimension, solving the problems of log continuity and reliability in high-concurrency business scenarios. Moreover, the isolation dimension can be custom-configured by the user according to the specific application scenario, selecting the required isolation dimension, with strong flexibility and good scenario applicability.

[0038] It can be understood that the above "business" can refer to the business of application scenarios such as banks, hospitals, enterprises, etc., or it can also be the sub-businesses divided according to functional modules under the above application scenarios. For example, form management, user management, etc. in enterprise business. On this basis, the logical isolation dimension configuration is carried out for the lower-level sub-businesses of the above business or sub-businesses.

[0039] The basic log information includes interface basic information, request information, response information, execution time, and user information. Specifically, the interface basic information includes the interface name, request method, and interface path. The request information includes the request header, request body, and request timestamp. The response information includes the response status code, response body, and response timestamp. The execution time includes the total time consumed from the interface being called to its return. The user information includes the user ID, IP address, etc. of the user who calls the interface.

[0040] The log context information is determined according to the custom logical isolation dimension configuration. The user can perform custom configuration according to the scenario needs. For example, the isolation dimension can be determined based on various factors such as business logic, user identity, service component, and geographical location. It can be understood that the above custom logical isolation dimension configuration can receive the user's manual configuration through the user interface, and the specific method will not be elaborated here.

[0041] In step S2, the log configuration information further includes log level configuration information, which is used to adjust the business log level, control the output detail of the log, and meet the requirements of different stages and environments. Specifically, the business log levels include, from lowest to highest severity: DEBUG, INFO, WARN, ERROR, FATAL. The higher the severity level of the log, the more detailed the record. Correspondingly, in step S3, the level of the current business log is also determined according to the log level configuration information and written into the log context information. By dynamically adjusting the log level to control the log detail, the log storage and performance can be optimized, and resource consumption can be reduced.

[0042] In the above method, by customizing the isolation dimension and increasing the log level to flexibly configure the context information of the business log, users can make flexible adjustments according to the requirements of logical isolation. However, this method also increases the context information of the log and the cache pressure. To solve the above problems, in one or more embodiments, a cache data removal mechanism is provided. For the log configuration information of the business interface, it also includes cache expiration time and / or cache maximum capacity configuration. Specifically, (1) Based on the cache expiration time configuration, the cache structure storage blocks that have not written new logs after exceeding the expiration time are automatically removed to avoid the long-term occupation of system resources by cached logs; (2) Based on the cache maximum capacity configuration, the parts of the cache log storage blocks that exceed the maximum capacity are automatically removed to avoid the situation of large cache occupying memory; (3) Define a cleaning thread for the log cache framework to periodically judge the current state of the cache structure and automatically clean the cache structures that have been idle for a long time to release the occupation of invalid memory.

[0043] Based on thread isolation, custom dimension isolation, and cache policy configuration, the problems of log continuity and reliability in high-concurrency business scenarios are solved. At the same time, the expiration listening and capacity listening services are used to further optimize the system resource overhead, ensuring that the system can maintain performance while controlling resource consumption when processing a large number of logs.

[0044] To further optimize cache storage and improve log persistence, and to avoid the deletion of important information due to the cache data removal mechanism when the access frequency of the business interface is very high in a short period. In some embodiments, for the log configuration information of the business interface, it also includes log compression depth configuration. In step S4, after generating the business log, the business log is compressed according to the log compression depth configuration and then written into the global shared cache.

[0045] Specifically, the log compression depth configuration includes at least one of the following compression strategy configurations:

[0046] (1) Fixed-word compression strategy: Words such as DEBUG and INFO representing log levels, words used for indexing such as User, RequestID, and isolation dimension index words in context information, etc., all belong to fixed words and can be simplified. Specifically, the fixed-word compression strategy is: set a unique encoding for each fixed word. For example, for the five log levels, they respectively correspond to the numbers 1 - 5, and for index fixed words such as User and RequestID, take their first letters.

[0047] (2) Fixed-string compression strategy: For example, "The current interface received...". For fixed strings, its compression strategy is: set a unique encoding for each fixed-word string. For example, set it to a string with a length of 4 bytes.

[0048] (3) Prefix timestamp compression strategy: For each cache space, record the time when the first log is written as the cache start time, and calculate the difference between the cache time of each log write and the start time as the prefix timestamp of the subsequent log, reducing the problem of the full time format occupying cache space. For example, the prefix timestamp of the first log in the cache space is "2024-10-13 14:23:56", and the prefix timestamp of the second log can be recorded as "+1min".

[0049] (4) Own stack path compression strategy: For the stack information in the log cache, the log cache framework only retains the complete stack path related to the platform and product, and only gives a simple hint for the native stack.

[0050] The above fixed words and fixed strings can be obtained through analysis of the historical business logs of the business interface. Specifically, compare and analyze the historical business logs to filter out fixed words or fixed strings.

[0051] When the user performs operations such as adding, deleting, or modifying the logical isolation dimension, if the isolation dimension index words are compressed, the log compression depth configuration also needs to be updated synchronously.

[0052] Through the hot word collection technology, the compression effect is optimized, the log persistence efficiency is improved, and the occupation of storage space is reduced.

[0053] In the above method, isolation dimensions, log levels, log compression policies, cache cleaning policies, etc. can all be customized and configured by users. On the one hand, the customized configuration of isolation dimensions can meet users' isolation requirements for diversified business systems, realizing the refined management of logs for specific businesses and improving retrieval and log analysis performance. On the other hand, the dynamic adjustment of log levels, the customized editing of log compression policies and cache cleaning policies enable users to flexibly configure in the face of different businesses, different business processes and other scenarios. On the basis of meeting their own requirements for log persistence and detail, the cache storage is optimized and the occupation of system resources in high-concurrency scenarios is reduced. For example, for a certain business, if the user needs to retain logs for a long time, then the log compression policy can be configured, multiple policies can be selected, and fixed words or fixed strings that can be compressed can be increased as much as possible, while relaxing the cache expiration time.

[0054] For the log configuration information of the business interface, it also includes the configuration of log storage methods, providing log storage methods of multiple media and multiple protocols, and supporting the storage of log information under different deployment architectures (single-machine deployment, distributed deployment, microservice architecture, etc.), different storage media (disks, databases, etc.), and different protocols (HTTP, FTP, NFS, etc.). Based on the storage method configuration of the configuration module, it supports multiple log storage methods under different deployment architectures and hardware environments, as well as the storage requirements compatible with different protocols and storage media.

[0055] According to different usage scenarios and business dimensions of the system, configure dimension parameters such as log levels, cache policies, compression depth, storage methods, etc., to more finely control the collection, caching and output of log information. The system can dynamically adjust and optimize its log processing mechanism, including key links such as log screening, log caching and log storage.

[0056] Based on the above method, one or more embodiments of the present invention also provide a business log processing device, including:

[0057] A business thread trigger module, configured to start a thread in response to a business interface call request, and obtain basic log information from the request;

[0058] An isolation dimension determination module, configured to obtain the log configuration information for the business interface, where the log configuration information includes the logical isolation dimension configuration for business logs; and configured to obtain the value corresponding to the logical isolation dimension from the request as the log context information according to the logical isolation dimension configuration;

[0059] A business log generation module, generates business logs according to the basic log information and the log context information, and writes them into the global shared cache according to the identification code of the thread.

[0060] To achieve dynamic adjustment of the log level and meet the control of the log detail level for different business functions in different services, the log configuration information further includes log level configuration information.

[0061] To balance the log storage requirement and cache pressure, the log configuration information further includes cache expiration time configuration and / or cache maximum capacity configuration.

[0062] To increase the persistence of logs on the basis of saving the cache as much as possible, the log configuration information further includes log compression depth configuration; after generating business logs, the business logs are compressed according to the log compression depth configuration and then written into the global shared cache.

[0063] Among them, the log compression depth configuration includes at least one of fixed word compression configuration, fixed string compression configuration, prefix timestamp compression configuration, and stack path compression configuration. The fixed word compression configuration includes setting a unique code for each fixed word; the fixed string compression configuration includes setting a unique code for each fixed string; the prefix timestamp compression configuration is for each cache space, recording the time when the first log is written as the cache start time, and calculating the difference between the time when each log is written into the cache and the start time as the prefix timestamp of the subsequent log; the stack path compression configuration is for the self-stack path, retaining the complete stack path related to the system, and briefly identifying the native stack path.

[0064] To achieve custom configuration of isolation dimensions, log levels, log compression policies, and cache cleaning policies, the device further includes a custom log configuration module for configuring the log configuration information.

[0065] One or more embodiments of the present invention further provide an electronic device that can be used to implement the business log processing method in the above embodiments. The electronic device includes one or more processors, one or more memories coupled to the processors, and a communication module coupled to the processors.

[0066] The memory may include one or more non-volatile memories and one or more volatile memories. Examples of non-volatile memories include, but are not limited to, at least one of the following: Read-Only Memory (ROM), Erasable Programmable Read Only Memory (EPROM), flash memory, hard disk, Compact Disc (CD), Digital Versatile Disc (DVD), or other magnetic storage and / or optical storage. Examples of volatile memories include, but are not limited to, at least one of the following: Random Access Memory (RAM), or other volatile memories that do not persist during a power outage duration. The computer program may be stored in the ROM. When the processor executes the computer program, the above-mentioned business log processing method is implemented.

[0067] In some embodiments, the program may be tangibly embodied in a computer-readable medium, which may be included in the device (such as in the memory) or other storage devices accessible by the device. The program may be loaded from the computer-readable medium into the RAM for execution. The computer-readable medium may include any type of tangible non-volatile memory, such as ROM, EPROM, flash memory, hard disk. The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the above-mentioned business log processing method is implemented.

[0068] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a server or a terminal, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial optical cable, optical fiber, digital subscriber line) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible by the server or the terminal, or a data storage device such as a server or a data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, and a magnetic tape, etc.), an optical medium (such as a digital video disk (DVD), etc.), or a semiconductor medium (such as a solid-state drive, etc.).

[0069] In addition, although the operations are depicted in a particular order, this should be understood to require that the operations be performed in the particular order shown or in a sequential order, or that all of the illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although a number of specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present application. Certain features described in the context of separate embodiments may also be implemented in combination in a single implementation. Conversely, the various features described in the context of a single implementation may also be implemented separately or in any suitable sub-combination in multiple implementations.

[0070] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms for implementing the claims.

Claims

1. A method for processing service logs, characterized in that, It includes the following steps: In response to a business interface call request, start a thread and obtain basic log information from the request; Obtain log configuration information for the business interface, where the log configuration information includes a logical isolation dimension configuration for business logs; According to the logical isolation dimension configuration, obtain the value corresponding to the logical isolation dimension from the request as log context information; Generate a business log according to the basic log information and the log context information, and write it into the global shared cache according to the identification code of the thread.

2. The service log processing method according to claim 1, wherein The log configuration information further includes log level configuration information.

3. The service log processing method according to claim 1, wherein The log configuration information further includes cache expiration time configuration and / or cache maximum capacity configuration.

4. The service log processing method according to claim 1, wherein The log configuration information further includes log compression depth configuration; after generating the business log, compress the business log according to the log compression depth configuration and then write it into the global shared cache.

5. The service log processing method according to claim 4, wherein The log compression depth configuration includes at least one of fixed word compression configuration, fixed string compression configuration, prefix timestamp compression configuration, and stack path compression configuration.

6. The business log processing method according to claim 5, wherein The fixed word compression configuration includes setting a unique code for each fixed word; The fixed string compression configuration includes setting a unique code for each fixed string; The prefix timestamp compression configuration is that for each cache space, record the time when the first log is written as the cache start time, and calculate the difference between the time when each log is written into the cache and the start time as the prefix timestamp of the subsequent log; The stack path compression configuration is to retain the complete stack path related to the system for the self-stack path and briefly identify the native stack path.

7. A business log processing device, characterized in that, It includes: A business thread trigger module configured to start a thread in response to a business interface call request and obtain basic log information from the request; An isolation dimension determination module configured to obtain log configuration information for the business interface, where the log configuration information includes a logical isolation dimension configuration for business logs; and configured to obtain the value corresponding to the logical isolation dimension from the request as log context information according to the logical isolation dimension configuration; A business log generation module that generates a business log according to the basic log information and the log context information and writes it into the global shared cache according to the identification code of the thread.

8. An electronic device, comprising a processor and a memory, wherein computer instructions are stored on the memory, characterized in that, When the computer instruction is executed by the processor, the electronic device is caused to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method according to any one of claims 1 to 6.

10. A computer program product, the computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the method according to any one of claims 1 to 6.

Images