Log processing method and device, equipment, medium and program product

By directly collecting and processing structured logs on the application framework of the application server, the problems of high resource consumption and low efficiency in traditional log processing methods are solved, and efficient and accurate log processing and analysis are achieved.

CN119025367AActive Publication Date: 2024-11-26TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202310606976.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-25
Publication Date
2024-11-26
Estimated Expiration
2043-05-25

AI Technical Summary

Technical Problem

In traditional log processing methods, the storage and transmission of log data consumes a large amount of resources, and the log parsing and structured extraction processes are inefficient, resulting in low processing efficiency and high cost.

Method used

Directly collect structured logs on the application framework of the application server, process them through the acquisition module, preprocessing module and write module, avoid the generation of intermediate files of the log, directly perform structured storage, reduce storage and processing resource consumption, and develop Seamlessly integrate application frameworks to reduce costs.

Benefits of technology

Improves the efficiency and accuracy of log processing, saves storage and processing resources, reduces log processing costs, and improves analysis capabilities through the comprehensiveness and accuracy of structured logs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025367A_ABST
    Figure CN119025367A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a log processing method and device, equipment, a medium and a program product, and relates to a natural language processing technology in an artificial intelligence technology, and the method comprises the following steps: through an acquisition module on an application framework of an application server, according to a configured log acquisition period in the operation process of the application server, acquiring a log in the application framework of the application server; and acquiring a value of a basic configuration type field from an operating system of an application server, acquiring a value of a framework type field from an application framework, acquiring a value of a business logic type field from a business logic code, forming a structured log, and performing structured storage on the structured log. The acquisition module on the application framework is utilized to directly acquire the structured log, no log intermediate file is generated, storage resources of log data are saved, a complex log analysis process and a structured extraction process are avoided, the log analysis efficiency is improved, the acquired log data are more comprehensive and accurate, and the user experience is improved. And the accuracy of log analysis is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence (AI), and in particular to a log processing method, apparatus, device, medium and program product. Background Art

[0002] Logs are an important tool for analyzing and handling application or operating system problems and exceptions. Logs record information such as the date, time, event type, and messages related to the application or system. During the game operation, it is usually necessary to obtain the log files of the background game server and develop, manage, and optimize the game based on the log files.

[0003] The traditional log processing method includes the following steps: collecting log data from the game server in the form of text files, and then performing operations such as storage, transmission, and data analysis on the collected log data. The log data can be stored in a database. When the log data needs to be analyzed, the text-based log data is read from the database, and the log data is parsed and data extracted through a pre-written log analysis program to obtain the structured information of the log, and then the structured information is analyzed.

[0004] In traditional log processing methods, the amount of data stored and transmitted is large, which requires a lot of storage and transmission resources. In addition, during the log analysis process, the parsing and extraction of logs also require a lot of processing resources, resulting in low log processing efficiency and high processing costs. Summary of the invention

[0005] The embodiments of the present application provide a log processing method, apparatus, device, medium and program product, which improve the efficiency and accuracy of log processing.

[0006] In a first aspect, an embodiment of the present application provides a log processing method, the method comprising: through a collection module on an application framework of an application server, during the operation of the application server, according to a configured log collection cycle, collecting the value of a basic configuration type field from the operating system of the application server, collecting the value of a framework type field from the application framework, and collecting the value of a business logic type field from the business logic code; combining the value of the basic configuration type field, the value of the collection framework type field, and the value of the business logic type field collected in the same collection cycle into a structured log; preprocessing the collected structured log through a preprocessing module on the application framework; and structured storing the preprocessed structured log through a write module on the application framework.

[0007] In a second aspect, an embodiment of the present application provides a log processing device, comprising: a collection module, configured on an application framework of an application server, for collecting values ​​of basic configuration type fields from the operating system of the application server, collecting values ​​of framework type fields from the application framework, and collecting values ​​of business logic type fields from business logic codes according to a configured log collection cycle during the operation of the application server; the collection module is also used to combine the values ​​of the basic configuration type fields, the values ​​of the collection framework type fields, and the values ​​of the business logic type fields collected in the same collection cycle into a structured log; a preprocessing module, configured on the application framework, for preprocessing the structured logs collected by the collection module; a writing module, configured on the application framework, for structured storage of the preprocessed structured logs.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the method described in the first aspect above.

[0009] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program enables a computer to execute the method described in the first aspect above.

[0010] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method described in the first aspect above.

[0011] The technical solution provided by the embodiment of the present application directly collects structured logs using the collection module on the application framework of the application server during the operation of the application server, and does not generate any log intermediate files, saving storage resources for log data, and avoiding complex log parsing processes and structured extraction processes, saving log processing resources, and improving the efficiency of log analysis. In addition, the application framework is used for log collection, and the application framework can be expanded when developing application services, so that log processing is seamlessly integrated with the application server, and no additional software and hardware are required for log processing, which reduces the log processing cost of the application server, and the structured logs collected by the application framework include basic configuration type fields, framework type fields, and business logic type fields, and the collected log data is more comprehensive and accurate, thereby being able to improve the accuracy of log analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0013] Figure 1 A schematic diagram of an implementation environment of the method provided in an embodiment of the present application;

[0014] Figure 2 Generate a schematic diagram for the log in the game scene;

[0015] Figure 3 is a flow chart of the log processing method provided in Example 1 of the present application;

[0016] Figure 4 This is a block diagram of the log processing method provided in Example 2 of the present application;

[0017] Figure 5 A schematic diagram of a writing method used by a writing module to write structured logs to a remote server;

[0018] Figure 6 A schematic diagram of the writing process of the writing module writing structured logs to a remote server;

[0019] Figure 7 A flow chart of the log processing method provided in Example 2 of the present application;

[0020] Figure 8 A schematic diagram of the structure of the log processing device provided in Example 3 of the present application;

[0021] Fig. 9 A structural schematic diagram of an electronic device provided in Example 4 of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0024] This application relates to artificial intelligence technology, in which artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0025] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0026] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graphs, and search-related technologies (such as sorting, keywords, recommendations, etc.).

[0027] In order to facilitate the understanding of the embodiments of the present application, before describing the various embodiments of the present application, some concepts involved in all the embodiments of the present application are first appropriately explained.

[0028] The present application embodiment provides a log processing method. Figure 1 A schematic diagram of an implementation environment of the method provided in the embodiment of the present application, such as Figure 1 As shown, the implementation environment includes: terminals 110, 120, 130, an application server 140 and a remote server 150. The terminals 110, 120, 130 communicate with the application server 140 via a wired or wireless network, and the application server 140 can also communicate with the remote server 150 via a wired or wireless network.

[0029] The terminal is not limited to the device shown in the figure, and the terminal can be a mobile phone, a tablet computer, a desktop computer, a portable laptop, an intelligent voice interaction device, a smart home appliance, and other devices in various forms. An application is running in the terminal. For example, the application can be a traditional application software, a cloud application software, a small program or an application module in a host application, or a web platform, which is not limited here. The application is connected to the application server and receives the service content provided by the application server.

[0030] Optionally, the above-mentioned application may be a game application, an e-commerce application, a short video application, an audio application, a novel application, a content interaction application, an instant messaging application, etc., and no specific limitation is made here.

[0031] The application server 140 is used to provide service content for the application. Different application servers 140 provide different service content. The application server 140 refers to a logical device that can logically and independently provide at least one service. It can be understood that the application server 140 does not correspond to a physical server. An application server 140 can be deployed on multiple physical servers, and multiple application servers can also be deployed on one physical server.

[0032] During operation, the application server 140 actively collects the structured logs of the application server 140 through the collection module on the application framework, and writes the structured logs to the remote server. Optionally, the application server 140 can also pre-process the structured logs, store them locally, or provide them to other interfaces.

[0033] Structured logs are composed of some meaningful fields, such as user account, timestamp, Internet Protocol (IP) address, etc. These fields have physical meanings, which are convenient for users to view and analyze logs. In the traditional way, unstructured logs are usually obtained. Unstructured logs are a string of unformatted characters. Usually, users cannot use unstructured logs directly. Log analysis methods are needed to extract structured information from unstructured logs and further analyze the structured information. In the embodiment of the present application, the application server 140 can directly collect structured log information, avoiding the collection, storage and extraction of structured information of unstructured logs, thereby improving the log processing efficiency and reducing the log processing cost.

[0034] The remote server 150 is used to perform structured storage on the structured logs and provide a query interface. Subsequently, the application server 140 or other devices can query the structured logs from the remote server 150 and further analyze and process the structured logs.

[0035] Optionally, the remote server uses Elasticsearch, which is a distributed, highly scalable, and highly real-time search and data analysis engine. Elasticsearch can efficiently store and query log data through its own distributed architecture and index optimization technology.

[0036] Optionally, the remote server may also use a database to store structured log data. The embodiment of the present application does not limit the specific storage architecture used by the remote server.

[0037] In this embodiment, the application server 140 and the remote server 150 can be deployed on an independent physical server, or on a server cluster or distributed system composed of multiple physical servers. They can also be deployed on a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud security, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0038] Among them, cloud technology refers to a hosting technology that unifies hardware, software, network and other resources in a wide area network or local area network to realize data computing, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which is used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of technical network systems require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, in the future, each item may have its own identification mark, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately. All kinds of industry data require strong system backing support, which can only be achieved through cloud computing.

[0039] Figure 1 In the implementation environment shown, the operation of writing structured logs to the remote server 150 and other processing operations can be performed by the application server 140. It can be understood that in the embodiment of the present application, the application server 140 can also be only responsible for the collection of structured logs, and the operation of writing structured logs to the remote server 150 and other processing operations can be performed by a separate log processing server.

[0040] The application server 140 generates logs during operation. Figure 2 Generate a schematic diagram for the log in the game scenario, such as Figure 2 As shown, the game terminal establishes a connection with the game server, a game program runs on the game terminal, and the game server establishes multiple business processes during operation. The business processes are used to handle specific game business. Exemplarily, the business processes include: room service process, game service process, directory service process, chat service process, etc.

[0041] During the operation of the game business, it not only needs the support of the game server, but also needs the support of some public components. The public components are used to provide some public services. The public services are independent of the game business development. The game server calls the public components through the interface. The public components can be understood as part of the game application. The public components will also generate log data. Exemplarily, the public components include but are not limited to: access components, distribution components and authentication components. The access components are used to process game requests, the distribution components are used to process the message forwarding of the game business, and the authentication components are used to provide account login, authentication, management, etc. during the game business access process.

[0042] During the game operation, not only the business process of the game server will generate log data, but also the above-mentioned public components will generate log data. The business process of the game server will generate log data and the log data generated by the public components will be output to the log processing module of the game server. The log processing module provides the following processing functions: pre-processing, local storage and local output, etc.

[0043] In this embodiment, the log data collected by the log processing module is a structured log. The preprocessing of the log data refers to the preprocessing of the structured log. The preprocessing includes at least one of the following processes: log format planning processing, deduplication processing, filtering processing, classification processing, etc.

[0044] The preprocessed log data needs to be persisted for subsequent query and analysis. The traditional approach is to save the log data in a local hard disk or database, also known as local storage. Local storage stores the log data locally in the form of unformatted strings, but this approach has security and stability issues. A better approach is to use remote storage to store the log data on one or more remote servers.

[0045] Remote storage of logs writes logs to a remote server through the local output function to ensure the security and reliability of logs. At the same time, remote storage can also realize cross-regional and cross-platform log sharing, facilitating collaboration and analysis among various teams.

[0046] The local output function can also provide storage functions in a variety of different paths, for example, log data can be stored in a remote server through different paths, provided to other data statistics devices, data security devices, etc. After the log is stored, the stored log data can be viewed in the remote server or application server through the terminal.

[0047] After introducing some concepts involved in the embodiments of the present application, a log processing method provided in the embodiments of the present application is specifically described below with reference to the accompanying drawings.

[0048] Figure 3 is a flow chart of the log processing method provided in the first embodiment of the present application. The method in this embodiment can be executed by an application server, such as Figure 3 As shown, the method of this embodiment includes the following steps.

[0049] S101. Through the collection module on the application framework of the application server, during the operation of the application server, according to the configured log collection cycle, the value of the basic configuration type field is collected from the operating system of the application server, the value of the framework type field is collected from the application framework, and the value of the business logic type field is collected from the business logic code.

[0050] The application framework is also called the application framework or program framework. The application framework is the foundation and core of application development. It is a convention used and followed by all developers. Developers make necessary extensions based on the convention, but the program always maintains the consistency of the main structure. The application framework itself can be understood as a piece of code running on the operating system. The application framework provides many application programming interfaces (Application Programming Interface, referred to as API) to the outside world. Developers can develop specific applications on the application framework through the API interface.

[0051] The method of this embodiment implements log collection of the application server through the application framework by extending the application framework of the application server. Extending the application framework means adding some programs and interfaces to implement the method of the embodiment of this application on the basis of the existing application framework of the application server. By extending the application framework, a collection module is added to the application framework, and the collection module is used to collect structured logs, so that the application framework has a log collection function.

[0052] In this embodiment, the collection module of the application framework can periodically and actively collect structured logs according to the configured log collection cycle during the operation of the application server, without being limited by user requests or triggering of specific business events (such as payment events, game interruption events), so that more and more comprehensive log data can be collected. The log collection cycle is, for example, 30 seconds, one minute, five minutes, etc., which is not limited in this embodiment. Optionally, the log collection cycle can be flexibly configured by the administrator, and a suitable log collection cycle can be set according to the application scenario of the application server.

[0053] The collection module can collect structured logs from the operating system, application framework and business logic code of the application server, so that the collected structured logs are more comprehensive and complete, and can meet the needs of subsequent log analysis. Among them, the collection module collects different log data from the operating system, application framework and business logic code. The module collects the value of the basic configuration type field from the operating system of the application server, the value of the framework type field from the application framework, and the value of the business logic type field from the business logic code.

[0054] S102: The value of the basic configuration type field, the value of the collection framework type field, and the value of the business logic type field collected in the same collection period are combined into a structured log.

[0055] The collection module will collect a large number of structured logs. A structured log is also called a structured log record. A structured log includes the following type fields: basic configuration type field, framework type field, and business logic type field. Each type of field includes at least one field, and the fields included in the type fields of different application servers may be different.

[0056] The basic configuration type field is used to record the hardware and / or software configuration information of the application server. Exemplarily, the basic configuration type field includes at least one of the following fields: a deployment environment field and an IP address. The deployment environment field is used to describe the environment information of the application server deployment. The application server can be deployed in different computer rooms, so the deployment environment field can be the name of the deployment computer room. In game applications, the application server can also be deployed in different partitions, so the deployment environment field can also include the name of the partition where the application server is located.

[0057] The framework type field is used to record some information in the application framework of the application server. Exemplarily, the framework type field includes at least one of the following fields: a trace ID field, a thread ID field, a file name field, a file line number field, a method name field, and a log level field.

[0058] The trace ID is used to identify a specific request. When a user's request enters the system, a globally unique trace ID is generated. During the request transmission process, the trace ID can completely connect the path of the request call in the system.

[0059] Optionally, a lightweight thread may be used in the application server. A lightweight thread is also called a coroutine. Therefore, the thread ID may be replaced by a coroutine ID.

[0060] The file name field and text line number refer to the file name of the source file called when this structured log is generated and the line number in the file.

[0061] The method name field refers to the name of the method called when this structured log is generated. The method can be understood as a special function. Therefore, the method here can also be replaced by a function, or the framework type field includes both the method name field and the function name field.

[0062] Exemplarily, the log level includes the following five levels: debug, info, warning, error, fatal. Different log levels print different log information. Optionally, the log level in this embodiment adopts the debug level.

[0063] The business logic type field is used to record specific business logic-related information of the application server. Exemplarily, the business logic type field includes at least one of the following fields: a time field, a user information field, an identification field of a first request type, an identification field of a second request type, and a custom field. The first request type is a request type between an application server and a client, and the second request type is a request type between an application server and another server.

[0064] The time field is used to record the collection timestamp or printing timestamp of each structured log. The timestamp recorded in the time field can be a microsecond-level timestamp.

[0065] The user information field is used to record information about users accessing the application server. The user information field may include a user account, user level, user type, and the like.

[0066] The client and the application server communicate using requests, which can be divided into Hyper Text Transfer Protocol (HTTP) requests, Transmission Control Protocol (TCP) requests, etc. Requests of the same protocol type can also be divided into multiple different types according to the content of the request. For example, HTTP requests can be divided into GET requests, POST requests, PUT requests, DELET requests, and PATCH requests.

[0067] The identification field of the first request type is used to record the identification of the request type between the application server and the client. The first request type can be a type based on the transmission protocol or a type based on the request content. This embodiment does not specify the request type.

[0068] During the operation of the application server, in addition to communicating with the client, it also communicates with other servers. For example, the application server can communicate with Figure 2 The second request type is the request type between the application server and other servers. The division of the second request type is similar to the division of the first request type, which will not be repeated here.

[0069] A custom field is a field reserved for users. Users can define the content of the field according to their own needs. For example, a custom field is a game item name.

[0070] Optionally, the collection module includes multiple collection units, each collection unit is used to collect at least one field, and during the operation of the application server, the structured logs of the application server are collected by the multiple collection units, and the collected structured logs are filled into a predefined log structure, which is used to temporarily store the collected structured logs.

[0071] Exemplarily, the collection unit includes three collection units: a first collection unit, a second collection unit, and a third collection unit. The first collection unit is used to call the operating system according to the log collection cycle, collect the value of the basic configuration type field from the operating system, and fill the collected value of the basic configuration type field into the predefined log structure; the second collection unit is used to collect the value of the framework type field from the application framework according to the log collection cycle, and fill the collected value of the framework type field into the log structure; the third collection unit is used to collect the business logic type field from the business logic code of the application server according to the log collection cycle, and fill the collected value of the business logic type field into the log structure. The value of the basic configuration type field, the value of the collection framework type field, and the value of the business logic type field collected in the same collection cycle stored in the log structure constitute a structured log.

[0072] In some commonly used log processing methods, usually only some business logic fields can be obtained to analyze the business logic. In this embodiment, the collection module not only collects business logic type fields, but also collects basic configuration type fields and framework type fields, so that the collected log data is more complete and comprehensive, and the basic configuration, framework and business logic of the application server can be comprehensively analyzed based on the log data, and the log data can be better used to manage and optimize the application server.

[0073] In this embodiment, during the operation of the application server, the collection module of the application framework is used to directly collect structured logs, and no intermediate log files are generated, which saves storage resources for log data, avoids complex log parsing processes and structured extraction processes, and saves parsing costs and structured extraction costs. In addition, when using the application framework for log collection, the application framework can be expanded when developing application services, and is seamlessly integrated with the application server, without the need for additional software and hardware for log processing, which reduces the log processing cost of the application server.

[0074] The application framework actively collects structured logs, making the content of the collected structured logs more complete and independent of some passive triggering conditions. For example, in existing solutions, log collection is triggered based on user requests. Log collection is only performed when user requests are detected, resulting in limited collected log data.

[0075] S103: Preprocess the collected structured logs through a preprocessing module on the application framework.

[0076] Optionally, before storing the structured log, the structured log may be preprocessed by a preprocessing module, and the preprocessing includes at least one of the following processes: log format planning processing, deduplication processing, filtering processing, and classification processing.

[0077] The preprocessing module reads the structured log from the log structure body, preprocesses the structured log, sends the preprocessed structured log to the writing module, and writes the structured log to the remote server through the writing module.

[0078] In this embodiment, the preprocessing module can be a module in the application server that is independent of the application framework, or a module on the application framework. By preprocessing the structured logs with the preprocessing module obtained by extending the application framework, no additional software modules are required, thus achieving seamless integration of log processing and the application server and reducing the cost of log processing.

[0079] Log format planning and specification processing refers to processing the values ​​of each field in the structured log so that the values ​​of each field in the structured log meet the preset specifications. The preset specifications include but are not limited to preset format specifications, value specifications, etc. For example, if the value of some fields exceeds the preset value range threshold, the value of the field needs to be modified to the correct value; or, if the format of the value of some fields does not meet the requirements, the format of the value of the field needs to be converted.

[0080] During the log collection process, some duplicate logs may be collected. Deduplication refers to removing some duplicate fields or duplicate logs to reduce the storage and transmission resources of subsequent logs.

[0081] Filtering processing refers to filtering out unnecessary fields, or filtering out fields with incorrect values, or filtering out some illegal fields, etc.

[0082] Classification processing is to classify the logs based on the values ​​of the fields in the collected structured logs. For example, the user level is determined based on the type of props in the business logic type field, and the user level information is added to the structured log. A classification field can be added based on the existing fields in the collected structured logs.

[0083] Optionally, when the business logic type field includes a custom field, the preprocessing of the structured log can also include word segmentation of the custom field value. The custom field value may be a text, and the text content is usually very long. The custom field is segmented into multiple entries through word segmentation. The purpose of word segmentation is to adapt to the inverted index search method during the log search phase, and avoid using prefix matching to match the custom field value during the log search phase.

[0084] Exemplarily, according to the configured word segmentation rules, the value of the custom field in the structured log is segmented to obtain at least one entry corresponding to the value of the custom field. When the text of the self-positioning field is very long, multiple entries can be obtained through word segmentation.

[0085] S104: The pre-processed structured log is structuredly stored through a writing module on the application framework.

[0086] Structured storage refers to storage according to the fields of structured logs, such as structured storage in the form of tables or key-value pairs.

[0087] In one implementation, the structured log is stored locally, that is, in the application server. In another implementation, the structured log is stored in a remote server, which is used to store and manage the structured log of the application server. The optional application server also provides other interfaces to store the structured log in statistical equipment, security equipment, etc.

[0088] When storing structured logs on a remote server, you need to consider the transmission efficiency and reliability of the log data.

[0089] In this embodiment, during the operation of the application server, the collection module on the application framework of the application server is used to directly collect structured logs, and no log intermediate files are generated, which saves storage resources for log data, avoids complex log parsing processes and structured extraction processes, saves log processing resources, and improves the efficiency of log analysis. In addition, the application framework is used for log collection, and the application framework can be expanded when developing application services, so that log processing is seamlessly integrated with the application server, and no additional software and hardware are required for log processing, which reduces the log processing cost of the application server, and the structured log collected by the application framework includes basic configuration type fields, framework type fields, and business logic type fields, and the collected log data is more comprehensive and accurate, thereby improving the accuracy of log analysis.

[0090] Based on the first embodiment, the second embodiment of the present application provides a log processing method, which is used to describe the complete processing process of storing structured logs in a remote server. Figure 4The principle block diagram of the log processing method provided in the second embodiment of the present application is as follows: Figure 4 As shown, the log processing method includes four processing stages: field definition, log collection, log preprocessing, and log writing.

[0091] Field definition refers to the fields that developers define according to log processing requirements. Business logic type fields can be defined by business developers, basic configuration type fields and framework type fields can be defined by framework providers to ensure the availability of logs. Figure 4 As shown, the cs cmd in the business logic type field refers to the identification field of the first request type, and the ss cmd refers to the identification field of the second request type.

[0092] After defining the log fields, the application framework of the application server is expanded to form a collection module. Log collection refers to the process of capturing the values ​​of the log fields defined in the field definition phase. The collection module is used for collection in the field collection phase. When the application server is started, the collection module collects the basic configuration type fields according to the resources and configuration of the application server. After the application server is started, the framework type fields and business logic type fields are collected during the running process, and the values ​​of the collected fields are written into the structure in the memory.

[0093] Log preprocessing refers to the preprocessing of structured logs collected by the collection module, which includes but is not limited to: log format planning, deduplication, filtering, and classification.

[0094] Log writing refers to the process of storing preprocessed structured logs. The log writing in the figure adopts multi-path writing, which includes: local log writing, remote log writing and diversion writing. Among them, local log writing refers to writing structured logs to the local log file of the application server, remote log writing refers to writing structured logs to the remote server, and diversion writing refers to providing structured logs to other devices that need log data.

[0095] In this embodiment, the structured log is written to the remote server through the writing module on the application server. The pre-writing module can be a module in the application server independent of the application framework, or a module on the application framework. The writing module obtained by extending the application framework performs writing operations on the structured log, and no additional software modules are required, thereby realizing seamless integration of log processing and application server and reducing the cost of log processing.

[0096] When the writing module writes structured logs to a remote server, it needs to consider the writing efficiency and writing reliability of the structured logs. The reliability can ensure the integrity and security of the structured log data.

[0097] Figure 5 The following is a schematic diagram of the writing method used by the writing module to write structured logs to the remote server. Figure 5 As shown in the figure, write efficiency is achieved through the following methods: asynchronous processing, data merging, and multi-threaded concurrent writing. Write reliability is achieved through the following methods: failure retry, graceful shutdown, and at least one successful retry.

[0098] Figure 6 This is a schematic diagram of the writing process of the writing module writing structured logs to the remote server. Figure 5 and Figure 6 The write module receives structured logs from the business log interface of the application server and writes them to the server through Figure 5 The write method writes to the remote server. The business log interface is an interface on the application framework. The structured log received by the write module may be a preprocessed log or an unpreprocessed log.

[0099] A large number of logs are generated during the business operation of the application server. In order to avoid the impact of a large amount of log data processing on the business, the log writing in this embodiment adopts asynchronous processing, that is, the log writing thread and the business thread use different threads, and the business thread and the writing thread run independently, so that the log writing will not affect the business. In order to realize asynchronous processing, a log data queue is set, and the log data generated by the application server is stored in the log data queue. The writing thread consumes data from the log data queue, that is, reads the structured log and writes it to the remote server.

[0100] When the writing thread writes the structured log to the remote server, it writes through the network input / output (IO), so the writing thread is also called the network IO writing thread. When the writing thread writes the structured log to the remote server, it can write one structured log at a time, or it can write multiple structured logs at a time.

[0101] Each time a structured log is written, a structured log is written through a write request. Each time multiple structured logs are written, multiple structured logs are written through a write request. The write module and the remote server communicate through requests. Each time data is written, a request needs to be established and released after the write is completed. The establishment and release of the request takes time. If each write request only carries a structured log, the log writing efficiency is very low, a lot of time is wasted on the establishment and release of the request, and the delay in log writing is also unbearable.

[0102] In this embodiment, multiple structured logs are read from the log data queue, and the multiple structured logs are carried in one write request and written to the remote server, that is, multiple structured logs can be written in one request, thereby improving the log writing efficiency.

[0103] Figure 5 and Figure 6 The data merging in the above process refers to merging multiple structured logs into one write request and writing them to the remote server. Optionally, the data merging also includes business merging of multiple structured logs according to business logic. Before writing multiple structured logs to the remote server through one write request, the write thread performs business merging of multiple structured logs according to business logic and writes the merged structured logs to the remote server through a write request.

[0104] According to the business logic, multiple structured logs can be merged into one structured log. The purpose of business logic merging is to reduce the amount of data transmission for write requests. For example, for a data movement operation, two structured logs may be generated. Most of the contents of the two structured logs are the same, and only the operation addresses are different. The operation address can be a custom field in the logical field. The two structured logs are merged into one structured log by merging. The merged structured log includes two operation addresses and the corresponding operation time.

[0105] Data merging reduces the number of write requests processed per unit time and reduces the latency of log writing. For example, the remote server uses Elasticsearch, and the write thread writes structured logs to Elasticsearch by calling the Platform as a Service (Paas) interface of Elasticsearch. When writing data, the write thread needs to establish a Paas request with the Paas interface. In order to reduce the query rate per second (QPS) of Paas requests, multiple structured logs are merged into one Paas request and written, thereby reducing the number of Paas requests processed per unit time.

[0106] Multi-threaded concurrent writing refers to using multiple concurrent writing threads to write structured logs to a remote server. The multi-threaded concurrent writing method provides a certain data processing capability, thereby ensuring the real-time reporting of log data.

[0107] When the writing thread is writing structured logs to the remote server, the writing may fail due to network instability or other reasons. For structured logs that fail to be written, in order to prevent the loss of structured logs that fail to be written, this embodiment adopts a failure retry mechanism to ensure that the structured logs that fail to be written can be rewritten to the remote server.

[0108] Specifically, when multiple structured logs fail to be written, the multiple structured logs that failed to be written are stored in the allocated storage space, and the multiple structured logs that failed to be written are put into the log data queue through the rewriting thread. By putting the structured logs that failed to be written back into the log data queue, the writing thread will rewrite the structured logs that failed to be written to the remote server. It can be understood that when the writing thread reads the structured logs from the log data queue, it does not distinguish whether the read structured logs are structured logs that failed to be written or structured logs that are written for the first time.

[0109] The storage space is a pre-allocated address space, which can be located in the cache of the operating system. The storage space can be used to store structured logs that failed to be written, and the storage space can be accessed by the rewriting thread, which reads the log data in the storage space into the log data queue.

[0110] Optionally, the storage space is also used to store the structured log of the overflow log data queue. When the log data queue receives the structured log from the log interface at a faster rate, the log data queue may overflow, that is, the log data queue is full and the structured log cannot be placed. In order to avoid data loss due to the log data queue overflow, in the present embodiment, when the log data queue overflows, the overflowed structured log is stored in the above-mentioned storage space; the overflowed structured log is placed in the log data queue by rewriting the thread.

[0111] During the operation of the application server, the device may be shut down due to system upgrade, expansion or other reasons. In this embodiment, when shutting down the application server, a graceful shutdown method is used to shut down the application server.

[0112] Graceful shutdown simply means that after sending a stop command to the application process, the business operations being executed are not affected, and the existing requests can continue to be processed, but new requests are stopped. When stopping a running application or process, the operating system will send a termination signal to the process. If no graceful shutdown mechanism is enabled, the application will terminate immediately after receiving the signal. At this time, some programs that have not been executed will exit directly, which may cause the business logic to fail to execute. In some business scenarios: data inconsistency will occur, and the transaction logic will not be rolled back.

[0113] In the present embodiment, if graceful shutdown is not adopted, when receiving the device shutdown instruction, shutdown is performed immediately, at this time, the log in the log data queue has not been processed, and when the device is turned on again, the unprocessed log in the log data queue will be cleared, thereby causing the unprocessed log in the log data queue to be lost. After adopting the graceful shutdown mode, the device will be shut down after the data in the log data queue is processed, and the log data queue will no longer receive new structured logs.

[0114] Specifically, after the graceful shutdown module sends a shutdown command to the log data queue, the log data queue no longer receives new structured logs, and the write thread continues to consume data from the log data queue. When all the data in the log data queue is consumed, the device is shut down, thereby ensuring that the log data will not be lost when the application server is shut down.

[0115] Optionally, in this embodiment, the retry thread uses the at least-once success method, and at least-once success means that at least one data retry is guaranteed to succeed. The retry thread may fail when writing the log in the storage space to the log data queue. If the at least-once success method is not used, the data will usually be considered as abnormal data after multiple attempts, and the data will be discarded. After the at least-once success method is used in this embodiment, it will continuously try to write the log to the log data queue, avoiding data loss, thereby increasing the reliability of the log data.

[0116] based on Figure 5 and Figure 6 The writing method and writing process shown, Figure 7 This is a flow chart of the log processing method provided in Example 2 of the present application. The method provided in this embodiment includes the following steps.

[0117] S201. Through the collection module of the application framework of the application server, during the operation of the application server, according to the configured log collection cycle, the value of the basic configuration type field is collected from the operating system of the application server, the value of the framework type field is collected from the application framework, and the value of the business logic type field is collected from the business logic code.

[0118] S202: The value of the basic configuration type field, the value of the collection framework type field, and the value of the business logic type field collected in the same collection period are combined into a structured log.

[0119] S203: The preprocessing module of the application framework collects structured logs from the collection module and preprocesses the structured logs.

[0120] The preprocessing includes at least one of the following processes: log format planning processing, deduplication processing, filtering processing, and classification processing.

[0121] S204: The writing module of the application framework puts the pre-processed structured log data into a log data queue.

[0122] Optionally, when the log data queue overflows, the overflowed structured logs are stored in a storage space, where the storage space is used to store the overflowed structured logs and the structured logs that failed to be written.

[0123] S205 . Read structured logs from the log data queue through multiple parallel writing threads, wherein each writing thread reads multiple structured logs from the log data queue.

[0124] S206: Each writing thread writes the read multiple structured logs to the remote server through a writing request.

[0125] S207: When the writing thread fails to write, the writing thread stores the multiple structured logs that failed to write into the allocated storage space.

[0126] S208. Put the structured log in the storage space into the log data queue through the rewriting thread.

[0127] S209: When a device shutdown instruction is received, an elegant shutdown method is adopted to shut down the device after the data in the log data queue is processed. The log data queue no longer receives new structured logs.

[0128] Steps S204-S209 are executed by the write module of the application framework. It can be understood that there is no sequence in the execution of steps S205-S207 and step S208. The write thread and the rewrite thread are two different threads, that is, the retry mechanism adopts asynchronous retry. The two threads are executed independently without affecting each other. They may be executed simultaneously or one after another.

[0129] After the application server writes the structured log to the remote server, the remote server uses a structured storage method to store the structured log, for example, by using Elasticsearch or a database to store the structured log. Later, log analysts can query the log data from the remote server. When querying the log data, the remote server uses a structured storage method to store the structured log. Structured storage can improve query efficiency. Users can enter fields in the structured log for query. The query results are highly readable and easy for analysts to perform log analysis.

[0130] Elasticsearch can efficiently store and query log data through its distributed architecture and index optimization technology.

[0131] In this embodiment, by expanding the application framework of the application server, a collection module, a preprocessing module and a writing module are added to the application framework. The collection module collects the structured logs of the application server during the operation of the application server, the preprocessing module preprocesses the structured logs collected by the collection module, and the writing module writes the preprocessed structured logs to the remote server. The writing module adopts asynchronous processing, data merging and multi-threaded concurrent writing methods to ensure the efficiency of log writing, and adopts failure retry, graceful shutdown and at least one successful retry method to ensure the reliability of log writing.

[0132] In the above method embodiment, the acquisition module, preprocessing module and writing module for processing log data on the application framework of the application server are functional modules defined from a functional perspective. It should be understood that the acquisition module, preprocessing module and writing module can also be implemented by one functional module, for example, a log processing module is added to the application framework, and the log processing module is used to collect the structured logs of the application server during the operation of the application server, preprocess the structured logs collected by the acquisition module, and write the preprocessed structured logs to the remote server.

[0133] The embodiment of the present application also provides a log processing device. Figure 8 This is a structural diagram of the log processing device provided in Example 3 of the present application, such as Figure 8 As shown, the log processing device 100 may include:

[0134] The collection module 11 is configured on the application framework of the application server and is used to collect the value of the basic configuration type field from the operating system of the application server, the value of the framework type field from the application framework, and the value of the business logic type field from the business logic code according to the configured log collection cycle during the operation of the application server;

[0135] The collection module 11 is further used to combine the value of the basic configuration type field, the value of the collection framework type field and the value of the business logic type field collected in the same collection period into a structured log;

[0136] A preprocessing module 12, configured on the application framework, for preprocessing the structured logs collected by the collection module;

[0137] The writing module 13 is configured on the application framework and is used for performing structured storage on the pre-processed structured log.

[0138] In some achievable embodiments, the acquisition module 11 includes a first acquisition unit, a second acquisition unit, and a third acquisition unit;

[0139] The first collection unit is used to call the operating system according to the log collection cycle, collect the value of the basic configuration type field from the operating system, and fill the collected value of the basic configuration type field into a predefined log structure;

[0140] The second collection unit is used to collect the value of the framework type field from the application framework according to the log collection period, and fill the collected value of the framework type field into the log structure;

[0141] The third collection unit is used to collect the value of the business logic type field from the business logic code according to the log collection cycle, and fill the collected value of the business logic type field into the log structure;

[0142] The value of the basic configuration type field, the value of the acquisition framework type field, and the value of the business logic type field collected in the same collection period and stored in the log structure constitute a structured log.

[0143] In some implementable embodiments, the basic configuration type field includes at least one of the following fields: a deployment environment field and an IP address; the framework type field includes at least one of the following fields: a trace identification field, a thread identification field, a file name field, a file line number field, a method name field, and a log level field; the business logic type field includes at least one of the following fields: a time field, a user information field, an identification field of a first request type, an identification field of a second request type, and a custom field, the first request type being a request type between the application server and the client, and the second request type being a request type between the application server and other servers.

[0144] In some achievable embodiments, the preprocessing includes at least one of the following processes: log format planning processing, deduplication processing, filtering processing, and classification processing.

[0145] In some implementable methods, when the business logic type field includes a custom field, the preprocessing also includes word segmentation processing; the preprocessing module 12 is specifically used to: according to the configured word segmentation rules, the value of the custom field in the structured log is segmented to obtain at least one entry corresponding to the value of the custom field.

[0146] In some feasible embodiments, the structured log is stored in a remote server; the writing module 13 is specifically used to: put the preprocessed structured log data into a log data queue; read multiple structured logs from the log data queue through a writing thread, and write the multiple structured logs to the remote server through a writing request.

[0147] In some achievable embodiments, the write module 13 is further used to: before writing the multiple structured logs to the remote server through a write request, perform business merging on the multiple structured logs according to business logic through the write thread, and write the merged structured logs to the remote server through the write request.

[0148] In some achievable embodiments, the writing module 13 is further used for: when the writing of the multiple structured logs to the remote server fails, storing the multiple structured logs that failed to be written into an allocated storage space through the writing thread, and the storage space is used to store the structured logs that failed to be written; and placing the structured logs in the storage space into the log data queue through the rewriting thread.

[0149] In some achievable manners, the writing module 13 is further configured to: when the log data queue overflows, store the overflowed structured log into the storage space.

[0150] In some achievable embodiments, the write module 13 is specifically used to: read structured logs from the log data queue through multiple parallel write threads, each of the write threads reads multiple structured logs from the log data queue; each of the write threads writes the read multiple structured logs to the remote server through a write request.

[0151] In some practicable embodiments, a shutdown module is further included, which is used to use an elegant shutdown method to shut down the device after processing the data in the log data queue when a device shutdown instruction is received, and the log data queue no longer receives new structured logs.

[0152] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here.

[0153] The above describes the device 100 of the embodiment of the present application from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

[0154] An embodiment of the present application also provides an electronic device. Fig. 9 A structural diagram of an electronic device provided in Embodiment 4 of the present application, such as Fig. 9 As shown, the electronic device 200 may include:

[0155] The memory 21 and the processor 22, the memory 21 is used to store the computer program and transmit the program code to the processor 22. In other words, the processor 22 can call and run the computer program from the memory 21 to implement the method in the embodiment of the present application.

[0156] For example, the processor 22 may be configured to execute the above method embodiments according to instructions in the computer program.

[0157] In some embodiments of the present application, the processor 22 may include but is not limited to:

[0158] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0159] In some embodiments of the present application, the memory 21 includes but is not limited to:

[0160] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DR RAM).

[0161] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 21 and executed by the processor 22 to complete the method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device 200.

[0162] like Fig. 9 As shown, the electronic device 200 further includes: a transceiver 23 , which can be connected to the processor 22 or the memory 21 .

[0163] The processor 22 may control the transceiver 23 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 23 may include a transmitter and a receiver. The transceiver 23 may further include an antenna, and the number of antennas may be one or more.

[0164] Understandable, although Fig. 9 Not shown, the electronic device 200 may also include other modules such as a positioning module, a Bluetooth module, a display, and a controller, which will not be described in detail here.

[0165] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0166] The present application also provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the present application embodiment also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

[0167] The present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. The processor of the electronic device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the electronic device executes the corresponding process in the above method embodiment, which will not be described here for brevity.

[0168] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0169] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0170] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A log processing method, characterized in that: include: Through a collection module on an application framework of an application server, during the operation of the application server, according to a configured log collection cycle, the value of a basic configuration type field is collected from the operating system of the application server, the value of a framework type field is collected from the application framework, and the value of a business logic type field is collected from the business logic code; The value of the basic configuration type field, the value of the collection framework type field, and the value of the business logic type field collected in the same collection period form a structured log; Preprocessing the collected structured logs through a preprocessing module on the application framework; The pre-processed structured log is stored in a structured manner through a writing module on the application framework.

2. The method according to claim 1, characterized in that: The acquisition module includes a first acquisition unit, a second acquisition unit and a third acquisition unit; The first collection unit is used to call the operating system according to the log collection cycle, collect the value of the basic configuration type field from the operating system, and fill the collected value of the basic configuration type field into a predefined log structure; The second collection unit is used to collect the value of the framework type field from the application framework according to the log collection period, and fill the collected value of the framework type field into the log structure; The third collection unit is used to collect the value of the business logic type field from the business logic code according to the log collection cycle, and fill the collected value of the business logic type field into the log structure; The value of the basic configuration type field, the value of the acquisition framework type field, and the value of the business logic type field collected in the same collection period and stored in the log structure constitute a structured log.

3. The method according to claim 1, characterized in that The basic configuration type field includes at least one of the following fields: a deployment environment field and an Internet Protocol IP address; The framework type field includes at least one of the following fields: a trace identification field, a thread identification field, a file name field, a file line number field, a method name field, and a log level field; The business logic type field includes at least one of the following fields: a time field, a user information field, an identification field of a first request type, an identification field of a second request type, and a custom field. The first request type is a request type between the application server and the client, and the second request type is a request type between the application server and other servers.

4. The method according to any one of claims 1 to 3, characterized in that: The preprocessing includes at least one of the following processes: log format planning processing, deduplication processing, filtering processing, and classification processing.

5. The method according to claim 4, characterized in that When the business logic type field includes a custom field, the preprocessing further includes word segmentation processing; The preprocessing of the collected structured logs includes: According to the configured word segmentation rule, the value of the custom field in the structured log is segmented to obtain at least one entry corresponding to the value of the custom field.

6. The method according to any one of claims 1 to 3, characterized in that: The structured log is stored in a remote server; The step of performing structured storage on the preprocessed structured logs includes: Putting the preprocessed structured log data into a log data queue; A plurality of structured logs are read from the log data queue through a write thread, and the plurality of structured logs are written to the remote server through a write request.

7. The method according to claim 6, characterized in that Before writing the plurality of structured logs to the remote server through a write request, the method further includes: The write thread performs business merging on the multiple structured logs according to business logic, and writes the merged structured logs to the remote server through the write request.

8. The method according to claim 6, characterized in that The method further comprises: When the writing of the plurality of structured logs to the remote server fails, the plurality of structured logs that failed to be written are stored in the allocated storage space through the writing thread, and the storage space is used to store the structured logs that failed to be written; The structured log in the storage space is put into the log data queue through the rewriting thread.

9. The method according to claim 8, characterized in that The step of placing the pre-processed structured log data into a log data queue includes: When the log data queue overflows, the overflowed structured log is stored in the storage space.

10. The method according to claim 6, characterized in that The step of reading a plurality of structured logs from the log data queue through a write thread and writing the plurality of structured logs to the remote server through a write request comprises: Reading structured logs from the log data queue through multiple parallel writing threads, each of the writing threads reading multiple structured logs from the log data queue; Each of the writing threads writes the read multiple structured logs to the remote server through a writing request.

11. The method according to claim 6, characterized in that Also includes: When a device shutdown instruction is received, an elegant shutdown mode is adopted to shut down the device after the data in the log data queue is processed, and the log data queue no longer receives new structured logs.

12. A log processing device, characterized in that: include: A collection module, configured on the application framework of the application server, for collecting the value of the basic configuration type field from the operating system of the application server, collecting the value of the framework type field from the application framework, and collecting the value of the business logic type field from the business logic code according to the configured log collection cycle during the operation of the application server; The collection module is further used to combine the value of the basic configuration type field, the value of the collection framework type field and the value of the business logic type field collected in the same collection period into a structured log; A preprocessing module, configured on the application framework, for preprocessing the structured logs collected by the collection module; A writing module is configured on the application framework and is used for performing structured storage on the preprocessed structured log.

13. An electronic device, characterized in that: include: A processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to execute the method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein the computer program causes a computer to execute the method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Citation Information

Patent Citations

  • Log management method and system and computer-readable storage medium

    CN108052675A

  • Log acquisition method and system, computer device and storage medium

    CN109218401A

  • Enterprise industrial control safety brain platform system and operation method

    CN114297020A

  • Matrix oriented application framework with interchangeable data source mapping and user interface

    US20060179065A1

  • Structured logging and instrumentation framework

    US20160041894A1