A log collection method and device, electronic equipment and storage medium

Abstract

The application provides a log collection method and device, electronic equipment and a storage medium. The method comprises the following steps: establishing a log entity for encapsulating log information; obtaining the log information; extracting at least one piece of sub-information from the log information by using a preset first analysis function; encapsulating each piece of extracted sub-information into a corresponding log class to obtain a target log entity; uploading the target log entity into a target partition of a message middleware; and when it is necessary to analyze the target log entity, analyzing the target log entity in the target partition by using a preset second analysis function to obtain analyzed target log information. The application can classify and store the sub-information in the log information during the log collection process, which facilitates subsequent queries.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a log collection method and device, electronic equipment and a storage medium. BACKGROUND

[0002] Traditional log collection methods collect logs through various log components that implement the log4j interface, such as log systems such as slf4j / log4j / logback / JDK14 Logging. Most of them flush logs to the local file system in an asynchronous manner, and then send the local log files or formatted data obtained through the tool to the corresponding log archiving center through a corresponding synchronous component such as Rsync or Beats. After aggregation and processing by the log archiving center, the log is provided to the application or project team for log query and problem positioning, or the local log file is collected through an agent, and the log system is sent to the remote Collector for log collection through the local Agent.

[0003] The above log collection methods not only have a delay, but also do not have a fine-grained division of the collected logs. When the log archiving center accesses a large number of projects, the query often becomes a relatively cumbersome matter. Different project teams focus on different log ranges, and often need to manually customize different query templates, which is relatively time-consuming. For a developer, traceability and quick positioning of logs are the most urgent needs of using a log system. The developer often has little energy to study how to use a log system and how to filter, and there is a risk of abnormal loss of logs. SUMMARY

[0004] Therefore, the embodiments of the present application provide a log collection method and device, electronic equipment and a storage medium, which can classify and store sub-information in log information during the collection process of the log, and facilitate subsequent queries.

[0005] The technical scheme of the embodiments of the present application is as follows:

[0006] In a first aspect, the embodiments of the present application provide a log collection method, including the following steps:

[0007] Establishing a log entity for encapsulating log information, wherein the log information includes at least one sub-information, each sub-information in the at least one sub-information is different in type, and the log entity includes at least one log class, each log class in the at least one log class is used for encapsulating a type of sub-information;

[0008] obtaining the log information, extracting the at least one piece of sub-information from the log information through a preset first analysis function, encapsulating each piece of sub-information after extraction into a corresponding log class, and obtaining a target log entity;

[0009] uploading the target log entity into a target partition of a message middleware, and when the target log entity needs to be analyzed, analyzing the target log entity in the target partition through a preset second analysis function to obtain analyzed target log information.

[0010] In a possible implementation, the establishing of the log entity for encapsulating log information comprises:

[0011] creating a target object and taking the target object as the log entity, wherein the target object comprises at least one of the following fields:

[0012] accessing a project name, a method call in code, a log level, a code class name, a log body, a log recording time, an application tracking ID, and an application module marker.

[0013] In a possible implementation, the obtaining of the log information comprises:

[0014] establishing an external interface for communicating with a local log storage service;

[0015] obtaining a local log file through the external interface, wherein the local log file carries the log information.

[0016] In a possible implementation, the extracting of the at least one piece of sub-information from the log information through a preset first analysis function and the encapsulating of each piece of sub-information after extraction into a corresponding log class to obtain a target log entity comprise:

[0017] reading the log information and storing the read log information in a cache;

[0018] performing traversal processing on the log information stored in the cache through a preset first analysis function to obtain the at least one piece of sub-information in the log information;

[0019] encapsulating each piece of sub-information in the at least one piece of sub-information into a corresponding log class to obtain the target log entity.

[0020] In a possible implementation, the uploading of the target log entity into a target partition of a message middleware comprises:

[0021] A producer client for the Kafka cluster is created, wherein the producer client is used to forward the target log entity to the target partition of the Kafka cluster, and the producer client carries a partition identifier of the target partition;

[0022] The target log entity is sent to the target partition of the Kafka cluster through the producer client.

[0023] In a possible implementation, the target log entity in the target partition is parsed through the preset second parsing function to obtain parsed target log information.

[0024] A consumer client for the Kafka cluster is created, wherein the consumer client is used to parse the target log entity in the target partition, and the second parsing function is preset in the consumer client;

[0025] The target log entity is parsed through the second parsing function in the consumer client to obtain the parsed target log information.

[0026] In a possible implementation, the method further includes:

[0027] The parsed target log information is stored in a log archiving center, wherein the log archiving center includes at least one of the following:

[0028] a database, clickHouse, and GrayLog.

[0029] In a second aspect, an embodiment of the present application further provides a log collection device, and the device includes:

[0030] A creating module is configured to create a log entity for encapsulating log information, wherein the log information includes at least one sub-information, each sub-information in the at least one sub-information is different in type, and the log entity includes at least one log class, each log class in the at least one log class is used to encapsulate a type of sub-information.

[0031] An obtaining module is configured to obtain the log information, extract the at least one sub-information from the log information through a preset first parsing function, encapsulate each sub-information after extraction into a corresponding log class, and obtain a target log entity.

[0032] A processing module is configured to upload the target log entity to a target partition of a message middleware, and when it is necessary to parse the target log entity, parse the target log entity in the target partition through a preset second parsing function to obtain parsed target log information.

[0033] In a third aspect, the embodiments of the present application further provide an electronic device, comprising a processor, a storage medium and a bus, the storage medium storing machine readable instructions executable by the processor, when the electronic device is running, the processor and the storage medium communicate through the bus, the processor executes the machine readable instructions to execute the log collection method of any one of the first aspect.

[0034] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, the computer readable storage medium storing a computer program, when the computer program is run by a processor, the log collection method of any one of the first aspect is executed.

[0035] The embodiments of the present application have the following beneficial effects:

[0036] 1. By establishing a log entity for packaging log information, the log information is acquired, which does not conflict with the original log component and is convenient to access.

[0037] 2. By connecting the target log entity with the kafka cluster, the target log entity is converted into a stream and pushed to a certain target partition (topic) on the kafka, which has relatively high real-time performance.

[0038] 3. Since kafka is used as a log stream transmission pipeline, the downstream storage is unlimited, various big data components such as ElasticSearch|ClickHouse|Flume can be accessed, and after cleaning the log, the associated log can be stored in a relational database for user behavior log analysis. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor.

[0040] Figure 1 is a flowchart of steps S101-S103 provided by the embodiments of the present application;

[0041] Figure 2 is a flowchart of steps S201-S202 provided by the embodiments of the present application;

[0042] Figure 3 is a flowchart of steps S301-S303 provided by the embodiments of the present application;

[0043] Figure 4 is a flowchart of steps S401-S402 provided by an embodiment of the present application;

[0044] Figure 5 is a flowchart of steps S501-S502 provided by an embodiment of the present application;

[0045] Figure 6 is a structural diagram of a log collection device provided by an embodiment of the present application;

[0046] Figure 7 is a structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0047] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of description and illustration, and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn according to the actual proportions. The flowcharts show the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can not be implemented in sequence, and the steps without logical context relationship can be reversed in sequence or implemented simultaneously. In addition, one or more other operations can be added to the flowcharts or one or more operations can be removed from the flowcharts by those skilled in the art under the guidance of the content of the present application.

[0048] In the following description, “some embodiments” are related to a subset of all possible embodiments, but it can be understood that “some embodiments” can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0049] In addition, the described embodiments are only some of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0050] In the following description, the terms "first", "second", "third" are merely used to distinguish similar objects, and do not represent a specific order of the objects. Understandably, the "first", "second", "third" can be interchanged in a specific order or sequence as allowed, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0051] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0053] Referring to Figure 1 , Figure 1 is a flowchart of the log collection method steps S101-S103 provided by the embodiments of the present application, which will be described in combination with Figure 1 steps S101-S103 shown in the figure.

[0054] Step S101, establishing a log entity for encapsulating log information, wherein the log information includes at least one sub-information, each sub-information in the at least one sub-information is different in type, and the log entity includes at least one log class, each log class in the at least one log class is used to encapsulate a type of sub-information;

[0055] Step S102, obtaining the log information, extracting the at least one sub-information from the log information through a preset first parsing function, and encapsulating each sub-information after extraction into a corresponding log class to obtain a target log entity;

[0056] Step S103, uploading the target log entity to a target partition of a message middleware, and when it is necessary to parse the target log entity, parsing the target log entity in the target partition through a preset second parsing function to obtain parsed target log information.

[0057] The above log collection method has the following beneficial effects:

[0058] 1. By establishing a log entity for encapsulating log information, the log information is obtained, which does not conflict with the original log component and is convenient to access.

[0059] 2. By connecting the target log entity with the kafka cluster, the target log entity is converted into a stream and pushed to a certain target partition (topic) on the kafka, which has high real-time performance.

[0060] 3. Since kafka is used as a log stream transmission pipeline, the downstream storage is unlimited, and various big data components such as ElasticSearch|ClickHouse|Flume can be accessed. After cleaning the log, the associated log can be stored in a relational database for user behavior log analysis.

[0061] The above exemplary steps of the embodiments of the present application will be described below.

[0062] In step S101, a log entity for encapsulating log information is established, wherein the log information includes at least one sub-information, each sub-information in the at least one sub-information is different in type, and the log entity includes at least one log class, each log class in the at least one log class is used for encapsulating a type of sub-information.

[0063] In some embodiments, the establishment of the log entity for encapsulating the log information includes:

[0064] A target object is created and used as the log entity, wherein the target object includes at least one of the following fields:

[0065] The project name, method call in the code, log level, code class name, log text, log recording time, application tracking ID, and application module marker are accessed.

[0066] Here, a target object LogMessage is first created as a log entity that can be freely expanded. For common attributes, default support is provided, and users can freely expand based on the entity. Common attributes include:

[0067] appName: Access the project name;

[0068] mMethod: Method call in the code;

[0069] mLevel: Log level;

[0070] className: Code class name;

[0071] message: Log text;

[0072] mDatetime: Log recording time;

[0073] traceId: Application tracking ID, used to concatenate log context;

[0074] marker: application module marker, used in multi-module development.

[0075] In step S102, the log information is acquired, the at least one piece of sub-information is extracted from the log information by using a preset first analysis function, each piece of the extracted sub-information is encapsulated into a corresponding log class, and a target log entity is obtained.

[0076] In some embodiments, referring to Figure 2 , Figure 2 is a flowchart of steps S201-S202 provided in the embodiments of the present application, and the acquisition of the log information can be implemented by steps S201-S202, which will be described in combination with the steps.

[0077] In step S201, an external interface for communicating with a local log storage service is established.

[0078] In step S202, a local log file is acquired through the external interface, and the log information is carried in the local log file.

[0079] Here, in order not to change the existing log system, an external interface for communicating with a local log storage service can be established in the existing log system, and then a local log file is acquired through the external interface.

[0080] In some embodiments, referring to Figure 3 , Figure 3 is a flowchart of steps S301-S303 provided in the embodiments of the present application, and the extraction of the at least one piece of sub-information from the log information by using a preset first analysis function and the encapsulation of each piece of the extracted sub-information into a corresponding log class to obtain a target log entity can be implemented by steps S301-S303, which will be described in combination with the steps.

[0081] In step S301, the log information is read, and the read log information is stored in a cache.

[0082] In step S302, the log information stored in the cache is processed by using a preset first analysis function, and the at least one piece of sub-information in the log information is obtained.

[0083] In step S303, each piece of the at least one piece of sub-information is encapsulated into a corresponding log class, and the target log entity is obtained.

[0084] Here, the KafkaAppender can parse the log event iLoggingEvent based on the UnsynchronizedAppenderBase abstract class of the logback-core, and can extract the sub-information in the iLoggingEvent by overriding the appender method, encapsulate the iLoggingEvent into the LogMessage, and encapsulate the appName and marker passed in the log configuration into the LogMessage. For some custom attributes such as traceId, the MDCPropertyMap can be used to obtain them.

[0085] In step S103, the target log entity is uploaded to the target partition of the message middleware, and when the target log entity needs to be parsed, the target log entity in the target partition is parsed by a preset second parsing function to obtain parsed target log information.

[0086] In some embodiments, referring to Figure 4 , Figure 4 is a flowchart of steps S401-S402 provided by the embodiments of the present application, and the uploading of the target log entity to the target partition of the message middleware can be implemented by steps S401-S402, and each step is described.

[0087] In step S401, a producer client for a kafka cluster is created, wherein the producer client is used to forward the target log entity to the target partition of the kafka cluster, and the producer client carries the partition identifier of the target partition.

[0088] In step S402, the target log entity is sent to the target partition of the kafka cluster through the producer client.

[0089] Here, the kafkaProducer client can be generated by the KafkaProducerProxy, and the serialized LogMessage can be sent to the specified Topic.

[0090] The kafkaProducer can be regarded as a layer of proxy service between the kafka cluster and the application system, which is responsible for forwarding the messages of the application system to the kafka cluster. The producer client is a connection established with a specific target partition, that is, the message is sent to a specific target partition, and it does not care which partition the message belongs to.

[0091] The above-mentioned method does not need to worry about occupying too many resources because the KafkaProducer is managed by pooling.

[0092] In some embodiments, referring to Figure 5 , Figure 5 is a flowchart of steps S501-S502 provided by the embodiments of the present application, and the target log information obtained by parsing the target log entity in the target partition through the preset second parsing function can be implemented by steps S501-S502. Each step will be described.

[0093] In step S501, a consumer client for a kafka cluster is created, wherein the consumer client is used to parse the target log entity in the target partition, and the second parsing function is preset in the consumer client.

[0094] In step S502, the target log entity is parsed through the second parsing function in the consumer client to obtain the parsed target log information.

[0095] Here, the downstream can parse the serialized LogMessage through the KafkaConsumer client to obtain the parsed target log information.

[0096] It should be noted that the Consumer is called a message subscriber or a message consumer, which is responsible for reading and consuming messages from the server. The Consumer Group refers to a class of Consumers, which usually receives and consumes the same type of messages, and the message consumption logic is consistent. The messages are subscribed and consumed from the target partition through the Pull mode.

[0097] In some embodiments, the method further comprises:

[0098] The parsed target log information is stored in a log archiving center, wherein the log archiving center comprises at least one of the following:

[0099] Database, clickHouse, GrayLog.

[0100] Here, the sub-information in the parsed target log information has been classified, and these sub-information can be stored respectively and stored in the log archiving center, thereby facilitating subsequent query.

[0101] In some embodiments, the target log information can be stored in a database, which can be a relational database or a non-relational database, such as MySql, Oracle, Nosql.

[0102] In some embodiments, the target log information can be stored to clickHouse, which is an open source columnar storage database (DBMS: Database Management System) for Online Analytical Processing query (OLAP: Online Analytical Processing) MPP architecture, and can generate analysis data reports in real time using SQL queries. The full name of ClickHouse is Click Stream, Data WareHouse.

[0103] In some embodiments, when searching for target log information, an Elasticsearch engine can be used, and Elasticsearch is a distributed, highly scalable, and highly real-time search and data analysis engine that can easily enable large amounts of data to have search, analysis, and exploration capabilities.

[0104] In some embodiments, GrayLog can also be used to manage target log information, and graylog is a simple and easy-to-use log management tool with relatively complete functions. Graylog also uses Elasticsearch as storage and indexing to ensure performance, MongoDB is used to store a small amount of self-configuration information, the master-node mode has good scalability, the basic query and analysis functions on the UI are relatively practical and efficient, supports LDAP, permission control, and has rich log types and standards (such as syslog, GELF) and supports log-based alarm.

[0105] In summary, the embodiments of the present application have the following beneficial effects:

[0106] 1. By establishing a log entity for encapsulating log information, the log information is acquired without conflict with the original log component, and the access is convenient.

[0107] 2. By connecting the target log entity with the kafka cluster, the target log entity is converted into a stream and pushed to a certain target partition (topic) on kafka, and the real-time performance is relatively high.

[0108] 3. Since kafka is used as a log stream transmission pipeline, the downstream storage is unlimited, various big data components such as ElasticSearch|ClickHouse|Flume can be accessed, and after cleaning the log, the associated log can be stored in a relational database for user behavior log analysis.

[0109] Based on the same inventive concept, the log collection device corresponding to the log collection method in the first embodiment is also provided in the embodiments of the present application. Since the principle of the device in the embodiments of the present application for solving the problem is similar to the log collection method, the implementation of the device can be referred to the implementation of the method, and the repeated parts will not be described here.

[0110] As shown in Figure 6 , Figure 6 is a structural schematic diagram of the log collection device 600 provided by the embodiments of the present application. The log collection device 600 comprises:

[0111] The creating module 601 is configured to establish a log entity for encapsulating log information, wherein the log information comprises at least one sub-information, each of the at least one sub-information is of different type, and the log entity comprises at least one log class, each of the at least one log class is used for encapsulating a type of sub-information.

[0112] The obtaining module 602 is configured to obtain the log information, extract the at least one sub-information from the log information by using a preset first analysis function, encapsulate each of the extracted sub-information into a corresponding log class, and obtain a target log entity.

[0113] The processing module 603 is configured to upload the target log entity to a target partition of a message middleware, and when it is needed to analyze the target log entity, analyze the target log entity in the target partition by using a preset second analysis function, and obtain analyzed target log information.

[0114] Those skilled in the art should understand that Figure 6 The implementation functions of each unit in the log collection device 600 shown in the above can be understood with reference to the related description of the log collection method. Figure 6 The functions of each unit in the log collection device 600 shown in the above can be implemented by a program running on a processor, or by a specific logic circuit.

[0115] In a possible implementation, the creating module 601 establishes a log entity for encapsulating log information, comprising:

[0116] A target object is created, and the target object is used as the log entity, wherein the target object comprises at least one of the following fields:

[0117] Access project name, code method call, log level, code class name, log body, log recording time, application tracking ID, and application module marker.

[0118] In a possible implementation, the obtaining module 602 obtains the log information, comprising:

[0119] establishing an external interface for communicating with the local log storage service;

[0120] acquiring a local log file through the external interface, wherein the local log file carries the log information.

[0121] In a possible implementation, the acquisition module 602 extracts the at least one piece of sub-information from the log information through a preset first parsing function, and encapsulates each piece of extracted sub-information into a corresponding log class to obtain a target log entity, including:

[0122] reading the log information and storing the read log information in a cache;

[0123] performing traversal processing on the log information stored in the cache through the preset first parsing function to obtain the at least one piece of sub-information in the log information;

[0124] encapsulating each piece of sub-information in the at least one piece of sub-information into a corresponding log class to obtain the target log entity.

[0125] In a possible implementation, the processing module 603 uploads the target log entity to a target partition of a message middleware, including:

[0126] creating a producer client for a kafka cluster, wherein the producer client is configured to forward the target log entity to the target partition of the kafka cluster, and the producer client carries a partition identifier of the target partition;

[0127] sending the target log entity to the target partition of the kafka cluster through the producer client.

[0128] In a possible implementation, the processing module 603 parses the target log entity in the target partition through a preset second parsing function to obtain parsed target log information, including:

[0129] creating a consumer client for the kafka cluster, wherein the consumer client is configured to parse the target log entity in the target partition, and the second parsing function is preset in the consumer client;

[0130] parsing the target log entity through the second parsing function in the consumer client to obtain the parsed target log information.

[0131] In a possible implementation, the processing module 603 further includes:

[0132] The parsed target log information is stored in a log archiving center, wherein the log archiving center includes at least one of the following:

[0133] Database, clickHouse, GrayLog.

[0134] The above-mentioned log collection device has the following beneficial effects:

[0135] 1. By establishing a log entity to encapsulate log information, log information can be obtained without conflicting with the original log component, and integration is convenient.

[0136] 2. By connecting the target log entity to the Kafka cluster, the target log entity is converted into a stream and pushed to a target partition (topic) on Kafka, which has high real-time performance.

[0137] 3. Since Kafka is used as the log stream transmission pipeline, there are no restrictions on downstream storage. Various big data components can be connected, such as ElasticSearch, ClickHouse, Flume, etc. Logs can also be cleaned and stored in relational databases for user behavior log analysis.

[0138] like Figure 7 As shown, Figure 7 This is a schematic diagram of the composition structure of the electronic device 700 provided in the embodiments of this application. The electronic device 700 includes:

[0139] The device 700 includes a processor 701, a storage medium 702, and a bus 703. The storage medium 702 stores machine-readable instructions that can be executed by the processor 701. When the electronic device 700 is running, the processor 701 communicates with the storage medium 702 via the bus 703. The processor 701 executes the machine-readable instructions to perform the steps of the log collection method described in the embodiments of this application.

[0140] In practical applications, the various components in the electronic device 700 are coupled together via a bus 703. It is understood that the bus 703 is used to achieve communication between these components. In addition to a data bus, the bus 703 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 7 The general designated all buses as Bus 703.

[0141] The above-mentioned electronic devices have the following beneficial effects:

[0142] 1. By establishing a log entity to encapsulate log information, log information can be obtained without conflicting with the original log component, and integration is convenient.

[0143] 2. By connecting the target log entity with the kafka cluster, the target log entity is converted into a stream and pushed to a certain target partition (topic) on the kafka, and the real-time performance is relatively high.

[0144] 3. Since kafka is used as a log stream transmission pipeline, the downstream storage is unlimited, and various big data components such as ElasticSearch, ClickHouse, Flume, etc. can be accessed. After cleaning the logs, the associated logs can be stored in a relational database for user behavior log analysis.

[0145] The embodiment of the present application also provides a computer readable storage medium, the storage medium stores executable instructions, when the executable instructions are executed by at least one processor 701, the log collection method is realized.

[0146] In some embodiments, the storage medium can be a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc. It can also be various devices including one or any combination of the above memories.

[0147] In some embodiments, the executable instructions can be in the form of programs, software, software modules, scripts or codes, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and can be deployed in any form, including being deployed as independent programs or as modules, components, subroutines or other units suitable for use in a computing environment.

[0148] As an example, executable instructions can correspond to a file in a file system, can be stored in a part of a file that is used by the operating system to store application program data (for example, one or more script files stored in a filesystem or directory on the computing device), can be stored as a single file (for example, a.jar file), or can be stored across multiple coordinated files (for example, files that store one or more modules, subprograms, or code portions).

[0149] As an example, the executable instructions can be deployed across one computing device, or across multiple computing devices located at one site, or distributed across multiple sites and interconnected by a communication network.

[0150] The computer-readable storage medium described above has the following beneficial effects:

[0151] 1. By establishing a log entity for encapsulating log information, the log information is acquired without conflict with the original log component, and the access is convenient.

[0152] 2. By connecting the target log entity with the kafka cluster, the target log entity is converted into a stream and pushed to a certain target partition (topic) on the kafka, and the real-time performance is relatively high.

[0153] 3. Since kafka is used as a log stream transmission pipeline, the downstream storage is unlimited, various big data components such as ElasticSearch|ClickHouse|Flume can be accessed, and the associated log can be stored in a relational database after being cleaned to perform user behavior log analysis.

[0154] In several embodiments provided in the present application, it should be understood that the disclosed method and electronic device can be implemented by other ways. The above-described device embodiments are only illustrative, for example, the division of the units is only a logical function division, and actual implementation can have another division mode, for example, multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed components can be through some interfaces, indirect coupling or communication connection of the devices or units, which can be electrical, mechanical or other forms.

[0155] The modules described as separate components may or may not be physically separate, and the components displayed as modules may or may not be physical units, i.e., may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0156] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0157] If the functions are realized in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a platform server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, and various program code storage media.

[0158] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A log collection method characterized by, The method comprises the following steps: establishing a log entity for encapsulating log information, wherein the log information comprises at least one sub-information, each of the at least one sub-information is of different type, the log entity comprises at least one log class, each of the at least one log class is used for encapsulating sub-information of one type; obtaining the log information, extracting the at least one sub-information from the log information through a preset first analysis function, and encapsulating each of the extracted sub-information into a corresponding log class to obtain a target log entity; wherein, based on the UnsynchronizedAppenderBase abstract class of logback-core, the KafkaAppender realizes the analysis of the log event iLoggingEvent; uploading the target log entity to a target partition of a message middleware, and when the target log entity needs to be analyzed, the target log entity in the target partition is analyzed through a preset second analysis function to obtain analyzed target log information; the establishing of the log entity for encapsulating the log information comprises: creating a target object LogMessage as a log entity that can be freely extended, wherein the target object comprises at least one of the following fields: accessing the project name, the method call in the code, the log level, the code class name, the log text, the log recording time, the application tracking ID, and the application module marker; the uploading of the target log entity to the target partition of the message middleware comprises: creating a producer client for a kafka cluster, wherein the producer client is used to forward the target log entity to the target partition of the kafka cluster, and the producer client carries the partition identifier of the target partition; sending the target log entity to the target partition of the kafka cluster through the producer client.

2. The method of claim 1, wherein, the obtaining of the log information comprises: establishing an external interface for communicating with a local log storage service; obtaining a local log file through the external interface, wherein the local log file carries the log information.

3. The method of claim 1, wherein, the extracting of the at least one sub-information from the log information through the preset first analysis function and the encapsulating of each of the extracted sub-information into a corresponding log class to obtain the target log entity comprises: reading the log information and storing the read log information in a cache; processing the log information stored in the cache through the preset first analysis function to obtain the at least one sub-information in the log information; encapsulating each of the at least one sub-information into a corresponding log class to obtain the target log entity.

4. The method of claim 1, wherein, the analyzing of the target log entity in the target partition through the preset second analysis function to obtain the analyzed target log information comprises: A consumer client for a Kafka cluster is created, wherein the consumer client is used to parse the target log entity in the target partition, and the second parsing function is preset in the consumer client; The target log entity is parsed by the second parsing function in the consumer client to obtain parsed target log information.

5. The method of claim 1, wherein, The method further comprises: The parsed target log information is stored in a log archiving center, wherein the log archiving center comprises at least one of the following: Database, clickHouse, GrayLog.

6. A log collection apparatus characterized by comprising: The device comprises: A creating module is configured to create a log entity for encapsulating log information, wherein the log information comprises at least one sub-information, each sub-information in the at least one sub-information is of a different type, the log entity comprises at least one log class, and each log class in the at least one log class is used to encapsulate a type of sub-information; the creating module is further configured to create a target object LogMessage as an implementation of a freely extensible log entity, wherein the target object comprises at least one of the following fields: an access project name, a method call in code, a log level, a code class name, a log body, a log recording time, an application tracking ID, and an application module marker; An obtaining module is configured to obtain the log information, extract the at least one sub-information from the log information by using a preset first parsing function, encapsulate each extracted sub-information into a corresponding log class, and obtain a target log entity; wherein, based on the UnsynchronizedAppenderBase abstract class of logback-core, the KafkaAppender parses the log event iLoggingEvent; A processing module is configured to upload the target log entity to a target partition of a message middleware, and when the target log entity needs to be parsed, the target log entity in the target partition is parsed by using a preset second parsing function to obtain parsed target log information; the uploading of the target log entity to the target partition of the message middleware comprises: creating a producer client for a Kafka cluster, wherein the producer client is used to forward the target log entity to the target partition of the Kafka cluster, and the producer client carries a partition identifier of the target partition; and sending the target log entity to the target partition of the Kafka cluster by using the producer client.

7. An electronic device, comprising: It comprises: A processor, a storage medium, and a bus, the storage medium stores machine readable instructions executable by the processor, when the electronic device is running, the processor and the storage medium communicate through the bus, the processor executes the machine readable instructions to execute the log collection method in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is run by the processor to perform the log collection method in any one of claims 1 to 5.

Images