Method and apparatus for log generation, log processing

Abstract

Embodiments of the present specification provide a log generation and log processing method and device. In the method, a first device can detect a predetermined event during the running of a service program. When the predetermined event is detected, the first device obtains a plurality of log data corresponding to a plurality of fields, respectively, from the predetermined event based on a plurality of fields of predetermined data to be obtained. When a predetermined output condition is met, the first device outputs the plurality of log data based on the correspondence between the plurality of fields and the plurality of log data, respectively.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and more particularly to a method and apparatus for generating and processing logs. Background Technology

[0002] Logs are data files used to record parameters such as user operations and equipment operating status, allowing relevant personnel to understand the equipment's operational status and business processing. As equipment continues to operate, a large amount of log data is often generated. Performing various analyses and processing on this log data is a common way to obtain useful information.

[0003] Therefore, we hope to find an improved solution that can process logs more efficiently. Summary of the Invention

[0004] This specification describes one or more embodiments of a log generation method and apparatus to achieve more efficient log processing. The specific technical solution is as follows.

[0005] In a first aspect, an embodiment provides a log generation method, executed by a first device, comprising:

[0006] During the operation of the service program, pre-defined events are detected;

[0007] Based on a number of preset fields for data to be acquired, several log element data corresponding to the preset fields are obtained from the predetermined event.

[0008] When the preset output conditions are met, the log element data is output in a structured manner based on the correspondence between the several fields and the several log element data.

[0009] In one implementation, the step of outputting the plurality of log element data in a structured manner includes:

[0010] The log elements arranged in a first preset order are used to form structured log data, which is then sent to a second device. The second device associates the log elements in the received structured log data with the fields arranged in the first preset order to obtain structured log records.

[0011] In one implementation, the preset output condition includes: when a preset number of log data groups are obtained; wherein, a log data group contains several log element data, and the preset number of log data groups are obtained from a preset number of predetermined events;

[0012] The step of forming structured log data from the plurality of log element data arranged in a first preset order includes:

[0013] The preset number of log data groups are used to form structured log data; wherein, the log element data contained in any log data group are arranged in the first preset order.

[0014] In one implementation, the step of outputting the plurality of log element data in a structured manner includes:

[0015] The fields are associated with the log element data respectively to obtain key-value pairs, and the key-value pairs are output in a structured manner.

[0016] In one implementation, the step of outputting the plurality of key-value pairs in a structured manner includes:

[0017] The structured key-value pairs are output to the log table of the specified file to obtain a log table record.

[0018] In one implementation, the step of outputting the plurality of log element data in a structured manner includes:

[0019] The log elements arranged in a second preset order are used to form structured log data. The structured log data is then output to a specified file, and the log elements contained in the structured log data are associated with several fields in the specified file. The fields in the specified file are arranged in the second preset order.

[0020] Secondly, the embodiment provides a log processing method, executed by a second device, including:

[0021] The system receives a plurality of log element data arranged in a first preset order, sent by a first device in a structured manner; wherein the plurality of log element data is obtained from a predetermined event and corresponds to a plurality of fields of data to be obtained.

[0022] The log element data is associated with several fields arranged in the first preset order to obtain structured log records.

[0023] In one embodiment, the step of receiving log element data arranged in a first preset order and sent in a structured manner by the first device includes:

[0024] The system receives a preset number of log data groups sent by the first device in a structured manner; wherein, any log data group contains several log element data arranged in the first preset order, and the preset number of log data groups are obtained from a preset number of predetermined events;

[0025] The step of associating the plurality of log element data with the plurality of fields arranged in the first preset order includes:

[0026] For any log data group, several log element data in the log data group are associated with several fields arranged in the first preset order to obtain the structured log record corresponding to the log data group.

[0027] In one implementation, after obtaining multiple structured log records, the method further includes:

[0028] Store the multiple structured log records in a database; and / or perform statistics on the multiple structured log records.

[0029] Thirdly, the embodiment provides a log generation apparatus, deployed in a first device, comprising:

[0030] The detection module is configured to detect predetermined events during the operation of the service program;

[0031] The acquisition module is configured to acquire, based on a preset number of fields of data to be acquired, a number of log element data corresponding to the preset fields from the predetermined event.

[0032] The output module is configured to output the log element data in a structured manner based on the correspondence between the several fields and the several log element data when the preset output conditions are met.

[0033] In one implementation, the output module is specifically configured as follows:

[0034] The log elements arranged in a first preset order are used to form structured log data, which is then sent to a second device. The second device associates the log elements in the received structured log data with the fields arranged in the first preset order to obtain structured log records.

[0035] In one implementation, the preset output condition includes: when a preset number of log data groups are obtained; wherein, a log data group contains several log element data, and the preset number of log data groups are obtained from a preset number of predetermined events;

[0036] The output module is specifically configured as follows:

[0037] The preset number of log data groups are used to form structured log data; wherein, the log element data contained in any log data group are arranged in the first preset order.

[0038] In one implementation, the output module is specifically configured as follows:

[0039] The fields are associated with the log element data respectively to obtain key-value pairs, and the key-value pairs are output in a structured manner.

[0040] Fourthly, the embodiment provides a log processing apparatus, deployed in a second device, comprising:

[0041] The receiving module is configured to receive a plurality of log element data arranged in a first preset order and sent by the first device in a structured manner; wherein the plurality of log element data is obtained from a predetermined event and corresponds to a plurality of fields of data to be obtained;

[0042] The association module is configured to associate the plurality of log element data with a plurality of fields arranged in the first preset order to obtain structured log records.

[0043] Fifthly, an embodiment provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method described in any one of the first and second aspects.

[0044] In a sixth aspect, an embodiment provides a computing device including a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of the first and second aspects.

[0045] In the methods and apparatus provided in the embodiments of this specification, when a predetermined event is detected, several log element data corresponding to several fields can be obtained from the predetermined event. When preset output conditions are met, several log element data are output in a structured manner based on the correspondence between fields and log element data. In generating logs, the embodiments of this specification use a structured method to output log element data, outputting structured log element data in the initial state of log generation. This makes subsequent operations such as recording, storing, statistically analyzing, and processing logs simpler and improves the efficiency of log processing. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0047] Figure 1 A flowchart illustrating a log generation method provided for an embodiment;

[0048] Figure 2 A flowchart illustrating a log processing method provided for an embodiment;

[0049] Figure 3 This is a schematic diagram illustrating log processing using the SDK provided in this embodiment;

[0050] Figure 4 A schematic block diagram of a log generation device provided in an embodiment;

[0051] Figure 5 This is a schematic block diagram of a log processing device provided for an embodiment. Detailed Implementation

[0052] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0053] When computer systems, services, and other applications are running, their logging components also operate. The logging component can be understood as a processing module that implements log collection functionality. It collects log data from the computer system and services during operation; this collected log data can also be called event records. The logging component can record different events that occur during the operation of the computer system or services. Generally, logs can be simply divided into operating system logs and service program logs. Operating system logs record events within the computer's operating system. Service program logs record events from various running services on the computer. A service program can be understood as an application program that provides various service functions.

[0054] The logging component can collect logs for different events. In different application scenarios, events can include different content. For example, in a blockchain transaction scenario, the event could be a transaction event; in a domain name resolution scenario, the event could be the server logs of the domain name resolution server, etc.

[0055] After collecting logs, you can perform processing operations such as storage and data analysis on the collected log records. For example, you can store the log records in a database. Detailed data analysis of a large number of log records can help locate problems and understand the operational status.

[0056] In practical applications, it is often necessary to perform data analysis on large amounts of continuously generated log records, making it crucial to improve log processing efficiency. In the embodiments of this specification, log processing may include processes such as log collection, log storage, log statistics and analysis, and log transmission, but is not limited to log analysis.

[0057] During the operation of a service program, a large amount of log data is generated. Due to the wide variety of service programs and the diverse log content that needs to be collected, the logging component can output the log data in text format and record it; or it can parse and process the text log data, extract key information, and perform statistical analysis on the key information.

[0058] To improve the efficiency of log processing, this specification provides a log generation method. In this method, predetermined events during the operation of a service program are detected. Based on several preset fields for data to be acquired, several log element data corresponding to each of the predetermined events are obtained. When preset output conditions are met, the log element data is output in a structured manner based on the correspondence between the fields and the log element data. When the log component outputs logs, it outputs the log element data in a structured manner, enabling more efficient recording, storage, statistics, and analysis of the log data. Detailed explanations are provided below through specific embodiments.

[0059] Figure 1 This is a flowchart illustrating a log generation method provided in an embodiment. The method can be executed by a first device A, which can be any device, equipment, platform, or device cluster with computing and processing capabilities. Specifically, it can be executed by a log component within the first device A. This embodiment of the method may include the following steps S110 to S130.

[0060] Step S110: During the operation of the service program, detect the predetermined event event1.

[0061] Among them, scheduled events are pre-defined events that require logging data. Scheduled events can be, for example, transaction events, fault events, change events, etc.

[0062] For example, the logging component in the execution body can run in the process or thread of the service program, continuously detecting whether a predetermined event has occurred. When a predetermined event is detected, step S120 is executed. The step of detecting the predetermined event can be performed using conventional methods.

[0063] Step S120: Based on a number of preset fields for data to be acquired, retrieve a number of log element data corresponding to the fields from the predefined event event1.

[0064] The fields in the data to be retrieved can be predefined, and the number of fields can be one, two, or more. Fields are used to identify the meaning and category of the data to be retrieved; fields can also be called attributes. For example, in a blockchain-based transaction scenario, the fields in the data to be retrieved could be the following: timestamp, log level, identifier (UUID), thread name, etc.

[0065] The scheduled event 'event1' contains various types of data. Based on several fields of data to be retrieved, several log element data corresponding to those fields can be obtained from the various types of data contained in 'event1'. One field can correspond to one or more log element data. The fields and their corresponding log element data can be understood as the key and value in a key-value pair.

[0066] Log element data consists of key data from a pre-defined event; it is scattered data within a complete log entry. A complete log entry contains one or more log element data pieces, and several log element data pieces can constitute a single log entry. Alternatively, several log element data pieces obtained from a pre-defined event can be called a log data group. A log data group contains several log element data pieces, and a log data group is obtained from a single pre-defined event. Log element data does not contain fields.

[0067] For example, the following log element data is obtained from a pre-defined event: 20210419010650, 1, 145, xxx. These log element data correspond to the following fields: timestamp, level, uuid, thread.

[0068] Multiple fields of data to be retrieved can be arranged in a list according to a first preset order, forming a log draft (schema). This schema can be preset by the developers. A log draft can be predefined for a given event. Different log drafts can be defined for different given events. Alternatively, a log draft can be defined for a single service application.

[0069] When retrieving log element data, the log element data corresponding to a field can be retrieved from the predefined event `event1` according to the first preset order of the fields in this schema. Alternatively, log element data corresponding to a field can be retrieved randomly based on this schema.

[0070] After obtaining several log element data, they can be cached. During caching, several log element data can be cached according to a first preset order.

[0071] Step S130: When the preset output conditions are met, the aforementioned log element data is output in a structured manner based on the correspondence between several fields and several log element data.

[0072] The preset output condition may include obtaining a preset number (i.e., n) of log data groups, where n is an integer. The preset number n can be a single value, and the preset output condition is obtaining a log data group. In this implementation, the first device A outputs one log data group each time it detects a predetermined event.

[0073] The preset quantity n can be other values, such as 2, 10, 100, etc. That is, steps S110 to S120 can be executed multiple times, for example, n times, to obtain n log data groups. After extracting log element data from n predetermined events based on steps S110 to S120, the first device A outputs the accumulated n log data groups in batches. Each of the n log data groups was obtained from one of the n predetermined events.

[0074] In step S130, it can be detected whether the current state meets the preset output conditions. If it does, the step of outputting the log element data in a structured manner based on the correspondence between several fields and several log element data is executed. If it does not meet the conditions, no log element data is output.

[0075] Step S130, which involves outputting the aforementioned log element data in a structured manner, can include various implementation methods. The implementation method differs depending on the direction of output; the structuring method also differs depending on the content of the output.

[0076] The system outputs a number of log elements in a structured manner. The output is structured log data. Structured log data has a defined data structure, each log entry has the same number of log elements, and the order of these log elements is fixed and predetermined.

[0077] In summary, in this embodiment, the log records are output as structured data during the log collection process (or generation process), which makes the storage, analysis, and transmission of log records more efficient and convenient.

[0078] In one embodiment of this specification, step S130 may be performed according to Figure 2 The flowchart shown is executed, and the log element data is sent to the second device B. Figure 2 This is a flowchart illustrating a log processing method provided in an embodiment. The second device B is different from the first device A. The first device A is used to collect and output log data, while the second device B is used to acquire log data. The second device B can also be used to store, statistically analyze, and process the log data. In this embodiment, when the first device A executes step S130, it can execute steps S131 and S132 accordingly.

[0079] Step S131: When the current state meets the preset output conditions, the first device A forms structured log data from the log elements arranged in the first preset order based on the correspondence between several fields and several log element data.

[0080] In step S132, the first device A sends structured log data to the second device B. The first device A can send structured log data to the second device B via a network. The second device B receives a number of log element data sent by the first device A in a structured manner, arranged in a first preset order.

[0081] The resulting structured log data does not contain fields. When several log elements are arranged in a first preset order, these log elements correspond to several fields arranged in the first preset order.

[0082] In one implementation, each log element in the resulting structured log data can occupy a fixed number of bytes. Different log elements can occupy the same or different numbers of bytes.

[0083] During the process of generating structured log data, the first device A can also convert log element data into binary data to obtain structured binary log data, which is then sent to the second device B. The second device B receives the structured binary log data sent by the first device.

[0084] When using the batch sending log data group output method, the first device A can accumulate multiple (e.g., n) log data groups. That is, after extracting log element data from the predetermined event n times, the n log data groups can be formed into structured log data, wherein the log element data contained in any log data group are arranged in a first preset order.

[0085] The second device B can receive n log data groups generated by the first device A in a structured manner.

[0086] In step S140, the second device B associates several log element data with several fields arranged in a first preset order to obtain structured log records.

[0087] The second device B holds several fields arranged in a first preset order. These fields may be pre-acquired. The second device B can use these fields as header fields and store them in a log table in the first preset order. When several log element data are received, the second device B can fill them into the log table in sequence to form a log record.

[0088] Log recording can be understood as the information obtained by recording log data in a certain format. When log data is recorded in a log table, one piece of log data forms one log record.

[0089] When the log element data received by the second device B is sent in batches, the second device B can receive n log data groups sent by the first device A in a structured manner. The second device B can parse the n log data groups from the received data. For any log data group G1, it associates several log element data points within G1 with several fields arranged in a first preset order to obtain the structured log record corresponding to that log data group G1. For n log data groups, the second device B can obtain n log records.

[0090] The second device B can sequentially fill the log element data from n log data groups into a log table, resulting in n log records. Log records obtained in this way are structured log records.

[0091] After obtaining multiple structured log records, the second device B can store these records in a database, or perform statistical analysis and other processing on them. One option is to store the structured log records in a relational database. Relational databases are well-suited for storing structured data, making them a more convenient option for storing large amounts of structured log records.

[0092] In this embodiment, log data can be transferred in batches between the first and second devices without transmitting fields; log element data can be sent directly. In scenarios involving large-volume log data transfers, this reduces the amount of data transmitted, streamlines the processing flow, and improves processing efficiency.

[0093] In another embodiment of this specification, when outputting several log element data in a structured manner in step S130, fields and log element data can also be output.

[0094] First device A can associate several fields with several log element data to obtain several key-value pairs, and output these key-value pairs in a structured manner. When outputting the key-value pairs in a structured manner, first device A can form structured key-value pairs from several key-value pairs arranged in a third preset order, and output the structured key-value pairs. The third preset order can be the same as or different from the first preset order.

[0095] In the output direction, structured key-value pairs can be output to a log table in a specified file, resulting in a single log table record. Each row in the log table represents one log record. Each log record contains multiple key-value pairs. In the column direction of the log table, each column contains key-value pairs corresponding to the same field. Therefore, the resulting log records are structured.

[0096] When outputting log element data to a specified file, the first device A can form structured log data from several log element data arranged in a second preset order, output the structured log data to the specified file, and associate the several log element data contained in the structured log data with several fields in the specified file.

[0097] In this configuration, several fields in the specified file are arranged in a second preset order. The fields in the specified file can be pre-defined. This way, only log element data needs to be entered into the specified file; no fields need to be entered. The specified file can be any file that supports storing structured tabular data.

[0098] The log generation process described in steps S110 to S130 above can be configured and coded using a Software Development Kit (SDK), for example, a Java version of the SDK. This allows the resulting SDK to be used by a wider range of service programs, adapting to different service programs without requiring the development of separate log components for each program, thus reducing development costs. Furthermore, the log generation method provided in this embodiment clearly defines the semantics of the log content (implementing the association between log element data and fields), thereby reducing the maintenance costs for the semantics of the log content across different service programs.

[0099] Figure 3 This is a schematic diagram illustrating log processing using the SDK provided in this embodiment. The SDK toolkit is developed using the log generation method provided in this embodiment and is used to collect logs. The SDK runs in the service process and, upon collecting log element data, can send structured log records to the data gateway. The data network can send the structured log records to a database (DB) for storage, enabling log persistence. Simultaneously, the data gateway can also send the structured log records to a pre-computation system, where a large number of structured log records are analyzed and processed, and the results are stored in the DB. Users can read the raw log records or the analysis results from the DB through the console. The service process is the process running the service program.

[0100] In this specification, the terms "first device," "first arrangement," and "second" are used merely for ease of distinction and description and do not have any limiting meaning.

[0101] The foregoing description describes specific embodiments of this specification; other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than those shown in the embodiments, and the desired result may still be achieved. Furthermore, the processes depicted in the drawings do not necessarily need to follow the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0102] Figure 4 This is a schematic block diagram of a log generation device provided for an embodiment. The device is deployed in a first device, which can be implemented using any device, equipment, platform, device cluster, etc., with computing and processing capabilities. This device embodiment is similar to... Figure 1 The method embodiment shown corresponds to this. The device 400 includes:

[0103] The detection module 410 is configured to detect predetermined events during the operation of the service program;

[0104] The acquisition module 420 is configured to acquire, based on a preset number of fields of data to be acquired, a number of log element data corresponding to the preset fields from the predetermined event.

[0105] The output module 430 is configured to output the log element data in a structured manner based on the correspondence between the several fields and the several log element data when the preset output conditions are met.

[0106] In one embodiment, the output module 430 is specifically configured as follows:

[0107] The log elements arranged in a first preset order are used to form structured log data, which is then sent to a second device B. The second device B associates the log elements in the received structured log data with the fields arranged in the first preset order to obtain structured log records.

[0108] In one implementation, the preset output condition includes: when a preset number of log data groups are obtained; wherein, a log data group contains several log element data, and the preset number of log data groups are obtained from a preset number of predetermined events;

[0109] The output module is specifically configured as follows:

[0110] The preset number of log data groups are used to form structured log data; wherein, the log element data contained in any log data group are arranged in the first preset order.

[0111] In one embodiment, the output module 430 is specifically configured as follows:

[0112] The fields are associated with the log element data respectively to obtain key-value pairs, and the key-value pairs are output in a structured manner.

[0113] In one implementation, when the output module 430 outputs the plurality of key-value pairs in a structured manner, it includes:

[0114] The structured key-value pairs are output to the log table of the specified file to obtain a log table record.

[0115] In one embodiment, the output module 430 is specifically configured as follows:

[0116] The log elements arranged in a second preset order are used to form structured log data. The structured log data is then output to a specified file, and the log elements contained in the structured log data are associated with several fields in the specified file. The fields in the specified file are arranged in the second preset order.

[0117] Figure 5 This is a schematic block diagram of a log processing device provided for an embodiment. The device is deployed in a second device, which can be implemented using any device, equipment, platform, device cluster, etc., with computing and processing capabilities. This device embodiment is similar to... Figure 2 The second device in the illustrated method embodiment corresponds to a portion of the device 500. The device 500 includes:

[0118] The receiving module 510 is configured to receive a plurality of log element data arranged in a first preset order and sent by the first device in a structured manner; wherein the plurality of log element data is obtained from a predetermined event and corresponds to a plurality of fields of data to be obtained.

[0119] The association module 520 is configured to associate the plurality of log element data with a plurality of fields arranged in the first preset order to obtain structured log records.

[0120] In one embodiment, the receiving module 510 is specifically configured as follows:

[0121] Receive a preset number of log data groups sent by the first device; wherein, any log data group contains several log data items arranged in the first preset order, and the preset number of log data groups are respectively obtained from a preset number of predetermined events;

[0122] The specific configuration of the associated module 520 is as follows:

[0123] For any log data group, several log element data in the log data group are associated with several fields arranged in the first preset order to obtain the log record corresponding to the log data group.

[0124] In one embodiment, the device further includes: a storage module and / or a statistics module (not shown in the figure);

[0125] The storage module is configured to store multiple structured log records into a database after obtaining them.

[0126] The statistics module is configured to perform statistics on multiple structured log records after they are obtained.

[0127] The above-described apparatus embodiments correspond to the method embodiments, and detailed descriptions can be found in the description of the method embodiments section, which will not be repeated here. The apparatus embodiments are derived based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments; detailed descriptions can be found in the corresponding method embodiments.

[0128] This specification also provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform... Figures 1 to 3 Any one of the methods described.

[0129] This specification also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement... Figures 1 to 3 Any one of the methods described.

[0130] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for storage media and computing devices are basically similar to the method embodiments, so they are described more simply; relevant parts can be referred to the descriptions of the method embodiments.

[0131] Those skilled in the art will recognize that the functions described in the embodiments of the present invention in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0132] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, or improvements made based on the technical solutions of the present invention should be included within the scope of protection of the present invention.

Claims

1. A log generation method, executed by a log component in a first device, wherein the log component is a processing module that implements log collection functionality; The logging component, when collecting log data, includes: During the operation of the service program, pre-defined events are detected; When a predetermined event is detected, based on a number of preset fields for data to be acquired, a number of log element data corresponding to the fields are retrieved from the predetermined event. After obtaining the log element data, the log element data is cached according to a first preset order; when the log element data is arranged in the first preset order, the log element data corresponds to the fields arranged in the first preset order. When the preset output conditions are met, based on the correspondence between the several fields and the several log element data respectively, the several log element data arranged in the first preset order are output in a structured manner.

2. The method according to claim 1, wherein the step of outputting the plurality of log element data arranged in the first preset order in a structured manner includes: The log elements arranged in the first preset order are formed into structured log data, and the structured log data is sent to the second device. The second device associates the received structured log data, which contains several log elements, with several fields arranged in the first preset order to obtain structured log records.

3. The method according to claim 2, wherein the preset output conditions include: When a preset number of log data groups are obtained; wherein, a log data group contains several log element data, and the preset number of log data groups are obtained from a preset number of predetermined events; The step of forming structured log data from the plurality of log element data arranged in the first preset order includes: The preset number of log data groups are used to form structured log data; wherein, the log element data contained in any log data group are arranged in the first preset order.

4. The method according to claim 1, wherein the step of outputting the plurality of log element data arranged in the first preset order in a structured manner includes: The fields are associated with the log element data respectively to obtain key-value pairs, and the key-value pairs are output in a structured manner.

5. The method according to claim 4, wherein the step of outputting the plurality of key-value pairs in a structured manner comprises: The structured key-value pairs are output to the log table of the specified file to obtain a log table record.

6. The method according to claim 1, wherein the step of outputting the plurality of log element data arranged in the first preset order in a structured manner includes: The log elements arranged in the first preset order are formed into structured log data, which is then output to a specified file. The log elements contained in the structured log data are associated with several fields in the specified file, wherein the fields in the specified file are arranged in the first preset order.

7. A log processing method, executed by a second device, the second device holding a plurality of fields arranged in a first preset order; the method comprising: The system receives a plurality of log element data arranged in a first preset order, sent in a structured manner by a log component in a first device; wherein, the log component is a processing module that implements the log collection function; the plurality of log element data is obtained by the log component from the predetermined event when the predetermined event is detected and cached in the first preset order, and when the plurality of log element data is arranged in the first preset order, the plurality of log element data corresponds to the plurality of fields arranged in the first preset order; The log element data is associated with several fields arranged in the first preset order to obtain structured log records.

8. The method according to claim 7, wherein the step of receiving log element data arranged in a first preset order and sent in a structured manner by the log component in the first device includes: The system receives a preset number of log data groups sent by the log component in a structured manner; wherein, any log data group contains several log element data arranged in the first preset order, and the preset number of log data groups are obtained from a preset number of predetermined events; The step of associating the plurality of log element data with the plurality of fields arranged in the first preset order includes: For any log data group, several log element data in the log data group are associated with several fields arranged in the first preset order to obtain the structured log record corresponding to the log data group.

9. The method according to claim 8, wherein, After obtaining multiple structured log records, it also includes: Store the multiple structured log records in a database; and / or perform statistics on the multiple structured log records.

10. A log generation device, deployed in a log component of a first device, wherein the log component is a processing module that implements log collection functionality; The logging component, when collecting log data, includes: The detection module is configured to detect predetermined events during the operation of the service program; The acquisition module is configured to, when a predetermined event is detected, acquire several log element data corresponding to several fields of data to be acquired from the predetermined event based on several preset fields of data to be acquired; The caching module is configured to cache the log element data in a first preset order after obtaining the log element data; when the log element data is arranged in the first preset order, the log element data corresponds to the fields arranged in the first preset order. The output module is configured to output the log element data in a structured manner based on the correspondence between the several fields and the several log element data when the preset output conditions are met.

11. The apparatus according to claim 10, wherein the output module is specifically configured as follows: The log elements arranged in a first preset order are used to form structured log data, which is then sent to a second device. The second device associates the log elements in the received structured log data with the fields arranged in the first preset order to obtain structured log records.

12. The apparatus according to claim 11, wherein the preset output conditions include: When a preset number of log data groups are obtained; wherein, a log data group contains several log element data, and the preset number of log data groups are obtained from a preset number of predetermined events; The output module is specifically configured as follows: The preset number of log data groups are used to form structured log data; wherein, the log element data contained in any log data group are arranged in the first preset order.

13. The apparatus according to claim 10, wherein the output module is specifically configured as follows: The fields are associated with the log element data respectively to obtain key-value pairs, and the key-value pairs are output in a structured manner.

14. A log processing apparatus, deployed in a second device, the second device holding a plurality of fields arranged in a first preset order; the apparatus comprising: The receiving module is configured to receive a plurality of log element data arranged in a first preset order, sent in a structured manner by the log component in the first device; wherein the log component is a processing module that implements the log collection function; the plurality of log element data is obtained by the log component from the predetermined event when the predetermined event is detected and cached in the first preset order, wherein when the plurality of log element data is arranged in the first preset order, the plurality of log element data corresponds to the plurality of fields arranged in the first preset order; The association module is configured to associate the plurality of log element data with a plurality of fields arranged in the first preset order to obtain structured log records.

15. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1-9.

16. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method of any one of claims 1-9.

Images