Log storage method and device, equipment and storage medium

Abstract

Embodiments of the invention provide a log storage method and apparatus, a device and a storage medium. The method comprises the steps of obtaining a log; judging a log format of the log; extracting target log information from the log according to the log format; the target log information comprises a field, and the field comprises field content and a field name; determining a target log type corresponding to the target log information; the target log type has a corresponding log compression scheme; compressing the field content and the field name by adopting a log compression scheme corresponding to the target log type to obtain a compressed log; and storing the compressed log. According to the embodiment of the invention, the space required by log storage is effectively reduced, and particularly, the space compression ratio can be further improved in the face of a specific target log type, so that efficient access of massive security software logs is realized.

Description

Technical Field

[0001] The present application relates to the field of cloud security technology, and in particular to a log storage method and device, an electronic device, and a storage medium. Background Art

[0002] Currently, when collecting logs, XDR (Extended Detection and Response) and SOAR (Security Orchestration, Automation and Response) systems usually need to connect with a large number of other security software. In particular, monitoring security software generates a large number of logs at a fast speed and occupies a large amount of storage space. These logs not only need to be stored, but also need to be frequently read and used. Simply compressing the log storage space is difficult to meet the dual needs of storage and reading.

[0003] Therefore, how to efficiently store and access massive security software logs has become an important issue that needs to be solved urgently. Summary of the invention

[0004] The embodiment of the present application provides a log storage method to solve the problem of how to efficiently access massive security software logs.

[0005] Correspondingly, the embodiment of the present application also provides a log storage device, an electronic device and a storage medium to ensure the implementation and application of the above method.

[0006] In order to solve the above problems, the embodiment of the present application discloses a log storage method, which includes:

[0007] Get logs;

[0008] Determining a log format of the log;

[0009] Extracting target log information from the log according to the log format; the target log information includes fields, and the fields include field content and field names;

[0010] Determine a target log type corresponding to the target log information; the target log type has a corresponding log compression scheme;

[0011] Compressing the field content and the field name using a log compression scheme corresponding to the target log type to obtain a compressed log;

[0012] The compressed log is stored.

[0013] Optionally, the log includes a key-value pair, and extracting target log information from the log according to the log format includes:

[0014] If the log format of the log is a non-standard format, extracting the key-value pair of the log to obtain a key-value pair extraction result and a first confidence level corresponding to the key-value pair extraction result;

[0015] Displaying the key-value pair extraction result and the first confidence level;

[0016] Receiving a selection operation on the key-value pair extraction result;

[0017] According to the selection operation, extracting the target log information from the log;

[0018] If the log format of the log is a standard format, an intelligent disassembly model is used to extract the target log information from the log according to the log format.

[0019] Optionally, the determining the target log type corresponding to the target log information includes:

[0020] Calculating a second confidence level that the target log information belongs to the log information type according to the field content and the field name;

[0021] The target log type corresponding to the target log information is determined according to the second confidence level.

[0022] Optionally, the field has a corresponding field type, and the log compression scheme corresponding to the target log type is used to compress the field content and the field name to obtain a compressed log, including:

[0023] If the target log type is a type with a consistent number of fields, the fields of the same type are arranged into field columns;

[0024] The field names and the field contents in the field columns are stored separately.

[0025] Optionally, the field has a corresponding field format, and the adopting of a log compression scheme corresponding to the target log type to compress the field content and the field name to obtain a compressed log includes:

[0026] If the target log type is a field format consistent type, and the field format is a Boolean type, then the field content in the target log information is obtained;

[0027] Positioning the field content to a target storage position of a preset byte;

[0028] Using the target storage bit of the preset byte to store the field content;

[0029] Stores the field name.

[0030] Optionally, the adopting a log compression scheme corresponding to the target log type to compress the field content and the field name to obtain a compressed log includes:

[0031] If the target log type is a field content-related type, obtaining a preset reference value;

[0032] Calculate the difference between the field content and the preset reference value to obtain a target storage value;

[0033] The field name and the target storage value are stored.

[0034] Optionally, the adopting a log compression scheme corresponding to the target log type to compress the field content and the field name to obtain a compressed log includes:

[0035] If the target log type is a field content boundary determination type, decomposing the field content to obtain a decomposed value within a preset range;

[0036] Using preset bytes to store the decomposed values ​​within the preset range;

[0037] Stores the field name.

[0038] The embodiment of the present application also discloses a log storage device, the device comprising:

[0039] Log acquisition module, used to obtain logs;

[0040] A log format determination module, used to determine the log format of the log;

[0041] A log disassembly module, used to extract target log information from the log according to the log format; the target log information includes fields, and the fields include field content and field names;

[0042] A log type determination module, used to determine a target log type corresponding to the target log information; the target log type has a corresponding log compression scheme;

[0043] A log compression module, used to compress the field content and the field name using a log compression scheme corresponding to the target log type to obtain a compressed log;

[0044] The log storage module is used to store the compressed logs.

[0045] The embodiments of the present application also disclose an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes the log storage method as described in one or more of the embodiments of the present application.

[0046] The embodiments of the present application also disclose one or more machine-readable media on which executable codes are stored. When the executable codes are executed, the processor executes the log storage method as described in one or more of the embodiments of the present application.

[0047] Compared with the prior art, the embodiments of the present application have the following advantages:

[0048] In an embodiment of the present application, a log is obtained; the log format of the log is determined; target log information is extracted from the log according to the log format; the target log information includes fields, and the fields include field content and field names; the target log type corresponding to the target log information is determined; the target log type has a corresponding log compression scheme; the field content and field name are compressed using the log compression scheme corresponding to the target log type to obtain a compressed log; and the compressed log is stored. The embodiment of the present application extracts target log information from logs of different formats and performs compression processing according to different target log types, thereby effectively reducing the space required for log storage, especially when facing a specific target log type, the space compression ratio can be further improved; after the storage space is compressed, subsequent log reading operations can still maintain high efficiency, thereby achieving efficient access to massive security software logs. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a flow chart of steps of an embodiment of a log storage method of the present application;

[0050] Figure 2 This is a schematic diagram of the system architecture of a log storage method embodiment of the present application;

[0051] Figure 3 This is a schematic diagram of a disassembly module of a log storage method embodiment of the present application;

[0052] Figure 4 This is a schematic diagram of a compression module of a log storage method embodiment of the present application;

[0053] Figure 5 This is a schematic diagram of a traceability display module of a log storage method embodiment of the present application;

[0054] Figure 6 It is a structural block diagram of an embodiment of a log storage device of the present application;

[0055] Figure 7It is a schematic diagram of the structure of a device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0057] Reference Figure 1 , is a flow chart of a log storage method embodiment of the present application, comprising the following steps:

[0058] Step 101: Get logs.

[0059] In the embodiment of the present application, the obtained logs may be logs generated by various software. In one example, the logs may be logs generated by monitoring security software.

[0060] Step 102: Determine the log format of the log.

[0061] Determine the log format of the obtained log. In one example, the log format can be divided into a standard format and a non-standard format. Among them, the standard format can be JSON (JavaScript Object Notation, a lightweight data exchange format), YAML (YAML Ain't Markup Language, a human-readable data serialization standard), XML (Extensible Markup Language, Extensible Markup Language), MessagePack (message pack), BSON (BinaryJSON, a binary data exchange format) and other formats.

[0062] Step 103: extract target log information from the log according to the log format; the target log information includes fields, and the fields include field content and field names.

[0063] In step 103, target log information is extracted from the log according to the log format, which is essentially converting the log into a unified format.

[0064] In one embodiment, a log in a standard format can directly use an intelligent disassembly model to extract target log information from the log; and a log in a non-standard format can first extract the key-value pairs of the log to obtain the key-value pair extraction results and the first confidence level corresponding to the key-value pair extraction results, and the user determines the key-value pair extraction results that are ultimately adopted to obtain the target log information.

[0065] Step 104: Determine a target log type corresponding to the target log information; the target log type has a corresponding log compression scheme.

[0066] After the target log information is obtained, a specific log compression scheme may be used to compress the target log information for the specific target log information.

[0067] In one embodiment, the target log type may include a field quantity consistent type, a field format consistent type, a field content associated type, and a field content boundary determined type. Each target log type may have a corresponding log compression scheme.

[0068] Step 105: compress the field content and the field name using a log compression scheme corresponding to the target log type to obtain a compressed log.

[0069] In step 105, the field content and field name are compressed using a log compression scheme corresponding to the target log type to obtain a compressed log.

[0070] In one embodiment, if the field names of each target log information are consistent, the field names of the target log information can be stored uniformly, and the field contents of the target log information can be compressed and stored separately. That is, for multiple target log information with the same field names, since the field names in each target log information are the same as the field names in the other target log information, only the field contents are different, so the same field names only need to be stored once, without repeated storage, further saving storage space.

[0071] Step 106: Store the compressed log.

[0072] After the log is compressed, store the compressed log

[0073] In one embodiment, after the storage is completed, the stored log data can be read and displayed to achieve the effect of traceability display.

[0074] In an embodiment of the present application, a log is obtained; the log format of the log is determined; target log information is extracted from the log according to the log format; the target log information includes fields, and the fields include field content and field names; the target log type corresponding to the target log information is determined; the target log type has a corresponding log compression scheme; the field content and field name are compressed using the log compression scheme corresponding to the target log type to obtain a compressed log; and the compressed log is stored. The embodiment of the present application extracts target log information from logs of different formats and performs compression processing according to different target log types, thereby effectively reducing the space required for log storage, especially when facing a specific target log type, the space compression ratio can be further improved; after the storage space is compressed, subsequent log reading operations can still maintain high efficiency, thereby achieving efficient access to massive security software logs.

[0075] Optionally, the log includes a key-value pair, and step 103 includes:

[0076] If the log format of the log is a non-standard format, extracting the key-value pair of the log to obtain a key-value pair extraction result and a first confidence level corresponding to the key-value pair extraction result;

[0077] Displaying the key-value pair extraction result and the first confidence level;

[0078] Receiving a selection operation on the key-value pair extraction result;

[0079] According to the selection operation, extracting the target log information from the log;

[0080] If the log format of the log is a standard format, an intelligent disassembly model is used to extract the target log information from the log according to the log format.

[0081] In this embodiment, different methods are used to extract target log information for logs in different log formats.

[0082] When the log format is non-standard, the log data is usually expressed in KV format (key-value pair format), that is, each key-value pair is an identifier plus a key-value separator plus the value represented by the identifier. The key-value separator may not be uniform. There is a corresponding element separator between each key-value pair.

[0083] In this embodiment, for logs in non-standard formats, the delimiter features supported by the delimiter library can be used to extract the key-value pairs of the logs to obtain the key-value pair extraction results and the first confidence corresponding to the key-value pair extraction results. That is, the basic key column (column K) and its corresponding value column (column V) and the first confidence corresponding to the key-value pair extraction results can be extracted. The key-value pair extraction results and the first confidence are displayed to the user, and the user makes the final decision.

[0084] In one example, if there is log data such as "SIP:192.168.1.100;fleName:file.txt", multiple key-value pair extraction results can be parsed, such as: the first type, ":" is used as the key-value separator of the key-value data, ";" is the element separator, "SIP" is the key, and "192.168.1.100" is the value; the second type, ";" is used as the key-value separator of the key-value data, "SIP:192.168.1.100" is the key, "fileName:file.txt" is the value, and so on. All possible key-value pair extraction results are calculated through the non-standard confidence model to obtain the first confidence, and the key-value pair extraction results and the first confidence are fed back to the user for final judgment.

[0085] Receive a user's selection operation for the key-value pair extraction result, and extract target log information from the log according to the selection operation. After the user selects a determined key-value pair extraction result, the user can extract the target log information from the log according to the key-value pair extraction result.

[0086] In this embodiment, for logs in a standard format, an intelligent disassembly model may be used to automatically determine the log format of the log by extracting key features of the log, thereby extracting target log information from the log according to the log format.

[0087] In one embodiment, the target log information extracted from the log can be returned to the user, who will confirm the format. For logs in non-standard formats, the manually confirmed format will be recorded to provide model training data for the non-standard confidence model and improve the calculation of the first confidence; for logs in standard formats, the user can perform correction operations on the extracted target log information, and the features of the marked correction items will be used as model data for training the intelligent disassembly model to optimize the performance of the subsequent intelligent disassembly model.

[0088] The embodiment of the present application can extract target log information in different ways for logs in different log formats, thereby intelligently extracting the target log information by combining automatic extraction in standard format and manual assisted determination in non-standard format for use in subsequent log compression steps.

[0089] Optionally, the step 104 includes:

[0090] Calculating a second confidence level that the target log information belongs to the log information type according to the field content and the field name;

[0091] The target log type corresponding to the target log information is determined according to the second confidence level.

[0092] In step 104, the target log type corresponding to the target log information may be determined for the target log information, thereby further determining a log compression scheme applicable to the target log information.

[0093] Specifically, as an example, assume that the following is a common database audit log.

[0094] {

[0095] "logType":"audit",

[0096] "startTime":"2023-06-28 17:17:53",

[0097] "sqlLen":"49",

[0098] "dvcAction":"SET",

[0099] "destMacAddress":"FA-16-3E-A6-D9-E2",

[0100] "srcMacAddress":"FA-16-3E-79-32-25",

[0101] "destAddress":"192.168.100.199",

[0102] "databaseType":"MYSQL",

[0103] "ruleName":"MySQL_Create a password-less user or connection",

[0104] "errorMessage":"ERROR 1133(4200):Can't find any matching row in theuser table",

[0105] "name":"sql operation log",

[0106] "message":"Source address / port: 10.25.200.215 / 39491, Destination address / port: 192.168.100.99 / 3306, Operation database object:, Operation type: SET, Execution result: 0, Whether to alarm: 1",

[0107] "deviceAddress":"192.168.100.114",

[0108] "alarmFlag":"1",

[0109] "responseCode":"0",

[0110] "deviceSendProductName":"DBAudit",

[0111] "deviceSendProductVersion":"v4.0.67",

[0112] "collectorReceiptTime":"2023-06-28 17:17:53",

[0113] "deviceReceiptTime":"2023-06-28 17:17:53",

[0114] "endTime":"2023-06-28 17:17:53",

[0115] "sendHostAddress":"192.168.100.114",

[0116] "eventId":"1673984084814073856"

[0117] }

[0118] First, extract the target log information from the log. Since the log is in a standard format, through format comparison, it can be confirmed that the log is in json format, that is, the target log information can be extracted from the log by json parsing.

[0119] After the target log information is extracted, the target log type corresponding to the target log information is predicted according to the format and field name of the field content, and the second confidence that the target log information belongs to a certain log information type is calculated.

[0120] For example, a field whose content format conforms to the "YYYY-MM-DD" type can be predicted as a time type, a field whose content is purely numeric can be predicted as a numeric type, a field whose content format conforms to the "XXX.XXX.XXX.XXX" type can be predicted as an IP type, and so on. The accuracy of the prediction is then weighted by the keywords of the field name, thereby calculating the second confidence that the target log information belongs to a certain log information type. For example, when the field name "startTime" contains "Time" and the value type conforms to the time format such as "YYYY-MM-DD", the confidence that the field is predicted to be a time type is increased.

[0121] The prediction results for the field type are sorted to obtain a second confidence level that the target log information belongs to a certain log information type, and the target log type corresponding to the target log information is determined according to the second confidence level.

[0122] The embodiment of the present application can determine the target log type corresponding to the target log information for the target log information, thereby further determining the log compression scheme applicable to the target log information for log compression storage.

[0123] Optionally, the field has a corresponding field type, and step 105 includes:

[0124] If the target log type is a type with a consistent number of fields, the fields of the same type are arranged into field columns;

[0125] The field names and the field contents in the field columns are stored separately.

[0126] In this embodiment, if the target log type is a type with a consistent number of fields, that is, the number of fields in the target log information is relatively consistent, the target log information can be compressed and stored in the following manner. Fields of the same type are arranged into field columns, thereby obtaining multiple field columns. For the same field names in the field columns, they can be stored only once; for the field contents, they are compressed and stored separately.

[0127] As an example, the field "logType":"audit" has a total of 17 characters. When there are two fields named "logType", they will form a field column. For this field column, according to the compression rate formula: compression rate = (1-compressed data size / original data size) × 100%, the compression rate can be calculated as follows:

[0128] Compression ratio = (1-(7+5×2)(characters) / 17×2(characters))×100% = 50%

[0129] Among them, the size of the uncompressed field is 17 characters, and the size of the two uncompressed fields is 17×2 characters; after compression, the field name stored separately is 7 characters, that is, the field name only needs to be stored once, the compressed and stored field content is 5 characters, and the two compressed and stored field contents are 5×2 characters. When this method is used for compression, the more fields of the same type in the target log information, the higher the compression rate.

[0130] The embodiment of the present application can use a corresponding log compression scheme to compress field content and field names for target log information with a consistent number of fields, thereby effectively reducing the space required for log storage.

[0131] Optionally, the field has a corresponding field format, and the step 105 may further include:

[0132] If the target log type is a field format consistent type, and the field format is a Boolean type, then the field content in the target log information is obtained;

[0133] Positioning the field content to a target storage position of a preset byte;

[0134] Using the target storage bit of the preset byte to store the field content;

[0135] Stores the field name.

[0136] In this embodiment, if the target log type is of a field format consistent type, that is, the field format of the target log information is relatively consistent, and the field format is a Boolean type, the target log information may be compressed and stored in the following manner.

[0137] For the field content in the target log information, obtain the field content in the target log information, and locate the field content to the target storage bit of the preset byte. For example, a byte contains eight bits, each bit can store a Boolean field content, and one byte can store eight Boolean field contents. When storing, you can use bitwise to anchor each field content in the target storage bit of the preset byte, so that the field content is stored using the target storage bit of the preset byte. After completing the storage of the field content, store the field name separately.

[0138] In one example, for a Boolean field such as "alarmFlag": "1", when stored using a "UTF-8" string, 3 bytes are required, and when stored using a Boolean type, 1 byte is required. However, in this embodiment, by arranging the contents of the Boolean field together, when there are 8 Boolean field contents, the specific value can be anchored by bit, that is, 1 8-bit byte can store 8 Boolean values. Therefore, for the 8 Boolean field contents, according to the compression rate formula: compression rate = (1-compressed data size / original data size) × 100%, the compression rate can be calculated as follows:

[0139] Compression ratio = (1-1×8(bits) / 8×3×8(bits))×100% = 95.83%

[0140] Among them, before compression, the content of each Boolean type field occupies 3 bytes, that is, the content of each Boolean type field occupies 3×8 bits, and the content of 8 Boolean type fields occupies 8×3×8 bits; after compression, the content of each Boolean type field occupies 1 bit, and the content of 8 Boolean type fields only occupies 1 8-bit byte.

[0141] The embodiment of the present application can use a corresponding log compression scheme to compress field content and field names for target log information with consistent field formats, thereby effectively reducing the space required for log storage.

[0142] Optionally, the step 105 may further include:

[0143] If the target log type is a field content-related type, obtaining a preset reference value;

[0144] Calculate the difference between the field content and the preset reference value to obtain a target storage value;

[0145] The field name and the target storage value are stored.

[0146] In this embodiment, if the target log type is a field content association type, that is, there is an association between the field contents of the target log information, the target log information can be compressed and stored in the following manner.

[0147] If the target log type is field content-related, the preset reference value is obtained, and then the difference between the field content and the preset reference value is calculated to obtain the target storage value. For example, for the field content of the time type, a timestamp can be set as the preset reference value, and the rest of the field content can be expressed as the difference based on the preset reference value. Only the difference needs to be stored to achieve compressed storage of these field contents.

[0148] In one example, for a time type field of "startTime": "2023-06-28 17:17:53", the field content of the field is converted into a timestamp identifier of "1687943873", and the difference between the field content of the remaining "startTime" fields and "1687943873" is recorded. For example, if the subsequent field is "startTime": "2023-06-29 17:17:53", then only the difference between the new timestamp and the reference timestamp needs to be recorded, that is, (1688030273-1687943873) = 86400.

[0149] The field name and the difference are stored to achieve compressed storage of logs.

[0150] In one embodiment, when displaying the log, the changes can be quickly displayed by compressing the correlation between the field contents of the target log information in the storage, which is also more convenient for the operator to find data mutations. For example, the baseline value set for the field "sqlLen":"49" is 0, and the change value is compressed and stored each time, that is, the field content of the field "sqlLen":"49" stores the difference 49. When this change value changes, for example, the change value suddenly changes to more than 1000, then this change value is worthy of the user's attention. Since the content of this type of field is compressed and stored based on a baseline value, the changed data can also be quickly found, achieving the effect of log traceability display.

[0151] The embodiment of the present application can use a corresponding log compression scheme to compress field content and field names for target log information associated with field content, thereby effectively reducing the space required for log storage, and can record the changing trend of field content, which helps to quickly filter out abnormally fluctuating data.

[0152] Optionally, the step 105 further includes:

[0153] If the target log type is a field content boundary determination type, decomposing the field content to obtain a decomposed value within a preset range;

[0154] Using preset bytes to store the decomposed values ​​within the preset range;

[0155] Stores the field name.

[0156] In this embodiment, if the target log type is a field content boundary-determined type, that is, when the value of the field content of the target log information has a definite boundary, the target log information can be compressed and stored in the following manner.

[0157] Decompose the field content, that is, decompose the field content into decomposed values ​​within a preset range. For example, if the field content is an IP address, the IP address can be divided into 4 parts of decomposed values, each of which is a decomposed value in the range of 0 to 255.

[0158] The preset bytes are used to store the decomposed values ​​within the preset range. For the above example, 0 to 255 is exactly what can be stored in 1 8-bit byte, so only 4 bytes, that is, 32 bits, are needed to complete the storage of the IP address.

[0159] After completing the storage compression of the field content, the field names are stored separately to complete the log compression.

[0160] In one example, the field content of "destAddress": "192.168.100.199" is an IP address. When the "UTF-8" string is used for storage before compression, the IP address requires 15 characters to be stored, 1 character occupies 3 bytes, 1 byte occupies 8 bits, and the uncompressed IP address requires 15×3×8 bits to be stored. After the IP address is divided into 4 parts and the decomposed values ​​are compressed, according to the compression rate formula: compression rate = (1-compressed data size / original data size) × 100%, the compression rate can be calculated as follows:

[0161] Compression ratio = (1-32 (bits) / 15 × 3 × 8 (bits)) × 100% = 91.11%

[0162] The embodiment of the present application can use a corresponding log compression scheme to compress field content and field names for target log information with determined field content boundaries, thereby effectively reducing the space required for log storage.

[0163] In order to enable those skilled in the art to better understand the technical solution of a log storage method embodiment of the present application, the following Figure 2 to Figure 5 A log storage method according to an embodiment of the present application is introduced.

[0164] Reference Figure 2 , is a system architecture diagram of a log storage method embodiment of the present application.

[0165] Figure 2A simple schematic diagram of the system architecture of a log storage method embodiment of the present application is shown in FIG. , including a disassembly module, a compression module, and a traceability display module.

[0166] Reference Figure 3 , is a schematic diagram of a disassembly module of a log storage method embodiment of the present application.

[0167] The sample log is input into the disassembly module. In the disassembly module, the intelligent disassembly model is used as the model support. The target log information is intelligently disassembled by combining the automatic extraction of the standard format and the manual assisted judgment of the non-standard format to extract the target log information. The user then marks and corrects the disassembled target log information to obtain the final target log information.

[0168] Reference Figure 4 , is a schematic diagram of a compression module of an embodiment of a log storage method of the present application.

[0169] The target log information obtained by disassembly is input into the compression module, and the field compression estimation model is used to estimate the field compression and evaluate the optimal compression scheme. After the user confirms the compression, the storage field compression information will be obtained. The storage field compression information is used to compress and store the target log information. At the same time, the storage field compression information can also be used to update the field compression estimation model.

[0170] Reference Figure 5 , is a schematic diagram of a traceability display module of an embodiment of a log storage method of the present application.

[0171] When it is necessary to perform source tracing display on the compressed and stored logs, the log type information of the logs is obtained, and the field information of the compressed logs is restored according to the log type information and the stored field compression information, so as to perform source tracing display according to the preset display format.

[0172] In an embodiment of the present application, a log is obtained; the log format of the log is determined; target log information is extracted from the log according to the log format; the target log information includes fields, and the fields include field content and field names; the target log type corresponding to the target log information is determined; the target log type has a corresponding log compression scheme; the field content and field name are compressed using the log compression scheme corresponding to the target log type to obtain a compressed log; and the compressed log is stored. The embodiment of the present application extracts target log information from logs of different formats and performs compression processing according to different target log types, thereby effectively reducing the space required for log storage, especially when facing a specific target log type, the space compression ratio can be further improved; after the storage space is compressed, subsequent log reading operations can still maintain high efficiency, thereby achieving efficient access to massive security software logs.

[0173] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0174] On the basis of the above embodiments, this embodiment further provides a log storage device, which is applied to electronic devices such as terminal devices and servers.

[0175] Reference Figure 6 , shows a structural block diagram of a log storage device embodiment of the present application, which may specifically include the following modules:

[0176] The log acquisition module 601 is used to acquire logs;

[0177] The log format determination module 602 is used to determine the log format of the log;

[0178] The log disassembly module 603 is used to extract target log information from the log according to the log format; the target log information includes fields, and the fields include field content and field name;

[0179] The log type determination module 604 is used to determine the target log type corresponding to the target log information; the target log type has a corresponding log compression scheme;

[0180] A log compression module 605, configured to compress the field content and the field name using a log compression scheme corresponding to the target log type to obtain a compressed log;

[0181] The log storage module 606 is used to store the compressed log.

[0182] Optionally, the log includes a key-value pair, and the log disassembly module 603 includes:

[0183] A non-standard format extraction submodule, configured to extract the key-value pair of the log to obtain a key-value pair extraction result and a first confidence level corresponding to the key-value pair extraction result if the log format of the log is a non-standard format; display the key-value pair extraction result and the first confidence level; receive a selection operation for the key-value pair extraction result; and extract the target log information from the log according to the selection operation;

[0184] The standard format extraction submodule is used to extract the target log information from the log according to the log format by using an intelligent disassembly model if the log format of the log is a standard format.

[0185] Optionally, the log type determination module 604 includes:

[0186] A second confidence calculation submodule, used for calculating a second confidence that the target log information belongs to the log information type according to the field content and the field name;

[0187] A log type determination submodule is used to determine the target log type corresponding to the target log information according to the second confidence level.

[0188] Optionally, the field has a corresponding field type, and the log compression module 605 includes:

[0189] A field column generation submodule is used to arrange fields of the same type into field columns if the target log type is a type with a consistent number of fields;

[0190] The first storage submodule is used to store the field name and the field content in the field column respectively.

[0191] Optionally, the field has a corresponding field format, and the log compression module 605 includes:

[0192] A field content acquisition submodule, used for acquiring the field content in the target log information if the target log type is a field format consistent type and the field format is a Boolean type;

[0193] A field content positioning submodule, used to position the field content to a target storage position of a preset byte;

[0194] A second storage submodule, used for storing the field content in a target storage bit of the preset byte;

[0195] The third storage submodule is used to store the field name.

[0196] Optionally, the log compression module 605 includes:

[0197] A reference value acquisition submodule, used for acquiring a preset reference value if the target log type is a field content-related type;

[0198] A difference calculation submodule, used to calculate the difference between the field content and the preset reference value to obtain a target storage value;

[0199] The fourth storage submodule is used to store the field name and the target storage value.

[0200] Optionally, the log compression module 605 includes:

[0201] A field content decomposition submodule, for decomposing the field content to obtain a decomposition value within a preset range if the target log type is a field content boundary determination type;

[0202] A fifth storage submodule, used to store the decomposed values ​​within the preset range using preset bytes;

[0203] The sixth storage submodule is used to store the field name.

[0204] In an embodiment of the present application, a log is obtained; the log format of the log is determined; target log information is extracted from the log according to the log format; the target log information includes fields, and the fields include field content and field names; the target log type corresponding to the target log information is determined; the target log type has a corresponding log compression scheme; the field content and field name are compressed using the log compression scheme corresponding to the target log type to obtain a compressed log; and the compressed log is stored. The embodiment of the present application extracts target log information from logs of different formats and performs compression processing according to different target log types, thereby effectively reducing the space required for log storage, especially when facing a specific target log type, the space compression ratio can be further improved; after the storage space is compressed, subsequent log reading operations can still maintain high efficiency, thereby achieving efficient access to massive security software logs.

[0205] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0206] The present application embodiment provides one or more machine-readable media on which instructions are stored, and when executed by one or more processors, an electronic device executes one or more of the methods described in the above embodiments. In the present application embodiment, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0207] The embodiments of the present disclosure may be implemented as a device configured as desired using any appropriate hardware, firmware, software, or any combination thereof, and the device may include electronic devices such as terminal devices and servers (clusters). Figure 7 An exemplary apparatus 700 that can be used to implement various embodiments described in this application is schematically illustrated.

[0208] For one embodiment, Figure 7An exemplary apparatus 700 is shown having one or more processors 702, a control module (chip set) 704 coupled to at least one of the processor(s) 702, a memory 706 coupled to the control module 704, a non-volatile memory (NVM) / storage device 708 coupled to the control module 704, one or more input / output devices 710 coupled to the control module 704, and a network interface 712 coupled to the control module 704.

[0209] The processor 702 may include one or more single-core or multi-core processors, and the processor 702 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 700 can be used as a terminal device, server (cluster), etc. described in the embodiments of the present application.

[0210] In some embodiments, the apparatus 700 may include one or more computer-readable media (e.g., memory 706 or NVM / storage device 708) having instructions 714 and one or more processors 702 combined with the one or more computer-readable media and configured to execute the instructions 714 to implement a module to perform the actions described in the present disclosure.

[0211] For one embodiment, the control module 704 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 702 and / or any suitable device or component in communication with the control module 704 .

[0212] The control module 704 may include a memory controller module to provide an interface to the memory 706. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0213] The memory 706 may be used, for example, to load and store data and / or instructions 714 for the device 700. For one embodiment, the memory 706 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 706 may include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0214] For one embodiment, control module 704 may include one or more input / output controllers to provide an interface to NVM / storage device 708 and input / output device(s) 710 .

[0215] For example, NVM / storage 708 may be used to store data and / or instructions 714. NVM / storage 708 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0216] NVM / storage device 708 may include storage resources that are physically part of the device on which apparatus 700 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 708 may be accessed via input / output device(s) 710 over a network.

[0217] (One or more) input / output devices 710 may provide an interface for the apparatus 700 to communicate with any other appropriate device, and the input / output device 710 may include a communication component, an audio component, a sensor component, etc. The network interface 712 may provide an interface for the apparatus 700 to communicate through one or more networks, and the apparatus 700 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof for wireless communication.

[0218] For one embodiment, at least one of the processor(s) 702 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 704. For one embodiment, at least one of the processor(s) 702 may be packaged together with the logic of one or more controllers of the control module 704 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 702 may be integrated on the same die with the logic of one or more controllers of the control module 704. For one embodiment, at least one of the processor(s) 702 may be integrated on the same die with the logic of one or more controllers of the control module 704 to form a system-on-chip (SoC).

[0219] In various embodiments, the apparatus 700 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the apparatus 700 may have more or fewer components and / or a different architecture. For example, in some embodiments, the apparatus 700 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0220] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in a memory or NVM / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0221] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0222] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0223] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable log storage terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable log storage terminal device generate instructions for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0224] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable log storage terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0225] These computer program instructions can also be loaded onto a computer or other programmable log storage terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0226] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0227] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0228] The above is a detailed introduction to a log storage method and device, an electronic device and a storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for a person skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A log storage method, characterized in that: The method comprises: Get logs; Determining a log format of the log; Extracting target log information from the log according to the log format; the target log information includes fields, and the fields include field content and field names; Determine a target log type corresponding to the target log information; the target log type has a corresponding log compression scheme; Compressing the field content and the field name using a log compression scheme corresponding to the target log type to obtain a compressed log; The compressed log is stored.

2. The method according to claim 1, characterized in that The log includes a key-value pair, and extracting target log information from the log according to the log format includes: If the log format of the log is a non-standard format, extracting the key-value pair of the log to obtain a key-value pair extraction result and a first confidence level corresponding to the key-value pair extraction result; Displaying the key-value pair extraction result and the first confidence level; Receiving a selection operation on the key-value pair extraction result; According to the selection operation, extracting the target log information from the log; If the log format of the log is a standard format, an intelligent disassembly model is used to extract the target log information from the log according to the log format.

3. The method according to claim 1, characterized in that The determining the target log type corresponding to the target log information includes: Calculating a second confidence level that the target log information belongs to the log information type according to the field content and the field name; The target log type corresponding to the target log information is determined according to the second confidence level.

4. The method according to claim 1, characterized in that The field has a corresponding field type, and the log compression scheme corresponding to the target log type is used to compress the field content and the field name to obtain a compressed log, including: If the target log type is a type with a consistent number of fields, the fields of the same type are arranged into field columns; The field names and the field contents in the field columns are stored separately.

5. The method according to claim 1, characterized in that The field has a corresponding field format, and the log compression scheme corresponding to the target log type is used to compress the field content and the field name to obtain a compressed log, including: If the target log type is a field format consistent type, and the field format is a Boolean type, then the field content in the target log information is obtained; Positioning the field content to a target storage position of a preset byte; Using the target storage bit of the preset byte to store the field content; Stores the field name.

6. The method according to claim 1, characterized in that The adopting the log compression scheme corresponding to the target log type to compress the field content and the field name to obtain a compressed log includes: If the target log type is a field content-related type, obtaining a preset reference value; Calculate the difference between the field content and the preset reference value to obtain a target storage value; The field name and the target storage value are stored.

7. The method according to claim 1, characterized in that The adopting the log compression scheme corresponding to the target log type to compress the field content and the field name to obtain a compressed log includes: If the target log type is a field content boundary determination type, decomposing the field content to obtain a decomposed value within a preset range; Using preset bytes to store the decomposed values ​​within the preset range; Stores the field name.

8. A log storage device, characterized in that: The device comprises: Log acquisition module, used to obtain logs; A log format determination module, used to determine the log format of the log; A log disassembly module, used to extract target log information from the log according to the log format; the target log information includes fields, and the fields include field content and field names; A log type determination module, used to determine a target log type corresponding to the target log information; the target log type has a corresponding log compression scheme; A log compression module, used to compress the field content and the field name using a log compression scheme corresponding to the target log type to obtain a compressed log; The log storage module is used to store the compressed logs.

9. An electronic device, characterized in that: include: processor; and A memory having executable codes stored thereon, which, when executed, enables the processor to execute the log storage method as described in one or more of claims 1-7.

10. One or more machine-readable media having executable codes stored thereon, which, when executed, enable a processor to execute the log storage method according to one or more of claims 1-7.

Images