A real-time storage method for log data

Abstract

The present application provides a real-time storage method for log data, which includes: based on a set real-time log data collector, real-time collection of log records output by a log production system is performed, and the collected log records are parsed to obtain a JSON object; the JSON object is encapsulated to obtain a log event object; the log event object is serialized to generate serialized log data; the serialized log data is mapped to a logical container to obtain a log data stream; according to a storage engine type and a storage cluster configuration parameter, the log data stream is consumed to generate a column storage table; based on a configured serialization component, the column storage table is binarized to generate a binary byte stream; according to a timestamp, a log level, and a log source, a partition storage key of the binary byte stream is generated; the binary byte stream is sent to a partition pointed to by the partition storage key for storage, thereby improving data writing performance, query performance, etc.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a real-time storage method for log data. Background Art

[0002] In the context of the rapid development of information technology and the Internet of Things, with the widespread deployment and continuous operation of IoT devices, the speed and scale of log data generation have shown an unprecedented growth trend. These log data not only contain the operating status of the device and user interaction information, but also contain important security audit traces, which are of immeasurable value for system optimization, rapid fault location, in-depth business analysis and real-time security supervision. However, traditional log processing and storage methods, such as text file-based storage or records relying on relational databases, are unable to cope with such large-scale and high-frequency log data, mainly reflected in limited data writing performance, slow query response and insufficient system scalability. Summary of the invention

[0003] In order to solve the above technical problems, the present application provides a real-time storage method for log data to at least solve or alleviate the above problems in the prior art.

[0004] A real-time storage method for log data comprises: collecting log records output by a log production system in real time based on a set real-time log data collector, parsing the collected log records to obtain a JSON object; encapsulating the JSON object to obtain a log event object; serializing the log event object to generate serialized log data; mapping the serialized log data to a logical container to obtain a log data stream; consuming the log data stream according to a storage engine type and a storage cluster configuration parameter to generate a column storage table; binarizing the column storage table based on a configured serialization component to generate a binary byte stream; generating a partition storage key for the binary byte stream according to a timestamp, a log level, and a log source; and sending the binary byte stream to the partition pointed to by the partition storage key for storage;

[0005] The method of generating a partition storage key of the binary byte stream according to the timestamp, log level and log source includes: performing hash mapping on the timestamp, log level and log source to obtain a timestamp hash value, a log level hash value and a log source hash value, respectively, and obtaining a hash value; performing bit truncation processing on the timestamp hash value, the log level hash value and the log source hash value respectively to obtain corresponding hash value fragments, and performing splicing processing on all hash value fragments to generate a partition storage key of the binary byte stream;

[0006] Among them, sending the binary byte stream to the partition pointed to by the partition storage key for storage includes: generating a storage request based on the partition storage key of the binary byte stream; according to the storage request, calling the Blob implementation class to encapsulate the binary byte stream into a Blob data type; establishing a mapping relationship between the Blob data type and the placeholder of the precompiled statement to send the Blob data type to the partition pointed to by the partition storage key for storage.

[0007] The technical solution provided by the embodiment of the present application has at least the following technical benefits: (1) A real-time log data collector captures the log records generated by each device in the IoT system in real time. The collector can efficiently parse these log records and convert them into standardized JSON objects, which is convenient for improving the data writing performance. (2) The parsed JSON objects are encapsulated to generate log event objects. In order to improve the efficiency of data transmission and storage, these log event objects are serialized to generate compact serialized log data, which is convenient for improving the data writing performance. (3) The serialized log data is mapped to a logical container to form a continuous log data stream. According to the pre-configured storage engine type and storage cluster parameters, the log data stream is consumed in real time and converted into an efficient column storage table. This step fully utilizes the performance advantage of column storage in processing analytical queries. (4) Based on the configured serialization component, the column storage table is binarized to generate a binary byte stream. Based on key information such as timestamp, log level and log source, a partition storage key for the binary byte stream is generated. Finally, these binary byte streams are sent to the corresponding partitions for storage, ensuring that the organization and management of data are both efficient and easy to query. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a flowchart of a method for real-time storage of log data according to an embodiment of the present application. DETAILED DESCRIPTION

[0009] Figure 1 This is a flow chart of a method for real-time storage of log data according to an embodiment of the present application. Figure 1 As shown, it includes:

[0010] S101, based on the set log data real-time collector, log records output by the log production system are collected in real time, and the collected log records are parsed to obtain a JSON object;

[0011] S102, encapsulating the JSON object to obtain a log event object;

[0012] S103, serializing the log event object to generate serialized log data;

[0013] S104, mapping the serialized log data into a logical container to obtain a log data stream;

[0014] S105, consuming the log data stream according to the storage engine type and storage cluster configuration parameters to generate a column storage table;

[0015] S106. Binarize the column storage table based on the configured serialization component to generate a binary byte stream;

[0016] S107, generating a partition storage key for the binary byte stream according to the timestamp, log level, and log source;

[0017] S108. Send the binary byte stream to the partition pointed to by the partition storage key for storage.

[0018] Optionally, the set-based real-time log data collector performs the following steps to collect the log records output by the log production system in real time, including:

[0019] Based on the index node ID corresponding to each file in the log production system, call the index node pointed to by the index node ID;

[0020] Based on the pointer of the index node, log records are collected from the log production system.

[0021] To this end, in the real-time collection of log data, the index node is called based on the index node ID corresponding to each file in the log production system, and log records are collected from the log production system based on the pointer of the index node. This has the following technical benefits: (1) The index node is a data structure in the file system that stores file metadata, including the file type, permissions, owner, size, and pointers to the actual data blocks of the file. By directly calling the index node ID, the conventional search process of the file system can be bypassed to quickly locate the data location of the file, thereby improving the efficiency of log record collection. Traditional log collection methods may need to traverse the entire file system or rely on the file system API to find and read log files, which consumes a lot of system resources. The collection method based on the index node directly locates the data location of the file, avoiding unnecessary file traversal and API calls, thereby reducing resource consumption. (2) In scenarios with high real-time requirements, such as IoT monitoring and financial transactions, the timely collection and processing of log data is crucial. Through the collection method based on the index node, log data can be obtained more quickly, shortening the time delay from data generation to data processing, and enhancing the real-time performance of the system. (3) As the scale of the log production system expands, the number and size of log files will also increase accordingly. Since the index node-based collection method directly locates the file data, it can be more easily expanded to the collection scenario of large-scale log files without significantly reducing the collection efficiency due to the increase in the number of files.

[0022] The following is a simplified example code to illustrate how to collect logs based on the index node ID.

[0023] #include<stdio.h>

[0024] #include<fcntl.h>

[0025] #include<unistd.h>

[0026] #include<sys / types.h>

[0027] #include<sys / stat.h>

[0028] #include<string.h>

[0029] / / Get the index node ID corresponding to the file

[0030] ino_t get_inode_id(const char *filename) {

[0031] struct stat statbuf;

[0032] if (stat(filename, &statbuf) == -1) {

[0033] perror("stat");

[0034] return -1;

[0035] }

[0036] return statbuf.st_ino;

[0037] }

[0038] / / Log collection function based on inode ID

[0039] void collect_logs_by_inode(ino_t inode_id) {

[0040] char *log_data = "Example log data from inode"; / / Log data

[0041] printf("Collected log: %s\n", log_data);

[0042] }

[0043] int main() {

[0044] const char *log_file = " / path / to / log / file.log";

[0045] ino_t inode_id = get_inode_id(log_file);

[0046] if (inode_id != -1) {

[0047] collect_logs_by_inode(inode_id);

[0048] } else {

[0049] fprintf(stderr, "Failed to get inode ID for file: %s\n", log_file);

[0050] }

[0051] return 0;

[0052] }

[0053] Optionally, the collecting log records from the log production system based on the pointer of the index node includes: based on the pointer of the index node, accessing its corresponding log file to determine whether the log file has changed, and if so, creating a corresponding index instance as a key and a file descriptor as a value to encapsulate them as a key-value pair; based on the key-value pair, collecting log records from the log file.

[0054] To this end, in the real-time collection of log data, through the above process, this technical solution has the following technical benefits: (1) By directly accessing the log file through the pointer of the index node, the data location of the file can be accurately located, avoiding the conventional search process of the file system, thereby improving the efficiency of log collection. When the file changes, by creating a key-value pair of the index instance and the file descriptor, the new log data to be collected can be quickly identified to ensure the accuracy and completeness of the collection. (2) Traditional log collection methods may need to frequently traverse the file system or rely on the file system API to check file changes, which consumes a lot of system resources. However, this solution can more effectively monitor file changes through the pointer of the index node and the file descriptor, reducing unnecessary resource consumption. By encapsulating the index instance and file descriptor in the form of key-value pairs, the processing flow of subsequent log collection can be simplified and the overall performance of the system can be improved. (3) In scenarios with high real-time requirements, such as IoT monitoring and financial transactions, the timely collection and processing of log data is crucial. This solution can quickly respond to file changes, ensure that newly generated log data can be collected and processed in a timely manner, and enhance the real-time performance of the system. By indexing the key-value pairs of instances and file descriptors, we can achieve continuous monitoring and reliable collection of log files, ensuring the integrity and accuracy of log data even when files change frequently or the system load is high. (4) As the scale of the log production system increases, the number and size of log files will also increase accordingly. This solution can be more easily expanded to large-scale log file collection scenarios through the index node pointer and key-value pairs, without significantly reducing the collection efficiency due to the increase in the number of files.

[0055] The following is a simplified example code to illustrate how to collect logs based on index node pointers and key-value pairs:

[0056] #include<stdio.h>

[0057] #include<fcntl.h>

[0058] #include<unistd.h>

[0059] #include<sys / types.h>

[0060] #include<sys / stat.h>

[0061] #include<string.h>

[0062] #include<stdlib.h>

[0063] / / Get the index node pointer corresponding to the file

[0064] typedef struct {

[0065] / / Index node pointer content

[0066] int dummy_pointer;

[0067] } InodePointer;

[0068] InodePointer get_inode_pointer(const char *filename) {

[0069] / / The process of getting the index node pointer

[0070] InodePointer pointer;

[0071] pointer.dummy_pointer = 42; / / just an example value

[0072] return pointer;

[0073] }

[0074] / / Determine if the file has changed

[0075] int is_file_changed(const char *filename) {

[0076] return rand() % 2;

[0077] }

[0078] / / Log collection function, based on index node pointer and key-value pair

[0079] void collect_logs_by_inode_pointer(InodePointer pointer, const char *filename) {

[0080] if (is_file_changed(filename)) {

[0081] / / Create a key-value pair of index instance and file descriptor

[0082] int fd = open(filename, O_RDONLY);

[0083] if (fd == -1) {

[0084] perror("open");

[0085] return;

[0086] }

[0087] / / Index instance

[0088] char *index_instance = strdup("example_index_instance");

[0089] / / Encapsulate as key-value pairs

[0090] printf("Index Instance: %s, File Descriptor: %d\n", index_instance,fd);

[0091] / / Close the file descriptor and release the memory

[0092] close(fd);

[0093] free(index_instance);

[0094] } else {

[0095] printf("File %s has not changed.\n", filename);

[0096] }

[0097] }

[0098] int main() {

[0099] const char *log_file = " / path / to / log / file.log";

[0100] InodePointer inode_pointer = get_inode_pointer(log_file);

[0101] collect_logs_by_inode_pointer(inode_pointer, log_file);

[0102] return 0;

[0103] }

[0104] Optionally, collecting log records from the log file based on the key-value pair includes: adding the key-value pair to an index event queue to verify the legitimacy of the key-value pair; generating a kernel event in response to passing the legitimacy verification; and collecting log records from the log file based on the kernel event.

[0105] To this end, the above-mentioned process of adding key-value pairs to the index event queue for legitimacy verification has the following technical benefits: (1) Legitimacy verification is an important part of ensuring data integrity and security. By adding key-value pairs to the index event queue and performing legitimacy verification, illegal or erroneous key-value pairs can be effectively prevented from being used for log collection, thereby enhancing the security and accuracy of the system. Legitimacy verification can also ensure that the index instance and file descriptor in the key-value pair are valid and matched, avoiding log collection failure or data confusion caused by incorrect matching. (2) By adding key-value pairs to the index event queue, asynchronous processing can be achieved, that is, the legitimacy of the key-value pairs is verified without blocking the main thread. This helps to improve the processing efficiency and response speed of the system, especially in scenarios with high concurrency or large data volumes. Once the legitimacy verification passes, a kernel event is immediately generated, and log records are collected based on the kernel event. This mechanism ensures the timely collection and processing of log data and reduces delays and waiting time. (3) Adding key-value pairs to the index event queue and performing legitimacy verification can simplify the log collection process. Through the unified event queue and kernel event mechanism, the log collection process can be more easily managed and monitored. In addition, this mechanism also facilitates the automation and intelligence of log collection, reducing the need for human intervention and manual configuration. (4) The design of the index event queue and kernel event mechanism makes it easier for the system to expand and adapt to different log collection needs. By adjusting the size of the event queue, optimizing the algorithm for legitimacy verification, or adding more processing logic, it can flexibly respond to different scenarios and needs. This mechanism also facilitates integration and collaboration with other systems, such as combining with log analysis systems, alarm systems, etc., to form a more complete log management and processing solution.

[0106] The following is a simplified example code to illustrate how to collect logs based on key-value pairs, indexed event queues, and kernel events:

[0107] import queue

[0108] import threading

[0109] import time

[0110] #Index event queue

[0111] index_event_queue = queue.Queue()

[0112] #Validity verification function

[0113] def validate_key_value_pair(key_value_pair):

[0114] #Logic for legitimacy verification

[0115] # Check if both the key and value are not empty

[0116] return key_value_pair['key'] is not None and key_value_pair['value'] is not None

[0117] #Kernel event generation function

[0118] def generate_kernel_event(key_value_pair):

[0119] #Generate kernel event logic

[0120] #Print a message to indicate the generation of a kernel event

[0121] print(f"Kernel event generated for key: {key_value_pair['key']},value: {key_value_pair['value']}")

[0122] #Log collection function

[0123] def collect_logs_from_file(file_descriptor):

[0124] #Log collection logic

[0125] #Print a message to indicate the start of log collection

[0126] print(f"Starting to collect logs from file descriptor: {file_descriptor}")

[0127] # Worker thread, used to process key-value pairs in the index event queue

[0128] def index_event_worker():

[0129] while True:

[0130] try:

[0131] # Get key-value pairs from the index event queue

[0132] key_value_pair = index_event_queue.get(block=True, timeout=1)

[0133] # Verify the legitimacy of key-value pairs

[0134] if validate_key_value_pair(key_value_pair):

[0135] # Legitimacy verification passed, generating kernel events

[0136] generate_kernel_event(key_value_pair)

[0137] # Start collecting log records based on kernel events

[0138] collect_logs_from_file(key_value_pair['value'])

[0139] else:

[0140] print("Invalid key-value pair detected.")

[0141] # Mark the task completed

[0142] index_event_queue.task_done()

[0143] except queue.Empty:

[0144] # The queue is empty. Wait for a while before trying to get the key-value pair.

[0145] time.sleep(0.1)

[0146] # Start a worker thread to process the index event queue

[0147] threading.Thread(target=index_event_worker, daemon=True).start()

[0148] #Add a key-value pair to the index event queue

[0149] index_event_queue.put({'key': 'example_index_instance', 'value':12345})

[0150] # Wait for a while to ensure that the worker thread has a chance to process the key-value pairs in the queue

[0151] time.sleep(2)

[0152] Optionally, the parsing of the collected log records to obtain a JSON object includes: performing string segmentation on the log records to obtain a number of fields, the several fields including a timestamp field, a log level field, a source field, a message field, a user information field, and a system information field, the timestamp field records the time when an event occurs, the log level field indicates the importance of the log entry, the source field includes file names, class names, method names, etc., which are used to indicate the location where the log entry is generated, the message field includes actual log information, including text descriptions, error codes, and stack traces; the user information field includes user ID and user name; the system information field includes IP address and host name; the several fields are verified and cleaned to form a valid nested data structure of a predetermined data type and added to a field list; the field list is traversed to map each valid nested data structure therein to a JSON object attribute to obtain a JSON object.

[0153] To this end, the collected log records are parsed to obtain JSON objects. This technical solution brings many technical benefits: (1) Parsing log records into JSON objects realizes the structuring and standardization of log data. This helps to store, retrieve and analyze log data more efficiently in the future. JSON format is a lightweight data exchange format that is easy for humans to read and write, and easy for machines to parse and generate. Therefore, it is very suitable for the representation and transmission of log data. (2) During the parsing process, each field of the log record is verified and cleaned to ensure the accuracy and integrity of the data. This helps to reduce the impact of erroneous data and redundant data on subsequent analysis. Through verification and cleaning, log data can be converted into a valid nested data structure of a predetermined data type, further improving the quality and availability of the data. (3) After parsing log records into JSON objects, log data can be more conveniently processed and analyzed in batches. For example, the JSON processing library can be used to merge, filter, sort and other operations on multiple JSON objects. The attribute mapping process of the JSON object makes data access and operation more intuitive and efficient, reducing complex data processing logic and code writing. (4) The properties of JSON objects can be expanded and modified as needed to adapt to different log collection and analysis scenarios. This enhances the scalability and flexibility of the system. By parsing log records to obtain JSON objects, it is easier to integrate and work with other systems, such as log analysis systems, visualization systems, alarm systems, etc. (5) Structured JSON objects make log data easier to perform advanced analytical operations such as pattern recognition, anomaly detection, and trend analysis. The various fields in the JSON object can be used as analysis dimensions or features to build machine learning models or perform data mining to discover potential associations, patterns, and anomalies.

[0154] The following is a simplified example code to illustrate how to parse the collected log records and obtain a JSON object:

[0155] import json

[0156] import re

[0157] #Logging

[0158] # "timestamp log_level source message user_info system_info"

[0159] # Define a function to parse log records

[0160] def parse_log_record(log):

[0161] # Use regular expressions to split log records

[0162] pattern = re.compile(r"(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (\w+)([\w\.\:]+) (.+?) ([\w\:]+) ([\d\.\:]+)")

[0163] match = pattern.match(log)

[0164] if not match:

[0165] raise ValueError("Invalid log format")

[0166] # Extract each field

[0167] timestamp, log_level, source, message, user_info, system_info =match.groups()

[0168] # Further parse and clean the fields

[0169] file_name, class_name, method_name = source.split('.')

[0170] # Split the user_info field into user ID and user name

[0171] user_id, username = user_info.split(':')

[0172] # Split the system_info field into IP address and host name

[0173] ip_address, hostname = system_info.split(':')

[0174] # Build JSON object

[0175] log_json = {

[0176] "timestamp": timestamp,

[0177] "log_level": log_level,

[0178] "source": {

[0179] "file_name": file_name,

[0180] "class_name": class_name,

[0181] "method_name": method_name

[0182] },

[0183] "message": message,

[0184] "user_info": {

[0185] "user_id": user_id,

[0186] "username": username

[0187] },

[0188] "system_info": {

[0189] "ip_address": ip_address,

[0190] "hostname": hostname

[0191] }

[0192] }

[0193] return log_json

[0194] # Parse the log record and get the JSON object

[0195] log_json_obj = parse_log_record(log_record)

[0196] # Print JSON object

[0197] print(json.dumps(log_json_obj, indent=4))

[0198] Optionally, encapsulating the JSON object to obtain a log event object includes: using a defined log event class as a container for encapsulating the JSON object; loading a set parsing library into the container for encapsulating the JSON object, and parsing the JSON object in the container using a decision tree log parsing model based on constant token features to create a log event instance; and encapsulating the log event instance based on a hierarchical invariant sequence search method to obtain a log event object.

[0199] To this end, the log event object is obtained by encapsulating the JSON object as described above. This technical solution brings the following technical benefits: (1) The defined log event class is used as a container to encapsulate the JSON object, which realizes the encapsulation and modularization of the log data. This helps to separate the log processing logic from the business logic and improve the readability and maintainability of the code. The encapsulated log event object can be stored, transmitted and processed as an independent module, which facilitates the subsequent use of the log data. (2) The set parsing library is loaded into the container, and the JSON object is parsed based on the decision tree log parsing model of the constant token feature, realizing an efficient and accurate log parsing process. This helps to quickly extract key information from the log and create the corresponding log event instance. The decision tree model can classify and parse the log according to the constant token feature, which improves the accuracy and efficiency of log parsing and reduces the occurrence of mis-parsing and missed parsing. (3) Based on the hierarchical invariant sequence search method, the log event instance is encapsulated to obtain a log event object with hierarchical structure and serialization characteristics. This helps to maintain the integrity and consistency of the log data and facilitates the subsequent retrieval, analysis and visualization of the log data. The hierarchical structure makes log data clearer and easier to understand, and also helps to discover the associations and patterns between log data. (4) The encapsulated log event object can be expanded and modified as needed to adapt to different log processing and analysis scenarios. This enhances the scalability and flexibility of the system. By adjusting the parameters and configuration of the decision tree model and the hierarchical invariant sequence search method, the parsing and encapsulation of log data of different types and formats can be achieved, meeting the diverse log processing needs. (5) The encapsulated log event object can be more conveniently processed and analyzed in batches, improving the processing efficiency of log data. At the same time, by reducing the steps of manual processing and parsing of log data, the risk of human error is reduced.

[0200] The following is a simplified example code to illustrate how to encapsulate a JSON object and obtain a log event object:

[0201] import json

[0202] # JSON object, representing log data

[0203] log_json = {

[0204] "timestamp": "2023-04-01 12:34:56",

[0205] "log_level": "INFO",

[0206] "source": {

[0207] "file_name": "file.py",

[0208] "class_name": "Class",

[0209] "method_name": "method"

[0210] },

[0211] "message": "Text description, error code, stack trace",

[0212] "user_info": {

[0213] "user_id": "user123",

[0214] "username": "username"

[0215] },

[0216] "system_info": {

[0217] "ip_address": "192.168.1.1",

[0218] "hostname": "hostname"

[0219] }

[0220] }

[0221] # Define the log event class

[0222] class LogEvent:

[0223] def __init__(self, timestamp, log_level, source, message, user_info,system_info):

[0224] self.timestamp = timestamp

[0225] self.log_level = log_level

[0226] self.source = source

[0227] self.message = message

[0228] self.user_info = user_info

[0229] self.system_info = system_info

[0230] #Parsing library and decision tree model

[0231] #Use JSON object to parse

[0232] # Create a log event instance

[0233] log_event_instance = LogEvent(

[0234] log_json["timestamp"],

[0235] log_json["log_level"],

[0236] log_json["source"],

[0237] log_json["message"],

[0238] log_json["user_info"],

[0239] log_json["system_info"] )

[0241] #Hierarchical invariant sequence search method

[0242] #Return the log event instance as the encapsulated log event object

[0243] log_event_object = log_event_instance#Encapsulation and processing

[0244] # Print log event object

[0245] print(f"LogEvent Object: {vars(log_event_object)}")

[0246] Output example:

[0247] LogEvent Object: {'timestamp': '2023-04-01 12:34:56', 'log_level': 'INFO', 'source': {'file_name': 'file.py', 'class_name': 'Class', 'method_name': 'method'}, 'message': 'Text description, error code, stack trace', 'user_info': {'user_id': 'user123', 'username': 'username'}, 'system_info': {'ip_address': '192.168.1.1', 'hostname': 'hostname'}}

[0248] Optionally, the set parsing library is loaded into a container that encapsulates the JSON object, and the JSON object is parsed in the container with a decision tree log parsing model based on constant token features to create a log event instance, including: extracting key fields in the JSON object as constant token features, the constant token features including words or phrases that frequently appear in log messages and have an impact on event classification greater than a set impact threshold; the decision tree log parsing model based on constant token features constructs a decision tree in the container to match the constant token features, each node in the decision tree represents a decision rule, and each leaf node represents a type of log event, and the decision rule is used to characterize the feature distribution and category label of the log event; and creating a log event instance according to the constant token features and the leaf nodes of the decision tree that have a matching relationship.

[0249] To this end, the set parsing library is loaded into the container, and the decision tree log parsing model based on constant token features is used to parse the JSON object to create log event instances, which has the following technical benefits: (1) By extracting key fields in the JSON object as constant token features, these features are usually words or phrases that frequently appear in log messages and have an important impact on event classification. This helps to accurately capture key information in the log and improve the accuracy of parsing. The decision tree model can accurately match these constant token features, thereby more accurately identifying the type of log event. This reduces the situation of mis-parsing and missed parsing and improves the reliability of log processing. (2) Each node in the decision tree model represents a decision rule, and each leaf node represents a type of log event. This structure makes the model easy to understand and interpret, and helps to discover patterns and regularities in log data. By analyzing the decision rules and leaf nodes in the decision tree, we can gain an in-depth understanding of the feature distribution and category labels of log events, providing strong support for subsequent log analysis and processing. (3) Using the decision tree model for log parsing can achieve fast matching and classification. Compared with the traditional one-by-one parsing method, the decision tree model can process multiple constant token features at the same time, improving the efficiency of log processing. By encapsulating the parsing library and decision tree model in a container, batch processing and analysis of log data can be achieved, further improving processing efficiency. (4) This method allows the extraction rules of constant token features and the parameter configuration of the decision tree model to be adjusted according to actual needs. This enhances the flexibility and scalability of the system, enabling it to adapt to different types of log data and parsing requirements. By continuously updating and optimizing the decision tree model, the accuracy and efficiency of log parsing can be continuously improved to meet the growing needs of log processing.

[0250] The following is a simplified example code to illustrate how to parse a JSON object based on a decision tree log parsing model with constant token features and create a log event instance:

[0251] import json

[0252] from sklearn.tree import DecisionTreeClassifier

[0253] # JSON object, representing log data

[0254] log_json = {

[0255] "timestamp": "2023-04-01 12:34:56",

[0256] "log_level": "INFO",

[0257] "message": "User user123 logged in successfully from IP 192.168.1.1",

[0258] }

[0259] # Extract constant token features

[0260] constant_tokens = ["logged in", "successfully", "from IP"]

[0261] #Pre-trained decision tree model

[0262] # DecisionTreeClassifier as a decision tree model

[0263] decision_tree_model = DecisionTreeClassifier()

[0264] log_event_type = "User Login Success"

[0265] # Create a log event instance)

[0266] class LogEvent:

[0267] def __init__(self, timestamp, log_level, message, event_type):

[0268] self.timestamp = timestamp

[0269] self.log_level = log_level

[0270] self.message = message

[0271] self.event_type = event_type

[0272] log_event_instance = LogEvent(

[0273] log_json["timestamp"],

[0274] log_json["log_level"],

[0275] log_json["message"],

[0276] log_event_type )

[0278] # Print log event instance

[0279] print(f"LogEvent Instance: {vars(log_event_instance)}")

[0280] Output example:

[0281] LogEvent Instance: {'timestamp': '2023-04-01 12:34:56', 'log_level':'INFO', 'message': 'User user123 logged in successfully from IP 192.168.1.1','event_type': 'User Login Success'}

[0282] Optionally, the method of encapsulating the log event instance based on the hierarchical invariant sequence search method to obtain a log event object includes: hierarchically processing the log event instance based on the hierarchical invariant sequence search method to obtain a hierarchical data structure and event metadata; extracting static features and dynamic features of the log event instance, the static features including fixed log message strings and error codes, and the dynamic features including time intervals, event frequencies, and parameter changes; serializing the static features and dynamic features based on the hierarchical data structure to generate a log event invariant sequence; encapsulating and packaging the log event invariant sequence and the event metadata to generate the log event object.

[0283] To this end, the above encapsulation of log event instances to obtain log event objects has the following technical benefits: (1) By hierarchically processing log event instances, a clear and orderly hierarchical data structure can be obtained. This structured data representation method is helpful for subsequent retrieval, analysis and visualization of log data. The hierarchical data structure can clearly represent the association and hierarchical structure between log events, which helps to discover patterns and laws in log data. (2) Extracting static and dynamic features of log event instances can comprehensively capture the key information of log events. Static features such as fixed log message strings and error codes are the basic attributes of log events; dynamic features such as time intervals, event frequencies, parameter changes, etc. reflect the changing trends and dynamic behaviors of log events. The extraction of these features helps to gain a deep understanding of the nature and laws of log events and provide strong support for subsequent log analysis and processing. (3) Based on the hierarchical data structure, static and dynamic features can be serialized to generate a stable and invariant sequence of log events. This invariant sequence is the unique identifier of the log event, which has stability and consistency and helps to accurately identify and locate specific events in a large amount of log data. The generation of invariant sequences can also help reduce the redundancy and noise of log data and improve the efficiency and quality of log processing. (4) Encapsulating and packaging the invariant sequence of log events with event metadata can generate a compact and efficient log event object. This object is easy to store and transmit, saving storage space and network bandwidth. Encapsulation and packaging can also help protect the integrity and security of log data and prevent data from being tampered with or lost during storage and transmission. (5) The encapsulated log event object has a unified data format and interface, which is convenient for subsequent log analysis, mining and visualization operations. This helps to discover potential problems and trends and provides strong support for system operation and optimization.

[0284] The following is a simplified example code to illustrate how to encapsulate a log event instance to obtain a log event object:

[0285] import json

[0286] from datetime import datetime

[0287] #Log event instance

[0288] log_event_instance = {

[0289] "timestamp": "2023-04-01 12:34:56",

[0290] "log_level": "INFO",

[0291] "message": "User user123 logged in successfully",

[0292] "error_code": None, #No error code

[0293] # Other fields...

[0294] }

[0295] #Hierarchical processing of log event instances

[0296] #Hierarchy log event instances by timestamp, log level, message, and error code

[0297] hierarchical_structure = {

[0298] "timestamp": log_event_instance["timestamp"],

[0299] "log_level": log_event_instance["log_level"],

[0300] "details": {

[0301] "message": log_event_instance["message"],

[0302] "error_code": log_event_instance["error_code"],

[0303] }

[0304] }

[0305] # Extract static and dynamic features

[0306] # Static features: fixed log message strings, error codes

[0307] static_features = {

[0308] "message": log_event_instance["message"],

[0309] "error_code": log_event_instance["error_code"],

[0310] }

[0311] # Dynamic features (calculation time interval, event frequency, etc.)

[0312] dynamic_features = {

[0313] "timestamp": log_event_instance["timestamp"], #Convert to timestamp format

[0314] }

[0315] # Serialize static and dynamic features based on hierarchical data structures

[0316] #Convert hierarchical data structures and features into JSON strings for serialization

[0317] serialized_data = json.dumps({

[0318] "hierarchical_structure": hierarchical_structure,

[0319] "static_features": static_features,

[0320] "dynamic_features": dynamic_features,

[0321] })

[0322] # Encapsulate and package the serialized data and event metadata

[0323] #Encapsulate the serialized data and event metadata into a dictionary object

[0324] event_metadata = {

[0325] "event_id": "some_unique_event_id",#Unique event ID

[0326] "source": "some_log_source",#Log source

[0327] # Other metadata...

[0328] }

[0329] log_event_object = {

[0330] "serialized_data": serialized_data,

[0331] "event_metadata": event_metadata,

[0332] }

[0333] # Print log event object

[0334] print(f"LogEvent Object: {json.dumps(log_event_object, indent=2)}")

[0335] Output example:

[0336] json

[0337] {

[0338] "serialized_data": "{...}", / / Hierarchical data structure and features after serialization

[0339] "event_metadata": {

[0340] "event_id": "some_unique_event_id",

[0341] "source": "some_log_source"

[0342] / / Other metadata...

[0343] }

[0344] }

[0345] Optionally, the log event object is serialized to generate serialized log data, including: converting the log event object into a vector and saving it in a co-occurrence matrix, and performing a vector convolution operation on the co-occurrence matrix to obtain a convolution word vector matrix; locally encoding and globally encoding the log event object to obtain a multi-stage encoding vector matrix; and serializing and fusing the convolution word vector matrix and the multi-stage encoding vector matrix to generate serialized log data.

[0346] To this end, the above process of serializing log event objects to generate serialized log data has the following technical benefits: (1) By converting log event objects into vectors and saving them in a co-occurrence matrix, the method can capture the co-occurrence relationship between log events, which is crucial for understanding the context and relevance of log events. The use of the co-occurrence matrix makes the representation of log data richer and more accurate. The co-occurrence matrix is ​​vector-convolved to obtain a convolutional word vector matrix. This step uses the powerful feature extraction capability of convolutional neural networks (CNNs) to automatically learn and extract important features from the co-occurrence matrix. This helps reduce the need for manual feature engineering and improves the robustness of feature extraction. (2) In the method, the log event objects are locally encoded and globally encoded to obtain a multi-stage encoding vector matrix. This multi-level encoding method can capture information about log events at different scales, thereby providing a more comprehensive representation of log data. Local encoding focuses on detailed information, while global encoding grasps the overall structure. The combination of the two makes the understanding of log data more in-depth. (3) By serializing and fusing the convolutional word vector matrix and the multi-stage encoding vector matrix, the method can comprehensively utilize the advantages of two different representation methods to generate more expressive serialized log data. This fusion processing not only improves the richness of the data, but also enhances the applicability of the data in different application scenarios. (4) The generation of serialized log data makes subsequent log processing and analysis more efficient. Serialized data is easy to store, transmit and process, and can be seamlessly connected to various log analysis systems and algorithms, thereby improving overall work efficiency.

[0347] The following is a simplified exemplary code to show how to implement the above process of serializing the log event object and generating serialized log data:

[0348] import numpy as np

[0349] from sklearn.preprocessing import StandardScaler

[0350] from keras.models import Sequential

[0351] from keras.layers import Conv2D, Flatten, Dense, Input, concatenate

[0352] #Log event objects are extracted and stored in a list

[0353] log_events = [...]

[0354] # Convert log event objects to vectors and save them in a co-occurrence matrix

[0355] co_occurrence_matrix = np.random.rand(len(log_events), len(log_events)) # Example co-occurrence matrix

[0356] # Perform vector convolution on the co-occurrence matrix

[0357] conv_model = Sequential()

[0358] conv_model.add(Conv2D(filters=32, kernel_size=(3, 3), activation='relu', input_shape=(len(log_events), len(log_events), 1)))

[0359] conv_model.add(Flatten())

[0360] conv_word_vector = conv_model.predict(np.expand_dims(co_occurrence_matrix, axis=-1))

[0361] # Locally and globally encode log event objects

[0362] local_encoding = np.random.rand(len(log_events), 64)#local encoding

[0363] global_encoding = np.random.rand(len(log_events), 64)#global encoding

[0364] multi_stage_encoding = np.concatenate([local_encoding, global_encoding], axis=1)

[0365] # Serialize and fuse the convolutional word vector matrix and the multi-stage encoding vector matrix

[0366] input_conv = Input(shape=(conv_word_vector.shape[1],))

[0367] input_multi = Input(shape=(multi_stage_encoding.shape[1],))

[0368] merged = concatenate([input_conv, input_multi])

[0369] dense = Dense(128, activation='relu')(merged)

[0370] output = Dense(64, activation='softmax')(dense)

[0371] model = Sequential([input_conv, input_multi, merged, dense, output])

[0372] serialized_log_data = model.predict([conv_word_vector, multi_stage_encoding])

[0373] print(serialized_log_data)

[0374] Optionally, mapping the serialized log data into a logical container to obtain a log data stream includes: based on a set sliding window, capturing features of the serialized log data according to a set sliding step to obtain corresponding feature items; storing the feature items based on a compressed prefix tree, and performing closure detection on the compressed prefix tree to map the compressed prefix tree when the closure degree reaches a preset closure degree threshold to the logical container to obtain the log data stream.

[0375] To this end, a sliding window and a compressed prefix tree are used to map the serialized log data into a logical container, thereby obtaining a log data stream, which has the following technical benefits: (1) The sliding window mechanism allows the serialized log data to be processed in segments, which helps to reduce the amount of data processed at a time and thus improve the processing speed. By setting a reasonable sliding step size, data can be processed quickly while ensuring data integrity. (2) Feature capture based on sliding windows can extract key information from log data, which is crucial for subsequent analysis and mining. At the same time, using a compressed prefix tree to store these feature items can greatly save storage space, because the compressed prefix tree reduces redundancy by sharing paths with the same prefix.

[0376] The following is a simplified example code to show how to implement the above process of using sliding windows and compressed prefix trees to map serialized log data into logical containers to obtain log data streams:

[0377] class CompressedPrefixTree:

[0378] # Implementation of compressed prefix tree

[0379] pass

[0380] def sliding_window_feature_extraction(serialized_log_data, window_size, step_size):

[0381] # Feature extraction based on sliding window

[0382] features = []

[0383] for i in range(0, len(serialized_log_data) - window_size + 1, step_size):

[0384] window_data = serialized_log_data[i:i + window_size]

[0385] # Extract features

[0386] feature = sum(window_data)

[0387] features.append(feature)

[0388] return features

[0389] def map_to_logical_container(features, closure_threshold):

[0390] cpt = CompressedPrefixTree()

[0391] for feature in features:

[0392] cpt.insert(feature)# Insert feature items into the compressed prefix tree

[0393] # Perform closure detection

[0394] closure_degree = calculate_closure_degree(cpt)#Calculate the degree of closure

[0395] if closure_degree>= closure_threshold:

[0396] # Mapping to logical containers

[0397] logical_container = {'data': cpt.get_compressed_data()}

[0398] return logical_container

[0399] else:

[0400] return None

[0401] Optionally, consuming the log data stream according to the storage engine type and storage cluster configuration parameters to generate a columnar storage table includes: generating a data stream consumption index according to the storage engine type and storage cluster configuration parameters; consuming the log data stream based on the data stream consumption index to generate a columnar storage table. Optionally, generating a data stream consumption index based on the storage engine type and storage cluster configuration parameters includes: parsing the storage engine type to determine the data model, compression algorithm, and index mechanism of the column-based storage engine; parsing the storage cluster configuration parameters to determine the number of nodes, storage capacity, and network bandwidth of the storage cluster; determining the mapping relationship of the log data stream after hash sharding, the consumption order of the log data stream after hash sharding, and the physical position of the log data stream in the storage engine after hash sharding based on the data model, compression algorithm, and index mechanism of the column-based storage engine, as well as the number of nodes, storage capacity, and network bandwidth of the storage cluster; generating a data stream consumption index based on the mapping relationship of the log data stream after hash sharding, the consumption order of the log data stream after hash sharding, and the physical position of the log data stream in the storage engine after hash sharding.

[0402] To this end, the above-mentioned technical processing of efficiently and orderly consuming log data streams according to the storage engine type and storage cluster configuration parameters to generate columnar storage tables has the following technical benefits: (1) By generating a data stream consumption index, efficient consumption of log data streams can be achieved. The index is optimized according to the specific configuration of the storage engine and storage cluster, ensuring the efficiency and accuracy of the data consumption process. (2) The storage engine type is parsed to determine the data model, compression algorithm and index mechanism of the columnar storage engine. This helps to select a suitable storage method based on the actual characteristics of the data, thereby optimizing the utilization of storage resources and improving storage efficiency. (3) By determining the mapping relationship of the log data stream being hashed, the consumption order and the physical location in the storage engine, the method ensures the consistency and integrity of the data during the consumption and storage process. This is crucial for subsequent data analysis and mining.

[0403] The following is a simplified example code that shows how to generate a data stream consumption index based on the storage engine type and storage cluster configuration parameters, and consume the log data stream to generate a columnar storage table:

[0404] class DataStreamConsumer:

[0405] def __init__(self, storage_engine_type, cluster_config):

[0406] self.storage_engine_type = storage_engine_type

[0407] self.cluster_config = cluster_config

[0408] self.consumption_index = self.generate_consumption_index()

[0409] def generate_consumption_index(self):

[0410] # Parse the storage engine type

[0411] data_model, compression_algo, indexing_mechanism = self.parse_storage_engine_type(self.storage_engine_type)

[0412] # Parse the storage cluster configuration parameters

[0413] node_count, storage_capacity, network_bandwidth = self.parse_cluster_config(self.cluster_config)

[0414] # Determine the mapping relationship, consumption order and physical location of hash shards

[0415] mapping_relation = self.determine_mapping_relation(node_count,storage_capacity)

[0416] consumption_order = self.determine_consumption_order(network_bandwidth)

[0417] physical_location = self.determine_physical_location(data_model,compression_algo, indexing_mechanism)

[0418] # Generate data stream consumption index

[0419] consumption_index = {

[0420] 'mapping_relation': mapping_relation,

[0421] 'consumption_order': consumption_order,

[0422] 'physical_location': physical_location

[0423] }

[0424] return consumption_index

[0425] def parse_storage_engine_type(self, storage_engine_type):

[0426] #Analysis logic

[0427] data_model = 'Columnar'

[0428] compression_algo = 'Snappy'

[0429] indexing_mechanism = 'Bitmap'

[0430] return data_model, compression_algo, indexing_mechanism

[0431] def parse_cluster_config(self, cluster_config):

[0432] #Analysis logic

[0433] node_count = 10

[0434] storage_capacity = '10TB'

[0435] network_bandwidth = '10Gbps'

[0436] return node_count, storage_capacity, network_bandwidth

[0437] def determine_mapping_relation(self, node_count, storage_capacity):

[0438] #Mapping relationship determination logic

[0439] mapping_relation = {'node_1': 'partition_1', 'node_2': 'partition_2',...}

[0440] return mapping_relation

[0441] def determine_consumption_order(self, network_bandwidth):

[0442] #Consumption order determination logic

[0443] consumption_order = ['partition_1', 'partition_2', ...]

[0444] return consumption_order

[0445] def determine_physical_location(self, data_model, compression_algo,indexing_mechanism):

[0446] #Physical location determination logic

[0447] physical_location = {'partition_1': 'disk_1', 'partition_2': 'disk_2', ...}

[0448] return physical_location

[0449] def consume_data_stream(self, log_data_stream):

[0450] # Consume the log data stream based on the data stream consumption index

[0451] columnar_table = []

[0452] for partition in self.consumption_index['consumption_order']:

[0453] node = self.consumption_index['mapping_relation'][partition]

[0454] location = self.consumption_index['physical_location'][partition]

[0455] #Consumption Logic

[0456] data = self.consume_from_node(node, location, log_data_stream)

[0457] columnar_table.append(data)

[0458] return columnar_table

[0459] def consume_from_node(self, node, location, log_data_stream):

[0460] #Logic for consuming data from nodes

[0461] data = log_data_stream.pop()

[0462] return data

[0463] Optionally, consuming the log data stream based on the data stream consumption index to generate a columnar storage table includes: generating a consumption naming resource according to the mapping relationship of the log data stream being hashed and the consumption order of the log data stream after being hashed in the data stream consumption index; consuming the log data stream based on the consumption naming resource, and identifying the consumed log data stream based on the physical position of the log data stream in the storage engine after being hashed to generate a columnar storage table.

[0464] To this end, the above method consumes the log data stream based on the data stream consumption index to generate a column storage table, which has the following technical benefits: (1) The consumption naming resource is generated according to the mapping relationship and consumption order of the log data stream in the data stream consumption index. This ensures that the data is consumed in a predetermined order and manner during the consumption process, avoids data confusion and erroneous consumption, and improves the accuracy and orderliness of data consumption. (2) By consuming the log data stream based on the consumption naming resource and identifying the consumed log data stream based on the physical location of the log data stream in the storage engine after being hashed, the method can ensure the correctness and consistency of the data during the storage process. At the same time, this identification method also facilitates subsequent data management and query. (3) Column storage tables are particularly suitable for processing and analysis of large amounts of data. By generating consumption naming resources and identifying the consumed log data stream, the method can efficiently convert the log data stream into a column storage table, providing strong support for subsequent data processing and analysis, and improving data processing efficiency. (4) The mapping relationship and consumption order of the hash shards are considered, which enables the system to support concurrent processing and distributed storage. When multiple nodes consume log data streams at the same time, they can coordinate and synchronize based on the consumption named resources to ensure data consistency and integrity. (5) Since the method consumes and stores data based on the data stream consumption index, when the system needs to be expanded or adjusted, it only needs to update the data stream consumption index accordingly. This makes the system more scalable and flexible, and can adapt to changing needs.

[0465] The following is a simplified example code that shows how to consume a log data stream based on a data stream consumption index to generate a columnar storage table:

[0466] class ColumnarStorageTableGenerator:

[0467] def __init__(self, consumption_index):

[0468] self.consumption_index = consumption_index

[0469] self.columnar_table = []

[0470] def generate_consumption_resources(self):

[0471] # Generate consumption naming resources according to the mapping relationship of the log data stream being hashed and the consumption order in the data stream consumption index

[0472] consumption_resources = []

[0473] for partition, node in self.consumption_index['mapping_relation'].items():

[0474] consumption_resource = {

[0475] 'partition': partition,

[0476] 'node': node,

[0477] 'order': self.consumption_index['consumption_order'].index(partition)

[0478] }

[0479] consumption_resources.append(consumption_resource)

[0480] return consumption_resources

[0481] def consume_and_store_data(self, log_data_stream):

[0482] # Consume the log data stream based on the consumption named resource, and identify the consumed log data stream based on the physical location of the log data stream in the storage engine after it is hashed.

[0483] consumption_resources = self.generate_consumption_resources()

[0484] for resource in sorted(consumption_resources, key=lambda x: x['order']):

[0485] partition_data = self.consume_partition_data(resource['node'], log_data_stream)

[0486] #Physical location identification logic

[0487] physical_location = self.determine_physical_location(resource['partition'])

[0488] self.store_columnar_data(partition_data, physical_location)

[0489] def consume_partition_data(self, node, log_data_stream):

[0490] #Logic for consuming partitioned data from nodes

[0491] partition_data = []

[0492] for i in range(len(log_data_stream) / / len(self.consumption_index['mapping_relation'])):

[0493] partition_data.append(log_data_stream.pop(0))#Get data based on nodes and partitions

[0494] return partition_data

[0495] def determine_physical_location(self, partition):

[0496] #Physical location determination logic

[0497] #Return the partition name as the physical location identifier

[0498] return partition

[0499] def store_columnar_data(self, partition_data, physical_location):

[0500] # Store partitioned data as part of a columnar table

[0501] self.columnar_table.append({

[0502] 'physical_location': physical_location,

[0503] 'data': partition_data

[0504] })

[0505] def get_columnar_table(self):

[0506] return self.columnar_table

[0507] Optionally, the configuration-based serialization component binarizes the column storage table to generate a binary byte stream, including: converting the column storage table into a metadata stream based on the configuration serialization component; recursively binarizing the metadata stream to generate a binary byte stream.

[0508] Optionally, generating the partition storage key of the binary byte stream according to the timestamp, log level and log source includes: performing hash mapping on the timestamp, log level and log source to obtain a timestamp hash value, a log level hash value and a log source hash value, respectively, to obtain a hash value; performing bit truncation processing on the timestamp hash value, the log level hash value and the log source hash value, respectively, to obtain corresponding hash value fragments, and splicing all the hash value fragments to generate the partition storage key of the binary byte stream.

[0509] To this end, the column storage table is binarized by a configuration-based serialization component to generate a binary byte stream, and the partition storage key of the binary byte stream is generated according to the timestamp, log level, and log source. The following technical benefits are achieved: (1) By converting the column storage table into a metadata stream based on the configuration-based serialization component and recursively binarizing the metadata stream to generate a binary byte stream, the storage space and transmission time of the data can be significantly reduced. Data in binary format is usually more compact than data in text format, so it can be transmitted and stored faster. (2) The binarization process can convert data into a format that is not easily understood by humans, thereby enhancing the security and privacy protection of the data to a certain extent. This is particularly important for the storage and transmission of log data that needs to protect sensitive information. (3) By generating a partition storage key for the binary byte stream based on the timestamp, log level, and log source, the data can be easily partitioned, stored, and managed. This partitioning method helps to quickly locate and retrieve data of a specific time period, log level, or log source, and improves the efficiency of data query and analysis. (4) The configuration-based serialization component and the partition storage key generation method enable the system to flexibly adapt to different data formats and storage requirements. When the system needs to be expanded or adjusted, only the configuration needs to be updated accordingly, without making major changes to the entire system. (5) Binary byte streams are usually easier to compress and decompress. By compressing the data, the storage space and transmission time can be further reduced, thus improving the overall performance of the system.

[0510] The following is a simplified example code that shows how to binarize a column storage table based on the configured serialization component and generate partition storage keys for binary byte streams based on timestamps, log levels, and log sources:

[0511] import hashlib

[0512] import struct

[0513] # Column storage table data

[0514] columnar_table = [

[0515] {"timestamp": "2023-04-01 12:00:00", "level": "INFO", "source": "module1", "message": "Log message 1"},

[0516] {"timestamp": "2023-04-01 12:01:00", "level": "WARN", "source": "module2", "message": "Log message 2"} ]

[0518] #Serialization configuration

[0519] serialization_config = {

[0520] "timestamp_format": "I",# Use 4-byte integer to represent timestamp

[0521] "level_mapping": {"INFO": 1, "WARN": 2}, # Log level mapping

[0522] "source_max_length": 10# Maximum length of the log source

[0523] }

[0524] def serialize_columnar_table(columnar_table, config):

[0525] binary_stream = bytearray()

[0526] for row in columnar_table:

[0527] # Serialization timestamp

[0528] timestamp = int(row["timestamp"].replace("-", "").replace(":", "").replace(" ", ""))

[0529] binary_stream.extend(struct.pack(config["timestamp_format"],timestamp))

[0530] #Serialize log level

[0531] level = config["level_mapping"].get(row["level"], 0) # The default value is 0, indicating an unknown level

[0532] binary_stream.extend(struct.pack("B", level))# Use 1 byte to represent the log level

[0533] #Serialize log source (truncate and pad to maximum length)

[0534] source = row["source"].encode('utf-8')[:config["source_max_length"]]

[0535] source_padded = source + bytearray(config["source_max_length"] - len(source))# Fill to the maximum length

[0536] binary_stream.extend(source_padded)

[0537] # Serializing log messages

[0538] return binary_stream

[0539] def generate_partition_key(row, config):

[0540] # Hash map timestamp, log level, and log source respectively

[0541] timestamp_hash = hashlib.md5(row["timestamp"].encode('utf-8')).hexdigest()

[0542] level_hash = hashlib.md5(row["level"].encode('utf-8')).hexdigest()

[0543] source_hash = hashlib.md5(row["source"].encode('utf-8')).hexdigest()

[0544] # Truncate the hash value

[0545] timestamp_hash_fragment = timestamp_hash[:8]

[0546] level_hash_fragment = level_hash[:8]

[0547] source_hash_fragment = source_hash[:8]

[0548] # Concatenate hash value fragments to generate partition storage keys

[0549] partition_key = timestamp_hash_fragment + level_hash_fragment +source_hash_fragment

[0550] return partition_key.encode('utf-8')# Return byte type

[0551] # Serialized column storage table

[0552] binary_stream = serialize_columnar_table(columnar_table,serialization_config)

[0553] print("Binary Stream:", binary_stream)

[0554] # Generate partition storage key

[0555] partition_key = generate_partition_key(columnar_table[0],serialization_config)

[0556] print("Partition Key:", partition_key)

[0557] Optionally, sending the binary byte stream to the partition pointed to by the partition storage key for storage includes: generating a storage request based on the partition storage key of the binary byte stream; according to the storage request, calling the Blob implementation class to encapsulate the binary byte stream into a Blob data type; establishing a mapping relationship between the Blob data type and a placeholder of a precompiled statement to send the Blob data type to the partition pointed to by the partition storage key for storage.

[0558] To this end, by sending the Blob data type to the partition pointed to by the partition storage key for storage, there are the following technical benefits: (1) By encapsulating the binary byte stream into the Blob data type, it can be easily stored in various databases or storage systems that support Blob storage. This encapsulation method makes storage operations more flexible and can adapt to different storage environments and requirements. (2) Using precompiled statements and placeholder mapping relationships can reduce repeated compilation and parsing operations in the storage process, thereby improving storage performance. Precompiled statements can also reduce security risks such as SQL injection and improve system security. (3) Through encapsulation and mapping relationships, the storage logic can be simplified, making the code clearer and easier to maintain. Developers only need to focus on how to encapsulate data into the Blob type and establish a mapping relationship with the precompiled statement without having to worry about the underlying storage details. (4) The Blob data type is usually used to store large amounts of binary data, such as images, audio, video, etc. Through this method, these large amounts of data can be efficiently stored in partitions to meet the big data storage needs of practical applications.

[0559] The following is a simplified example code that shows how to generate a storage request based on the partition storage key of a binary byte stream and encapsulate the binary byte stream as a Blob data type for storage:

[0560] import java.sql.*;

[0561] / / Established a database connection and obtained the Connection object: Connection conn

[0562] / / Binary byte stream and partition storage key

[0563] byte[] binaryStream = ...; / / binary byte stream data

[0564] String partitionKey = ...; / / Partition storage key

[0565] / / Blob implementation class (using the Blob interface provided by JDBC)

[0566] Blob blob = null;

[0567] try {

[0568] / / Generate storage request based on partition storage key (SQL insert statement)

[0569] String sql = "INSERT INTO logs_partition (partition_key, log_data)VALUES (?, ?)";

[0570] / / Prepare precompiled statements

[0571] PreparedStatement pstmt = conn.prepareStatement(sql);

[0572] / / Set the partition storage key parameters

[0573] pstmt.setString(1, partitionKey);

[0574] / / Encapsulate the binary byte stream into a Blob data type

[0575] / / Using JDBC's Blob interface

[0576] blob = conn.createBlob();

[0577] blob.setBytes(1, binaryStream);

[0578] / / Establish a mapping relationship between the Blob data type and the placeholder of the precompiled statement

[0579] pstmt.setBlob(2, blob);

[0580] / / Perform storage operation

[0581] pstmt.executeUpdate();

[0582] / / Commit the transaction

[0583] / / conn.commit();

[0584] } catch (SQLException e) {

[0585] e.printStackTrace();

[0586] / / Handle SQL exceptions (such as rolling back transactions, etc.)

[0587] } finally {

[0588] / / Close resources (Blob, PreparedStatement, Connection, etc.)

[0589] try {

[0590] if (blob != null) {

[0591] blob.close();

[0592] }

[0593] / / Close PreparedStatement and Connection

[0594] } catch (SQLException e) {

[0595] e.printStackTrace();

[0596] / / Handle SQL exception when resources are closed

[0597] }

[0598] }

[0599] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A real-time storage method for log data, characterized in that: include: Based on the set log data real-time collector, the log records output by the log production system are collected in real time, and the collected log records are parsed to obtain JSON objects; Use the defined log event class as a container to encapsulate the JSON object; Loading the set parsing library into a container encapsulating the JSON object, parsing the JSON object in the container with a decision tree log parsing model based on constant token features to create a log event instance; Based on the hierarchical invariant sequence search method, the log event instance is encapsulated to obtain a log event object; Serializing the log event object to generate serialized log data; Mapping the serialized log data into a logical container to obtain a log data stream; According to the storage engine type, the storage engine type is parsed to determine the data model, compression algorithm, and index mechanism of the column storage engine; According to the storage cluster configuration parameters, the storage cluster configuration parameters are parsed to determine the number of nodes, storage capacity, and network bandwidth of the storage cluster; Determine the mapping relationship of the log data stream after being hashed, the consumption order of the log data stream after being hashed, and the physical location of the log data stream in the storage engine after being hashed according to the data model, compression algorithm, index mechanism of the column storage engine, and the number of nodes, storage capacity, and network bandwidth of the storage cluster; Generate a data stream consumption index according to the mapping relationship of the log data stream being hashed, the consumption order of the log data stream after being hashed, and the physical position of the log data stream in the storage engine after being hashed; Consume the log data stream based on the data stream consumption index to generate a column storage table; Based on the configured serialization component, binarize the column storage table to generate a binary byte stream; Generate a partition storage key for the binary byte stream according to the timestamp, log level, and log source; Sending the binary byte stream to the partition pointed to by the partition storage key for storage; The step of generating a partition storage key of the binary byte stream according to the timestamp, log level, and log source includes: Perform hash mapping on the timestamp, log level, and log source to obtain a timestamp hash value, a log level hash value, and a log source hash value, respectively, to obtain a hash value; Performing digit truncation processing on the timestamp hash value, log level hash value, and log source hash value respectively to obtain corresponding hash value fragments, and performing splicing processing on all hash value fragments to generate a partition storage key for the binary byte stream; The step of sending the binary byte stream to the partition pointed to by the partition storage key for storage includes: Generate a storage request based on a partition storage key of the binary byte stream; According to the storage request, calling the Blob implementation class to encapsulate the binary byte stream into a Blob data type; A mapping relationship between the Blob data type and the placeholder of the precompiled statement is established to send the Blob data type to the partition pointed to by the partition storage key for storage.

2. The method according to claim 1, characterized in that The set-based real-time log data collector performs the following steps to collect the log records output by the log production system in real time, including: Based on the index node ID corresponding to each file in the log production system, call the index node pointed to by the index node ID; Based on the pointer of the index node, start collecting log records from the log production system; The step of collecting log records from the log production system based on the pointer of the index node includes: Based on the pointer of the index node, access its corresponding log file to determine whether the log file has changed. If it has changed, create a corresponding index instance as a key and a file descriptor as a value to encapsulate it into a key-value pair; Based on the key-value pair, log records are collected from the log file.

3. The method according to claim 2, characterized in that: The collecting log records from the log file based on the key-value pair includes: Add the key-value pair to the index event queue to verify the legitimacy of the key-value pair; In response to the legitimacy verification being passed, a kernel event is generated; Based on the kernel event, log records are collected from the log file.

4. The method according to claim 3, characterized in that: The collected log records are parsed to obtain a JSON object, including: Performing string segmentation on the log record to obtain a number of fields, the several fields including a timestamp field, a log level field, a source field, a message field, a user information field, and a system information field. The timestamp field records the time when the event occurs, the log level field indicates the importance of the log entry, the source field includes the file name, class name, and method name, which are used to indicate the location where the log entry is generated, the message field includes the actual log information, including a text description, an error code, and a stack trace; the user information field includes a user ID and a user name; and the system information field includes an IP address and a host name; Verifying and cleaning the plurality of fields to form a valid nested data structure of a predetermined data type and adding the structure to a field list; The field list is traversed to map each valid nested data structure therein to a JSON object attribute to obtain a JSON object.

5. The method according to claim 4, characterized in that: The step of loading the set parsing library into a container encapsulating the JSON object, and parsing the JSON object in the container using a decision tree log parsing model based on constant token features to create a log event instance includes: Extracting key fields in the JSON object as constant token features, wherein the constant token features include words or phrases that frequently appear in log messages and have an impact on event classification greater than a set impact threshold; A decision tree log parsing model based on constant token features builds a decision tree in the container to match the constant token features, wherein each node in the decision tree represents a decision rule, and each leaf node represents a type of log event, and the decision rule is used to characterize the feature distribution and category label of the log event; A log event instance is created according to the constant token features having a matching relationship and the leaf nodes of the decision tree.

6. The method according to claim 5, characterized in that The hierarchical invariant sequence search method is based on encapsulating the log event instance to obtain a log event object, including: Based on the hierarchical invariant sequence search method, the log event instance is hierarchically processed to obtain a hierarchical data structure and event metadata; Extracting static features and dynamic features of the log event instance, wherein the static features include fixed log message strings and error codes, and the dynamic features include time intervals, event frequencies, and parameter changes; Serializing the static features and dynamic features based on the hierarchical data structure to generate an invariant sequence of log events; The log event invariant sequence and the event metadata are encapsulated and packaged to generate the log event object.

7. The method according to claim 6, characterized in that The step of serializing the log event object to generate serialized log data includes: Convert the log event object into a vector and save it in a co-occurrence matrix, and perform a vector convolution operation on the co-occurrence matrix to obtain a convolution word vector matrix; Performing local encoding and global encoding on the log event object to obtain a multi-stage encoding vector matrix; The convolution word vector matrix and the multi-stage encoding vector matrix are serialized and fused to generate serialized log data.

8. The method according to claim 7, characterized in that: Mapping the serialized log data into a logical container to obtain a log data stream includes: Based on the set sliding window, the serialized log data is subjected to feature capture according to the set sliding step to obtain corresponding feature items; The feature items are stored based on a compressed prefix tree, and a closure detection is performed on the compressed prefix tree to map the compressed prefix tree when the closure degree reaches a preset closure degree threshold to the logical container to obtain a log data stream.

Images