Real-time storage method for log data

By collecting and serializing log data in real time, and generating partition storage keys for binary byte streams, it solves the problems of limited performance and insufficient scalability in traditional log storage methods, and realizes efficient log data storage and query.

CN118964916AActive Publication Date: 2024-11-15BEIJING TRADITIONAL CHINESE MEDICINE XINCHUANG TECH DEV CO LTD

Patent Information

Application Number
CN202411421289.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-12
Publication Date
2024-11-15
Estimated Expiration
2044-10-12

AI Technical Summary

Technical Problem

When facing large-scale and high-frequency log data, traditional log processing and storage methods show problems such as limited data writing performance, slow query response and insufficient system scalability.

Method used

Using the method of collecting log data in real time, the log records are parsed and encapsulated through the real-time log data collector, JSON objects are generated, and serialized, and mapped to a logical container to form a log data stream, according to the storage engine type and cluster The configuration parameters are converted into columnar storage tables, and partition storage keys for binary byte streams are generated based on timestamps, log levels and log sources for partition storage.

Benefits of technology

It improves the writing performance and query efficiency of log data, enhances the scalability and real-time nature of the system, and ensures efficient organization and management of data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118964916A_ABST
    Figure CN118964916A_ABST
Patent Text Reader

Abstract

The invention provides a log data real-time storage method, which comprises the following steps of: collecting log records output by a log production system in real time based on a set log data real-time collector, and analyzing the collected log records to obtain a JSON (JavaScript Object Notation) object; packaging the JSON object to obtain a log event object; serializing the log event object to generate serialized log data; mapping the serialized log data into a logic container to obtain a log data stream; according to the storage engine type and the storage cluster configuration parameters, consuming the log data stream to generate a column type storage table; based on a configured serialization component, binarizing the column type storage table to generate a binary byte stream; according to the timestamp, the log level and the log source, generating a partition storage key of the binary byte stream; and the binary byte stream is sent to the partition pointed by the partition storage key for storage, so that the writing performance, the query performance and the like of the data are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a real-time storage method for log data. Background Art

[0002] In the context of the rapid development of information technology and the Internet of Things, with the widespread deployment and continuous operation of IoT devices, the speed and scale of log data generation have shown an unprecedented growth trend. These log data not only contain the operating status of the device and user interaction information, but also contain important security audit traces, which are of immeasurable value for system optimization, rapid fault location, in-depth business analysis and real-time security supervision. However, traditional log processing and storage methods, such as text file-based storage or records relying on relational databases, are unable to cope with such large-scale and high-frequency log data, mainly reflected in limited data writing performance, slow query response and insufficient system scalability. Summary of the invention

[0003] In order to solve the above technical problems, the present application provides a real-time storage method for log data to at least solve or alleviate the above problems in the prior art.

[0004] A real-time storage method for log data comprises: collecting log records output by a log production system in real time based on a set real-time log data collector, parsing the collected log records to obtain a JSON object; encapsulating the JSON object to obtain a log event object; serializing the log event object to generate serialized log data; mapping the serialized log data to a logical container to obtain a log data stream; consuming the log data stream according to a storage engine type and a storage cluster configuration parameter to generate a column storage table; binarizing the column storage table based on a configured serialization component to generate a binary byte stream; generating a partition storage key for the binary byte stream according to a timestamp, a log level, and a log source; and sending the binary byte stream to the partition pointed to by the partition storage key for storage; The method of generating a partition storage key of the binary byte stream according to the timestamp, log level and log source includes: performing hash mapping on the timestamp, log level and log source to obtain a timestamp hash value, a log level hash value and a log source hash value, respectively, and obtaining a hash value; performing bit truncation processing on the timestamp hash value, the log level hash value and the log source hash value respectively to obtain corresponding hash value fragments, and performing splicing processing on all hash value fragments to generate a partition storage key of the binary byte stream; Among them, sending the binary byte stream to the partition pointed to by the partition storage key for storage includes: generating a storage request based on the partition storage key of the binary byte stream; according to the storage request, calling the Blob implementation class to encapsulate the binary byte stream into a Blob data type; establishing a mapping relationship between the Blob data type and the placeholder of the precompiled statement to send the Blob data type to the partition pointed to by the partition storage key for storage.

[0005] The technical solution provided by the embodiment of the present application has at least the following technical benefits: (1) A real-time log data collector captures the log records generated by each device in the IoT system in real time. The collector can efficiently parse these log records and convert them into standardized JSON objects, which is convenient for improving the data writing performance. (2) The parsed JSON objects are encapsulated to generate log event objects. In order to improve the efficiency of data transmission and storage, these log event objects are serialized to generate compact serialized log data, which is convenient for improving the data writing performance. (3) The serialized log data is mapped to a logical container to form a continuous log data stream. According to the pre-configured storage engine type and storage cluster parameters, the log data stream is consumed in real time and converted into an efficient column storage table. This step fully utilizes the performance advantage of column storage in processing analytical queries. (4) Based on the configured serialization component, the column storage table is binarized to generate a binary byte stream. Based on key information such as timestamp, log level and log source, a partition storage key for the binary byte stream is generated. Finally, these binary byte streams are sent to the corresponding partitions for storage, ensuring that the organization and management of data are both efficient and easy to query. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 It is a flowchart of a method for real-time storage of log data according to an embodiment of the present application. DETAILED DESCRIPTION

[0007] Figure 1 This is a flow chart of a method for real-time storage of log data according to an embodiment of the present application. Figure 1 As shown, it includes: S101, based on the set log data real-time collector, log records output by the log production system are collected in real time, and the collected log records are parsed to obtain a JSON object; S102, encapsulating the JSON object to obtain a log event object; S103, serializing the log event object to generate serialized log data; S104, mapping the serialized log data into a logical container to obtain a log data stream; S105, consuming the log data stream according to the storage engine type and storage cluster configuration parameters to generate a column storage table; S106. Binarize the column storage table based on the configured serialization component to generate a binary byte stream; S107, generating a partition storage key for the binary byte stream according to the timestamp, log level, and log source; S108. Send the binary byte stream to the partition pointed to by the partition storage key for storage.

[0008] Optionally, the set-based real-time log data collector performs the following steps to collect the log records output by the log production system in real time, including: Based on the index node ID corresponding to each file in the log production system, call the index node pointed to by the index node ID; Based on the pointer of the index node, log records are collected from the log production system.

[0009] To this end, in the real-time collection of log data, the index node is called based on the index node ID corresponding to each file in the log production system, and log records are collected from the log production system based on the pointer of the index node. This has the following technical benefits: (1) The index node is a data structure in the file system that stores file metadata, including the file type, permissions, owner, size, and pointers to the actual data blocks of the file. By directly calling the index node ID, the conventional search process of the file system can be bypassed to quickly locate the data location of the file, thereby improving the efficiency of log record collection. Traditional log collection methods may need to traverse the entire file system or rely on the file system API to find and read log files, which consumes a lot of system resources. The collection method based on the index node directly locates the data location of the file, avoiding unnecessary file traversal and API calls, thereby reducing resource consumption. (2) In scenarios with high real-time requirements, such as IoT monitoring and financial transactions, the timely collection and processing of log data is crucial. Through the collection method based on the index node, log data can be obtained more quickly, shortening the time delay from data generation to data processing, and enhancing the real-time performance of the system. (3) As the scale of the log production system expands, the number and size of log files will also increase accordingly. Since the index node-based collection method directly locates the file data, it can be more easily expanded to the collection scenario of large-scale log files without significantly reducing the collection efficiency due to the increase in the number of files.

[0010] The following is a simplified example code to illustrate how to collect logs based on the index node ID. #include<stdio.h> #include<fcntl.h> #include<unistd.h> #include<sys / types.h> #include<sys / stat.h> #include<string.h> / / Get the index node ID corresponding to the file ino_t get_inode_id(const char *filename) { struct stat statbuf; if (stat(filename,&statbuf) == -1) { perror("stat"); return -1; } return statbuf.st_ino; } / / Log collection function, based on index node ID void collect_logs_by_inode(ino_t inode_id) { char *log_data = "Example log data from inode"; / / Log data printf("Collected log: %s\n", log_data); } int main() { const char *log_file = " / path / to / log / file.log"; ino_t inode_id = get_inode_id(log_file); if (inode_id != -1) { collect_logs_by_inode(inode_id); } else { fprintf(stderr, "Failed to get inode ID for file: %s\n", log_file);} return 0; } Optionally, the collecting log records from the log production system based on the pointer of the index node includes: based on the pointer of the index node, accessing its corresponding log file to determine whether the log file has changed, and if so, creating a corresponding index instance as a key and a file descriptor as a value to encapsulate them as a key-value pair; based on the key-value pair, collecting log records from the log file.

[0011] To this end, in the real-time collection of log data, through the above process, this technical solution has the following technical benefits: (1) By directly accessing the log file through the pointer of the index node, the data location of the file can be accurately located, avoiding the conventional search process of the file system, thereby improving the efficiency of log collection. When the file changes, by creating a key-value pair of the index instance and the file descriptor, the new log data to be collected can be quickly identified to ensure the accuracy and completeness of the collection. (2) Traditional log collection methods may need to frequently traverse the file system or rely on the file system API to check file changes, which consumes a lot of system resources. However, this solution can more effectively monitor file changes through the pointer of the index node and the file descriptor, reducing unnecessary resource consumption. By encapsulating the index instance and file descriptor in the form of key-value pairs, the processing flow of subsequent log collection can be simplified and the overall performance of the system can be improved. (3) In scenarios with high real-time requirements, such as IoT monitoring and financial transactions, the timely collection and processing of log data is crucial. This solution can quickly respond to file changes, ensure that newly generated log data can be collected and processed in a timely manner, and enhance the real-time performance of the system. By indexing the key-value pairs of instances and file descriptors, we can achieve continuous monitoring and reliable collection of log files, ensuring the integrity and accuracy of log data even when files change frequently or the system load is high. (4) As the scale of the log production system increases, the number and size of log files will also increase accordingly. This solution can be more easily expanded to large-scale log file collection scenarios through the index node pointer and key-value pairs, without significantly reducing the collection efficiency due to the increase in the number of files.

[0012] The following is a simplified example code to illustrate how to collect logs based on index node pointers and key-value pairs: #include<stdio.h> #include<fcntl.h> #include<unistd.h> #include<sys / types.h> #include<sys / stat.h> #include<string.h> #include<stdlib.h> / / Get the index node pointer corresponding to the file typedef struct { / / Index node pointer content int dummy_pointer; } InodePointer; InodePointer get_inode_pointer(const char *filename) { / / The process of getting the index node pointer InodePointer pointer; pointer.dummy_pointer = 42; / / just an example value return pointer; } / / Determine if the file has changed int is_file_changed(const char *filename) { return rand() % 2; } / / Log collection function, based on index node pointer and key-value pair void collect_logs_by_inode_pointer(InodePointer pointer, const char *filename) { if (is_file_changed(filename)) { / / Create a key-value pair of index instance and file descriptor int fd = open(filename, O_RDONLY); if (fd == -1) { perror("open"); return; } / / Index instance char *index_instance = strdup("example_index_instance"); / / Encapsulate as key-value pairs printf("Index Instance: %s, File Descriptor: %d\n", index_instance,fd); / / Close the file descriptor and release the memory close(fd); free(index_instance); } else { printf("File %s has not changed.\n", filename); } } int main() { const char *log_file = " / path / to / log / file.log"; InodePointer inode_pointer = get_inode_pointer(log_file); collect_logs_by_inode_pointer(inode_pointer, log_file); return 0; } Optionally, collecting log records from the log file based on the key-value pair includes: adding the key-value pair to an index event queue to verify the legitimacy of the key-value pair; generating a kernel event in response to passing the legitimacy verification; and collecting log records from the log file based on the kernel event.

[0013] To this end, the above-mentioned process of adding key-value pairs to the index event queue for legitimacy verification has the following technical benefits: (1) Legitimacy verification is an important part of ensuring data integrity and security. By adding key-value pairs to the index event queue and performing legitimacy verification, illegal or erroneous key-value pairs can be effectively prevented from being used for log collection, thereby enhancing the security and accuracy of the system. Legitimacy verification can also ensure that the index instance and file descriptor in the key-value pair are valid and matched, avoiding log collection failure or data confusion caused by incorrect matching. (2) By adding key-value pairs to the index event queue, asynchronous processing can be achieved, that is, the legitimacy of the key-value pairs is verified without blocking the main thread. This helps to improve the processing efficiency and response speed of the system, especially in scenarios with high concurrency or large data volumes. Once the legitimacy verification passes, a kernel event is immediately generated, and log records are collected based on the kernel event. This mechanism ensures the timely collection and processing of log data and reduces delays and waiting time. (3) Adding key-value pairs to the index event queue and performing legitimacy verification can simplify the log collection process. Through the unified event queue and kernel event mechanism, the log collection process can be more easily managed and monitored. In addition, this mechanism also facilitates the automation and intelligence of log collection, reducing the need for human intervention and manual configuration. (4) The design of the index event queue and kernel event mechanism makes it easier for the system to expand and adapt to different log collection needs. By adjusting the size of the event queue, optimizing the algorithm for legitimacy verification, or adding more processing logic, it can flexibly respond to different scenarios and needs. This mechanism also facilitates integration and collaboration with other systems, such as combining with log analysis systems, alarm systems, etc., to form a more complete log management and processing solution.

[0014] The following is a simplified example code to illustrate how to collect logs based on key-value pairs, indexed event queues, and kernel events: import queue import threading import time #Index event queue index_event_queue = queue.Queue() #Validity verification function def validate_key_value_pair(key_value_pair): #Logic for legitimacy verification # Check if both the key and value are not empty return key_value_pair['key'] is not None and key_value_pair['value'] is not None #Kernel event generation function def generate_kernel_event(key_value_pair): #Generate kernel event logic #Print a message to indicate the generation of a kernel event print(f"Kernel event generated for key: {key_value_pair['key']},value: {key_value_pair['value']}") #Log collection function def collect_logs_from_file(file_descriptor): #Log collection logic #Print a message to indicate the start of log collection print(f"Starting to collect logs from file descriptor: {file_descriptor}") # Worker thread, used to process key-value pairs in the index event queue def index_event_worker(): while True: try: # Get key-value pairs from the index event queue key_value_pair = index_event_queue.get(block=True, timeout=1) # Verify the legitimacy of key-value pairs if validate_key_value_pair(key_value_pair): # Legitimacy verification passed, generating kernel events generate_kernel_event(key_value_pair) # Start collecting log records based on kernel events collect_logs_from_file(key_value_pair['value']) else: print("Invalid key-value pair detected.") # Mark the task completed index_event_queue.task_done() except queue.Empty: # The queue is empty. Wait for a while before trying to get the key-value pair. time.sleep(0.1) # Start a worker thread to process the index event queue threading.Thread(target=index_event_worker, daemon=True).start() #Add a key-value pair to the index event queue index_event_queue.put({'key': 'example_index_instance', 'value':12345}) # Wait for a while to ensure that the worker thread has a chance to process the key-value pairs in the queue time.sleep(2) Optionally, the parsing of the collected log records to obtain a JSON object includes: performing string segmentation on the log records to obtain a number of fields, the several fields including a timestamp field, a log level field, a source field, a message field, a user information field, and a system information field, the timestamp field records the time when an event occurs, the log level field indicates the importance of the log entry, the source field includes file names, class names, method names, etc., which are used to indicate the location where the log entry is generated, the message field includes actual log information, including text descriptions, error codes, and stack traces; the user information field includes user ID and user name; the system information field includes IP address and host name; the several fields are verified and cleaned to form a valid nested data structure of a predetermined data type and added to a field list; the field list is traversed to map each valid nested data structure therein to a JSON object attribute to obtain a JSON object.

[0015] To this end, the collected log records are parsed to obtain JSON objects. This technical solution brings many technical benefits: (1) Parsing log records into JSON objects realizes the structuring and standardization of log data. This helps to store, retrieve and analyze log data more efficiently in the future. JSON format is a lightweight data exchange format that is easy for humans to read and write, and easy for machines to parse and generate. Therefore, it is very suitable for the representation and transmission of log data. (2) During the parsing process, each field of the log record is verified and cleaned to ensure the accuracy and integrity of the data. This helps to reduce the impact of erroneous data and redundant data on subsequent analysis. Through verification and cleaning, log data can be converted into a valid nested data structure of a predetermined data type, further improving the quality and availability of the data. (3) After parsing log records into JSON objects, log data can be more conveniently processed and analyzed in batches. For example, the JSON processing library can be used to merge, filter, sort and other operations on multiple JSON objects. The attribute mapping process of the JSON object makes data access and operation more intuitive and efficient, reducing complex data processing logic and code writing. (4) The properties of JSON objects can be expanded and modified as needed to adapt to different log collection and analysis scenarios. This enhances the scalability and flexibility of the system. By parsing log records to obtain JSON objects, it is easier to integrate and work with other systems, such as log analysis systems, visualization systems, alarm systems, etc. (5) Structured JSON objects make log data easier to perform advanced analytical operations such as pattern recognition, anomaly detection, and trend analysis. The various fields in the JSON object can be used as analysis dimensions or features to build machine learning models or perform data mining to discover potential associations, patterns, and anomalies.

[0016] The following is a simplified example code to illustrate how to parse the collected log records and obtain a JSON object: import json import re #Logging # "timestamp log_level source message user_info system_info" # Define a function to parse log records def parse_log_record(log): # Use regular expressions to split log records pattern = re.compile(r"(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (\w+)([\w\.\:]+) (.+?) ([\w\:]+) ([\d\.\:]+)") match = pattern.match(log) if not match: raise ValueError("Invalid log format") # Extract each field timestamp, log_level, source, message, user_info, system_info =match.groups() # Further parse and clean the fields file_name, class_name, method_name = source.split('.') # Split the user_info field into user ID and user name user_id, username = user_info.split(':') # Split the system_info field into IP address and host name ip_address, hostname = system_info.split(':') # Build JSON object log_json = { "timestamp": timestamp, "log_level": log_level, "source": { "file_name": file_name, "class_name": class_name, "method_name": method_name }, "message": message, "user_info": { "user_id": user_id, "username": username }, "system_info": { "ip_address": ip_address, "hostname": hostname } } return log_json # Parse the log record and get the JSON object log_json_obj = parse_log_record(log_record) # Print JSON object print(json.dumps(log_json_obj, indent=4)) Optionally, encapsulating the JSON object to obtain a log event object includes: using a defined log event class as a container for encapsulating the JSON object; loading a set parsing library into the container for encapsulating the JSON object, and parsing the JSON object in the container using a decision tree log parsing model based on constant token features to create a log event instance; and encapsulating the log event instance based on a hierarchical invariant sequence search method to obtain a log event object.

[0017] To this end, the log event object is obtained by encapsulating the JSON object as described above. This technical solution brings the following technical benefits: (1) The defined log event class is used as a container to encapsulate the JSON object, which realizes the encapsulation and modularization of the log data. This helps to separate the log processing logic from the business logic and improve the readability and maintainability of the code. The encapsulated log event object can be stored, transmitted and processed as an independent module, which facilitates the subsequent use of the log data. (2) The set parsing library is loaded into the container, and the JSON object is parsed based on the decision tree log parsing model of the constant token feature, realizing an efficient and accurate log parsing process. This helps to quickly extract key information from the log and create the corresponding log event instance. The decision tree model can classify and parse the log according to the constant token feature, which improves the accuracy and efficiency of log parsing and reduces the occurrence of mis-parsing and missed parsing. (3) Based on the hierarchical invariant sequence search method, the log event instance is encapsulated to obtain a log event object with hierarchical structure and serialization characteristics. This helps to maintain the integrity and consistency of the log data and facilitates the subsequent retrieval, analysis and visualization of the log data. The hierarchical structure makes log data clearer and easier to understand, and also helps to discover the associations and patterns between log data. (4) The encapsulated log event object can be expanded and modified as needed to adapt to different log processing and analysis scenarios. This enhances the scalability and flexibility of the system. By adjusting the parameters and configuration of the decision tree model and the hierarchical invariant sequence search method, the parsing and encapsulation of log data of different types and formats can be achieved, meeting the diverse log processing needs. (5) The encapsulated log event object can be more conveniently processed and analyzed in batches, improving the processing efficiency of log data. At the same time, by reducing the steps of manual processing and parsing of log data, the risk of human error is reduced.

[0018] The following is a simplified example code to illustrate how to encapsulate a JSON object and obtain a log event object: import json # JSON object, representing log data log_json = { "timestamp": "2023-04-01 12:34:56", "log_level": "INFO", "source": { "file_name": "file.py", "class_name": "Class", "method_name": "method" }, "message": "Text description, error code, stack trace", "user_info": { "user_id": "user123", "username": "username" }, "system_info": { "ip_address": "192.168.1.1", "hostname": "hostname" } } # Define log event class class LogEvent: def __init__(self, timestamp, log_level, source, message, user_info,system_info): self.timestamp = timestamp self.log_level = log_level self.source = source self.message = message self.user_info = user_info self.system_info = system_info #Parsing library and decision tree model #Use JSON object to parse # Create a log event instance log_event_instance = LogEvent( log_json["timestamp"], log_json["log_level"], log_json["source"], log_json["message"], log_json["user_info"], log_json["system_info"] ) #Hierarchical invariant sequence search method #Return the log event instance as the encapsulated log event object log_event_object = log_event_instance#Encapsulation and processing # Print log event object print(f"LogEvent Object: {vars(log_event_object)}") Output example: LogEvent Object: {'timestamp': '2023-04-01 12:34:56', 'log_level': 'INFO', 'source': {'file_name': 'file.py', 'class_name': 'Class', 'method_name': 'method'}, 'message': 'Text description, error code, stack trace', 'user_info': {'user_id': 'user123', 'username': 'username'}, 'system_info': {'ip_address': '192.168.1.1', 'hostname': 'hostname'}} Optionally, the set parsing library is loaded into a container that encapsulates the JSON object, and the JSON object is parsed in the container with a decision tree log parsing model based on constant token features to create a log event instance, including: extracting key fields in the JSON object as constant token features, the constant token features including words or phrases that frequently appear in log messages and have an impact on event classification greater than a set impact threshold; the decision tree log parsing model based on constant token features constructs a decision tree in the container to match the constant token features, each node in the decision tree represents a decision rule, and each leaf node represents a type of log event, and the decision rule is used to characterize the feature distribution and category label of the log event; and creating a log event instance according to the constant token features and the leaf nodes of the decision tree that have a matching relationship.

[0019] To this end, the set parsing library is loaded into the container, and the decision tree log parsing model based on constant token features is used to parse the JSON object to create log event instances, which has the following technical benefits: (1) By extracting key fields in the JSON object as constant token features, these features are usually words or phrases that frequently appear in log messages and have an important impact on event classification. This helps to accurately capture key information in the log and improve the accuracy of parsing. The decision tree model can accurately match these constant token features, thereby more accurately identifying the type of log event. This reduces the situation of mis-parsing and missed parsing and improves the reliability of log processing. (2) Each node in the decision tree model represents a decision rule, and each leaf node represents a type of log event. This structure makes the model easy to understand and interpret, and helps to discover patterns and regularities in log data. By analyzing the decision rules and leaf nodes in the decision tree, we can gain an in-depth understanding of the feature distribution and category labels of log events, providing strong support for subsequent log analysis and processing. (3) Using the decision tree model for log parsing can achieve fast matching and classification. Compared with the traditional one-by-one parsing method, the decision tree model can process multiple constant token features at the same time, improving the efficiency of log processing. By encapsulating the parsing library and decision tree model in a container, batch processing and analysis of log data can be achieved, further improving processing efficiency. (4) This method allows the extraction rules of constant token features and the parameter configuration of the decision tree model to be adjusted according to actual needs. This enhances the flexibility and scalability of the system, enabling it to adapt to different types of log data and parsing requirements. By continuously updating and optimizing the decision tree model, the accuracy and efficiency of log parsing can be continuously improved to meet the growing needs of log processing.

[0020] The following is a simplified example code to illustrate how to parse a JSON object based on a decision tree log parsing model with constant token features and create a log event instance: import json from sklearn.tree import DecisionTreeClassifier # JSON object, representing log data log_json = { "timestamp": "2023-04-01 12:34:56", "log_level": "INFO", "message": "User user123 logged in successfully from IP 192.168.1.1", } # Extract constant token features constant_tokens = ["logged in", "successfully", "from IP"] #Pre-trained decision tree model # DecisionTreeClassifier as a decision tree model decision_tree_model = DecisionTreeClassifier() log_event_type = "User Login Success" # Create a log event instance) class LogEvent: def __init__(self, timestamp, log_level, message, event_type): self.timestamp = timestamp self.log_level = log_level self.message = message self.event_type = event_type log_event_instance = LogEvent( log_json["timestamp"], log_json["log_level"], log_json["message"], log_event_type ) # Print log event instance print(f"LogEvent Instance: {vars(log_event_instance)}") Output example: LogEvent Instance: {'timestamp': '2023-04-01 12:34:56', 'log_level':'INFO', 'message': 'User user123 logged in successfully from IP 192.168.1.1','event_type': 'User Login Success'} Optionally, the method of encapsulating the log event instance based on the hierarchical invariant sequence search method to obtain a log event object includes: hierarchically processing the log event instance based on the hierarchical invariant sequence search method to obtain a hierarchical data structure and event metadata; extracting static features and dynamic features of the log event instance, the static features including fixed log message strings and error codes, and the dynamic features including time intervals, event frequencies, and parameter changes; serializing the static features and dynamic features based on the hierarchical data structure to generate a log event invariant sequence; encapsulating and packaging the log event invariant sequence and the event metadata to generate the log event object.

[0021] To this end, the above encapsulation of log event instances to obtain log event objects has the following technical benefits: (1) By hierarchically processing log event instances, a clear and orderly hierarchical data structure can be obtained. This structured data representation method is helpful for subsequent retrieval, analysis and visualization of log data. The hierarchical data structure can clearly represent the association and hierarchical structure between log events, which helps to discover patterns and laws in log data. (2) Extracting static and dynamic features of log event instances can comprehensively capture the key information of log events. Static features such as fixed log message strings and error codes are the basic attributes of log events; dynamic features such as time intervals, event frequencies, parameter changes, etc. reflect the changing trends and dynamic behaviors of log events. The extraction of these features helps to gain a deep understanding of the nature and laws of log events and provide strong support for subsequent log analysis and processing. (3) Based on the hierarchical data structure, static and dynamic features can be serialized to generate a stable and invariant sequence of log events. This invariant sequence is the unique identifier of the log event, which has stability and consistency and helps to accurately identify and locate specific events in a large amount of log data. The generation of invariant sequences can also help reduce the redundancy and noise of log data and improve the efficiency and quality of log processing. (4) Encapsulating and packaging the invariant sequence of log events with event metadata can generate a compact and efficient log event object. This object is easy to store and transmit, saving storage space and network bandwidth. Encapsulation and packaging can also help protect the integrity and security of log data and prevent data from being tampered with or lost during storage and transmission. (5) The encapsulated log event object has a unified data format and interface, which is convenient for subsequent log analysis, mining and visualization operations. This helps to discover potential problems and trends and provides strong support for system operation and optimization.

[0022] The following is a simplified example code to illustrate how to encapsulate a log event instance to obtain a log event object: import json from datetime import datetime #Log event instance log_event_instance = { "timestamp": "2023-04-01 12:34:56", "log_level": "INFO", "message": "User user123 logged in successfully", "error_code": None, #No error code # Other fields... } #Hierarchical processing of log event instances #Hierarchy log event instances by timestamp, log level, message, and error code hierarchical_structure = { "timestamp": log_event_instance["timestamp"], "log_level": log_event_instance["log_level"], "details": { "message": log_event_instance["message"], "error_code": log_event_instance["error_code"], } } # Extract static and dynamic features # Static features: fixed log message strings, error codes static_features = { "message": log_event_instance["message"], "error_code": log_event_instance["error_code"], } # Dynamic features (calculation time interval, event frequency, etc.) dynamic_features = { "timestamp": log_event_instance["timestamp"], #Convert to timestamp format } # Serialize static and dynamic features based on hierarchical data structures #Convert hierarchical data structures and features into JSON strings for serialization serialized_data = json.dumps({ "hierarchical_structure": hierarchical_structure, "static_features": static_features, "dynamic_features": dynamic_features, }) # Encapsulate and package the serialized data and event metadata #Encapsulate the serialized data and event metadata into a dictionary object event_metadata = { "event_id": "some_unique_event_id",#Unique event ID "source": "some_log_source",#Log source # Other metadata... } log_event_object = { "serialized_data": serialized_data, "event_metadata": event_metadata, } # Print log event object print(f"LogEvent Object: {json.dumps(log_event_object, indent=2)}") Output example: json { "serialized_data": "{...}", / / Hierarchical data structure and features after serialization "event_metadata": { "event_id": "some_unique_event_id", "source": "some_log_source" / / Other metadata... } } Optionally, the log event object is serialized to generate serialized log data, including: converting the log event object into a vector and saving it in a co-occurrence matrix, and performing a vector convolution operation on the co-occurrence matrix to obtain a convolution word vector matrix; locally encoding and globally encoding the log event object to obtain a multi-stage encoding vector matrix; and serializing and fusing the convolution word vector matrix and the multi-stage encoding vector matrix to generate serialized log data.

[0023] To this end, the above process of serializing log event objects to generate serialized log data has the following technical benefits: (1) By converting log event objects into vectors and saving them in a co-occurrence matrix, the method can capture the co-occurrence relationship between log events, which is crucial for understanding the context and relevance of log events. The use of the co-occurrence matrix makes the representation of log data richer and more accurate. The co-occurrence matrix is ​​vector-convolved to obtain a convolutional word vector matrix. This step uses the powerful feature extraction capability of convolutional neural networks (CNNs) to automatically learn and extract important features from the co-occurrence matrix. This helps reduce the need for manual feature engineering and improves the robustness of feature extraction. (2) In the method, the log event objects are locally encoded and globally encoded to obtain a multi-stage encoding vector matrix. This multi-level encoding method can capture information about log events at different scales, thereby providing a more comprehensive representation of log data. Local encoding focuses on detailed information, while global encoding grasps the overall structure. The combination of the two makes the understanding of log data more in-depth. (3) By serializing and fusing the convolutional word vector matrix and the multi-stage encoding vector matrix, the method can comprehensively utilize the advantages of two different representation methods to generate more expressive serialized log data. This fusion processing not only improves the richness of the data, but also enhances the applicability of the data in different application scenarios. (4) The generation of serialized log data makes subsequent log processing and analysis more efficient. Serialized data is easy to store, transmit and process, and can be seamlessly connected to various log analysis systems and algorithms, thereby improving overall work efficiency.

[0024] The following is a simplified exemplary code to show how to implement the above process of serializing the log event object and generating serialized log data: import numpy as np from sklearn.preprocessing import StandardScaler from keras.models import Sequential from keras.layers import Conv2D, Flatten, Dense, Input, concatenate #Log event objects are extracted and stored in a list log_events = [...] # Convert log event objects to vectors and save them in a co-occurrence matrix co_occurrence_matrix = np.random.rand(len(log_events), len(log_events)) # Example co-occurrence matrix # Perform vector convolution on the co-occurrence matrix conv_model = Sequential() conv_model.add(Conv2D(filters=32, kernel_size=(3, 3), activation='relu', input_shape=(len(log_events), len(log_events), 1))) conv_model.add(Flatten()) conv_word_vector = conv_model.predict(np.expand_dims(co_occurrence_matrix, axis=-1)) # Locally and globally encode log event objects local_encoding = np.random.rand(len(log_events), 64)#local encoding global_encoding = np.random.rand(len(log_events), 64)#global encoding multi_stage_encoding = np.concatenate([local_encoding, global_encoding], axis=1) # Serialize and fuse the convolutional word vector matrix and the multi-stage encoding vector matrix input_conv = Input(shape=(conv_word_vector.shape[1],)) input_multi = Input(shape=(multi_stage_encoding.shape[1],)) merged = concatenate([input_conv, input_multi]) dense = Dense(128, activation='relu')(merged) output = Dense(64, activation='softmax')(dense) model = Sequential([input_conv, input_multi, merged, dense, output]) serialized_log_data = model.predict([conv_word_vector, multi_stage_encoding]) print(serialized_log_data) Optionally, mapping the serialized log data into a logical container to obtain a log data stream includes: based on a set sliding window, capturing features of the serialized log data according to a set sliding step to obtain corresponding feature items; storing the feature items based on a compressed prefix tree, and performing closure detection on the compressed prefix tree to map the compressed prefix tree when the closure degree reaches a preset closure degree threshold to the logical container to obtain the log data stream.

[0025] To this end, a sliding window and a compressed prefix tree are used to map the serialized log data into a logical container, thereby obtaining a log data stream, which has the following technical benefits: (1) The sliding window mechanism allows the serialized log data to be processed in segments, which helps to reduce the amount of data processed at a time and thus improve the processing speed. By setting a reasonable sliding step size, data can be processed quickly while ensuring data integrity. (2) Feature capture based on sliding windows can extract key information from log data, which is crucial for subsequent analysis and mining. At the same time, using a compressed prefix tree to store these feature items can greatly save storage space, because the compressed prefix tree reduces redundancy by sharing paths with the same prefix.

[0026] The following is a simplified example code to show how to implement the above process of using sliding windows and compressed prefix trees to map serialized log data into logical containers to obtain log data streams: class CompressedPrefixTree: # Implementation of compressed prefix tree pass def sliding_window_feature_extraction(serialized_log_data, window_size, step_size): # Feature extraction based on sliding window features = [] for i in range(0, len(serialized_log_data) - window_size + 1, step_size): window_data = serialized_log_data[i:i + window_size] # Extract features feature = sum(window_data) features.append(feature) return features def map_to_logical_container(features, closure_threshold): cpt = CompressedPrefixTree() for feature in features: cpt.insert(feature)# Insert feature items into the compressed prefix tree # Perform closure detection closure_degree = calculate_closure_degree(cpt)#Calculate the degree of closure if closure_degree>= closure_threshold: # Mapping to logical containers logical_container = {'data': cpt.get_compressed_data()} return logical_container else: return None Optionally, consuming the log data stream according to the storage engine type and storage cluster configuration parameters to generate a columnar storage table includes: generating a data stream consumption index according to the storage engine type and storage cluster configuration parameters; consuming the log data stream based on the data stream consumption index to generate a columnar storage table. Optionally, generating a data stream consumption index based on the storage engine type and storage cluster configuration parameters includes: parsing the storage engine type to determine the data model, compression algorithm, and index mechanism of the column-based storage engine; parsing the storage cluster configuration parameters to determine the number of nodes, storage capacity, and network bandwidth of the storage cluster; determining the mapping relationship of the log data stream after hash sharding, the consumption order of the log data stream after hash sharding, and the physical position of the log data stream in the storage engine after hash sharding based on the data model, compression algorithm, and index mechanism of the column-based storage engine, as well as the number of nodes, storage capacity, and network bandwidth of the storage cluster; generating a data stream consumption index based on the mapping relationship of the log data stream after hash sharding, the consumption order of the log data stream after hash sharding, and the physical position of the log data stream in the storage engine after hash sharding.

[0027] To this end, the above-mentioned technical processing of efficiently and orderly consuming log data streams according to the storage engine type and storage cluster configuration parameters to generate columnar storage tables has the following technical benefits: (1) By generating a data stream consumption index, efficient consumption of log data streams can be achieved. The index is optimized according to the specific configuration of the storage engine and storage cluster, ensuring the efficiency and accuracy of the data consumption process. (2) The storage engine type is parsed to determine the data model, compression algorithm and index mechanism of the columnar storage engine. This helps to select a suitable storage method based on the actual characteristics of the data, thereby optimizing the utilization of storage resources and improving storage efficiency. (3) By determining the mapping relationship of the log data stream being hashed, the consumption order and the physical location in the storage engine, the method ensures the consistency and integrity of the data during the consumption and storage process. This is crucial for subsequent data analysis and mining.

[0028] The following is a simplified example code that shows how to generate a data stream consumption index based on the storage engine type and storage cluster configuration parameters, and consume the log data stream to generate a columnar storage table: class DataStreamConsumer: def __init__(self, storage_engine_type, cluster_config): self.storage_engine_type = storage_engine_type self.cluster_config = cluster_config self.consumption_index = self.generate_consumption_index() def generate_consumption_index(self): # Parse the storage engine type data_model, compression_algo, indexing_mechanism = self.parse_storage_engine_type(self.storage_engine_type) # Parse the storage cluster configuration parameters node_count, storage_capacity, network_bandwidth = self.parse_cluster_config(self.cluster_config) # Determine the mapping relationship, consumption order and physical location of hash shards mapping_relation = self.determine_mapping_relation(node_count,storage_capacity) consumption_order = self.determine_consumption_order(network_bandwidth) physical_location = self.determine_physical_location(data_model,compression_algo, indexing_mechanism) # Generate data stream consumption index consumption_index = { 'mapping_relation': mapping_relation, 'consumption_order': consumption_order, 'physical_location': physical_location } return consumption_index def parse_storage_engine_type(self, storage_engine_type): # Parsing logic data_model = 'Columnar' compression_algo = 'Snappy' indexing_mechanism = 'Bitmap' return data_model, compression_algo, indexing_mechanism def parse_cluster_config(self, cluster_config): # Parsing logic node_count = 10 storage_capacity = '10TB' network_bandwidth = '10Gbps' return node_count, storage_capacity, network_bandwidth def determine_mapping_relation(self, node_count, storage_capacity): # Mapping relation determination logic mapping_relation = {'node_1': 'partition_1', 'node_2': 'partition_2',...} return mapping_relation def determine_consumption_order(self, network_bandwidth): # Consumption order determination logic consumption_order = ['partition_1', 'partition_2',...] return consumption_order def determine_physical_location(self, data_model, compression_algo,indexing_mechanism): #Physical location determination logic physical_location = {'partition_1': 'disk_1', 'partition_2': 'disk_2', ...} return physical_location def consume_data_stream(self, log_data_stream): # Consume the log data stream based on the data stream consumption index columnar_table = [] for partition in self.consumption_index['consumption_order']: node = self.consumption_index['mapping_relation'][partition] location = self.consumption_index['physical_location'][partition] #Consumption Logic data = self.consume_from_node(node, location, log_data_stream) columnar_table.append(data) return columnar_table def consume_from_node(self, node, location, log_data_stream): #Logic for consuming data from nodes data = log_data_stream.pop() return data Optionally, consuming the log data stream based on the data stream consumption index to generate a columnar storage table includes: generating a consumption naming resource according to the mapping relationship of the log data stream being hashed and the consumption order of the log data stream after being hashed in the data stream consumption index; consuming the log data stream based on the consumption naming resource, and identifying the consumed log data stream based on the physical position of the log data stream in the storage engine after being hashed to generate a columnar storage table.

[0029] To this end, the above method consumes the log data stream based on the data stream consumption index to generate a column storage table, which has the following technical benefits: (1) The consumption naming resource is generated according to the mapping relationship and consumption order of the log data stream in the data stream consumption index. This ensures that the data is consumed in a predetermined order and manner during the consumption process, avoids data confusion and erroneous consumption, and improves the accuracy and orderliness of data consumption. (2) By consuming the log data stream based on the consumption naming resource and identifying the consumed log data stream based on the physical location of the log data stream in the storage engine after being hashed, the method can ensure the correctness and consistency of the data during the storage process. At the same time, this identification method also facilitates subsequent data management and query. (3) Column storage tables are particularly suitable for processing and analysis of large amounts of data. By generating consumption naming resources and identifying the consumed log data stream, the method can efficiently convert the log data stream into a column storage table, providing strong support for subsequent data processing and analysis, and improving data processing efficiency. (4) The mapping relationship and consumption order of the hash shards are considered, which enables the system to support concurrent processing and distributed storage. When multiple nodes consume log data streams at the same time, they can coordinate and synchronize based on the consumption named resources to ensure data consistency and integrity. (5) Since the method consumes and stores data based on the data stream consumption index, when the system needs to be expanded or adjusted, it only needs to update the data stream consumption index accordingly. This makes the system more scalable and flexible, and can adapt to changing needs.

[0030] The following is a simplified example code that shows how to consume a log data stream based on a data stream consumption index to generate a columnar storage table: class ColumnarStorageTableGenerator: def __init__(self, consumption_index): self.consumption_index = consumption_index self.columnar_table = [] def generate_consumption_resources(self): # Generate consumption naming resources according to the mapping relationship of the log data stream being hashed and the consumption order in the data stream consumption index consumption_resources = [] for partition, node in self.consumption_index['mapping_relation'].items(): consumption_resource = { 'partition': partition, 'node': node, 'order': self.consumption_index['consumption_order'].index(partition) } consumption_resources.append(consumption_resource) return consumption_resources def consume_and_store_data(self, log_data_stream): # Consume the log data stream based on the consumption named resource, and identify the consumed log data stream based on the physical location of the log data stream in the storage engine after it is hashed. consumption_resources = self.generate_consumption_resources() for resource in sorted(consumption_resources, key=lambda x: x['order']): partition_data = self.consume_partition_data(resource['node'], log_data_stream) #Physical location identification logic physical_location = self.determine_physical_location(resource['partition']) self.store_columnar_data(partition_data, physical_location) def consume_partition_data(self, node, log_data_stream): #Logic for consuming partitioned data from nodes partition_data = [] for i in range(len(log_data_stream) / / len(self.consumption_index['mapping_relation'])): partition_data.append(log_data_stream.pop(0))#Get data based on nodes and partitions return partition_data def determine_physical_location(self, partition): #Physical location determination logic #Return the partition name as the physical location identifier return partition def store_columnar_data(self, partition_data, physical_location): # Store partitioned data as part of a columnar table self.columnar_table.append({ 'physical_location': physical_location, 'data': partition_data }) def get_columnar_table(self): return self.columnar_table Optionally, the configuration-based serialization component binarizes the column storage table to generate a binary byte stream, including: converting the column storage table into a metadata stream based on the configuration serialization component; recursively binarizing the metadata stream to generate a binary byte stream.

[0031] Optionally, generating the partition storage key of the binary byte stream according to the timestamp, log level and log source includes: performing hash mapping on the timestamp, log level and log source to obtain a timestamp hash value, a log level hash value and a log source hash value, respectively, to obtain a hash value; performing bit truncation processing on the timestamp hash value, the log level hash value and the log source hash value, respectively, to obtain corresponding hash value fragments, and splicing all the hash value fragments to generate the partition storage key of the binary byte stream.

[0032] To this end, the column storage table is binarized by a configuration-based serialization component to generate a binary byte stream, and the partition storage key of the binary byte stream is generated according to the timestamp, log level, and log source. The following technical benefits are achieved: (1) By converting the column storage table into a metadata stream by a configuration-based serialization component and recursively binarizing the metadata stream to generate a binary byte stream, the storage space and transmission time of the data can be significantly reduced. Data in binary format is usually more compact than data in text format, so it can be transmitted and stored faster. (2) Binarization processing can convert data into a format that is not easily understood by humans, thereby enhancing data security and privacy protection to a certain extent. This is particularly important for the storage and transmission of log data that needs to protect sensitive information. (3) By generating a partition storage key for the binary byte stream according to the timestamp, log level, and log source, data can be easily partitioned, stored, and managed. This partitioning method helps to quickly locate and retrieve data of a specific time period, log level, or log source, and improves the efficiency of data query and analysis. (4) The configuration-based serialization component and the partition storage key generation method enable the system to flexibly adapt to different data formats and storage requirements. When the system needs to be expanded or adjusted, only the configuration needs to be updated accordingly, without making major changes to the entire system. (5) Binary byte streams are usually easier to compress and decompress. By compressing the data, the storage space and transmission time can be further reduced, thus improving the overall performance of the system.

[0033] The following is a simplified example code that shows how to binarize a column storage table based on the configured serialization component and generate partition storage keys for binary byte streams based on timestamps, log levels, and log sources: import hashlib import struct # Column storage table data columnar_table = [ {"timestamp": "2023-04-01 12:00:00", "level": "INFO", "source": "module1", "message": "Log message 1"}, {"timestamp": "2023-04-01 12:01:00", "level": "WARN", "source": "module2", "message": "Log message 2"} ] #Serialization configuration serialization_config = { "timestamp_format": "I",# Use 4-byte integer to represent timestamp "level_mapping": {"INFO": 1, "WARN": 2}, # Log level mapping "source_max_length": 10# Maximum length of the log source } def serialize_columnar_table(columnar_table, config): binary_stream = bytearray() for row in columnar_table: # Serialization timestamp timestamp = int(row["timestamp"].replace("-", "").replace(":", "").replace(" ", "")) binary_stream.extend(struct.pack(config["timestamp_format"],timestamp)) #Serialize log level level = config["level_mapping"].get(row["level"], 0) # The default value is 0, indicating an unknown level binary_stream.extend(struct.pack("B", level))# Use 1 byte to represent the log level #Serialize log source (truncate and pad to maximum length) source = row["source"].encode('utf-8')[:config["source_max_length"]] source_padded = source + bytearray(config["source_max_length"] - len(source))# Fill to the maximum length binary_stream.extend(source_padded) # Serializing log messages return binary_stream def generate_partition_key(row, config): # Hash map timestamp, log level, and log source respectively timestamp_hash = hashlib.md5(row["timestamp"].encode('utf-8')).hexdigest() level_hash = hashlib.md5(row["level"].encode('utf-8')).hexdigest() source_hash = hashlib.md5(row["source"].encode('utf-8')).hexdigest() # Truncate the hash value timestamp_hash_fragment = timestamp_hash[:8] level_hash_fragment = level_hash[:8] source_hash_fragment = source_hash[:8] # Concatenate hash value fragments to generate partition storage keys partition_key = timestamp_hash_fragment + level_hash_fragment +source_hash_fragment return partition_key.encode('utf-8')# Return byte type # Serialized column storage table binary_stream = serialize_columnar_table(columnar_table,serialization_config) print("Binary Stream:", binary_stream) # Generate partition storage key partition_key = generate_partition_key(columnar_table[0],serialization_config) print("Partition Key:", partition_key) Optionally, sending the binary byte stream to the partition pointed to by the partition storage key for storage includes: generating a storage request based on the partition storage key of the binary byte stream; according to the storage request, calling the Blob implementation class to encapsulate the binary byte stream into a Blob data type; establishing a mapping relationship between the Blob data type and a placeholder of a precompiled statement to send the Blob data type to the partition pointed to by the partition storage key for storage.

[0034] To this end, by sending the Blob data type to the partition pointed to by the partition storage key for storage, there are the following technical benefits: (1) By encapsulating the binary byte stream into the Blob data type, it can be easily stored in various databases or storage systems that support Blob storage. This encapsulation method makes storage operations more flexible and can adapt to different storage environments and requirements. (2) Using precompiled statements and placeholder mapping relationships can reduce repeated compilation and parsing operations in the storage process, thereby improving storage performance. Precompiled statements can also reduce security risks such as SQL injection and improve system security. (3) Through encapsulation and mapping relationships, the storage logic can be simplified, making the code clearer and easier to maintain. Developers only need to focus on how to encapsulate data into the Blob type and establish a mapping relationship with the precompiled statement without having to worry about the underlying storage details. (4) The Blob data type is usually used to store large amounts of binary data, such as images, audio, video, etc. Through this method, these large amounts of data can be efficiently stored in partitions to meet the big data storage needs of practical applications.

[0035] The following is a simplified example code that shows how to generate a storage request based on the partition storage key of a binary byte stream and encapsulate the binary byte stream as a Blob data type for storage: import java.sql.*; / / Established a database connection and obtained the Connection object: Connection conn / / Binary byte stream and partition storage key byte[] binaryStream = ...; / / binary byte stream data String partitionKey = ...; / / Partition storage key / / Blob implementation class (using the Blob interface provided by JDBC) Blob blob = null; try { / / Generate storage request based on partition storage key (SQL insert statement) String sql = "INSERT INTO logs_partition (partition_key, log_data)VALUES (?, ?)"; / / Prepare precompiled statements PreparedStatement pstmt = conn.prepareStatement(sql); / / Set the partition storage key parameters pstmt.setString(1, partitionKey); / / Encapsulate the binary byte stream into a Blob data type / / Using JDBC's Blob interface blob = conn.createBlob(); blob.setBytes(1, binaryStream); / / Establish a mapping relationship between the Blob data type and the placeholder of the precompiled statement pstmt.setBlob(2, blob); / / Perform storage operation pstmt.executeUpdate(); / / Commit the transaction / / conn.commit(); } catch (SQLException e) { e.printStackTrace(); / / Handle SQL exceptions (such as rolling back transactions, etc.) finally / / Close resources (Blob, PreparedStatement, Connection, etc.) try { if (blob != null) { blob.close(); } / / Close PreparedStatement and Connection } catch (SQLException e) { e.printStackTrace(); / / Handle SQL exception when resources are closed } } The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A real-time storage method for log data, characterized in that: include: Based on the set log data real-time collector, the log records output by the log production system are collected in real time, and the collected log records are parsed to obtain JSON objects; Encapsulate the JSON object to obtain a log event object; Serializing the log event object to generate serialized log data; Mapping the serialized log data into a logical container to obtain a log data stream; Consuming the log data stream according to the storage engine type and storage cluster configuration parameters to generate a columnar storage table; Based on the configured serialization component, binarize the column storage table to generate a binary byte stream; Generate a partition storage key for the binary byte stream according to the timestamp, log level, and log source; Sending the binary byte stream to the partition pointed to by the partition storage key for storage; The step of generating a partition storage key of the binary byte stream according to the timestamp, log level, and log source includes: Perform hash mapping on the timestamp, log level, and log source to obtain a timestamp hash value, a log level hash value, and a log source hash value, respectively, to obtain a hash value; Performing digit truncation processing on the timestamp hash value, log level hash value, and log source hash value respectively to obtain corresponding hash value fragments, and performing splicing processing on all hash value fragments to generate a partition storage key for the binary byte stream; The step of sending the binary byte stream to the partition pointed to by the partition storage key for storage includes: Generate a storage request based on a partition storage key of the binary byte stream; According to the storage request, calling the Blob implementation class to encapsulate the binary byte stream into a Blob data type; A mapping relationship between the Blob data type and the placeholder of the precompiled statement is established to send the Blob data type to the partition pointed to by the partition storage key for storage.

2. The method according to claim 1, characterized in that The set-based real-time log data collector performs the following steps to collect the log records output by the log production system in real time, including: Based on the index node ID corresponding to each file in the log production system, call the index node pointed to by the index node ID; Based on the pointer of the index node, start collecting log records from the log production system; The step of collecting log records from the log production system based on the pointer of the index node includes: Based on the pointer of the index node, access its corresponding log file to determine whether the log file has changed. If it has changed, create a corresponding index instance as a key and a file descriptor as a value to encapsulate it into a key-value pair; Based on the key-value pair, log records are collected from the log file.

3. The method according to claim 2, characterized in that: The collecting log records from the log file based on the key-value pair includes: Add the key-value pair to the index event queue to verify the legitimacy of the key-value pair; In response to the legitimacy verification being passed, a kernel event is generated; Based on the kernel event, log records are collected from the log file.

4. The method according to claim 3, characterized in that: The collected log records are parsed to obtain a JSON object, including: Performing string segmentation on the log record to obtain a number of fields, the several fields including a timestamp field, a log level field, a source field, a message field, a user information field, and a system information field. The timestamp field records the time when the event occurs, the log level field indicates the importance of the log entry, the source field includes the file name, class name, and method name, which are used to indicate the location where the log entry is generated, the message field includes the actual log information, including a text description, an error code, and a stack trace; the user information field includes a user ID and a user name; and the system information field includes an IP address and a host name; Verifying and cleaning the plurality of fields to form a valid nested data structure of a predetermined data type and adding the structure to a field list; The field list is traversed to map each valid nested data structure therein to a JSON object attribute to obtain a JSON object.

5. The method according to claim 4, characterized in that: The step of encapsulating the JSON object to obtain a log event object includes: Use the defined log event class as a container to encapsulate the JSON object; Loading the set parsing library into a container encapsulating the JSON object, parsing the JSON object in the container with a decision tree log parsing model based on constant token features to create a log event instance; Based on the hierarchical invariant sequence search method, the log event instance is encapsulated to obtain a log event object.

6. The method according to claim 5, characterized in that: The step of loading the set parsing library into a container encapsulating the JSON object, and parsing the JSON object in the container using a decision tree log parsing model based on constant token features to create a log event instance includes: Extracting key fields in the JSON object as constant token features, wherein the constant token features include words or phrases that frequently appear in log messages and have an impact on event classification greater than a set impact threshold; A decision tree log parsing model based on constant token features builds a decision tree in the container to match the constant token features, wherein each node in the decision tree represents a decision rule, and each leaf node represents a type of log event, and the decision rule is used to characterize the feature distribution and category label of the log event; A log event instance is created according to the constant token features having a matching relationship and the leaf nodes of the decision tree.

7. The method according to claim 6, characterized in that The hierarchical invariant sequence search method is based on encapsulating the log event instance to obtain a log event object, including: Based on the hierarchical invariant sequence search method, the log event instance is hierarchically processed to obtain a hierarchical data structure and event metadata; Extracting static features and dynamic features of the log event instance, wherein the static features include fixed log message strings and error codes, and the dynamic features include time intervals, event frequencies, and parameter changes; Serializing the static features and dynamic features based on the hierarchical data structure to generate an invariant sequence of log events; The log event invariant sequence and the event metadata are encapsulated and packaged to generate the log event object.

8. The method according to claim 7, characterized in that The step of serializing the log event object to generate serialized log data includes: Convert the log event object into a vector and save it in a co-occurrence matrix, and perform a vector convolution operation on the co-occurrence matrix to obtain a convolution word vector matrix; Performing local encoding and global encoding on the log event object to obtain a multi-stage encoding vector matrix; The convolution word vector matrix and the multi-stage encoding vector matrix are serialized and fused to generate serialized log data.

9. The method according to claim 8, characterized in that: Mapping the serialized log data into a logical container to obtain a log data stream includes: Based on the set sliding window, the serialized log data is subjected to feature capture according to the set sliding step to obtain corresponding feature items; The feature items are stored based on a compressed prefix tree, and a closure detection is performed on the compressed prefix tree to map the compressed prefix tree when the closure degree reaches a preset closure degree threshold to the logical container to obtain a log data stream.

10. The method according to claim 9, characterized in that The consuming the log data stream according to the storage engine type and the storage cluster configuration parameters to generate a column storage table includes: Generate a data stream consumption index based on the storage engine type and storage cluster configuration parameters; Consume the log data stream based on the data stream consumption index to generate a column storage table; The step of generating a data stream consumption index according to the storage engine type and storage cluster configuration parameters includes: Parsing the storage engine type to determine the data model, compression algorithm, and indexing mechanism of the column storage engine; Parsing the storage cluster configuration parameters to determine the number of nodes, storage capacity, and network bandwidth of the storage cluster; Determine the mapping relationship of the log data stream after being hashed, the consumption order of the log data stream after being hashed, and the physical location of the log data stream after being hashed in the storage engine according to the data model, compression algorithm, index mechanism of the column storage engine, as well as the number of nodes, storage capacity, and network bandwidth of the storage cluster; A data stream consumption index is generated according to the mapping relationship of the log data stream being hashed, the consumption order of the log data stream after being hashed, and the physical position of the log data stream in the storage engine after being hashed.

Citation Information

Patent Citations

  • Log data processing method and device thereof, computer equipment and storage medium

    CN112148674A

  • Online analysis method and system of multi-source log, electronic equipment and storage medium

    CN115437877A

  • Service log acquisition method and device and electronic equipment

    CN116627751A

  • Log data storage method and device, log data query method and device, equipment and medium

    CN116894021A

  • Methods and systems for deterministic classification of log messages

    US20220100780A1

Cited By

  • Log recording method and device during data synchronization between systems, equipment and medium

    CN119474235A

  • Log storage system construction method and system based on multidimensional logic relation interaction

    CN119621672A

  • Method and System for Constructing a Log Storage System Based on Multi-Dimensional Logical Relationship Interaction

    CN119621672B

  • Equipment data collection method, device and system based on multi-source heterogeneity and storage medium

    CN119782404A

  • A method, device, system and storage medium for collecting device data based on multi-source heterogeneity

    CN119782404B