Log gathering and preprocessing system based on distributed architecture

Abstract

The invention provides a log gathering and preprocessing system based on a distributed architecture, which comprises a distributed acquisition module, a preprocessing module, a cache module, a standardized conversion module, a storage module, a unified management service module and an application module, and is characterized in that lightweight agents are deployed in a multi-network area through the distributed acquisition module to realize efficient data gathering; a plug-in assembly line of a preprocessing module and a hybrid processing architecture of a standardized conversion module are combined, and data management is optimized by utilizing a hierarchical storage strategy, so that the problem of data islands in multi-source heterogeneous log convergence and preprocessing is effectively solved, the network bandwidth pressure is reduced, the single-point fault risk is avoided, and the data processing efficiency is improved. The reliability and expansibility of the system are improved, standardized processing of data is realized, and a high-quality and consistent data basis is provided for security analysis.

Description

Technical Field

[0001] This invention relates to the field of network security data processing technology, and in particular to a log aggregation and preprocessing system based on a distributed architecture. Background Technology

[0002] As enterprises deepen their digital transformation, the network architecture of large corporations is becoming increasingly complex, forming a multi-layered and widely distributed network system encompassing headquarters, IT companies, pilot units, and grassroots enterprises. Against this backdrop, network security devices and systems (such as firewalls, intrusion detection systems, and endpoint protection software) generate massive amounts of multi-source, heterogeneous security logs and data. This data is typically stored in separate security data silos using different formats and protocols, making it difficult for the group level to form a unified and clear understanding of the overall network security posture.

[0003] In existing technologies, security data collection and processing often employ centralized or point-to-point methods, which have significant drawbacks: First, traditional centralized collection faces high network bandwidth pressure, a high risk of single-point failures, and poor system scalability when dealing with distributed deployments across regions and multiple network areas. Second, due to the lack of unified data standards and preprocessing specifications, data from different vendors and types of devices varies greatly in format and field meanings, making subsequent data fusion, correlation analysis, and in-depth mining extremely difficult. This not only affects the efficiency of security threat detection but also hinders the group from building a unified, proactive cybersecurity defense system.

[0004] Therefore, there is an urgent need for a system that can efficiently aggregate, process in real time, and standardize multi-source heterogeneous security data in order to break down data silos and provide high-quality, consistent underlying data support for high-level security analysis and control. Summary of the Invention

[0005] This invention provides a log aggregation and preprocessing system based on a distributed architecture, which effectively solves the data silo problem in the process of multi-source heterogeneous log aggregation, reduces network bandwidth pressure, avoids the risk of single point of failure, improves system reliability and scalability, and realizes standardized data processing, providing a high-quality and consistent data foundation for security analysis.

[0006] On the one hand, the present invention provides a log aggregation and preprocessing system based on a distributed architecture, which includes: a distributed collection module for deploying lightweight collection agents in multiple network areas to collect raw data from different types of security devices and systems with a unified configuration strategy; The preprocessing module is used to preprocess the original data to obtain preprocessed data; The caching module is used to send the preprocessed data into a message queue for caching; The standardization conversion module is used to call a preset unified data model to perform field mapping, type conversion and enrichment on the preprocessed data obtained from the message queue to obtain standardized structured data. The storage module is used to hierarchically store the standardized structured data into hot storage, cold storage, and archive storage; The unified management service module is used to read data from hot storage, cold storage, and archive storage, and generate data service interfaces; The application module is used to obtain the required data by calling the data service interface and to perform security analysis, detection and response operations.

[0007] According to the present invention, a log aggregation and preprocessing system based on a distributed architecture is provided, which further includes a version control module and a central management platform; The version control module is used to store the configuration information of the distributed acquisition module, including acquisition path, parsing rules and tags; The central management platform is used to acquire the configuration information and distribute it to the distributed acquisition module; The distributed collection module is also used to integrate a preset cloud environment interface to discover dynamic log sources and apply preset collection strategies, and to set resource usage thresholds, and to perform automatic degradation or circuit breaker operations when the current value of resource usage exceeds the resource usage threshold.

[0008] According to the present invention, a log aggregation and preprocessing system based on a distributed architecture is provided, wherein the preprocessing module is specifically used for: Construct a modular preprocessing pipeline, which contains multiple independent processing modules; Based on the type of the original data, the corresponding processing plugin is dynamically loaded from the pipeline, and preprocessing operations are performed to obtain preprocessed data; The preprocessing operations include: The raw data is processed by extracting key fields using a preset model, followed by data cleaning, filtering, and parsing. Query preset external data sources and add asset information, geographic location, or threat tags to the raw data; Sensitive fields in the original data are processed using reversible encryption or irreversible desensitization methods, with the encryption key coming from a preset KMS system.

[0009] According to the log aggregation and preprocessing system based on a distributed architecture provided by the present invention, the standardization transformation module is specifically used for: Build a hybrid processing architecture that includes a declarative field mapping rule base, a dynamic SQL parsing engine, and an intelligent transformation recommender; For preprocessed data of known types, the predefined rules in the declarative field mapping rule base are called to force the data to be converted into a standardized format that is strictly consistent with the unified data model before the data is written to storage, and a format verification report is generated. For preprocessed data of unknown type, the original structure is retained and stored as semi-structured data. When the semi-structured data is queried, it is parsed by a dynamic SQL parsing engine. The dynamic SQL parsing engine performs real-time parsing and field projection of the original structure of the semi-structured data using standard SQL syntax during the data query stage, and automatically feeds back the frequently queried field patterns to the declarative field mapping rule base. When new log data is accessed, structured transformation rules are recommended based on a pre-defined log template library using a text similarity algorithm.

[0010] According to the log aggregation and preprocessing system based on a distributed architecture provided by the present invention, the standardization conversion module is further used for: Based on a pre-defined log template library, a dual similarity matching process is performed using an optimized edit distance algorithm and a BERT-based semantic vector similarity algorithm to obtain the matching results. Based on the matching results, the optimal set of candidate structured transformation rules is recommended from the log template library; A rule simulation execution environment is provided for users to verify and adjust the recommended rules in the optimal structured transformation rule candidate set. Confirmed valid rules will be automatically included in the declarative field mapping rule base and marked as originating from intelligent recommendation.

[0011] According to the log aggregation and preprocessing system based on a distributed architecture provided by the present invention, the matching result includes distance similarity obtained based on the edit distance algorithm and semantic similarity obtained based on the semantic vector similarity algorithm of BERT; The standardization conversion module is also used for: The distance similarity and the semantic similarity are weighted and summed to obtain the comprehensive similarity. Based on the comprehensive similarity scores, the top N structured transformation rules are selected to form an initial candidate set. By comparing the field mapping logic of each structured transformation rule in the initial candidate set, redundant structured transformation rules with consistent field mapping logic are eliminated to obtain a deduplicated candidate set. The candidate set for deduplication is sorted a second time by combining the historical application indicators of the rules to obtain the optimal candidate set of structured transformation rules.

[0012] According to the log aggregation and preprocessing system based on a distributed architecture provided by the present invention, the standardization conversion module is further used for: By using a preset variable field rule base, the variable fields of the preprocessed data are identified and extracted, and the variable fields are replaced with corresponding preset placeholders to obtain mask data. Based on the mask data and the text content of each template in the log template library, the Levenshtein algorithm is used to calculate the edit distance value between the mask data and each template; The calculated edit distance values ​​are normalized to obtain the distance similarity between the preprocessed data and each template.

[0013] According to the log aggregation and preprocessing system based on a distributed architecture provided by the present invention, the standardization conversion module is further used for: The text fragments of the preprocessed data and the text fragments of each template in the log template library are respectively input into the pre-built BERT model for semantic recognition, so as to obtain the first semantic vector corresponding to the preprocessed data and the second semantic vector corresponding to each template. Calculate the cosine similarity between the first semantic vector and each of the second semantic vectors to obtain the semantic similarity between the preprocessed data and each template.

[0014] According to the log aggregation and preprocessing system based on a distributed architecture provided by the present invention, the storage module is further used for: Based on data time, access frequency, and compliance requirements, and combined with predefined data category labels, a multi-dimensional migration strategy is formed; Based on a multidimensional migration strategy, standardized structured data is automatically migrated between different storage layers.

[0015] According to the log aggregation and preprocessing system based on a distributed architecture provided by the present invention, a time-series prediction model is constructed to predict the access probability of the standardized structured data in a future preset time period based on historical access frequency, business cycle data and security event patterns. When the access probability is lower than a preset prediction threshold, a pre-migration operation is triggered for the standardized structured data to move from the current storage layer to the next lower storage layer.

[0016] The log aggregation and preprocessing system based on a distributed architecture provided by this invention collects raw data from different types of security devices and systems by deploying lightweight collection agents in multiple network areas with a unified configuration strategy. After processing by the preprocessing module, the data is sent to a message queue cache. Then, a preset unified data model is called to perform field mapping, type conversion, and enrichment to obtain standardized structured data. Subsequently, the data is stored hierarchically in hot storage, cold storage, and archive storage. A unified management service module generates a data service interface for application modules to call and perform security analysis, detection, and response operations. This effectively solves the data silo problem in the process of multi-source heterogeneous log aggregation, reduces network bandwidth pressure, avoids single point of failure risk, improves system reliability and scalability, and realizes standardized data processing, providing a high-quality and consistent data foundation for security analysis. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the structure of a log aggregation and preprocessing system based on a distributed architecture provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] Figure 1 This is a schematic diagram of the structure of a log aggregation and preprocessing system based on a distributed architecture provided in an embodiment of the present invention. For example... Figure 1 As shown, the log aggregation and preprocessing system based on a distributed architecture in this embodiment may include a distributed acquisition module 11, a preprocessing module 12, a caching module 13, a standardization conversion module 14, a storage module 15, a unified management service module 16, and an application module 17.

[0021] Among them, the distributed acquisition module 11 is used to deploy lightweight acquisition agents in multiple network areas to collect raw data from different types of security devices and systems with unified configuration strategies; Preprocessing module 12 is used to preprocess the raw data to obtain preprocessed data; Cache module 13 is used to send preprocessed data into a message queue for caching; The standardization conversion module 14 is used to call the preset unified data model to perform field mapping, type conversion and enrichment on the preprocessed data obtained from the message queue to obtain standardized structured data. Storage module 15 is used to hierarchically store standardized structured data into hot storage, cold storage and archive storage; The unified management service module 16 is used to read data from hot storage, cold storage and archive storage and generate data service interfaces; Application module 17 is used to obtain required data by calling the data service interface and to perform security analysis, detection and response operations.

[0022] In practical applications, the distributed acquisition module 11 can be understood as a component that deploys lightweight acquisition agents across multiple network areas to achieve data acquisition functionality. Specifically, the lightweight acquisition agent can be implemented using independently running scripts or embedded acquisition tools, such as acquisition scripts written in Python or agent-based acquisition tools. Its main purpose is to reduce the network bandwidth pressure caused by centralized acquisition and improve system scalability. Furthermore, a unified configuration policy can be distributed to each acquisition agent via configuration files or remote commands to ensure consistency in acquisition behavior.

[0023] The preprocessing module 12 can be understood as a functional unit that cleans, filters, and parses the raw data to generate preprocessed data. Specifically, the preprocessing process can be implemented through regular expression matching, string replacement, or simple logical judgments, such as deleting invalid characters, extracting key fields, and filtering duplicate data. Its main purpose is to provide structured input for subsequent standardization.

[0024] The caching module 13 can be understood as a component that buffers and stores preprocessed data through a message queue. Specifically, the message queue can be implemented using open-source message middleware, such as Kafka or RabbitMQ. Its main purpose is to smooth data flow and avoid data loss caused by differences in processing speed.

[0025] The standardization transformation module 14 can be understood as a functional unit that performs field mapping, type conversion, and enrichment on preprocessed data through a unified data model. Specifically, field mapping can be achieved by configuring mapping tables or writing transformation rules, such as mapping log fields from different sources to unified field names; type conversion can be achieved through data format validation and forced conversion, such as converting string timestamps to standard time formats; enrichment can be achieved by associating with external databases or calling API interfaces, such as supplementing asset information or threat intelligence tags. Its main purpose is to achieve seamless integration of multi-source heterogeneous data.

[0026] Storage module 15 can be understood as a functional unit that stores standardized structured data in hierarchical layers to different storage tiers. Specifically, hierarchical storage can be achieved by setting storage strategies, such as storing frequently accessed data on high-performance storage media and frequently accessed data on low-cost storage media based on data access frequency. Its main purpose is to optimize the utilization of storage resources.

[0027] The unified management service module 16 can be understood as a component that reads data from storage at all levels and generates data service interfaces. Specifically, the data service interfaces can be implemented through RESTful APIs or GraphQL, for example, providing a unified query interface for upper-layer applications to call. Its main purpose is to shield the complexity of the underlying storage and simplify the data access path.

[0028] Application module 17 can be understood as a functional unit that obtains required data by calling data service interfaces and performs security analysis, detection, and response operations. Specifically, security analysis can be achieved through correlation analysis or multi-dimensional statistics, such as analyzing abnormal behavior based on time series; detection can be achieved through rule matching or pattern recognition, such as detecting known threat patterns; and response can be achieved through automated scripts or manual intervention, such as triggering alarms or isolating infected devices. Its main purpose is to support the construction of a proactive defense system.

[0029] Specifically, the system achieves efficient aggregation, real-time processing, and standardization of massive heterogeneous log data across geographically dispersed and network-wide environments through the collaborative operation of multiple modules. The distributed acquisition module 11 deploys lightweight acquisition agents across multiple network regions. These agents collect raw data from different types of security devices and systems using a unified configuration strategy, avoiding the network bandwidth pressure and single-point-of-failure risks associated with centralized architectures, while ensuring consistency in acquisition behavior and adapting to the complexity of distributed networks. The preprocessing module 12 receives the collected raw data and performs cleaning, filtering, and parsing operations to obtain preprocessed data, thus solving the problem of diverse raw data formats and noise content, providing clean and structured input for subsequent standardization. The caching module 13 sends the preprocessed data to a message queue for caching. Due to the burstiness and imbalance of data streams, the message queue acts as a buffer layer to smooth traffic fluctuations, ensuring stable system operation under high load and avoiding data loss caused by differences in processing speed.

[0030] The standardization conversion module 14 invokes a preset unified data model to perform field mapping, type conversion, and enrichment operations on the preprocessed data obtained from the message queue, ultimately generating standardized structured data. Field mapping resolves the semantic differences between data from different sources, type conversion ensures data consistency, and enrichment supplements asset information or threat intelligence, thereby achieving seamless integration of multi-source data. The storage module 15, based on differences in data timeliness and access requirements, hierarchically stores the standardized structured data in hot storage, cold storage, and archive storage, optimizing storage resource utilization efficiency and meeting different needs for real-time analysis and long-term preservation. The unified management service module 16 reads data from each level of storage and generates a unified data service interface, shielding the complexity of the underlying storage and providing an efficient data access path for the application module 17. The application module 17 obtains the required data by calling the data service interface, performs security analysis, detection, and response operations, realizing rapid correlation of multi-source information and threat identification, supporting the construction of a proactive defense system.

[0031] Thus, through its distributed architecture and modular design, the system effectively addresses the challenges of aggregating and processing massive amounts of heterogeneous log data in cross-regional and multi-network environments, breaking down data silos and laying the foundation for unified security situation awareness.

[0032] The log aggregation and preprocessing system based on a distributed architecture of the present invention solves the problem of efficient aggregation and processing of massive heterogeneous log data in cross-regional and multi-network environments through distributed architecture and modular design. Furthermore, the present invention reduces network bandwidth pressure and single-point failure risk through the distributed acquisition module 11, solves the fusion difficulties caused by data heterogeneity through the preprocessing module 12, and achieves seamless fusion of multi-source data through the standardization conversion module 14, ultimately providing high-quality and consistent underlying data support for unified security situation awareness.

[0033] In some embodiments, the present invention further proposes a scheme that includes a version control module and a central management platform. The version control module is used to store configuration information for the distributed collection module 11, including collection paths, parsing rules, and tags. The central management platform is used to obtain the configuration information and distribute it to the distributed collection module 11. The distributed collection module 11 is also used to integrate preset cloud environment interfaces to discover dynamic log sources and apply preset collection strategies, and to set resource usage thresholds, performing automatic degradation or circuit breaker operations when the current resource usage value exceeds the resource usage threshold.

[0034] In practical applications, the version control module refers to a functional unit that centrally stores and manages configuration information, using Git or other version control systems to ensure the traceability and consistency of configuration data. Configuration information can be understood as a set of key parameters including collection paths, parsing rules, and tags, stored in JSON or YAML format to provide a unified configuration standard. The central management platform can be a microservice-based management component that distributes configuration information uniformly via message queues or RESTful APIs to improve operational efficiency and policy execution accuracy in multi-regional environments. The distributed collection module integrates a cloud environment interface, such as a Kubernetes API or a cloud vendor's SDK, to automatically identify new or changed log sources and dynamically adjust collection strategies. The resource usage threshold refers to the upper limit of resource consumption set during system operation. It can be monitored in real-time using tools like Prometheus, and protection mechanisms are executed when triggered to ensure system stability and robustness.

[0035] Specifically, this solution effectively addresses the issues of fragmented configuration management, insufficient adaptability to dynamic environments, and resource overload risks by integrating a version control module, a central management platform, and an enhanced distributed acquisition module 11. The version control module stores configuration information and ensures data consistency through centralized version management, preventing data collection interruptions caused by configuration errors or update delays during cross-network deployments. The central management platform acquires configuration information and distributes it to the distributed acquisition module 11, achieving unified distribution and real-time synchronization of configuration policies, reducing the need for manual intervention and improving operational efficiency. The distributed acquisition module 11 integrates cloud environment interfaces, utilizing automation capabilities to discover dynamic log sources and apply preset acquisition policies. This allows the system to adapt to topology changes in the cloud environment without manual reconfiguration, enhancing the flexibility and coverage of data collection. Simultaneously, the distributed acquisition module 11 sets resource usage thresholds and performs automatic degradation or circuit breaker operations when the current resource usage exceeds the threshold, preventing system crashes due to resource exhaustion and ensuring service continuity. These features work together to not only strengthen the centralization and automation capabilities of configuration management but also improve the system's adaptability and stability in complex and dynamic network environments.

[0036] In some embodiments, the present invention further proposes a preprocessing module 12 specifically used for: constructing a pluggable preprocessing pipeline, the pipeline containing multiple independent processing pluggins; dynamically loading the corresponding processing pluggins from the pipeline according to the type of the original data, and performing preprocessing operations to obtain preprocessed data; wherein, the preprocessing operations include: extracting key fields from the original data using a preset model, and cleaning, filtering, and parsing the data; the preset model includes Grok model or NLP model; querying preset external data sources to add asset information, geographical location, or threat tags to the original data; the external data sources include CMDB and threat intelligence IoC database; and processing sensitive fields in the original data using reversible encryption or irreversible desensitization methods, with the encryption key coming from a preset KMS system.

[0037] Specifically, a pluggable preprocessing pipeline refers to a modular architecture that can be implemented using microservices or a component-based framework. Its purpose is to improve system flexibility and scalability by decoupling individual processing logic units. Processing plug-ins are functional units capable of independently completing specific preprocessing tasks. They can be developed using scripting languages ​​(such as Python and Lua) or compiled languages ​​(such as C++ and Java) to support diverse data formats and parsing requirements. Dynamic loading refers to a mechanism that automatically selects and activates the corresponding plug-in based on the characteristics of the input data. This can be implemented based on rule engines or machine learning classifiers, aiming to ensure that the preprocessing process accurately adapts to the characteristics of different data sources. Pre-defined models refer to pre-defined and trained algorithmic tools, which can use regular expression templates (such as Grok patterns) or deep learning models (such as NLP models). Their purpose is to efficiently extract key fields from log data and understand contextual semantics. External data sources refer to third-party systems that store supplementary information. These can be configuration management databases (CMDBs) or threat intelligence platforms (IoC libraries), aiming to add rich contextual information to the raw data and enhance the depth of subsequent analysis. Reversible encryption or irreversible desensitization refers to security processing methods adopted for sensitive fields. These methods can be implemented using AES encryption algorithms or hash functions. The purpose is to ensure data privacy compliance while reducing security risks by centrally managing keys through a KMS system.

[0038] Specifically, the above solution constructs a pluggable preprocessing pipeline, decomposing the preprocessing function into multiple independent processing plug-ins, each focusing on a specific type of log data processing task. When the system receives heterogeneous raw data from multiple sources, it first dynamically loads the corresponding processing plug-in based on the data type. For example, for structured firewall logs, a parsing plug-in based on the Grok pattern is loaded first, while for unstructured terminal logs, an NLP model plug-in is called. In the key field extraction stage, the Grok pattern uses regular expression templates to quickly capture fields, while the NLP model uses semantic analysis to deeply understand the text content, thereby ensuring the accuracy and contextual relevance of field extraction. Subsequently, the system queries external data sources to add contextual information such as asset information, geographical location, or threat tags to the raw data, giving the log data richer semantic support. For sensitive fields, the system automatically selects reversible encryption or irreversible desensitization strategies based on the data sensitivity level, and centrally manages encryption keys through a KMS system, ensuring both privacy compliance and improved operational security. Overall, this solution, through dynamic adaptation and modular design, significantly enhances the flexibility and adaptability of the preprocessing process, providing a high-quality and consistent data foundation for subsequent standardized transformation and security analysis.

[0039] Furthermore, the design of the preprocessing module 12 works closely with functional modules such as the distributed acquisition module 11 and the standardization conversion module 14 to jointly solve the problem of efficient aggregation and processing of multi-source heterogeneous log data. Through a dynamic plugin loading mechanism, the system can flexibly respond to the access requirements of new log sources, avoiding the rigid processing problems caused by traditional fixed rules. At the same time, combining the context information of external data sources and the security handling strategies for sensitive fields not only improves the quality of the preprocessing results but also lays a solid foundation for the standardization conversion and in-depth analysis of subsequent modules.

[0040] In some embodiments, the present invention further proposes a standardized conversion module 14 specifically used for: constructing a hybrid processing architecture including a declarative field mapping rule base, a dynamic SQL parsing engine, and an intelligent conversion recommender; for preprocessed data of known types, calling predefined rules in the declarative field mapping rule base to forcibly convert the data into a standardized format strictly consistent with the unified data model before writing the data to storage, and generating a format verification report; for preprocessed data of unknown types, configuring it to retain the original structure and store it as semi-structured data, and parsing it through the dynamic SQL parsing engine when querying semi-structured data; wherein, during the data query stage, the dynamic SQL parsing engine performs real-time parsing and field projection of the original structure of the semi-structured data through standard SQL syntax, and automatically feeds back the frequently queried field patterns to the declarative field mapping rule base; when new log data is accessed, recommending structured conversion rules based on a preset log template library using a text similarity algorithm.

[0041] Specifically, a declarative field mapping rule base refers to a set of rules that define field mapping relationships declaratively. It can be implemented using JSON Schema or YAML format, aiming to provide flexible and scalable rule management capabilities. A dynamic SQL parsing engine can be understood as a parsing tool that supports standard SQL syntax. It can be implemented by integrating open-source SQL parsing libraries such as Apache Calcite, aiming to provide efficient query capabilities for semi-structured data. In practical applications, an intelligent transformation recommender is a recommendation system based on machine learning or rule matching. It can be implemented using collaborative filtering algorithms or content-based recommendation algorithms, aiming to accelerate the adaptation process to new log formats.

[0042] In detail, this solution achieves efficient processing of multi-source heterogeneous log data through a hybrid processing architecture. A declarative field mapping rule base handles the standardization transformation of known data types, ensuring data meets uniform format requirements before storage and generating format verification reports to guarantee data quality. A dynamic SQL parsing engine provides flexible query capabilities for unknown data types, avoiding the rigidity issues of traditional forced conversions, and continuously optimizes the rule base through a feedback mechanism based on high-frequency query patterns. An intelligent conversion recommender, combined with a log template library, uses text similarity algorithms to quickly generate conversion rules for new log formats, significantly reducing manual configuration costs. Furthermore, this solution works organically with the distributed acquisition module 11 and the preprocessing module 12, ensuring fully automated processing from data acquisition to standardization transformation, effectively solving the problem of processing unknown log data and improving the system's adaptability and reliability.

[0043] In some embodiments, the present invention further proposes that the standardization conversion module 14 is also used for: performing dual similarity matching based on a preset log template library using an optimized edit distance algorithm and a BERT-based semantic vector similarity algorithm to obtain matching results; recommending an optimal set of structured conversion rule candidates from the log template library based on the matching results; providing a rule simulation execution environment for users to verify and adjust the recommended rules in the optimal set of structured conversion rule candidates, and automatically including the confirmed valid rules in the declarative field mapping rule library and marking their source as intelligent recommendation.

[0044] Specifically, dual similarity matching refers to a comprehensive matching method that combines character-level difference quantification with deep semantic understanding. It uses the edit distance algorithm to accurately calculate differences in character arrangement within log text, while simultaneously combining it with the BERT semantic vector similarity algorithm to capture the deeper meaning of log content through contextual semantic association. This aims to overcome the limitations of single algorithms when handling complex log variations. The optimal structured transformation rule candidate set refers to a highly adaptable set of rules selected through comprehensive similarity scores. It achieves efficient rule selection through logical operations such as weighted summation, sorting, and deduplication, aiming to reduce the workload of manual rule selection and improve the targeting of recommendations. The rule simulation execution environment refers to a testing platform that supports real-time verification and dynamic adjustment. It can use sandbox technology or a virtualized environment to implement secure rule testing, aiming to ensure the accuracy and robustness of rule logic and avoid the risk of data processing failures due to rule errors.

[0045] In detail, the standardization conversion module 14 employs a dual similarity matching mechanism. First, it analyzes new log data based on a log template library. The edit distance algorithm quantifies character-level differences in the log text, while the BERT semantic vector similarity algorithm delves into the contextual semantic relationships of the log content. The combination of these two approaches generates matching results. Subsequently, based on the comprehensive similarity score in the matching results, high-scoring structured conversion rules are selected to form an initial candidate set. Redundant rules are then eliminated through field mapping logic comparison, ultimately yielding the optimal candidate set of structured conversion rules. On this basis, a rule simulation execution environment allows users to verify and adjust the recommended rules in the candidate set, testing the actual effectiveness of the rules against real data streams to ensure their logical accuracy and applicability. Confirmed valid rules are automatically added to the declarative field mapping rule library and marked as intelligent recommendation sources, thereby achieving automated accumulation and traceable management of rule knowledge. This process not only significantly improves the accuracy of structured conversion rule recommendations when new logs are accessed but also greatly reduces manual maintenance costs and enhances the system's self-learning capabilities. Furthermore, when combined with the aforementioned log aggregation and preprocessing system based on a distributed architecture, this solution can effectively address the issues of inaccurate rule recommendations and low implementation efficiency in processing multi-source heterogeneous log data, providing high-quality and consistent underlying data support for advanced security analysis and control.

[0046] In some embodiments, the present invention further proposes that the standardization conversion module 14 is also used for: weighted summation of distance similarity and semantic similarity to obtain a comprehensive similarity; sorting the comprehensive similarity scores in descending order and selecting the top N structured conversion rules to form an initial candidate set; comparing the field mapping logic of each structured conversion rule in the initial candidate set and eliminating redundant structured conversion rules with consistent field mapping logic to obtain a deduplicated candidate set; and sorting the deduplicated candidate set a second time in combination with the historical application indicators of the rules to obtain the optimal structured conversion rule candidate set.

[0047] In practical applications, the comprehensive similarity score refers to the numerical value calculated by fusing the results of the edit distance algorithm (which focuses on evaluating differences at the text character level) and the BERT semantic vector similarity algorithm (which captures deep semantic relationships) through weighted factors. This score can be achieved by setting different weight ratios. The initial candidate set can be understood as a set of highly matching rules selected after global ranking based on the comprehensive similarity score. Its purpose is to quickly focus on key options and balance computational efficiency with coverage. The deduplicated candidate set is a simplified set after eliminating duplicate rules by analyzing the consistency of field mapping logic. This can be achieved using techniques such as field mapping relationship comparison or hash verification, aiming to eliminate redundant items and ensure the diversity and uniqueness of the candidate set. Historical application metrics can be the success rate, response efficiency, or other quantitative feedback data of past applications. These metrics reflect the performance of the rules in real-world scenarios, thereby improving the practicality and adaptability of the recommendation results.

[0048] Specifically, this scheme first effectively integrates the complementary advantages of distance similarity and semantic similarity by weighted summation, avoiding evaluation bias caused by a single perspective and providing a more comprehensive and reliable matching basis for subsequent rule selection. Next, the scheme sorts the rules in descending order based on the comprehensive similarity score and selects the top N rules to form an initial candidate set. This operation not only reduces the burden of processing invalid data but also ensures that the candidate set is both efficient and retains key options. Subsequently, redundant rules are eliminated by comparing the field mapping logic of each rule in the initial candidate set, eliminating resource waste and decision interference caused by identical mapping logic, further optimizing the quality of the candidate set. Finally, a second sorting is performed based on the historical application indicators of the rules, prioritizing rules that have been verified and are reliable, significantly improving the accuracy and efficiency of novel log data transformation. The entire process, through refined processing of dual similarity data and optimization of the rule candidate set generation steps, significantly enhances the accuracy and practicality of structured transformation rule recommendations, while solving the problems of one-sided similarity evaluation, interference from redundant rules, and the inability of recommendation results to reflect actual application effects.

[0049] In some embodiments, the present invention further proposes a standardization conversion module 14, which is also used to: identify and extract variable fields of the preprocessed data through a preset variable field rule base, replace the variable fields with corresponding preset placeholders to obtain mask data; calculate the edit distance value between the mask data and each template using the Levenshtein algorithm based on the text content of each template in the log template library; and normalize the calculated edit distance value to obtain the distance similarity between the preprocessed data and each template.

[0050] Specifically, a variable field rule base refers to a database that stores dynamic fields that may appear in log data and their corresponding placeholder mapping relationships. It can be implemented using regular expression matching, keyword lists, or machine learning classification models, with the aim of accurately locating volatile content in logs. Masked data can be understood as data generated by uniformly replacing placeholders in the variable fields of the original log data. It can be generated using string replacement algorithms or template rendering tools, aiming to eliminate the impact of variable content differences on structural similarity assessment. The Levenshtein algorithm is a classic algorithm for calculating the edit distance between two strings. It can be implemented using dynamic programming methods or optimized matrix calculation techniques, aiming to quantify the textual structural differences between logs and templates. Normalization refers to mapping edit distance values ​​to a fixed range, which can be achieved through linear transformations or nonlinear functions, aiming to ensure that similarity indicators have comparability and standardization characteristics.

[0051] In detail, the above scheme, through the introduction of a variable field rule base, first identifies and extracts dynamic variable fields from log data, and replaces them with uniform preset placeholders to generate mask data. This process effectively avoids structural similarity assessment bias caused by frequent changes in variable content. Subsequently, based on the mask data and the text content of each template in the log template library, the Levenshtein algorithm is used to calculate the edit distance value, ensuring that the difference only comes from the structural features of the log and the template, rather than fluctuations in variable content. Finally, by normalizing the edit distance value, the influence of log length on the original distance is eliminated, making the distance similarity have a standardized range and comparability. The above scheme solves the problem of interference from dynamic variable fields in log data on structural similarity calculation, ensuring the reliability of template matching. At the same time, combined with the function of the aforementioned standardization conversion module 14, this scheme further improves the efficiency and accuracy of log data processing, laying a solid foundation for subsequent intelligent recommendation and rule optimization.

[0052] In some embodiments, the present invention further proposes the following technical solution: the standardization conversion module 14 is also used to input the text fragments of the preprocessed data and the text fragments of each template in the log template library into the pre-built BERT model for semantic recognition, to obtain the first semantic vector corresponding to the preprocessed data and the second semantic vector corresponding to each template; calculate the cosine similarity between the first semantic vector and each second semantic vector to obtain the semantic similarity between the preprocessed data and each template.

[0053] In practical applications, the BERT model refers to a deep learning language model based on the Transformer architecture, which can be implemented through pre-training and fine-tuning. By pre-training on large-scale corpora, the BERT model can capture context-related semantic information, thereby generating high-quality semantic vector representations for log text. The purpose of introducing the BERT model is to overcome the sensitivity of traditional methods to surface character differences, ensuring more accurate semantic matching. The first and second semantic vectors refer to vector representations generated from preprocessed data and templates in the log template library, respectively, which can be obtained by extracting features from the encoder output layer of the BERT model. The purpose of introducing these vectors is to provide a computable semantic representation, enabling a quantitative basis for evaluating the similarity between logs and templates. Furthermore, cosine similarity is a mathematical method for measuring the directional consistency of two vectors. It can be achieved by calculating the ratio of the vector dot product to its magnitude, focusing on the core consistency of semantic content rather than surface character differences.

[0054] Specifically, this solution addresses the issue of inaccurate semantic matching in log data by constructing a deep semantic recognition and quantitative evaluation mechanism. First, text fragments from preprocessed data and text fragments from various templates in the log template library are input into a pre-built BERT model, utilizing the BERT model's context-aware capabilities to generate semantic vectors. This process effectively overcomes the blind spots of traditional methods in recognizing technical terminology abbreviations, synonym variants, or dynamic fields, significantly improving the accuracy of semantic feature capture. Second, the generated first and second semantic vectors map the log text into a continuous semantic space, making previously difficult-to-comparate discrete text computable, particularly suitable for threat description variants frequently appearing in security logs. Finally, by calculating cosine similarity, the system can quantify the degree of semantic similarity based on the consistency of vector directions, avoiding misjudgments caused by differences in fixed patterns such as timestamps and IP addresses, thus providing a high-confidence semantic matching basis for structured transformation rule recommendations. The above solution, combined with the dual similarity matching mechanism in the pre-processed information, further improves the quality and reliability of structured transformation rule generation when new logs are accessed.

[0055] In some embodiments, the present invention further proposes that the storage module 15 is also used to: form a multi-dimensional migration strategy based on data time, access frequency and compliance requirements, and in combination with predefined category labels of the data; and automatically migrate standardized structured data between different storage layers according to the multi-dimensional migration strategy.

[0056] Specifically, a multi-dimensional migration strategy refers to a dynamic decision-making mechanism that comprehensively considers multiple factors to determine the data storage hierarchy. This can be implemented using rule engines, machine learning models, or hybrid algorithms. Among these, data time refers to the timestamp information of data generation or the last access, reflecting the timeliness of the data; access frequency refers to the number of times data is read or written per unit of time, measuring the data's usage frequency; compliance requirements refer to the legal, regulatory, or industry standard constraints related to data storage, ensuring that the migration process complies with dynamically changing legal boundaries; and predefined category labels refer to identification information that categorizes data types or uses, allowing for customized migration logic for different types of data.

[0057] In detail, this solution integrates information from multiple dimensions to construct a migration decision-making basis, thereby achieving intelligent allocation of storage resources. First, a multi-dimensional migration strategy is formed based on data time, access frequency, and compliance requirements, combined with predefined data category tags. This operation uses the time dimension to reflect the freshness of the data, access frequency to determine the data's activity level, compliance requirements to constrain the legality of migration behavior, and category tags to differentiate the characteristic requirements of different data types. For example, frequently accessed old data may be identified as hot data and retained in the high-performance storage layer, while infrequently accessed new data may be migrated to the cold storage layer to free up resources. Second, standardized structured data is automatically migrated between different storage layers according to the multi-dimensional migration strategy. This process uses a strategy engine to parse migration instructions in real time and trigger data flow, achieving dynamic optimization at the storage layer level. The automatic migration mechanism, based on the comprehensive judgment of the multi-dimensional strategy, retains active data in the hot storage layer to maintain low-latency access, and orderly migrates low-frequency data to the cold storage or archive layer to reduce storage costs, thus balancing performance and cost without manual intervention.

[0058] Furthermore, the aforementioned technical solution works seamlessly with other modules in the distributed log aggregation and preprocessing system. For example, after the standardization transformation module 14 completes the field mapping and type conversion of the data, the storage module 15 can dynamically allocate storage for the standardized structured data based on a multi-dimensional migration strategy, thereby ensuring that the flow of data between storage layers is both efficient and compliant. This design not only improves the overall performance of the system but also significantly reduces the operational risks and compliance vulnerabilities caused by manual operations.

[0059] In some embodiments, the present invention further proposes to construct a time-series prediction model, which predicts the access probability of standardized structured data in the future within a preset time period based on historical access frequency, business cycle data and security event patterns; when the access probability is lower than a preset prediction threshold, a pre-migration operation of standardized structured data from the current storage layer to the next level storage layer is triggered.

[0060] Specifically, time series forecasting models are mathematical tools capable of modeling and predicting time series data, which can be implemented using ARIMA models, LSTM neural networks, or the Prophet algorithm. In practical applications, historical access frequency refers to the statistical results of the number of times standardized structured data has been accessed over a past period. This can be extracted using log analysis tools and stored in a dedicated database for modeling purposes. Business cycle data can be understood as a set of data reflecting the daily operational patterns of an enterprise, such as the difference in access frequency between weekdays and holidays. Its purpose is to capture the impact of periodic fluctuations on access patterns. Security incident patterns refer to the trends in data access triggered by cybersecurity threats or abnormal behavior, which can be generated by correlating security alert logs with access records.

[0061] The preset time period refers to a time window set according to system performance requirements and business characteristics, such as 24 hours or 7 days. Its purpose is to refine the time dimension to more accurately predict access probabilities. The preset prediction threshold can be a dynamically adjusted value based on historical data distribution, and its purpose is to serve as the basis for triggering pre-migration operations. Standardized structured data refers to data processed by a unified data model, which has consistent field mapping rules and format specifications, facilitating efficient migration between different storage layers.

[0062] In detail, by constructing a time-series prediction model, historical access frequency, business cycle data, and security event patterns are incorporated into a unified analysis framework to form a comprehensive predictive capability. This model not only reflects regular access patterns but also captures potential dynamic evolution patterns, thus providing forward-looking evidence. During the prediction process, the system divides time windows according to preset time periods and generates access probability prediction results by combining multi-dimensional data. When the access probability falls below a preset prediction threshold, the system automatically triggers a pre-migration operation, migrating standardized structured data from the current storage layer to the next lower storage layer. This mechanism ensures that the migration action strictly matches the data value decay curve, avoiding the invalidation of hot storage space by low-frequency data, while reducing the risk of query response latency in cold storage.

[0063] Furthermore, the above solution, combined with storage module 15, significantly enhances the dynamic scheduling capability of storage resources. By introducing a prediction-driven migration mechanism, the system can proactively identify data that is about to enter a low-access period before peak access times, avoiding interference with real-time query performance by performing migration operations during high-load periods. This design not only optimizes the accuracy and intelligence of data flow between storage layers but also enhances the resilience and efficiency of the overall storage architecture, providing high-quality, consistent underlying data support for advanced security analysis and control.

[0064] It should be noted that all relevant information that may be involved in the various embodiments of the present invention is processed in strict accordance with the requirements of laws and regulations, following the principles of legality, legitimacy, and necessity, based on the reasonable purpose of the business scenario, and is information that users actively provide or generate during the use of the product / service, as well as information obtained with user authorization.

[0065] The information processed by this invention may vary depending on the specific product / service scenario and should be based on the specific scenario in which the user uses the product / service. This may involve user account information, device information, or other related information. This invention will treat the relevant information and its processing with the utmost diligence.

[0066] This invention places great importance on the security of related information and has adopted reasonable and feasible security protection measures that comply with industry standards to protect related information and prevent unauthorized access, public disclosure, use, modification, damage or loss of related information.

[0067] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A log aggregation and preprocessing system based on a distributed architecture, characterized in that, include: The distributed data collection module is used to deploy lightweight data collection agents in multiple network areas to collect raw data from different types of security devices and systems with a unified configuration strategy. The preprocessing module is used to preprocess the original data to obtain preprocessed data; The caching module is used to send the preprocessed data into a message queue for caching; The standardization conversion module is used to call a preset unified data model to perform field mapping, type conversion and enrichment on the preprocessed data obtained from the message queue to obtain standardized structured data. The storage module is used to hierarchically store the standardized structured data into hot storage, cold storage, and archive storage; The unified management service module is used to read data from hot storage, cold storage, and archive storage, and generate data service interfaces; The application module is used to obtain the required data by calling the data service interface and to perform security analysis, detection and response operations.

2. The log aggregation and preprocessing system based on a distributed architecture according to claim 1, characterized in that, It also includes a version control module and a central management platform; The version control module is used to store the configuration information of the distributed acquisition module, including acquisition path, parsing rules and tags; The central management platform is used to acquire the configuration information and distribute it to the distributed acquisition module; The distributed collection module is also used to integrate a preset cloud environment interface to discover dynamic log sources and apply preset collection strategies, and to set resource usage thresholds, and to perform automatic degradation or circuit breaker operations when the current value of resource usage exceeds the resource usage threshold.

3. The log aggregation and preprocessing system based on a distributed architecture according to claim 1, characterized in that, The preprocessing module is specifically used for: Construct a modular preprocessing pipeline, which contains multiple independent processing modules; Based on the type of the original data, the corresponding processing plugin is dynamically loaded from the pipeline, and preprocessing operations are performed to obtain preprocessed data; The preprocessing operations include: The raw data is processed by extracting key fields using a preset model, followed by data cleaning, filtering, and parsing. Query preset external data sources and add asset information, geographic location, or threat tags to the raw data; Sensitive fields in the original data are processed using reversible encryption or irreversible desensitization methods, with the encryption key coming from a preset KMS system.

4. The log aggregation and preprocessing system based on a distributed architecture according to claim 1, characterized in that, The standardization conversion module is specifically used for: Build a hybrid processing architecture that includes a declarative field mapping rule base, a dynamic SQL parsing engine, and an intelligent transformation recommender; For preprocessed data of known types, the predefined rules in the declarative field mapping rule base are called to force the data to be converted into a standardized format that is strictly consistent with the unified data model before the data is written to storage, and a format verification report is generated. For preprocessed data of unknown type, the original structure is retained and stored as semi-structured data. When the semi-structured data is queried, it is parsed by a dynamic SQL parsing engine. The dynamic SQL parsing engine performs real-time parsing and field projection of the original structure of the semi-structured data using standard SQL syntax during the data query stage, and automatically feeds back the frequently queried field patterns to the declarative field mapping rule base. When new log data is accessed, structured transformation rules are recommended based on a pre-defined log template library using a text similarity algorithm.

5. The log aggregation and preprocessing system based on a distributed architecture according to claim 4, characterized in that, The standardization conversion module is also used for: Based on a pre-defined log template library, a dual similarity matching process is performed using an optimized edit distance algorithm and a BERT-based semantic vector similarity algorithm to obtain the matching results. Based on the matching results, the optimal set of candidate structured transformation rules is recommended from the log template library; A rule simulation execution environment is provided for users to verify and adjust the recommended rules in the optimal structured transformation rule candidate set. Confirmed valid rules will be automatically included in the declarative field mapping rule base and marked as originating from intelligent recommendation.

6. The log aggregation and preprocessing system based on a distributed architecture according to claim 5, characterized in that, The matching results include distance similarity obtained based on the edit distance algorithm and semantic similarity obtained based on the BERT semantic vector similarity algorithm; The standardization conversion module is also used for: The distance similarity and the semantic similarity are weighted and summed to obtain the comprehensive similarity. Based on the comprehensive similarity scores, the top N structured transformation rules are selected to form an initial candidate set. By comparing the field mapping logic of each structured transformation rule in the initial candidate set, redundant structured transformation rules with consistent field mapping logic are eliminated to obtain a deduplicated candidate set. The candidate set for deduplication is sorted a second time by combining the historical application indicators of the rules to obtain the optimal candidate set of structured transformation rules.

7. The log aggregation and preprocessing system based on a distributed architecture according to claim 6, characterized in that, The standardization conversion module is also used for: By using a preset variable field rule base, the variable fields of the preprocessed data are identified and extracted, and the variable fields are replaced with corresponding preset placeholders to obtain mask data. Based on the mask data and the text content of each template in the log template library, the Levenshtein algorithm is used to calculate the edit distance value between the mask data and each template; The calculated edit distance values ​​are normalized to obtain the distance similarity between the preprocessed data and each template.

8. The log aggregation and preprocessing system based on a distributed architecture according to claim 6, characterized in that, The standardization conversion module is also used for: The text fragments of the preprocessed data and the text fragments of each template in the log template library are respectively input into the pre-built BERT model for semantic recognition, so as to obtain the first semantic vector corresponding to the preprocessed data and the second semantic vector corresponding to each template. Calculate the cosine similarity between the first semantic vector and each of the second semantic vectors to obtain the semantic similarity between the preprocessed data and each template.

9. The log aggregation and preprocessing system based on a distributed architecture according to any one of claims 1-8, characterized in that, The storage module is also used for: Based on data time, access frequency, and compliance requirements, and combined with predefined data category labels, a multi-dimensional migration strategy is formed; Based on a multidimensional migration strategy, standardized structured data is automatically migrated between different storage layers.

10. The log aggregation and preprocessing system based on a distributed architecture according to claim 9, characterized in that, A time-series prediction model is constructed to predict the probability of accessing the standardized structured data in the future within a preset time period, based on historical access frequency, business cycle data, and security event patterns. When the access probability is lower than a preset prediction threshold, a pre-migration operation is triggered for the standardized structured data to move from the current storage layer to the next lower storage layer.

Images