Method, device, equipment and program product for processing heterogeneous log data

Abstract

This invention discloses a method, apparatus, device, and program product for processing heterogeneous log data, relating to the field of log data processing technology. The method includes: acquiring raw log data and determining the log format type and sensitive information characteristics of each raw log data; filtering candidate parsers from a preset parser set; determining a target parser from the candidate parsers based on the target log format type and target sensitive information characteristics of the target raw log data; and performing de-sensitization processing on the target raw log data using the target parser to obtain de-sensitized target structured log data; outputting the target structured log data to a log storage system, and storing the correspondence between the target log format type, target sensitive information characteristics, and target parser in a parser selection record table. The solution of this invention can achieve a dynamic balance between log parsing security and processing performance.

Description

Technical Field

[0001] This invention relates to the field of log data processing technology, and in particular to a method, apparatus, device, and program product for processing heterogeneous log data. Background Technology

[0002] In modern information systems, log data serves as a core data source for system monitoring, troubleshooting, security auditing, and business analysis. Its processing efficiency and quality directly impact system stability and compliance. However, with increasing system architecture complexity and data scale expansion, log data exhibits characteristics such as heterogeneous formats, diverse sources, high risks of sensitive information, and stringent real-time requirements, posing a challenge to parsing technologies in balancing security, performance, and functionality.

[0003] Currently, all logs are mainly processed through a single regular expression engine, such as the PCRE2 engine. Although it can provide complete regular expression functionality, it is vulnerable to ReDoS (Regular Expression Denial of Service) attacks. This results in a failure to achieve a balance between functionality and security, and an inability to dynamically adapt to log characteristics and business scenario requirements. Summary of the Invention

[0004] This invention provides a method, apparatus, device, and program product for processing heterogeneous log data, so as to achieve a dynamic balance between log parsing security and processing performance, and can ensure the high availability of the parsing process and the effectiveness of quality assessment.

[0005] According to one aspect of the present invention, a method for processing heterogeneous log data is provided, the method comprising: Obtain raw log data and determine the log format type and sensitive information characteristics of each piece of raw log data; Based on the log format type, the sensitive information characteristics, and the current operating status of the log processing system, candidate parsers are selected from a preset parser set; wherein, the preset parser set contains multiple parsers adapted to different log formats; Based on the target log format type and target sensitive information characteristics of the target raw log data, a target parser is determined from each of the candidate parsers, and the target parser is used to perform desensitization processing on the target raw log data to obtain desensitized target structured log data; The target structured log data is output to the log storage system, and the correspondence between the target log format type, the target sensitive information features and the target parser is stored in the parser selection record table.

[0006] According to another aspect of the present invention, a processing apparatus for heterogeneous log data is provided, the apparatus comprising: The acquisition module is used to acquire raw log data and determine the log format type and sensitive information characteristics of each piece of raw log data. The candidate parser determination module is used to filter candidate parsers from a preset parser set based on the log format type, the sensitive information characteristics, and the current operating status of the log processing system; wherein, the preset parser set includes multiple parsers adapted to different log formats; The desensitization module is used to determine the target parser from the candidate parsers based on the target log format type and target sensitive information characteristics of the target original log data, and to perform desensitization processing on the target original log data through the target parser to obtain the desensitized target structured log data. The storage module is used to output the target structured log data to the log storage system and store the correspondence between the target log format type, the target sensitive information features and the target parser in the parser selection record table.

[0007] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the heterogeneous log data processing method according to any embodiment of the present invention.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the heterogeneous log data processing method according to any embodiment of the present invention.

[0009] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method for processing heterogeneous log data as described in any embodiment of the present invention.

[0010] The technical solution of this invention involves acquiring raw log data and determining the log format type and sensitive information characteristics of each raw log data. Based on the log format type, the sensitive information characteristics, and the current operating status of the log processing system, candidate parsers are selected from a preset parser set. The preset parser set includes multiple parsers adapted to different log formats. A target parser is determined from the candidate parsers according to the target log format type and target sensitive information characteristics of the target raw log data. The target parser is then used to de-identify the target raw log data, resulting in de-identified target structured log data. The target structured log data is output to a log storage system, and the correspondence between the target log format type, the target sensitive information characteristics, and the target parser is stored in a parser selection record table. This achieves a dynamic balance between log parsing security and processing performance, ensuring high availability and effective quality assessment during the parsing process.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a method for processing heterogeneous log data according to Embodiment 1 of the present invention; Figure 2 This is a flowchart of a method for processing heterogeneous log data according to Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the structure of a heterogeneous log data processing system provided in Embodiment 2 of the present invention; Figure 4 This is a schematic diagram of the structure of a heterogeneous log data processing device according to Embodiment 3 of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device that implements the heterogeneous log data processing method of the present invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] Example 1 Figure 1 This is a flowchart of a method for processing heterogeneous log data according to Embodiment 1 of the present invention. This embodiment is applicable to the processing of heterogeneous log data within the same log system. The method can be executed by a heterogeneous log data processing device, which can be implemented in hardware and / or software and can be configured in electronic devices such as computers, servers, or tablet computers. Figure 1 As shown, the method includes: Step 110: Obtain the raw log data and determine the log format type and sensitive information characteristics of each raw log data.

[0017] The raw log data can be a massive collection of records generated by different components, applications or network devices in a distributed system during operation. These data come from a wide range of sources, have different formats, and are in an initial state without standardization processing. For example, server access logs or application logs in the same system.

[0018] Log format type refers to the structural type of the log, such as JSON, CSV, XML, or custom key-value pair format. Sensitive information characteristics can be the data attributes in the log that need to be de-identified, including the category of sensitive information, its location in the log, and its risk level. For example, in the log "id_card=10000019xx03072316", the sensitive information characteristics are: category "ID number", location [9, 27], and risk level "high".

[0019] Optionally, in this embodiment, raw log data output by the target system can be received. This raw log data is highly heterogeneous and may include structured logs (e.g., key-value pairs enclosed in curly braces), semi-structured logs (e.g., row and column data separated by commas or tabs), and unstructured logs (e.g., system messages in free text format). In this embodiment, each log entry independently enters the subsequent processing flow.

[0020] In this embodiment, for each piece of raw log data, log format type determination and sensitive information feature extraction can be performed simultaneously. In log format type determination, a multi-path discrimination strategy can be adopted. First, the log content can be structurally validated, including checking whether it possesses the curly brace-enclosed structure and valid key-value pair syntax required for JSON format, or the closing tag and document root node required for XML format. Second, the consistency distribution of delimiters is analyzed to identify CSV / TSV type logs separated by commas, tabs, or vertical bars. Simultaneously, the entire log text is matched against a set of preset standard log templates. Each discrimination path independently calculates a confidence score, which comprehensively considers factors such as structural integrity, field quantity stability, and keyword frequency. Finally, the format with the highest confidence score is selected as the log format type.

[0021] For sensitive information feature extraction, a multi-layered identification mechanism can be adopted. The first layer is a global scan based on regular expressions, covering typical sensitive patterns such as ID card numbers, mobile phone numbers, bank card numbers, and email addresses. The second layer is field semantic analysis, which triggers enhanced verification of field values ​​by parsing keywords in field names. The third layer introduces contextual co-occurrence relationship evaluation; for example, when a numerical sequence appears near user documents or identity authentication contexts, its weight in being judged as sensitive information is increased. For each confirmed sensitive information, its start and end character positions in the original log string are precisely recorded, and its risk level is assessed as low, medium, or high based on factors such as information type, length, and whether it appears in combination with other sensitive fields. Finally, the category, location coordinates, and risk level of the sensitive information are combined to form a complete sensitive information feature, providing a basis for subsequent parser screening and de-identification processing.

[0022] Optionally, in this embodiment, determining the log format type and sensitive information characteristics of each original log data may include: performing character encoding detection on each original log data, and transcoding the log content of each log data into a unified character encoding format based on the detection results; processing the transcoded target log data through multiple preset format discrimination rules to obtain the matching confidence level between the target log data and each candidate log format, and determining the candidate log format corresponding to the maximum confidence level as the log format type of the target log data; wherein, the multiple preset format discrimination rules include: structural legality verification, delimiter pattern analysis, preset log template matching, and field width detection.

[0023] In another optional implementation of this embodiment, determining the log format type and sensitive information features of each original log data may include: processing the transcoded target log data through multiple preset sensitive information identification rules to obtain the sensitive information category, location coordinates, and risk level contained in the target log data; combining the sensitive information category, location coordinates, and risk level to determine the sensitive information features of the target log data; wherein the multiple preset sensitive information identification rules include: sensitive pattern matching, field name keyword identification, and sensitive information co-occurrence relationship evaluation.

[0024] Character encoding detection identifies the character set type used in the original log text, which can be achieved through parallel attempts with multiple encodings and confidence quantification. Unifying the character encoding format converts all log content to a standard character encoding, eliminating parsing errors or garbled text caused by encoding differences. Match confidence is a value between 0 and 1, quantifying the degree of match between the log and candidate formats; a higher value indicates a more reliable match. Structural validity verification checks whether the log conforms to the syntax of a specific format, such as verifying the balance of nested brackets in JSON or the closure of tags in XML. Delimiter pattern analysis analyzes the occurrence and consistency of delimiters (e.g., commas, tabs) in the log, used to identify row-column logs such as CSV / TSV. Preset log template matching compares the log content with a pre-stored standard log format template, calculating the match degree. Field width detection, for fixed-length logs, verifies whether each field strictly adheres to the preset byte length.

[0025] Sensitive information features are a set of features composed of sensitive information category, location coordinates, and risk level. Sensitive pattern matching identifies typical sensitive data using regular expressions and includes validation logic. Field name keyword recognition parses keywords in field names and triggers enhanced validation of field values. Sensitive information co-occurrence relationship evaluation analyzes the correlation of sensitive information in the log context to improve recognition accuracy.

[0026] Optionally, in this embodiment, for each piece of original log data, character encoding detection is performed first. This step employs a multi-encoding parallel detection mechanism, sequentially attempting to decode the log content using various common character sets. Based on character distribution statistics and the number of illegal character sequences, the recognition confidence score for each character set is calculated. The character set with the highest confidence score is selected as the original character set, and the log content is uniformly converted into a standard character encoding format. The conversion process synchronously records the original character set type, conversion operation, and confidence score, forming metadata.

[0027] Subsequently, multiple preset format discrimination rules are applied to the transcoded log content. The structure validity verification module checks whether the log conforms to specific syntax specifications; the delimiter pattern analysis module statistically analyzes the consistency of delimiter positional distribution in the log to identify row-column logs; the preset log template matching module compares the log with pre-stored standard log templates; and the field width detection module verifies whether the fields meet preset byte length constraints. Each rule independently outputs a matching confidence score in the range of 0 to 1, and finally, the candidate format corresponding to the highest confidence score is selected as the log format type.

[0028] Simultaneously, multiple preset sensitive information identification rules are applied to the transcoded log content. The sensitive pattern matching module identifies typical sensitive data through predefined rules; the field name keyword identification module parses keywords in field names, triggering enhanced field value verification; and the sensitive information co-occurrence relationship evaluation module analyzes contextual relevance to improve the identification rate of edge cases. For each confirmed sensitive piece of information, its category, the starting and ending character positions in the log string are precisely recorded, and a risk score is calculated based on the information type, quantity, and combination complexity. A risk score below 30 indicates low risk, 30 to 70 indicates medium risk, and above 70 indicates high risk. Finally, the category, location coordinates, and risk level are combined to form a complete sensitive information feature.

[0029] In one optional implementation of this embodiment, the raw log data is received and buffered through multiple input interfaces. After initial verification and filtering of invalid or malicious data, character encoding detection is performed: a multi-encoding parallel trial mechanism is adopted, and the identification confidence is calculated based on character distribution statistics and the proportion of illegal characters, uniformly converting the log content into a standard character encoding format. Subsequently, a preset format discrimination rule is applied to perform multi-dimensional detection on the transcoded log, calculate the matching confidence of each candidate format, and select the format corresponding to the highest confidence as the log format type. Based on the identified format type, an intelligent data sharding strategy is executed: structured data is sharded according to complete object boundaries, row-based data is sharded according to logical row boundaries and retains the table header, and streaming data is sharded according to a preset size threshold, with secondary semantic integrity optimization performed on shards exceeding the threshold. After sharding, data cleaning is performed to remove invalid characters, standardize newline characters, and process escape sequences. On this basis, structural features are extracted: the nesting depth is calculated, the number of fields is counted, the diversity of data types is analyzed, the complexity of reference relationships is evaluated, and data density evaluation and regularity checks are performed simultaneously. Synchronous content feature extraction: Typical sensitive data is identified through sensitive pattern matching, enhanced verification is triggered by keyword recognition of field names, and the accuracy of edge cases is improved by utilizing co-occurrence relationships. Sensitive information categories, location coordinates, and risk levels are accurately recorded to form sensitive information features. Finally, business feature extraction is performed: a business dictionary is loaded for keyword matching and importance assessment, business scenarios are categorized based on keyword combinations, timestamp formats are parsed and time interval distributions are calculated, and performance indicators are pre-extracted to provide structured feature input for subsequent dynamic parser selection and quality assessment.

[0030] Step 120: Based on the log format type, sensitive information characteristics, and the current running status of the log processing system, select candidate parsers from the preset parser set.

[0031] The preset parser set includes multiple parsers adapted to different log formats, such as a security parser, a high-performance parser, and a general-purpose parser.

[0032] The system operating status refers to the real-time resource usage of the log processing system, including CPU utilization, memory utilization, current throughput, and queue length. For example, CPU utilization is 85%, memory utilization is 70%, and current throughput is 2000 records / second.

[0033] Optionally, in this embodiment, after determining the log format type and sensitive information characteristics of each raw log data, the real-time operating status parameters of the current log processing system can be further obtained, including CPU utilization, memory usage, and current throughput. Subsequently, dynamic filtering logic is executed: when the sensitive information risk level is high and the system CPU utilization exceeds a preset threshold, a secure parser is preferentially selected from a preset parser set; when the risk level is low and the CPU utilization is below a preset threshold, a high-performance parser is preferentially selected; in other cases, a weighted score is calculated based on a comprehensive evaluation of risk level and system load, and the parser with the highest score is selected as the target parser. This filtering mechanism dynamically adjusts the parser selection strategy by real-time sensing of log characteristics and system status, ensuring that high-risk logs automatically use secure parsers in high-load systems to avoid attack risks, and low-risk logs preferentially use high-performance parsers in low-load systems to improve processing efficiency, thereby achieving an adaptive balance between security and performance.

[0034] For example, for JSON format logs containing high-risk sensitive information, when the processing load exceeds a preset threshold, a secure parser is automatically selected from a preset parser set; for CSV format logs containing low-risk sensitive information, when the processing load is below a preset threshold, a high-performance parser is prioritized, ensuring a dynamic adaptive match between security and performance.

[0035] Step 130: Determine the target parser from among the candidate parsers based on the target log format type and target sensitive information characteristics of the target original log data, and use the target parser to perform de-sensitization processing on the target original log data to obtain the de-sensitized target structured log data.

[0036] The target raw log data can be any raw log data, and this embodiment does not limit its log format type or the sensitive information features it contains.

[0037] Optionally, in this embodiment, after determining each candidate parser, a target parser can be further determined from each candidate parser based on the target log format type and target sensitive information characteristics of the target original log data, and the target parser can be used to perform desensitization processing on the target original log data to obtain desensitized target structured log data.

[0038] In this embodiment, the target log format type and target sensitive information characteristics of the target raw log data can be obtained first. Then, the parser matching logic is executed: if the target log format type is a structured format and the risk level of the target sensitive information characteristics is higher than a preset threshold, a secure parser is selected; if the target log format type is a line format and the risk level of the target sensitive information characteristics is lower than a preset threshold, a high-performance parser is selected; otherwise, a weighted comprehensive score is calculated based on the risk level and format compatibility, and the parser with the highest score is selected as the target parser.

[0039] After determining the target parser, further de-identification processing can be performed: the parser performs structured parsing of the original log data, locates the start and end positions of sensitive information in the log string, applies preset de-identification rules for secure transformation, and generates de-identified structured log data. This process simultaneously verifies field integrity, ensuring that non-sensitive fields remain unchanged, while only sensitive fields are securely processed, ultimately outputting structured logs that conform to business specifications.

[0040] For example, if the target original log data is a JSON format log containing sensitive information such as ID card number and a risk level of 75, the system determines that the target parser is a secure parser. The de-identification process converts the ID card number 123456789 into 1234****9, generating a de-identified structured log {"timestamp":"2025-03-01T14:20:10","user_id":12345,"id_card":"1234****9","action":"login"}.

[0041] Step 140: Output the target structured log data to the log storage system, and store the correspondence between the target log format type, the target sensitive information characteristics and the target parser in the parser selection record table.

[0042] The target structured log data consists of log data that has undergone anonymization processing. The log storage system is a platform used to persistently store structured logs, such as a distributed log database or message queue system. The parser selection record table is a structured data table that stores the correspondence between log format types, sensitive information characteristics, and parsers. For example, it might record "JSON format, ID number, risk level 75, parser RE2".

[0043] Optionally, in this embodiment, after de-identifying the target raw log data to obtain target structured log data, the target structured log data can be output to the log storage system. The de-identified log content can be written to the storage platform in a standard protocol format via a preset interface, ensuring data integrity during transmission and real-time access by subsequent systems. Simultaneously, the target log format type, target sensitive information features, and target parser identifier are extracted to construct record entries. These entries are stored in a parser selection record table via a transactional write mechanism. This table uses a key-value pair storage structure, where the key is a combination of format type and risk level, and the value is the parser identifier.

[0044] In one optional implementation of this embodiment, data validation can be performed before storage: for example, verifying whether the risk level is within the range of 0 to 100, and whether the parser identifier exists in a preset set. The writing process supports batch processing, and each record contains a unique timestamp and operation ID to ensure record traceability. Finally, the log storage system stores the log content, and the parser selects a record table to store the parsing decision basis; together, they constitute the complete data foundation of the log processing chain.

[0045] The technical solution of this embodiment obtains raw log data and determines the log format type and sensitive information characteristics of each raw log data. Based on the log format type, sensitive information characteristics, and the current operating status of the log processing system, candidate parsers are selected from a preset parser set. The preset parser set contains multiple parsers adapted to different log formats. A target parser is determined from the candidate parsers according to the target log format type and target sensitive information characteristics of the target raw log data. The target raw log data is then de-identified by the target parser to obtain de-identified target structured log data. The target structured log data is output to the log storage system, and the correspondence between the target log format type, target sensitive information characteristics, and target parser is stored in the parser selection record table. This achieves a dynamic balance between log parsing security and processing performance, ensuring high availability and effective quality assessment of the parsing process.

[0046] Example 2 Figure 2 This is a flowchart of a method for processing heterogeneous log data according to Embodiment 2 of the present invention. This embodiment is a further refinement of the above technical solution, and the technical solution in this embodiment can be combined with various optional solutions in one or more of the above embodiments. Figure 2 As shown, the method includes: Step 210: Obtain the raw log data and determine the log format type and sensitive information characteristics of each raw log data.

[0047] Step 220: Based on the log format type, select parsers that support the log format type from the preset parser set to obtain a format-compatible parser subset; based on the risk level in the sensitive information characteristics, select parsers with corresponding security features from the format-compatible parser subset to obtain a security-compliant parser subset; based on the data complexity and data volume of the original log data, select parsers with matching performance from the security-compliant parser subset to obtain a performance-adapted parser subset; based on the current log processing system's operating status, select parsers whose resource consumption meets the current system load constraints from the performance-adapted parser subset to obtain a system-state-adapted parser subset; prune the number of system-state-adapted parsers to obtain each candidate parser.

[0048] In one optional implementation of this embodiment, a subset of format-compatible parsers can first be selected from a preset parser set based on the log format type. For example, when the log format is JSON, parsers that support JSON are selected. Then, a subset of security-compliant parsers is selected based on the risk level of sensitive information characteristics: a secure parser is selected when the risk level exceeds 70; a high-performance parser is selected when the risk level is below 30; and a balanced parser is selected when the risk level is between 30 and 70. Further, a subset of performance-adaptive parsers can be selected based on data complexity and data volume. For example, the performance adaptation score formula can be nesting depth × 0.4 + number of records × 0.6, selecting the parser with the highest score. Then, a subset of system-state-adaptive parsers is selected based on system operating status: a low-resource-consumption parser is selected when CPU utilization exceeds 70%; a high-performance parser is selected when it is below 50%; and other cases are comprehensively evaluated based on the load index. Finally, the subset of system-state-adaptive parsers is pruned, for example, retaining the top three parsers to obtain a candidate parser list.

[0049] In practical implementation, the parser set includes parsers that support multiple formats. When the log format is JSON, all parsers that support JSON are selected. Furthermore, a subset of security-compliant parsers is selected based on the risk level of sensitive information characteristics: parsers with only security features are retained when the risk level exceeds 70; parsers with only high-performance features are retained when the risk level is below 30; and parsers with both security and performance features are retained when the risk level is between 30 and 70. Further, a subset of performance-adaptive parsers is selected based on data complexity and data volume, calculating a performance adaptation score (nesting depth × 0.4 + number of records × 0.6), and selecting the parser with the highest score. Furthermore, a subset of system state-adaptive parsers is selected based on system operating status (CPU utilization, memory usage): when CPU utilization exceeds 70%, parsers with resource consumption below a preset threshold are retained; when CPU utilization is below 50%, parsers with higher resource consumption are retained; other cases are comprehensively evaluated based on the system load index (CPU utilization × 0.5 + memory usage × 0.3 + throughput × 0.2). Finally, the subset of system state-adaptive parsers is pruned, for example, the top three parsers are retained after sorting by score from highest to lowest, forming a candidate parser list for subsequent parsing processing.

[0050] In one example of this embodiment, for a log data in JSON format, its sensitive information risk level is 75, the data complexity nesting depth is 3, the data volume is 500 records, and the system CPU utilization is 85%. First, based on the log format type JSON, parsers that support JSON are selected from a preset parser set, resulting in a subset of format-compatible parsers including parser A (secure), parser B (high-performance), and parser C (balanced). Then, based on the risk level of 75 exceeding the threshold of 70, a subset of security-compliant parsers is selected, retaining only parser A (which has security features and prevents regular expression denial-of-service attacks). Next, based on the data complexity nesting depth of 3 and the data volume of 500 records, the performance adaptation score is calculated: nesting depth × 0.4 + number of records × 0.6 = 3 × 0.4 + 500 × 0.6 = 1.2 + 300 = 301.2. Parser A has the highest score, therefore, parser A is determined to be the performance-adapted parser subset. Then, based on the threshold that the system CPU utilization rate exceeds 70% (85%), a subset of system state-adaptive parsers is selected, retaining only parser A whose resource consumption is below a preset threshold. Finally, the number of system state-adaptive parser subsets is pruned, and since only parser A remains, it is directly used as the sole candidate parser.

[0051] Step 230: Determine the target parser from among the candidate parsers based on the target log format type and target sensitive information characteristics of the target original log data, and use the target parser to perform desensitization processing on the target original log data to obtain the desensitized target structured log data.

[0052] Optionally, in this embodiment, a target parser is determined from each candidate parser based on the target log format type and the target sensitive information characteristics of the target original log data. The target parser is then used to de-identify the target original log data to obtain de-identified target structured log data. This may include: determining the scores of each candidate parser in different performance dimensions, and determining the comprehensive score of each candidate parser based on each score; determining the candidate parser with the highest comprehensive score as the target parser, and generating configuration parameters for the target parser. The configuration parameters include at least one of the following: security parameters, performance parameters, and functional parameters that match the security score, performance score, functional score, and resource consumption score, respectively; and parsing and de-identifying the target original log data based on the target parser and configuration parameters, outputting the de-identified target structured log data.

[0053] The performance dimension score refers to the parser's quantitative score across four dimensions: security, performance, functionality, and resource consumption, ranging from 0 to 100. For example, the security score is based on the ability to prevent regular expression denial-of-service attacks, with a score of 90 indicating high security; the performance score is based on processing speed testing, with a score of 85 indicating high throughput.

[0054] The overall score is a weighted average, with weights set according to business requirements, such as security weight 0.3, performance weight 0.4, functionality weight 0.2, and resource consumption weight 0.1. For example, if parser A scores 90 in security, 85 in performance, 80 in functionality, and 75 in resource consumption, its overall score is 90 × 0.3 + 85 × 0.4 + 80 × 0.2 + 75 × 0.1 = 27 + 34 + 16 + 7.5 = 84.5 points.

[0055] Configuration parameters are optimization settings for the target parser, and can include security parameters (e.g., maximum regular expression depth limit), performance parameters (e.g., number of concurrent threads), and feature parameters (e.g., list of supported regular expression features). For example, the security parameter is set to a maximum depth of 5, and the performance parameter is set to a number of concurrent threads of 10.

[0056] In an optional implementation of this embodiment, after determining the candidate parsers, the scores of each candidate parser can be further determined in four dimensions: security, performance, functionality, and resource consumption. Specifically, a security test is performed on each candidate parser, simulating a regular expression denial-of-service attack, and the number of successful defenses is recorded to calculate the security score; a performance test is performed, counting the number of log entries processed per second to calculate the performance score; the number of regular expression features supported by the parser is counted to calculate the functionality score; and CPU and memory usage are measured to calculate the resource consumption score. Subsequently, the comprehensive score is calculated using the formula: Comprehensive Score = Security Score × 0.3 + Performance Score × 0.4 + Functionality Score × 0.2 + Resource Consumption Score × 0.1, and the candidate parser with the highest comprehensive score is selected as the target parser. Configuration parameters are generated: the security parameter sets the maximum regular expression depth based on the security score (e.g., depth is 5 when the score is 85), the performance parameter sets the number of concurrent threads based on the performance score (e.g., number of threads is 10 when the score is 85), and the functionality parameter configures the list of supported regular expression features based on the functionality score (e.g., backreferences are included when the score is 80). Finally, the target parser and configuration parameters are used to parse the target raw log data, locate the start and end positions of sensitive information in the log string, apply de-identification rules, and generate de-identified structured log data.

[0057] Optionally, in this embodiment, determining the scores of each candidate parser across different performance dimensions and determining the comprehensive score of each candidate parser based on these scores may include: scoring each candidate parser on a security dimension, where the security dimension score is calculated based on at least one of the following: whether the response time of the candidate parser in processing malicious input exceeds a first threshold under a preset regular expression attack mode; whether the candidate parser has a memory boundary checking mechanism to prevent out-of-bounds access; and the accuracy of the candidate parser in identifying and de-identifying simulated sensitive information. Scoring each candidate parser on a performance dimension, where the performance dimension score is calculated based on at least one of the following: the unit data parsing time estimated based on a log data complexity model; the peak memory usage recorded in historical operations; and the number of log processing records per second under concurrent stress testing. The following steps are performed: First, each candidate parser is scored on a functional dimension, calculated based on at least one of the following: the number of supported log formats; support for backreferences, zero-width assertions, and advanced regular expression features; and error recovery success rate in case of input anomalies. Second, each candidate parser is scored on a resource consumption dimension, calculated based on at least one of the following: processor utilization measured in benchmark tests; total static and dynamic memory usage; and the number of disk read / write operations per thousand log entries. Third, the weights of the security, performance, functional, and resource consumption dimensions are adjusted based on the security risk level of the target raw log data, the current system load, and business priorities. Finally, each dimension score is multiplied by its corresponding adjusted weight, and the results are summed to obtain the overall score for each candidate parser.

[0058] In this embodiment, the system can be scored based on several factors, including: whether the response time for processing malicious input under a preset regular expression attack mode exceeds a preset threshold; whether a memory boundary check mechanism is available; and the accuracy of identifying and de-identifying simulated sensitive information. Performance is assessed based on the estimated unit data parsing time using a log data complexity model, the peak memory usage recorded in historical operations, and the number of log entries processed per second under concurrent stress testing. Functionality is assessed based on the number of supported log formats, support for advanced regular expression features such as backreferences, and the success rate of error recovery when input anomalies occur. Resource consumption is assessed based on processor utilization measured in benchmark tests, the total static and dynamic memory usage, and the number of disk read / write operations per thousand log entries. Subsequently, the weights of each dimension are dynamically adjusted according to the security risk level of the target raw log data, the current system load, and business priority. For example, when the security risk level is higher than 70, the security weight increases to 0.4; when the system CPU utilization exceeds 70%, the performance weight decreases to 0.3; and when the business priority is high, the functionality weight increases to 0.2. Finally, the scores for each dimension are multiplied by their corresponding adjusted weights and summed to obtain the comprehensive score of each candidate parser. The candidate parser with the highest comprehensive score is selected as the target parser.

[0059] Optionally, in this embodiment, this step may further include a circuit breaker protection mechanism. Specifically, the target parser may be initialized, and its security parameters, performance parameters, and functional parameters may be loaded. In an isolated execution environment, parsing processing with circuit breaker protection may be performed on the target raw log data, and execution time, memory usage, and processor utilization may be monitored in real time. When parsing execution timeout, memory usage exceeding a preset limit, or empty or invalid results are detected, a preset downgraded parser is invoked to re-execute the parsing processing on the target raw log data, obtaining a first parsing result and a second parsing result. The first and second parsing results are then subjected to quality assessments, including integrity scoring, accuracy scoring, consistency scoring, and timeliness scoring. Based on the quality assessment results, the parsing result with the highest quality score is selected as the target structured log data.

[0060] In one optional implementation of this embodiment, after determining the target parser, it can be initialized. During initialization, security parameters, performance parameters, and functional parameters can be extracted from a preset configuration library. The security parameter sets the maximum regular expression depth to 5 to prevent overly deep regular expressions from triggering regular expression denial-of-service attacks. The performance parameter configures the number of concurrent threads to 10 to improve log processing throughput. The functional parameter enables backtracking support to enhance regular expression matching capabilities. Subsequently, the target parser is deployed in a containerized isolated execution environment, and the parsing task is started. During parsing, execution time, memory usage, and processor utilization are monitored in real time: execution time is recorded using timestamps, memory usage is obtained through the operating system interface, and processor utilization is collected through CPU monitoring tools. When the execution time exceeds 500 milliseconds, memory usage exceeds 80%, or the parsing result is an empty string or has an incorrect format, a circuit breaker protection mechanism is triggered. The system automatically calls a preset downgrade parser to re-execute the parsing of the original target log data, generating a first parsing result and a second parsing result. The two parsing results were evaluated for quality as follows: Completeness score was calculated based on the number of missing fields, using the formula: (Total number of fields - Number of missing fields) / Total number of fields × 100; Accuracy score was calculated based on the correctness of data masking, using the formula: (Number of correctly masked fields / Total number of sensitive fields) × 100; Consistency score was calculated based on the formatting error rate, using the formula: (Number of error fields / Total number of fields) × 100; Timeliness score was calculated based on the ratio of time taken to the threshold, using the formula: (500 milliseconds / Actual time taken) × 100. Finally, the completeness score, accuracy score, consistency score, and timeliness score were multiplied by weights of 0.3, 0.3, 0.2, and 0.2 respectively, and then summed. The parsing result with the highest overall score was selected as the target structured log data.

[0061] Step 240: Output the target structured log data to the log storage system, and store the correspondence between the target log format type, the target sensitive information characteristics and the target parser in the parser selection record table.

[0062] To better understand the heterogeneous log data processing method involved in this embodiment, Figure 3 This is a schematic diagram of the structure of a heterogeneous log data processing system according to Embodiment 2 of the present invention, as shown below. Figure 3 As shown, the system may include: a preprocessing layer, a feature recognition layer, an intelligent routing layer, a parsing and execution layer, and a post-processing layer.

[0063] The preprocessing layer can include the following steps: First, raw log data is received via HTTP API, file upload, or message queue. A memory buffer queue is used for temporary data storage, supporting backpressure control. Then, character encoding detection is performed on the raw log data. A multi-encoding parallel trial mechanism is used to test common character sets such as UTF-8, GBK, and GB2312 sequentially. Encoding recognition confidence is calculated based on character distribution statistics and the number of illegal character sequences. Special encoding formats such as Base64 and Hex are identified and processed. All log content is uniformly converted to the standard UTF-8 format, and the original encoding type, conversion operation, and confidence score are recorded. Next, multi-format parallel detection is applied to the transcoded log content, including verifying the balance of key-value pair nested structures, analyzing the consistency of delimiter distribution, matching preset industry-standard format templates, identifying fixed field width patterns, calculating the matching confidence of each candidate format, extracting format features such as delimiter type, nesting depth, and field pattern, and selecting the format with the highest confidence as the primary processing type. Finally, intelligent data fragmentation is performed based on the identified format type. For structured data, fragmentation is performed according to complete object boundaries to ensure syntactic integrity. For row-based data, fragmentation is performed according to logical row boundaries and header information is retained. For streaming data, fragmentation is performed according to a 1-megabyte threshold. For fragments exceeding the threshold, secondary intelligent segmentation is performed to ensure semantic integrity. A unique identifier is generated for each fragment and its size, number of rows, and integrity status are recorded. At the same time, invalid characters are removed, newline characters are standardized to a uniform format, and backslash escape sequences are processed to complete data preprocessing.

[0064] The feature recognition layer can include the following processing steps: In structural feature extraction, the nesting depth is calculated to assess the complexity of the data structure, the number of fields is counted and the diversity of data types is analyzed, and the complexity of the reference relationships between data is assessed; the ratio of valid data to total data volume is calculated, the distribution of null and default values ​​is analyzed, and the degree of data redundancy is assessed; the consistency of field order is verified, the consistency of data types is checked, and the standardization of delimiter usage is confirmed; the Shannon entropy algorithm is used to quantify the randomness and information content of the data. In content feature extraction and sensitive information recognition, ID card numbers, mobile phone numbers, bank card numbers, email addresses, and IP addresses are matched through preset rules, and the recognition accuracy is improved by combining field name keywords and contextual environment. A risk score of 0 to 100 is calculated based on the information type and quantity, and the risk level is divided. The start and end positions of sensitive information in the text are accurately recorded. In business feature extraction and scenario recognition, a predefined business dictionary is loaded to support synonym expansion, the TF-IDF algorithm is used to assess the importance of keywords and perform business relevance scoring, and scenarios are classified into transaction, authentication, query, or system scenarios based on keyword combinations. Multiple timestamp formats are identified and the time interval distribution is calculated, and time-consuming, status, and count-related indicators are pre-extracted.

[0065] The intelligent routing layer can include the following processing steps: First, based on the identified log format type, parsers supporting that format are selected. Parsers with high security features are selected based on risk scores higher than 70. Parsers with matching performance are selected based on data complexity and size. Resource-friendly parsers are selected considering current processor utilization exceeding 80% or memory usage. Parsers that performed poorly in similar scenarios in the past are excluded based on historical performance data, ensuring the number of candidate parsers remains between 3 and 5. Then, each candidate parser is quantitatively scored across four dimensions: security, performance, functionality, and resource consumption. The security score assesses the ability to prevent regular expression denial-of-service attacks, memory boundary checks, and support for sensitive information masking. The performance score assesses parsing speed, memory usage efficiency, and concurrent processing capabilities. The functionality score assesses the types of formats supported, advanced regular expression support, and error handling capabilities. The resource consumption score assesses processor utilization, memory usage, and disk I / O requirements. Next, the weights are dynamically adjusted based on the real-time context. In high-risk scenarios, the security weight is increased to 0.45 and the performance weight is decreased to 0.35. In high-load scenarios, the performance weight is increased to 0.45 and the resource weight is increased to 0.35. In critical business scenarios, the function weight is increased to 0.35 and the accuracy weight is increased to 0.25. The weights are fine-tuned based on the recent decision-making effects to ensure that the weight sum is 1.0. Finally, the comprehensive score is calculated. An exploration and exploitation balance strategy is adopted. In the exploitation mode, the parser with the highest comprehensive score is selected. In the exploration mode, the parser ranked 2nd to 4th is randomly selected to collect performance data. Configuration parameters are generated, including maximum backtracking depth, cache size, and concurrency settings. Degradation trigger conditions and switching strategies are defined and prepared. The decision-making process is fully recorded, including the candidate set, score, weight, and reasons for selection.

[0066] The parsing execution layer can include the following processing steps: First, obtain and configure an instance of the selected parser, set the maximum backtracking depth to prevent regular expression denial-of-service attacks, configure a memory usage limit to prevent memory overflow, and enable boundary checks and overflow detection; set the cache size to optimize data processing efficiency, configure pre-compilation options to improve parsing speed, and adjust the number of concurrent threads to support multi-task parallel processing; enable or disable specific advanced regular expression features, configure the error handling mode as strict or lenient, and set result formatting options to ensure output standardization; set the execution timeout to 5000 milliseconds, configure the memory usage alarm threshold to 80%, and set the CPU usage limit to 90%; perform a warm-up operation to fill the cache and verify the configuration validity. Then, start the parsing task in the isolated execution environment, and monitor the execution time, memory usage, and processor usage in real time. When the execution time exceeds 5000 milliseconds, the memory usage exceeds 80%, or an empty result or format error is returned, the circuit breaker protection mechanism is triggered; the circuit breaker state is dynamically updated according to the execution result, remaining closed on success and entering a half-open or open state on failure. Simultaneously, performance indicators such as execution time, peak memory usage, and CPU time are collected to verify the basic validity of the results. If the main parser fails, a pre-defined fallback parser is invoked to re-execute the parsing and generate alternative results. Quality assessments are then performed on both the main parser results and the alternative results. The integrity score is calculated based on the required field fill rate, the accuracy score is calculated based on the accuracy of business rule verification, the consistency score is based on internal data logic consistency assessment, and the timeliness score is calculated based on the ratio of processing latency to a threshold. Based on the overall quality scores, the result with the highest score is selected as the target structured log data. If the quality is similar, the result with better performance is prioritized. The parser used, configuration parameters, execution timeline, problem details, and quality assessment results are fully recorded.

[0067] The post-processing layer may include the following steps: Selecting anonymization strategies based on sensitive information type, risk level, and business scenario; retaining the first six and last four digits of ID card numbers, the first three and last four digits of mobile phone numbers, the first six and last four digits of bank card numbers, and the first character before the @ symbol and the complete domain name of email addresses, achieving context-aware anonymization; fully anonymizing production logs to protect privacy, partially anonymizing debug logs to retain debug information, and storing audit logs with complete information encrypted, recording the anonymization operation type, location coordinates, and original value hash, and verifying that the anonymization results comply with GDPR and personal information protection laws. Subsequently, field standardization is performed, converting strings to number or date types, unifying the timestamp format to the ISO 8601 standard, standardizing the enumeration value to true or false, unifying numerical units, unifying field naming conventions, ensuring all text fields use UTF-8 encoding, standardizing newline characters to \n, and marking missing fields, outliers, and the severity of data quality issues. Next, intelligent calculation of business metrics is performed. The timeline is reconstructed by associating relevant log entries with transaction IDs, calculating the total processing time and the time spent at each stage, and statistically analyzing the average time spent, P50, P95, and P99 percentiles. Business success rates and failure reasons are analyzed, performance bottlenecks are identified, and optimization suggestions are provided. Metrics are aggregated by minute, hour, day time windows, and business type. Finally, integrity checks are performed to ensure required fields are filled, and consistency checks verify the internal logical consistency of the data. Data quality scores from 0 to 100, processing time, the parser used, and business insight information are appended, generating standardized JSON output, marking low-quality data, and providing remediation suggestions.

[0068] The solution of this invention automatically completes encoding conversion and format recognition through a preprocessing layer, extracts structural features and sensitive information through a feature recognition layer, dynamically optimizes parser selection through an intelligent routing decision module, supports seamless switching between multiple engines and has circuit breaker protection through a parsing execution layer, and implements context-aware desensitization and intelligent calculation of business indicators through a post-processing layer. Algorithm innovation ensures a dynamic balance between security and performance, data processing innovation improves the accuracy of sensitive information identification, and the execution framework enhances system robustness. Post-processing outputs high-quality structured logs, significantly reducing parsing failure rates, mitigating the risk of regular expression denial-of-service attacks, improving the accuracy of business indicator extraction, and providing a reliable data foundation for system monitoring and security auditing.

[0069] Example 3 Figure 4 This is a schematic diagram of a heterogeneous log data processing device according to Embodiment 3 of the present invention. Figure 4 As shown, the device includes: an acquisition module 410, a candidate parser determination module 420, a desensitization module 430, and a storage module 440.

[0070] The acquisition module 410 is used to acquire raw log data and determine the log format type and sensitive information characteristics of each piece of raw log data. The candidate parser determination module 420 is used to filter candidate parsers from a preset parser set based on the log format type, the sensitive information characteristics, and the current running status of the log processing system; wherein, the preset parser set includes multiple parsers adapted to different log formats; The desensitization module 430 is used to determine the target parser from each of the candidate parsers based on the target log format type and target sensitive information characteristics of the target original log data, and to perform desensitization processing on the target original log data through the target parser to obtain the desensitized target structured log data. The storage module 440 is used to output the target structured log data to the log storage system and store the correspondence between the target log format type, the target sensitive information features and the target parser in the parser selection record table.

[0071] In an optional implementation of this embodiment, the acquisition module 410 is specifically used to perform character encoding detection on each of the original log data, and to convert the log content of each of the log data into a unified character encoding format according to the detection result; The transcoded target log data is processed by multiple preset format discrimination rules to obtain the matching confidence of the target log data with each candidate log format, and the candidate log format corresponding to the maximum confidence is determined as the log format type of the target log data. The preset multiple format discrimination rules include: structural validity verification, delimiter pattern analysis, preset log template matching, and field width detection.

[0072] In an optional implementation of this embodiment, the acquisition module 410 is further specifically used to process the transcoded target log data through multiple preset sensitive information identification rules to obtain the sensitive information category, location coordinates and risk level contained in the target log data; The sensitive information category, location coordinates, and risk level are combined to determine the sensitive information features of the target log data; The preset multiple sensitive information identification rules include: sensitive pattern matching, field name keyword identification, and sensitive information co-occurrence relationship evaluation.

[0073] In an optional implementation of this embodiment, the candidate parser determination module 420 is specifically used to filter parsers that support the log format type from the preset parser set based on the log format type, and obtain a subset of format-compatible parsers; Based on the risk level in the sensitive information features, parsers with corresponding security features are selected from the subset of format-compatible parsers to obtain a subset of security-compliant parsers; Based on the data complexity and data volume of the original log data, a performance-matched parser subset is obtained by selecting parsers from the security compliance parser subset; Based on the current operating status of the log processing system, a subset of parsers whose resource consumption meets the current system load constraints is selected from the performance-adaptive parser subset to obtain the system state-adaptive parser subset. The number of the system state adaptation parser subset is reduced to obtain each of the candidate parsers.

[0074] In an optional implementation of this embodiment, the desensitization module 430 is specifically used to determine the scores of each candidate parser in different performance dimensions, and to determine the comprehensive score of each candidate parser based on each score; The candidate parser with the highest comprehensive score is determined as the target parser, and the configuration parameters of the target parser are generated. The configuration parameters include at least one of the following: security parameters, performance parameters, and functional parameters that match the security score, performance score, functional score, and resource consumption score, respectively. Based on the target parser and the configuration parameters, the target raw log data is parsed and sensitive information is de-identified, and the de-identified target structured log data is output.

[0075] In an optional implementation of this embodiment, the desensitization module 430 is further configured to score each of the candidate parsers on a security dimension, wherein the security dimension score is calculated based on at least one of the following: Under the preset regular expression attack mode, does the response time of the candidate parser in processing malicious input exceed the first threshold? Does the candidate parser have a memory boundary checking mechanism to prevent out-of-bounds access? The candidate parser's accuracy in identifying simulated sensitive information and its desensitization accuracy; Each candidate parser is scored on a performance dimension, and the performance dimension score is calculated based on at least one of the following: The unit data parsing time estimated based on the log data complexity model; Peak memory usage recorded during historical operation; Log messages processed per second under concurrent stress testing; Each candidate parser is scored on a functional dimension, and the functional dimension score is calculated based on at least one of the following: Number of supported log formats; Status of support for backreferences and advanced regular expression features such as zero-width assertions; Error recovery success rate when input errors occur; Each candidate parser is scored in terms of resource consumption, and the resource consumption score is calculated based on at least one of the following: Processor utilization as measured in benchmark tests; Total usage of static and dynamic memory; The number of disk read / write operations generated per thousand log entries; The weights of security, performance, functionality, and resource consumption dimensions are adjusted based on the security risk level of the target raw log data, the current system load status, and business priorities. The scores for each dimension are multiplied by their corresponding adjusted weights, and then summed to obtain the overall score for each candidate parser.

[0076] In an optional implementation of this embodiment, the heterogeneous log data processing device further includes: a target parser initialization module, used to initialize the target parser, load the security parameters, performance parameters and functional parameters of the target parser, perform parsing processing with circuit breaker protection on the target raw log data in an isolated execution environment, and monitor the execution time, memory usage and processor utilization in real time; When a parsing execution timeout is detected, memory usage exceeds the preset limit, or an empty or invalid result is returned, a preset downgrade parser is invoked to re-execute the parsing process on the target original log data to obtain a first parsing result and a second parsing result. The first and second parsing results are respectively subjected to quality assessment, which includes completeness score, accuracy score, consistency score and timeliness score; Based on the quality assessment results, the parsing result with the highest quality score is selected as the target structured log data.

[0077] The heterogeneous log data processing apparatus provided in the embodiments of the present invention can execute the heterogeneous log data processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0078] Example 4 Figure 5A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0079] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from the storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0080] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0081] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods described above, such as methods for processing heterogeneous log data.

[0082] In some embodiments, the method for processing heterogeneous log data may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the heterogeneous log data processing method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the heterogeneous log data processing method by any other suitable means (e.g., by means of firmware).

[0083] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0084] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0085] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM), optical fibers, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0086] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device for displaying information to the user (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0087] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0088] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system. It addresses the shortcomings of traditional physical hosts and Virtual Private Servers (VPS) in terms of management difficulty and weak business scalability.

[0089] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0090] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

[0091] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements a database detection method as provided in any embodiment of this application.

[0092] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including LANs or WANs—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0093] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, they do not mean that the solution has been or necessarily used.

[0094] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A method for processing heterogeneous log data, characterized in that, The method includes: Obtain raw log data and determine the log format type and sensitive information characteristics of each piece of raw log data; Based on the log format type, the sensitive information characteristics, and the current operating status of the log processing system, candidate parsers are selected from a preset parser set; wherein, the preset parser set contains multiple parsers adapted to different log formats; Based on the target log format type and target sensitive information characteristics of the target raw log data, a target parser is determined from each of the candidate parsers, and the target parser is used to perform desensitization processing on the target raw log data to obtain desensitized target structured log data; The target structured log data is output to the log storage system, and the correspondence between the target log format type, the target sensitive information features and the target parser is stored in the parser selection record table.

2. The method for processing heterogeneous log data according to claim 1, characterized in that, The step of determining the log format type and sensitive information characteristics of each of the original log data includes: Character encoding detection is performed on each of the original log data, and the log content of each log data is converted into a unified character encoding format based on the detection results; The transcoded target log data is processed by multiple preset format discrimination rules to obtain the matching confidence of the target log data with each candidate log format, and the candidate log format corresponding to the maximum confidence is determined as the log format type of the target log data. The preset multiple format discrimination rules include: structural validity verification, delimiter pattern analysis, preset log template matching, and field width detection.

3. The method for processing heterogeneous log data according to claim 2, characterized in that, The step of determining the log format type and sensitive information characteristics of each of the original log data includes: The transcoded target log data is processed by multiple preset sensitive information identification rules to obtain the sensitive information categories, location coordinates and risk levels contained in the target log data; The sensitive information category, location coordinates, and risk level are combined to determine the sensitive information features of the target log data; The preset multiple sensitive information identification rules include: sensitive pattern matching, field name keyword identification, and sensitive information co-occurrence relationship evaluation.

4. The method for processing heterogeneous log data according to claim 1, characterized in that, The step of selecting candidate parsers from a preset parser set based on the log format type, the sensitive information characteristics, and the current operating status of the log processing system includes: Based on the log format type, a subset of format-compatible parsers is obtained by filtering parsers that support the log format type from the preset parser set. Based on the risk level in the sensitive information features, parsers with corresponding security features are selected from the subset of format-compatible parsers to obtain a subset of security-compliant parsers; Based on the data complexity and data volume of the original log data, a performance-matched parser subset is obtained by selecting parsers from the security compliance parser subset; Based on the current operating status of the log processing system, a subset of parsers whose resource consumption meets the current system load constraints is selected from the performance-adaptive parser subset to obtain the system state-adaptive parser subset. The number of the system state adaptation parser subset is reduced to obtain each of the candidate parsers.

5. The method for processing heterogeneous log data according to claim 1, characterized in that, The process involves determining a target parser from among the candidate parsers based on the target log format type and target sensitive information characteristics of the target original log data, and then using the target parser to perform de-sensitization processing on the target original log data to obtain de-sensitized target structured log data, including: Each candidate parser is scored on different performance dimensions, and a comprehensive score for each candidate parser is determined based on each score. The candidate parser with the highest comprehensive score is determined as the target parser, and the configuration parameters of the target parser are generated. The configuration parameters include at least one of the following: security parameters, performance parameters, and functional parameters that match the security score, performance score, functional score, and resource consumption score, respectively. Based on the target parser and the configuration parameters, the target raw log data is parsed and sensitive information is de-identified, and the de-identified target structured log data is output.

6. The method for processing heterogeneous log data according to claim 5, characterized in that, The step of determining the scores of each candidate parser on different performance dimensions and determining the comprehensive score of each candidate parser based on each score includes: Each candidate parser is scored on a security dimension, and the security dimension score is calculated based on at least one of the following: Under the preset regular expression attack mode, does the response time of the candidate parser in processing malicious input exceed the first threshold? Does the candidate parser have a memory boundary checking mechanism to prevent out-of-bounds access? The candidate parser's accuracy in identifying simulated sensitive information and its desensitization accuracy; Each candidate parser is scored on a performance dimension, and the performance dimension score is calculated based on at least one of the following: The unit data parsing time estimated based on the log data complexity model; Peak memory usage recorded during historical operation; Log messages processed per second under concurrent stress testing; Each candidate parser is scored on a functional dimension, and the functional dimension score is calculated based on at least one of the following: Number of supported log formats; Status of support for backreferences and advanced regular expression features such as zero-width assertions; Error recovery success rate when input errors occur; Each candidate parser is scored in terms of resource consumption, and the resource consumption score is calculated based on at least one of the following: Processor utilization as measured in benchmark tests; Total usage of static and dynamic memory; The number of disk read / write operations generated per thousand log entries; The weights of security, performance, functionality, and resource consumption dimensions are adjusted based on the security risk level of the target raw log data, the current system load status, and business priorities. The scores for each dimension are multiplied by their corresponding adjusted weights, and then summed to obtain the overall score for each candidate parser.

7. The method for processing heterogeneous log data according to claim 1, characterized in that, After determining the target parser from the candidate parsers based on the target log format type and target sensitive information characteristics of the target raw log data, the method further includes: The target parser is initialized, and its security parameters, performance parameters, and functional parameters are loaded. In an isolated execution environment, the target raw log data is parsed with circuit breaker protection, and the execution time, memory usage, and processor utilization are monitored in real time. When a parsing execution timeout is detected, memory usage exceeds the preset limit, or an empty or invalid result is returned, a preset downgrade parser is invoked to re-execute the parsing process on the target original log data to obtain a first parsing result and a second parsing result. The first and second parsing results are respectively subjected to quality assessment, which includes completeness score, accuracy score, consistency score and timeliness score; Based on the quality assessment results, the parsing result with the highest quality score is selected as the target structured log data.

8. A processing apparatus for heterogeneous log data, characterized in that, include: The acquisition module is used to acquire raw log data and determine the log format type and sensitive information characteristics of each piece of raw log data. The candidate parser determination module is used to filter candidate parsers from a preset parser set based on the log format type, the sensitive information characteristics, and the current operating status of the log processing system; wherein, the preset parser set includes multiple parsers adapted to different log formats; The desensitization module is used to determine the target parser from the candidate parsers based on the target log format type and target sensitive information characteristics of the target original log data, and to perform desensitization processing on the target original log data through the target parser to obtain the desensitized target structured log data. The storage module is used to output the target structured log data to the log storage system and store the correspondence between the target log format type, the target sensitive information features and the target parser in the parser selection record table.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the method for processing heterogeneous log data according to any one of claims 1-7.

10. A computer program product comprising a computer program that, when executed by a processor, implements a method for processing heterogeneous log data according to any one of claims 1-7.

Images