Log data processing method and device, equipment and medium

Abstract

The embodiment of the invention provides a log data processing method and device, equipment and a medium. The method comprises the following steps: firstly, acquiring log data; then, text features, metadata features and behavior pattern features of the log data are extracted, and multi-dimensional feature vectors are constructed; then, the multi-dimensional feature vectors are input into a trained log data classification and grading model for processing, and a classification result and a risk grading result of the log data are obtained; and finally, based on the multi-dimensional feature vector, adjusting a risk grading result in a mode of combining a rule engine and a risk assessment model to obtain a classification grading result of the log data. By means of the method, the accuracy and practicability of log classification and grading are achieved.

Description

Technical Field

[0001] This application relates to the field of log data processing, and in particular to a log data processing method, apparatus, device and medium. Background Technology

[0002] In complex system environments characterized by multi-source heterogeneity, explosive growth, and dynamic evolution, classifying and grading log data is of paramount importance. Classification enables structured management based on log source, type, and content characteristics (e.g., security logs, application logs, system logs), improving the efficiency of log retrieval, analysis, and tracing. Grading allows for differentiated storage, access, and protection strategies based on the logs' importance, sensitivity, and impact, thereby ensuring the traceability of critical security information while reducing storage and computation costs. This classification and grading mechanism not only supports security audits and compliance requirements but also provides a fundamental guarantee for the stable, secure, and efficient operation of the system.

[0003] In existing technologies, common log data processing methods use manual rules to classify log data. Specifically, fixed rules are defined based on expert experience (such as regular expression matching keywords, log template extraction, field feature division, etc.) to map logs from different sources or types to preset categories, and importance levels are set according to the rules for subsequent auditing, analysis and storage management.

[0004] However, existing log data processing methods struggle to handle massive, multi-source, and dynamically changing log data based on fixed rules, leading to a decrease in classification accuracy. Summary of the Invention

[0005] This application provides log data processing methods, apparatus, devices, and media to solve the problem in the prior art that it is difficult to handle massive, multi-source, and dynamically changing log data according to fixed rules, resulting in a decrease in classification accuracy.

[0006] In a first aspect, embodiments of this application provide a log data processing method, including:

[0007] Retrieve log data;

[0008] Extract text features, metadata features, and behavioral pattern features from the log data to construct a multi-dimensional feature vector;

[0009] The multi-dimensional feature vectors are input into the trained log data classification and grading model for processing to obtain the classification results and risk grading results of the log data.

[0010] Based on the multi-dimensional feature vectors, the risk classification results are adjusted by combining a rule engine with a risk assessment model to obtain the classification and grading results of the log data.

[0011] In one possible implementation, the step of extracting text features, metadata features, and behavioral pattern features from the log data to construct a multi-dimensional feature vector includes:

[0012] The log data is vectorized using a trained language model to generate text features. The language model is obtained by learning the semantic relationships and structured expressions in the log data through unsupervised or self-supervised learning of historical log data.

[0013] Extract the structured fields from the log data and convert the structured fields into numerical features;

[0014] Based on the historical behavior baseline model, the behavioral pattern features corresponding to the log data are extracted. The historical behavior baseline model is based on the same type of log data collected within a preset time range, and a normal behavior distribution model of user operations, system events or resource access is constructed through statistical analysis.

[0015] The text features, numerical features, and behavioral pattern features are fused to generate a multi-dimensional feature vector.

[0016] In one possible implementation, calculating the behavioral pattern features corresponding to the log data based on the historical behavioral baseline model includes:

[0017] Based on the historical behavior baseline model, the time series features, resource access features, operation instruction sequences, and event triggering chains in the log data are compared and analyzed to calculate at least one feature value used to quantify the degree of deviation of the log data from the normal pattern, which is then used as the behavior pattern feature.

[0018] In one possible implementation, the training process of the log data classification and grading model includes:

[0019] Obtain a training dataset, which contains multiple sample log data labeled with log type and risk level labels;

[0020] For each of the sample log data, extract the multi-dimensional feature vector of the sample log data as the model input feature;

[0021] The log data classification and grading model is trained using the training dataset. The log classification and grading model is a neural network model that includes a shared parameter layer and a multi-task output layer. The shared parameter layer learns a general feature representation, and one sub-task output layer of the multi-task output layer learns to predict log type labels, while another sub-task output layer learns to predict risk level labels.

[0022] In one possible implementation, the step of adjusting the risk classification result based on the multi-dimensional feature vector by combining a rule engine with a risk assessment model to obtain the classification and grading result of the log data includes:

[0023] The multi-dimensional feature vector is input into the rule engine, which outputs an initial risk level. The rule engine performs condition matching and risk scoring on the log data based on the text content features, metadata features, and behavioral pattern features in the multi-dimensional feature vector, combined with a preset rule base, to determine the initial risk level.

[0024] Real-time risk indicators are obtained and input into a real-time risk assessment model to obtain risk correction coefficients. The risk indicators include the frequency of abnormal operations by the same user or the same address within a preset time period, system load rate, and abnormal access score of target resources. The real-time risk assessment model includes an input layer, at least one hidden layer, and an output layer. The input layer receives feature vectors used to characterize the real-time risks of log data. The hidden layer performs linear transformation and activation function calculation on the input vectors to generate hidden representations. The output layer outputs risk correction coefficients.

[0025] Based on the classification results, the initial risk level, and the risk correction coefficient, the classification and grading results of the log data are obtained.

[0026] In one possible implementation, the method further includes:

[0027] Collect feedback data, which includes both correctly graded and incorrectly graded results manually labeled;

[0028] The log data classification and grading model is incrementally trained based on the feedback data.

[0029] The rules in the rule engine are updated based on the feedback data.

[0030] In one possible implementation, prior to acquiring the log data, the method further includes:

[0031] Preprocessing operations are performed on multi-source log data that are structured, semi-structured, and unstructured. The preprocessing operations include log format standardization, missing field completion, and sensitive information de-identification.

[0032] Logs across services and nodes are sequence-aligned according to timestamps to generate log data that can be used for feature extraction and model processing.

[0033] Secondly, embodiments of this application provide a log data processing apparatus, comprising:

[0034] The first acquisition module is used to acquire log data;

[0035] The construction module is used to extract text features, metadata features, and behavioral pattern features from the log data and construct a multi-dimensional feature vector.

[0036] The second acquisition module is used to input the multi-dimensional feature vector into the trained log data classification and grading model for processing, and to obtain the classification result and risk grading result of the log data.

[0037] The processing module is used to adjust the risk classification results based on the multi-dimensional feature vector by combining a rule engine with a risk assessment model, so as to obtain the classification and grading results of the log data.

[0038] In one possible implementation, the building module is specifically used for:

[0039] The log data is vectorized using a trained language model to generate text features. The language model is obtained by learning the semantic relationships and structured expressions in the log data through unsupervised or self-supervised learning of historical log data.

[0040] Extract the structured fields from the log data and convert the structured fields into numerical features;

[0041] Based on the historical behavior baseline model, the behavioral pattern features corresponding to the log data are extracted. The historical behavior baseline model is based on the same type of log data collected within a preset time range, and a normal behavior distribution model of user operations, system events or resource access is constructed through statistical analysis.

[0042] The text features, numerical features, and behavioral pattern features are fused to generate a multi-dimensional feature vector.

[0043] In one possible implementation, the building module is specifically used for:

[0044] Based on the historical behavior baseline model, the time series features, resource access features, operation instruction sequences, and event triggering chains in the log data are compared and analyzed to calculate at least one feature value used to quantify the degree of deviation of the log data from the normal pattern, which is then used as the behavior pattern feature.

[0045] In one possible implementation, the log data processing apparatus further includes a training module for:

[0046] Obtain a training dataset, which contains multiple sample log data labeled with log type and risk level labels;

[0047] For each of the sample log data, extract the multi-dimensional feature vector of the sample log data as the model input feature;

[0048] The log data classification and grading model is trained using the training dataset. The log classification and grading model is a neural network model that includes a shared parameter layer and a multi-task output layer. The shared parameter layer learns a general feature representation, and one sub-task output layer of the multi-task output layer learns to predict log type labels, while another sub-task output layer learns to predict risk level labels.

[0049] In one possible implementation, the processing module is specifically used for:

[0050] The multi-dimensional feature vector is input into the rule engine, which outputs an initial risk level. The rule engine performs condition matching and risk scoring on the log data based on the text content features, metadata features, and behavioral pattern features in the multi-dimensional feature vector, combined with a preset rule base, to determine the initial risk level.

[0051] Real-time risk indicators are obtained and input into a real-time risk assessment model to obtain risk correction coefficients. The risk indicators include the frequency of abnormal operations by the same user or the same address within a preset time period, system load rate, and abnormal access score of target resources. The real-time risk assessment model includes an input layer, at least one hidden layer, and an output layer. The input layer receives feature vectors used to characterize the real-time risks of log data. The hidden layer performs linear transformation and activation function calculation on the input vectors to generate hidden representations. The output layer outputs risk correction coefficients.

[0052] Based on the classification results, the initial risk level, and the risk correction coefficient, the classification and grading results of the log data are obtained.

[0053] In one possible implementation, the log data processing device further includes a second processing module for:

[0054] Collect feedback data, which includes both correctly graded and incorrectly graded results manually labeled;

[0055] The log data classification and grading model is incrementally trained based on the feedback data.

[0056] The rules in the rule engine are updated based on the feedback data.

[0057] In one possible implementation, before acquiring the log data, the second processing module is further configured to:

[0058] Preprocessing operations are performed on multi-source log data that are structured, semi-structured, and unstructured. The preprocessing operations include log format standardization, missing field completion, and sensitive information de-identification.

[0059] Logs across services and nodes are sequence-aligned according to timestamps to generate log data that can be used for feature extraction and model processing.

[0060] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0061] The memory stores computer-executed instructions;

[0062] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0063] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0064] The log data processing method, apparatus, device, and medium provided in this application first acquire log data to provide basic information for subsequent analysis. Then, multi-dimensional feature extraction is performed on the acquired log data, including text feature extraction, metadata feature extraction, and behavioral pattern feature extraction. Next, the extracted features are constructed into a multi-dimensional feature vector and input into a trained log data classification and grading model. This model uses machine learning algorithms to process the feature vector and automatically outputs the classification results and preliminary risk grading results of the log data, achieving efficient and intelligent analysis of massive logs. Finally, to further improve accuracy, the preliminary risk grading results are dynamically adjusted based on the multi-dimensional feature vector, combined with a rule engine and a risk assessment model. This allows for refined modification of known security policies or business rules using the rule engine, and captures complex potential risk patterns through the risk assessment model, ultimately generating accurate and actionable log classification and risk grading results. Therefore, this method, through a combination of multi-dimensional feature fusion, model inference, and rule engine optimization, achieves a comprehensive understanding, accurate classification, and intelligent grading of log data, improving the efficiency and accuracy of security monitoring, while also supporting timely detection and risk warning of abnormal behavior. Attached Figure Description

[0065] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0066] Figure 1 A flowchart illustrating the log data processing method provided in this application embodiment. Figure 1 ;

[0067] Figure 2 A flowchart illustrating the log data processing method provided in this application embodiment. Figure 2 ;

[0068] Figure 3 This is a schematic diagram of the structure of the log data processing device provided in the embodiments of this application;

[0069] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0070] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0071] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0072] In complex system environments characterized by multi-source heterogeneity, explosive growth, and dynamic evolution, classifying and grading log data is of paramount importance. Classification enables structured management based on log source, type, and content characteristics (e.g., security logs, application logs, system logs), improving the efficiency of log retrieval, analysis, and tracing. Grading allows for differentiated storage, access, and protection strategies based on the logs' importance, sensitivity, and impact, thereby ensuring the traceability of critical security information while reducing storage and computation costs. This classification and grading mechanism not only supports security audits and compliance requirements but also provides a fundamental guarantee for the stable, secure, and efficient operation of the system.

[0073] In existing technologies, common log data processing methods use manual rules to classify log data. Specifically, fixed rules are defined based on expert experience (such as regular expression matching keywords, log template extraction, field feature division, etc.) to map logs from different sources or types to preset categories, and importance levels are set according to the rules for subsequent auditing, analysis and storage management.

[0074] However, existing log data processing methods rely on expert experience, which incurs high update and maintenance costs. Rules are prone to becoming invalid when system architecture or log formats change. Secondly, fixed rules struggle to handle massive, multi-source, and dynamically changing log data, leading to decreased classification accuracy and coarse-grained hierarchical granularity, making it difficult to support automated and intelligent analysis needs. Furthermore, rule matching fails to capture implicit semantics or abnormal patterns, making it difficult to discover potential security risks or complex event correlations, thus limiting the effectiveness of log data in security auditing and risk prediction.

[0075] Based on this, this application proposes a log data processing method. Addressing the problems of existing log data processing methods relying on expert experience, high cost of rule updates and maintenance, and susceptibility to failure when system architecture or log format changes, the inventors, based on multidimensional feature analysis and machine learning principles, first acquire log data and vectorize the log text content. Simultaneously, they extract metadata features such as log time, source, and event type, as well as behavioral pattern features based on a historical behavior baseline model, constructing a multidimensional feature vector. Subsequently, this feature vector is input into a trained log classification and grading model to obtain preliminary log classification and risk grading results. Finally, by combining a rule engine and a risk assessment model, the risk grading results are dynamically adjusted, achieving fine-grained optimization of known rules and the capture of complex potential risk patterns. This method not only effectively solves the limitations of fixed rules in handling massive, multi-source, and dynamically changing logs, improving classification accuracy and grading precision, but also identifies implicit semantics and abnormal behavior, significantly enhancing the application effect of log data in security auditing, risk prediction, and intelligent analysis, achieving automated, efficient, and accurate log management.

[0076] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0077] Figure 1 A flowchart illustrating the log data processing method provided in this application embodiment. Figure 1 ;like Figure 1 As shown, the method includes:

[0078] S101. Obtain log data.

[0079] It should be noted that the raw multi-source log data needs to be processed before it can be obtained. Specifically:

[0080] First, preprocessing operations are performed on the multi-source log data, including structured, semi-structured, and unstructured data. Then, logs from different services and nodes are aligned according to their timestamps to generate log data that can be used for feature extraction and model processing.

[0081] The preprocessing operations include log format standardization, missing field completion, and sensitive information de-identification.

[0082] It should be understood that, in this embodiment, step S101 aims to acquire log data from a complex information system environment and perform structured processing to ensure the effectiveness and accuracy of subsequent feature extraction and model inference. Since real-world systems typically contain multiple types and formats of log sources, such as server logs, application logs, database access logs, and security audit logs, directly using them as model input not only results in inconsistent formats but also risks such as missing fields and exposure of sensitive information, leading to model malfunctions or other risks. Therefore, a preprocessing mechanism is needed to unify, standardize, and analyze multi-source logs, laying a reliable data foundation for the entire classification and grading process.

[0083] In the specific implementation, the first step is to perform log format standardization on the acquired structured, semi-structured, and unstructured logs. For structured logs such as JSON or XML, fields are completed using a field mapping table, and field names are standardized. For semi-structured logs such as those from Apache and Nginx, key fields, such as timestamps, URLs, and status codes, are extracted using regular expression templates. For unstructured logs (such as debug output or user-defined text), keyword matching, syntactic analysis, or template generation algorithms (such as the Drain and Spell methods based on log template extraction) can be used to convert them into a semi-structured format. For example, "User 123 delete file / tmp / a.txt failed" can be converted to "User [ID] performed [operation] accessed [resource] result [status]".

[0084] Through standardized processing, all types of logs have a unified field structure, which not only facilitates the model to extract textual and numerical features, but also avoids model misjudgment due to format differences, thereby significantly improving the accuracy of subsequent classification and grading tasks.

[0085] In some log sources, due to differences in system design or recording strategies, situations may occur such as missing source service fields, unlabeled operation types, and missing request link IDs. This embodiment automatically completes key fields through field inference algorithms and context completion strategies. For example, when a log entry does not record a service name, the service to which it belongs can be mapped based on the IP of the node that generated it; when the operation type is missing, its operation intent can be inferred through the operation path or text keywords (such as GET / POST, delete, update); when a transaction ID is not included, the link identifier can be reconstructed based on time proximity and call link information.

[0086] Through the above-described completion process, it can be ensured that all log entries can fully express the operation behavior, avoid deviations in behavioral pattern characteristics due to missing fields, and ensure the stable and reliable quality of model input data.

[0087] Furthermore, to prevent the leakage of user privacy information in the logs, this embodiment performs anonymization processing on phone numbers, ID card numbers, email addresses, access tokens, etc. in the logs. For example, the phone number "13812341234" can be processed into "138****1234"; the ID card number can be hashed using SHA-256 for identity association but cannot be reverse-parsed; the access token can be replaced with a fixed-length random string. Through anonymization measures, the risk of sensitive information leakage is effectively avoided without affecting the model's recognition of behavioral patterns and user association, thereby improving system security and compliance and meeting data protection regulations.

[0088] Finally, in distributed systems, the same business process often spans multiple services and nodes, so a single log entry cannot fully describe the system behavior. This embodiment aligns logs from different sources according to their time sequence based on the precise timestamps (accurate to milliseconds) and transaction IDs (such as TraceID, SpanID, RequestID) carried in the logs. This sequence alignment not only enhances the model's understanding of the behavioral context, making behavioral pattern features more representative, but also effectively identifies anomalies in the chain, improving the accuracy of risk detection.

[0089] It is understood that, through the above preprocessing steps, this embodiment of the application achieves unified processing of heterogeneous, multi-source, and dynamically changing logs, eliminating noise factors caused by format differences and field inconsistencies, and ensuring that all log data has a standard structure and can be used for model analysis. Furthermore, sensitive information desensitization and time series alignment further enhance data security and analytical accuracy, providing high-quality input data for subsequent feature extraction and classification / grading models, thereby significantly improving the stability, accuracy, and security of the entire log processing method.

[0090] S102. Extract text features, metadata features, and behavioral pattern features from log data to construct a multi-dimensional feature vector.

[0091] Understandably, in this embodiment, multi-type, highly expressive feature information is extracted from preprocessed log data, and a unified multi-dimensional feature vector is constructed to provide sufficient input for subsequent classification and risk grading models. Since log content simultaneously contains various information such as textual semantics, structured attributes, and behavioral patterns, relying solely on a single type of feature often fails to accurately depict the behavioral essence of the log. Therefore, this embodiment utilizes a three-dimensional fusion structure of textual features, metadata features, and behavioral pattern features to enable the feature vector to comprehensively reflect the semantic, structural information, and anomaly degree of the log, thereby improving the accuracy and robustness of the model's judgment.

[0092] It should be noted that the specific implementation method in this embodiment will be... Figure 2 The embodiments are described in detail, and will not be repeated in this embodiment.

[0093] S103. Input the multi-dimensional feature vectors into the trained log data classification and grading model for processing to obtain the classification results and risk grading results of the log data.

[0094] In one feasible approach, the training process of the log data classification and grading model includes:

[0095] First, obtain the training dataset; then, for each sample log data, extract the multi-dimensional feature vector of the sample log data as the input features of the model; finally, use the training dataset to train the log data classification and grading model.

[0096] The training dataset contains multiple sample log data labeled with log type and risk level labels. The log classification and grading model is a neural network model containing a shared parameter layer and a multi-task output layer. The shared parameter layer learns a general feature representation, and one sub-task output layer of the multi-task output layer learns to predict the log type label, while the other sub-task output layer learns to predict the risk level label.

[0097] It should be understood that in actual implementation, the first step is to construct a training dataset for model training. The training dataset includes multiple sample log entries that have been manually reviewed or labeled using empirical rules. Each sample contains two labels: a log type label, indicating whether the log belongs to a category such as system log, access log, error log, or security event log; and a risk level label, indicating the risk level of the log event in terms of security risk, business impact, and stability impact, such as low risk, medium risk, high risk, or severe risk. By providing dual labels for each log entry in the training dataset, the model can simultaneously learn log type recognition and risk classification tasks, thereby enhancing the model's understanding of log semantics and improving the reliability of risk judgment.

[0098] During training, for each sample log in the training dataset, text features, metadata features, and behavioral pattern features are extracted according to step S102, and these three types of features are fused into a unified multi-dimensional feature vector. This feature vector serves as the input data for the model and reflects the comprehensive characteristics of the log in terms of semantics, structure, contextual behavior, and temporal relationships. For example, for a log related to access anomalies, its text features can reflect key information fields (such as error messages, request parameter structures, etc.); metadata features can reflect the log's source host, generating module, priority, etc.; and behavioral pattern features can reflect the degree of deviation between this access behavior and past normal behavior, and whether the access frequency is abnormal, etc.

[0099] It should be noted that the log classification and grading model employs a neural network structure containing a shared parameter layer and a multi-task output layer. The shared parameter layer is used to learn a general feature representation of the input log data, typically composed of multiple neural networks, such as convolutional neural networks, multilayer perceptrons, and Transformer encoders. The shared parameter layer can simultaneously provide basic feature representations for both log classification and risk grading tasks. Based on the shared feature representation, two independent output sub-task layers are set: a log type prediction sub-task output layer and a risk level prediction sub-task output layer. The log type prediction sub-task output layer receives features from the shared layer, learns the classification rules for different log categories, and outputs log type labels through Softmax. The risk level prediction sub-task output layer, also based on features from the shared layer, learns the risk level patterns corresponding to log events and outputs risk level labels through Softmax or regression.

[0100] The main steps of the model training phase are as follows: First, a training dataset containing log type labels and risk level labels is obtained, and text features, metadata features, and behavioral pattern features are extracted for each sample log to construct a multi-dimensional feature vector as the model input. Then, the feature vector is input into a neural network model containing a shared parameter layer and a multi-task output layer. The shared parameter layer learns a general feature representation of the log data, and the log type prediction sub-task output layer and the risk level prediction sub-task output layer output the corresponding prediction results, respectively. During training, a joint loss is constructed based on the log type label loss function and the risk level label loss function. The parameters of the shared parameter layer and each task output layer are optimized simultaneously through backpropagation, enabling the model to achieve feature sharing and mutual promotion between the two task objectives. Through multiple rounds of iterative training, the model gradually converges on the training dataset, ultimately forming a classification and grading model that can simultaneously perform log classification and risk grading.

[0101] It is understood that, through the above training method, the embodiments of this application enable the model to simultaneously learn log category differentiation patterns and risk level judgment patterns on the basis of a unified multi-dimensional feature representation, realizing multi-task collaborative optimization of log analysis tasks. This not only reduces the computational cost and parameter redundancy caused by training multiple models individually, but also fully utilizes the semantic association between log type and risk level to improve the model's generalization ability. In addition, the feature representation based on multi-source feature fusion and shared parameter layer learning enables the model to have a more comprehensive understanding of the semantics, structure, and behavioral patterns of log events, thereby significantly improving the accuracy and stability of log classification and risk grading, providing more reliable initial prediction results for the subsequent risk adjustment stage, and improving the overall processing performance and practical value of the log intelligent analysis system.

[0102] S104. Based on multi-dimensional feature vectors, the risk classification results are adjusted by combining a rule engine with a risk assessment model to obtain the classification and grading results of the log data.

[0103] In one feasible approach, firstly, multi-dimensional feature vectors are input into a rule engine to output an initial risk level; then, real-time risk indicators are obtained and input into a real-time risk assessment model to obtain risk correction coefficients; finally, based on the classification results, the initial risk level, and the risk correction coefficients, the classification and grading results of the log data are obtained.

[0104] The rule engine uses text content features, metadata features, and behavioral pattern features from multi-dimensional feature vectors, combined with a pre-defined rule base, to perform conditional matching and risk scoring on log data to determine the initial risk level. Risk indicators include the frequency of abnormal operations by the same user or address within a preset time period, system load rate, and abnormal access scores for target resources. The risk assessment model includes an input layer, at least one hidden layer, and an output layer. The input layer receives feature vectors that characterize the real-time risk of log data. The hidden layer performs linear transformations and activation function calculations on the input vectors to generate a hidden representation. The output layer outputs risk correction coefficients.

[0105] It should be understood that the static semantic features, structural features, and behavioral pattern features of log data can be interpreted logically through a rule system, while the real-time status of the system, the immediate risk of user behavior, and short-term trend changes are more suitable for dynamic evaluation through lightweight models. Therefore, this step integrates the initial classification results of the rule engine with the risk correction coefficients output by the risk assessment model, so that the final risk level can simultaneously reflect static risk characteristics and real-time behavioral risks, achieving a classification strategy that combines stability and sensitivity.

[0106] In the implementation process, the multi-dimensional feature vector constructed by S102 is first input into the rule engine for conditional matching and risk scoring based on the rule base. Specifically, the rule engine parses text content features, metadata features, and behavioral pattern features from the multi-dimensional feature vector, and performs logical judgments on the log data according to the preset rule base. For example, if the text features contain high-risk keywords such as "authenticationfailed" and "SQLinjection," the metadata features show that the source IP is a risk blacklist address, and the behavioral pattern features reflect abnormal request frequency, then the rule engine can accumulate scores based on multiple rules and output a high initial risk level. Through the interpretable design of the rule base, it is ensured that the basic judgment of log risks has clear and traceable basis.

[0107] After the rule engine outputs the initial risk level, real-time risk indicators associated with log data are further obtained to assess the real-time risk level of log events under the current system state. For example, if a source address triggers multiple incorrect password logins within a short period, and the system is under high load, the risk level reflected by the real-time risk indicators will be relatively higher, making it more suitable for integration into the dynamic assessment model for real-time correction. Subsequently, the aforementioned real-time risk indicators are constructed into feature vectors to characterize real-time risks and input into the risk assessment model for processing. This embodiment uses a lightweight neural network model as the risk assessment model, whose structure includes an input layer, at least one hidden layer, and an output layer. The input layer receives the real-time risk vector; the hidden layer calculates a deep risk representation through linear transformation and activation functions, thereby extracting the coupling relationship between risk indicators; the output layer outputs risk correction coefficients based on the deep representation, reflecting the magnitude of the increase or decrease in the initial risk level due to real-time risk. This approach enables the risk assessment model to effectively capture the nonlinear correlation between anomaly frequency, system pressure, and abnormal resource behavior while maintaining computational lightweightness.

[0108] After obtaining the classification results, the initial risk level provided by the rule engine, and the risk correction coefficient output by the risk assessment model, this step calculates the final log classification and grading results according to a preset fusion strategy. The fusion strategy may include a linear weighting strategy, an interval enhancement strategy, or a threshold triggering strategy. For example, if the initial risk level is medium risk but the risk correction coefficient exceeds a set threshold, the final risk level can be increased to high risk; if the initial risk level is high but the correction coefficient is low, the original level can be maintained or appropriately lowered. This fusion strategy ensures that the final grading results balance the stability of rule-based judgments with the real-time sensitivity of model predictions, thereby improving the accuracy and practicality of log risk analysis.

[0109] Understandably, by integrating the above-mentioned rules and models, the final risk score becomes more comprehensive and accurate. It can not only identify explicit risks but also capture implicit risks that traditional rules cannot detect, thereby significantly enhancing the adaptability and robustness of the log monitoring system in multiple scenarios.

[0110] The log data processing method, apparatus, device, and medium provided in this application first acquire log data to provide basic information for subsequent analysis. Then, multi-dimensional feature extraction is performed on the acquired log data, including text feature extraction, metadata feature extraction, and behavioral pattern feature extraction. Next, the extracted features are constructed into a multi-dimensional feature vector and input into a trained log data classification and grading model. This model uses machine learning algorithms to process the feature vector and automatically outputs the classification results and preliminary risk grading results of the log data, achieving efficient and intelligent analysis of massive logs. Finally, to further improve accuracy, the preliminary risk grading results are dynamically adjusted based on the multi-dimensional feature vector, combined with a rule engine and a risk assessment model. This allows for refined modification of known security policies or business rules using the rule engine, and captures complex potential risk patterns through the risk assessment model, ultimately generating accurate and actionable log classification and risk grading results. Therefore, this method, through a combination of multi-dimensional feature fusion, model inference, and rule engine optimization, achieves a comprehensive understanding, accurate classification, and intelligent grading of log data, improving the efficiency and accuracy of security monitoring, while also supporting timely detection and risk warning of abnormal behavior.

[0111] It should also be noted that, in order to improve the accuracy of log data classification and grading results, continuous model optimization and rule updates are necessary. Specifically:

[0112] First, feedback data is collected, including both correctly and incorrectly classified results from manual annotation. Then, the log data classification and grading model is incrementally trained based on the feedback data. Finally, the rules in the rule engine are updated based on the feedback data.

[0113] It should be understood that as system architecture, user behavior patterns, and attack methods continue to change, the effectiveness of the initial model and rule base will gradually decrease. Therefore, it is necessary to introduce a feedback loop to reuse the labeled data generated in actual operation for updating the model and rules, thereby ensuring that the log classification and grading system can continuously adapt to the dynamically changing risk environment.

[0114] In one feasible approach, feedback data generated during system operation is first collected. For example, when the system automatically identifies a log entry as low-risk, but manual inspection confirms it as an "abnormal login" event, the log entry and its correct risk level are recorded as feedback samples for subsequent training and rule updates. By continuously accumulating feedback data, the risk change trends in the real operating environment of the system can be comprehensively reflected. After collecting feedback data, multi-dimensional feature vectors corresponding to the feedback data are extracted and combined with existing training samples to construct an incremental training set. Subsequently, the shared parameter layer and multi-task output layer of the log classification and grading model are locally updated using the incremental training set, enabling the model to learn the latest risk patterns and log representations. For example, when new attack methods (such as API brute-force probing and container escape attempts) gradually increase, incremental training allows the model to automatically enhance its sensitivity to relevant text features and behavioral pattern features, improving detection accuracy and recall. This process does not require complete model retraining, significantly reducing training costs and ensuring the system quickly adapts to changes in the business environment.

[0115] It should also be noted that, while updating the model, this embodiment also dynamically updates the rules in the rule engine based on feedback data. Specifically, the misjudgments and omissions in the rule matching process can be statistically analyzed based on the results of manual annotation, and the rule content, thresholds, scoring weights, and matching conditions can be adjusted. For example, if multiple misclassified samples show that a certain type of abnormal access is not identified in the current rule, identification conditions can be added to the rule base; if a rule is triggered too frequently but the actual risk is low, its risk score can be reduced or the trigger threshold can be increased. By continuously calibrating the rule base, the interpretability and accuracy of rule judgments can be maintained, and erroneous judgments caused by rule aging can be avoided.

[0116] It is understandable that by introducing a continuous model optimization and rule update mechanism, the embodiments of this application can achieve the long-term evolution capability of the log processing system, improving its adaptability and robustness in complex scenarios. This approach not only improves the overall accuracy of log classification and grading, but also reduces the pressure of manual inspection, while ensuring that the system can quickly respond to emerging threat patterns and abnormal behaviors, achieving continuous, efficient, and intelligent log risk management.

[0117] Figure 2 A flowchart illustrating the log data processing method provided in this application embodiment. Figure 2 ,like Figure 2 As shown, in this embodiment... Figure 1 Based on the examples, the process of constructing multi-dimensional feature vectors is described in detail. The method includes:

[0118] S201. The log data is vectorized using a trained language model to generate text features.

[0119] The language model is obtained by learning the semantic relationships and structured expressions in historical log data through unsupervised or self-supervised methods.

[0120] It should be understood that, in this embodiment, for log data containing natural language descriptions or semi-structured descriptions, a language model trained on historical log data is used to perform deep semantic vectorization processing on the log text. Specifically, the language model automatically learns the semantic relevance, contextual dependencies, and implicit patterns between operational behaviors from a large number of historical logs through unsupervised or self-supervised learning methods; for example, it learns the difference between deleting user data and modifying user data through a Masked Language Model or autoregression, thereby generating high-dimensional semantic vectors that can distinguish operational intentions. This vectorization process maps the original free text or templated text into fixed-length feature representations that can be used for model analysis, improving the semantic expressiveness of the log text and the accuracy of classification and grading tasks.

[0121] S202. Extract the structured fields from the log data and convert the structured fields into numerical features.

[0122] It should be understood that in this embodiment, structured fields natively present in the log data, such as timestamps, source IP addresses, user identifiers, operation interfaces, target resource paths, and operation result status codes, are extracted through a field parsing module and converted into numerical features using different methods according to the field type. For example, timestamps can be parsed into hours, peak period information, etc.; IP addresses can be converted into network segment or geographical area codes; and status codes can be mapped to risk level labels (e.g., 5xx indicates high risk).

[0123] Understandably, by quantifying structured fields, key contextual information from log metadata can be effectively incorporated into the model, improving the model's ability to identify factors such as access sources, time patterns, and operation categories, thus achieving a full expression of structured information and improving the accuracy of classification and grading.

[0124] S203. Based on the historical behavior baseline model, extract the behavioral pattern features corresponding to the log data.

[0125] Among them, the historical behavior baseline model is based on the same type of log data collected within a preset time range, and constructs a normal behavior distribution model of user operations, system events or resource access through statistical analysis.

[0126] In one feasible approach, based on a historical behavior baseline model, the time series features, resource access features, operation instruction sequences, and event triggering chains in the log data are compared and analyzed to calculate at least one feature value used to quantify the degree of deviation between the log data and the normal pattern, which serves as the behavior pattern feature.

[0127] It should be understood that, in this embodiment, in order to identify potential abnormal behavior, a historical behavior baseline model is constructed based on a large number of similar logs collected within a preset time range. This model forms a statistical distribution of normal behavior by statistically analyzing user operation sequences, system event patterns, and resource access frequencies. For example, the model is used to model the login time distribution of user accounts, interface call frequency, and resource access paths to form a range of normal behavior. Subsequently, based on the content of the logs to be analyzed, their time series characteristics, resource access characteristics, operation command sequences, and event triggering links are compared with the behavior baseline to calculate their degree of deviation, such as at least one deviation feature value, such as access frequency surge index, operation sequence anomaly score, and cross-service link difference.

[0128] This method quantifies the differences between log behavior and normal patterns, thereby generating behavioral pattern features that are highly sensitive to anomaly detection, enabling early identification of potential attack behaviors or abnormal business behaviors.

[0129] S204. Feature fusion is performed on text features, numerical features, and behavioral pattern features to generate a multi-dimensional feature vector.

[0130] It should be understood that after obtaining text features, numerical features, and behavioral pattern features respectively, the feature fusion module encodes and concatenates the three types of features in a unified manner to construct a complete multi-dimensional feature vector.

[0131] It should be noted that feature fusion can employ concatenation layers, attention mechanisms, or linear projection to jointly represent features from different sources and with different dimensions within a shared vector space. For example, a 768-dimensional text vector can be combined with numerical features and behavioral deviation features to form a unified high-dimensional feature representation, which can then be input into a log classification and grading model. Through feature fusion, the model can simultaneously utilize semantic, environmental, and behavioral pattern information for judgment, thereby significantly improving the accuracy, robustness, and generalization ability of log classification and risk grading.

[0132] Figure 3 This is a schematic diagram of the structure of the log data processing device provided in the embodiments of this application; as shown below. Figure 3 As shown, the device includes:

[0133] The first acquisition module 301 is used to acquire log data;

[0134] Module 302 is used to extract text features, metadata features, and behavioral pattern features from log data to construct a multi-dimensional feature vector.

[0135] The second acquisition module 303 is used to input multi-dimensional feature vectors into the trained log data classification and grading model for processing, and to obtain the classification results and risk grading results of the log data.

[0136] Processing module 304 is used to adjust the risk classification results based on multi-dimensional feature vectors by combining a rule engine with a risk assessment model, so as to obtain the classification and grading results of log data.

[0137] In one possible implementation, the construction module 302 is specifically used for:

[0138] The trained language model is used to vectorize the log data and generate text features. The language model is obtained by learning the semantic relationships and structured expressions in the log data in an unsupervised or self-supervised manner.

[0139] Extract structured fields from log data and convert them into numerical features;

[0140] Based on the historical behavior baseline model, behavioral pattern features corresponding to log data are extracted. The historical behavior baseline model is based on the same type of log data collected within a preset time range. Through statistical analysis, a normal behavior distribution model of user operations, system events, or resource access is constructed.

[0141] By fusing textual features, numerical features, and behavioral pattern features, a multi-dimensional feature vector is generated.

[0142] In one possible implementation, the construction module 302 is specifically used for:

[0143] Based on the historical behavior baseline model, the time series features, resource access features, operation instruction sequences, and event triggering chains in the log data are compared and analyzed to calculate at least one feature value used to quantify the degree of deviation between the log data and the normal pattern, which serves as the behavior pattern feature.

[0144] In one possible implementation, the log data processing apparatus further includes a training module for:

[0145] Obtain the training dataset, which contains multiple sample log data labeled with log type and risk level labels;

[0146] For each sample log data, extract the multi-dimensional feature vector of the sample log data as the input features of the model;

[0147] The log data classification and grading model is trained using the training dataset. The log classification and grading model is a neural network model that includes a shared parameter layer and a multi-task output layer. The shared parameter layer learns a general feature representation, and one sub-task output layer of the multi-task output layer learns to predict the log type label, while the other sub-task output layer learns to predict the risk level label.

[0148] In one possible implementation, the processing module 304 is specifically used for:

[0149] The multi-dimensional feature vector is input into the rule engine, which outputs the initial risk level. The rule engine determines the initial risk level by performing condition matching and risk scoring on the log data based on the text content features, metadata features, and behavioral pattern features in the multi-dimensional feature vector, combined with a preset rule base.

[0150] Real-time risk indicators are obtained and input into a real-time risk assessment model to obtain risk correction coefficients. Risk indicators include the frequency of abnormal operations by the same user or the same address within a preset time period, system load rate, and abnormal access score of target resources. The real-time risk assessment model includes an input layer, at least one hidden layer, and an output layer. The input layer receives feature vectors that characterize the real-time risks of log data. The hidden layer performs linear transformation and activation function calculation on the input vectors to generate hidden representations. The output layer outputs risk correction coefficients.

[0151] Based on the classification results, initial risk level, and risk correction coefficient, the classification and grading results of the log data are obtained.

[0152] In one possible implementation, the log data processing apparatus further includes a second processing module for:

[0153] Collect feedback data, including both correctly and incorrectly graded results manually labeled;

[0154] Incremental training of the log data classification and grading model is performed based on feedback data;

[0155] The rules in the rule engine are updated based on the feedback data.

[0156] In one possible implementation, before acquiring log data, the second processing module is further configured to:

[0157] Perform preprocessing operations on structured, semi-structured, and unstructured multi-source log data. The preprocessing operations include log format standardization, missing field completion, and sensitive information de-identification.

[0158] Logs across services and nodes are sequence-aligned according to timestamps to generate log data that can be used for feature extraction and model processing.

[0159] The log data processing device provided in this application embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0160] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 40 provided in this embodiment includes at least one processor 401 and a memory 402. Optionally, the device 40 further includes a communication component 403. The processor 401, memory 402, and communication component 403 are connected via a bus 404.

[0161] In a specific implementation, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to perform the above-described method.

[0162] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0163] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0164] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0165] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0166] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0167] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0168] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0169] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0170] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0171] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0172] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0173] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0174] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A log data processing method, characterized in that, include: Retrieve log data; Extract text features, metadata features, and behavioral pattern features from the log data to construct a multi-dimensional feature vector; The multi-dimensional feature vectors are input into the trained log data classification and grading model for processing to obtain the classification results and risk grading results of the log data. Based on the multi-dimensional feature vectors, the risk classification results are adjusted by combining a rule engine with a risk assessment model to obtain the classification and grading results of the log data.

2. The method according to claim 1, characterized in that, The step of extracting text features, metadata features, and behavioral pattern features from the log data to construct a multi-dimensional feature vector includes: The log data is vectorized using a trained language model to generate text features. The language model is obtained by learning the semantic relationships and structured expressions in the log data through unsupervised or self-supervised learning of historical log data. Extract the structured fields from the log data and convert the structured fields into numerical features; Based on the historical behavior baseline model, the behavioral pattern features corresponding to the log data are extracted. The historical behavior baseline model is based on the same type of log data collected within a preset time range, and a normal behavior distribution model of user operations, system events or resource access is constructed through statistical analysis. The text features, numerical features, and behavioral pattern features are fused to generate a multi-dimensional feature vector.

3. The method according to claim 2, characterized in that, The calculation of behavioral pattern features corresponding to the log data based on the historical behavioral baseline model includes: Based on the historical behavior baseline model, the time series features, resource access features, operation instruction sequences, and event triggering chains in the log data are compared and analyzed to calculate at least one feature value used to quantify the degree of deviation of the log data from the normal pattern, which is then used as the behavior pattern feature.

4. The method according to claim 1, characterized in that, The training process of the log data classification and grading model includes: Obtain a training dataset, which contains multiple sample log data labeled with log type and risk level labels; For each of the sample log data, extract the multi-dimensional feature vector of the sample log data as the model input feature; The log data classification and grading model is trained using the training dataset. The log classification and grading model is a neural network model that includes a shared parameter layer and a multi-task output layer. The shared parameter layer learns a general feature representation, and one sub-task output layer of the multi-task output layer learns to predict log type labels, while another sub-task output layer learns to predict risk level labels.

5. The method according to claim 1, characterized in that, The risk classification results are adjusted based on the multi-dimensional feature vectors by combining a rule engine and a risk assessment model to obtain the classification and grading results of the log data, including: The multi-dimensional feature vector is input into the rule engine, which outputs an initial risk level. The rule engine performs condition matching and risk scoring on the log data based on the text content features, metadata features, and behavioral pattern features in the multi-dimensional feature vector, combined with a preset rule base, to determine the initial risk level. Real-time risk indicators are obtained and input into a real-time risk assessment model to obtain risk correction coefficients. The risk indicators include the frequency of abnormal operations by the same user or the same address within a preset time period, system load rate, and abnormal access score of target resources. The real-time risk assessment model includes an input layer, at least one hidden layer, and an output layer. The input layer receives feature vectors used to characterize the real-time risks of log data. The hidden layer performs linear transformation and activation function calculation on the input vectors to generate hidden representations. The output layer outputs risk correction coefficients. Based on the classification results, the initial risk level, and the risk correction coefficient, the classification and grading results of the log data are obtained.

6. The method according to claim 1, characterized in that, The method further includes: Collect feedback data, which includes both correctly graded and incorrectly graded results manually labeled; The log data classification and grading model is incrementally trained based on the feedback data. The rules in the rule engine are updated based on the feedback data.

7. The method according to claim 1, characterized in that, Before acquiring the log data, the method further includes: Preprocessing operations are performed on multi-source log data that are structured, semi-structured, and unstructured. The preprocessing operations include log format standardization, missing field completion, and sensitive information de-identification. Logs across services and nodes are sequence-aligned according to timestamps to generate log data that can be used for feature extraction and model processing.

8. A log data processing device, characterized in that, include: The first acquisition module is used to acquire log data; The construction module is used to extract text features, metadata features, and behavioral pattern features from the log data and construct a multi-dimensional feature vector. The second acquisition module is used to input the multi-dimensional feature vector into the trained log data classification and grading model for processing, and to obtain the classification result and risk grading result of the log data. Processing module, used for; Based on the multi-dimensional feature vectors, the risk classification results are adjusted by combining a rule engine with a risk assessment model to obtain the classification and grading results of the log data.

9. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

Images