Log denoising methods, devices, equipment, and media based on large models
By using a log denoising method based on a large model, leveraging real-time stream processing and contextual semantic analysis of the large model, and combining log metadata to construct log fingerprints for clustering and deduplication, the problem of excessive alarms in massive log data is solved, achieving efficient log alarm management.
Patent Information
- Application Number
- CN202611283115.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-21
- Publication Date
- 2026-11-10
AI Technical Summary
When faced with massive amounts of log data, existing technologies struggle to effectively reduce the number of alerts, making it difficult for operations and maintenance personnel to quickly identify critical risk information when confronted with an alert storm.
A log denoising method based on a large model is adopted. Rule detection is performed through a real-time stream processing engine, and contextual semantic analysis is performed in combination with a pre-trained large model to extract log features. These features are then fused with log metadata to construct a log fingerprint. Based on the log fingerprint, clustering and deduplication are performed to reduce the number of alarms.
It improves the accuracy of log feature extraction, can accurately distinguish similar anomalies in different environments, effectively reduces the number of alarms from massive log data, and improves the response efficiency of operation and maintenance personnel.
Smart Images

Figure CN123068404A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of data processing technology and is applicable to the financial and medical fields. In particular, it relates to a log noise reduction method, device, equipment and medium based on a large model. Background Technology
[0002] In the financial and healthcare sectors, the widespread adoption of distributed systems and microservice architectures results in massive amounts of log data being generated daily. For example, a medium-sized internet healthcare platform might generate hundreds of thousands of log entries per minute during peak periods, covering various types such as appointment booking logs, online consultation logs, prescription processing logs, and payment settlement logs. Similarly, an online financial platform processing user loan applications and insurance claims involves multiple stages in the transaction chain, including identity authentication, risk assessment, and electronic signing. An anomaly in any stage can trigger a chain reaction, resulting in a large volume of logs being generated in a short period. Operations and maintenance personnel need to monitor logs to promptly identify system anomalies, pinpoint the root cause of faults, and take appropriate action.
[0003] However, current log governance systems generally employ a rule-based, line-by-line inspection mechanism. When an underlying service malfunctions, upstream business modules that depend on it will output a large number of similar logs within a short period, triggering alarm notifications line by line. This means that a single point of failure can trigger tens of thousands of alarms within minutes. For example, a timeout in a consultation information synchronization interface in a medical platform may cause multiple related modules, such as the registration system, doctor workstation, and pharmacy system, to simultaneously output alarms; an anomaly in a payment callback interface in a financial platform may trigger concentrated alarms in the order system, clearing system, and reconciliation system. Faced with this alarm storm, operations and maintenance personnel struggle to quickly identify the issues that truly require attention, and critical risk information is easily overwhelmed by duplicate alarms.
[0004] Therefore, how to effectively reduce the number of alerts when dealing with massive amounts of log data is a technical problem that urgently needs to be solved. Summary of the Invention
[0005] This invention provides a log noise reduction method, apparatus, device, and medium based on a large model, to solve the technical problem of how to effectively reduce the number of alarms when dealing with massive log data.
[0006] In a first aspect, the present invention provides a log denoising method based on a large model, comprising: Obtain the first log to be analyzed and the corresponding log metadata. Perform rule detection on the first log to be analyzed through the real-time stream processing engine to obtain the second log to be analyzed that matches the rule. The second log to be analyzed is sent to a pre-trained large model for contextual semantic analysis, and log features of the second log to be analyzed are extracted. The log features of the second log to be analyzed are fused with the log metadata, and a log fingerprint is constructed based on the fused information. Based on the log fingerprint, the second log to be analyzed generated within the preset time window is clustered, and the second log to be analyzed under the same cluster is deduplicated. The log fingerprint retained after deduplication is used as an alarm message for the corresponding cluster and pushed to the log security management platform.
[0007] Secondly, the present invention provides a log denoising device based on a large model, comprising: The acquisition module is used to acquire the first log to be analyzed and the log metadata corresponding to the first log to be analyzed, and to perform rule detection on the first log to be analyzed through the real-time stream processing engine to obtain the second log to be analyzed that matches the rule. The analysis module is used to send the second log to be analyzed to a pre-trained large model for contextual semantic analysis and to extract the log features of the second log to be analyzed. A construction module is used to fuse the log features of the second log to be analyzed with the log metadata, and construct a log fingerprint based on the fused information; The clustering module is used to cluster the second logs to be analyzed generated within a preset time window based on the log fingerprint, and to deduplicate the second logs to be analyzed under the same cluster. The log fingerprints retained after deduplication are used as an alarm message for the corresponding cluster and pushed to the log security management platform.
[0008] Thirdly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described log denoising method based on a large model.
[0009] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described log denoising method based on a large model.
[0010] The aforementioned log denoising method, apparatus, device, and medium based on a large model, implements a scheme in which a first log to be analyzed and its corresponding log metadata are obtained through a client. A real-time stream processing engine performs rule detection on the first log to be analyzed to obtain a second log to be analyzed that matches the rules. The second log to be analyzed is then sent to a pre-trained large model for contextual semantic analysis, and log features of the second log to be analyzed are extracted. The log features of the second log to be analyzed are fused with the log metadata, and a log fingerprint is constructed based on the fused information. Based on the log fingerprint, the second logs to be analyzed generated within a preset time window are clustered, and duplicate logs within the same cluster are deduplicated. The deduplicated log fingerprint is used as an alarm message for the corresponding cluster and pushed to the log security management platform. In this invention, by performing contextual semantic analysis and extracting log features through a pre-trained large model, key business information can be accurately identified from semantically complex logs, improving the accuracy of feature extraction. Meanwhile, the extracted log features are integrated with log metadata such as deployment environment to construct a log fingerprint that includes both semantic anomaly description and technical context. This can accurately distinguish similar anomalies in different environments, providing a reliable basis for clustering and noise reduction, and effectively reducing the number of alarms from massive log data. Attached Figure Description
[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0012] Figure 1 This is a schematic diagram of an application environment for a log denoising method based on a large model, according to an embodiment of the present invention.
[0013] Figure 2 This is a flowchart illustrating a log denoising method based on a large model in one embodiment of the present invention.
[0014] Figure 3 yes Figure 2 A schematic diagram of a specific implementation method for step S10.
[0015] Figure 4 yes Figure 2 A flowchart illustrating a specific implementation of step S20.
[0016] Figure 5 yes Figure 2 A flowchart illustrating a specific implementation of step S30.
[0017] Figure 6 yes Figure 2 A schematic diagram of a specific implementation of step S40.
[0018] Figure 7 This is a schematic diagram of a log noise reduction device based on a large model in one embodiment of the present invention.
[0019] Figure 8 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention.
[0020] Figure 9 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation
[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0022] The log denoising method based on a large model provided in this invention can be applied to, for example... Figure 1 In the application environment, Figure 1This is a schematic diagram of an application environment for a log denoising method based on a large model, according to an embodiment of the present invention. The client communicates with the server via a network. The server can obtain a first log to be analyzed and its corresponding log metadata through the client. It then performs rule detection on the first log to be analyzed using a real-time stream processing engine to obtain a second log to be analyzed that matches the rules. The second log to be analyzed is sent to a pre-trained large model for contextual semantic analysis, and log features of the second log to be analyzed are extracted. The log features of the second log to be analyzed are fused with the log metadata, and a log fingerprint is constructed based on the fused information. Based on the log fingerprint, the second logs to be analyzed generated within a preset time window are clustered, and duplicate logs within the same cluster are deduplicated. The deduplicated log fingerprint is used as an alarm message for the corresponding cluster and pushed to the log security management platform. In this invention, by performing contextual semantic analysis and extracting log features using a pre-trained large model, key business information can be accurately identified from semantically complex logs, improving the accuracy of feature extraction. Simultaneously, the extracted log features are integrated with log metadata such as the deployment environment to construct a log fingerprint that includes both semantic anomaly descriptions and technical context. This fingerprint can accurately distinguish similar anomalies in different environments, providing a reliable basis for clustering and noise reduction, and effectively reducing the number of alarms from massive log data. The invention will now be described in detail through specific embodiments.
[0023] Please see Figure 2 As shown, Figure 2 This is a flowchart illustrating a log denoising method based on a large model provided in an embodiment of the present invention. The log denoising method based on a large model specifically includes the following steps: S10: Obtain the first log to be analyzed, along with the corresponding log metadata. Perform rule detection on the first log to be analyzed using the real-time stream processing engine to obtain the second log to be analyzed that matches the rules. Specifically, such as... Figure 3 The above, Figure 3 yes Figure 2 A schematic flowchart of a specific implementation of step S10 includes the following steps S11-S14: S11: Obtain the real-time generated raw log data stream, and use each log in the raw log data stream as the first log to be analyzed, and simultaneously collect the log metadata associated with the first log to be analyzed. Specifically, in this embodiment of the invention, log metadata is structured information describing the context of log generation, which is an inherent attribute of the log. Binding metadata as soon as the log enters the processing pipeline provides consistent basic data support for all subsequent processing stages. This avoids the overhead of backtracking or cross-system calls due to missing metadata, improves overall processing efficiency, and ensures the accurate correspondence between metadata and log text. For example, in a financial scenario, when a user initiates a loan application, the log output by the identity authentication service will simultaneously carry metadata such as the system name being the core transaction system, the deployment environment being the production environment, and the log level being the information level. In a medical scenario, for example, a log output by the prescription circulation service in an online consultation platform carries metadata such as the system type being an internet medical platform, the deployment environment being the production environment, and the log level being the error level when it is generated. This log is used as the first log to be analyzed, and metadata collection is completed simultaneously.
[0024] S12: Load the preset rule base, call at least one detection rule stored in the rule base, and extract the matching conditions and action strategies corresponding to the detection rule. Specifically, in this embodiment of the invention, each detection rule in the rule base includes two parts: the matching conditions define the field features that the trigger rule must satisfy, and the action strategy defines the handling method after the rule is hit. The system preloads all rules into memory before processing logs, so that subsequent matching operations do not need to repeatedly read the storage or parse the configuration file. Operation and maintenance personnel can add, delete, or modify detection rules at any time without restarting the system, which improves the flexibility and maintainability of log detection. For example, in the financial scenario, the rule base can be configured with a payment callback timeout detection rule, whose matching condition is that the log level is equal to the error level and the log content contains the keyword "payment timeout", and the action strategy is to mark it as a candidate log and send it to the large model; it can also be configured with a database connection failure detection rule, etc. In the medical scenario, the rule base can be configured with a consultation service anomaly detection rule, whose matching condition is that the log level is equal to the error level and the log content contains the keyword "consultation synchronization failure".
[0025] S13: The real-time stream processing engine parses the first log to be analyzed, extracts the log fields of the first log to be analyzed, and compares the log fields with the matching conditions in the rule base. Specifically, in this embodiment of the invention, the real-time stream processing engine performs field-level parsing on the log text, extracts structured log fields, and then compares these fields one by one with the matching conditions of each detection rule in the rule base. Efficient and accurate rule matching is achieved through precise field-level comparison. For example, in a financial scenario, taking a payment callback log as an example, the engine parses fields from the log text such as the log level being error level, the log content containing payment callback timeout, and the service name being payment service, and then compares the payment callback timeout field with the matching conditions of the payment callback timeout detection rule.
[0026] S14: If the log field satisfies the matching condition of any of the detection rules, the corresponding first log to be analyzed is determined to have hit the rule, and the first log to be analyzed that has hit the rule is marked as the second log to be analyzed. Specifically, in this embodiment of the invention, the original logs are divided into two categories: logs that have hit the rule and logs that have not hit the rule and low-value logs. Logs that have hit the rule are marked as the second log to be analyzed and sent to the subsequent large model analysis stage, while logs that have not hit the rule are directly discarded or archived, and no longer consume subsequent computing resources. This effectively reduces the number of logs sent to the large model, reduces the model call cost and computing resource consumption, and at the same time ensures that all potentially abnormal logs that have hit the rule enter the subsequent deep analysis stage, avoiding the omission of key risk events. For example, in a financial scenario, a log recording a payment callback timeout, because its log level is error level and the log content contains payment timeout, meets the matching condition of the payment callback timeout detection rule and is determined to have hit the rule. The system marks this log as the second log to be analyzed and adds it to the queue to be sent.
[0027] S20: The second log to be analyzed is sent to a pre-trained large model for contextual semantic analysis, and log features of the second log to be analyzed are extracted. Specifically, such as... Figure 4 The above, Figure 4 yes Figure 2 A schematic flowchart of a specific implementation of step S20 includes the following steps S21-S24: S21: The second log to be analyzed is formatted and converted according to a preset log template to generate structured log data. Specifically, in this embodiment of the invention, after the second log to be analyzed has undergone preliminary rule detection and screening, its original format may vary significantly due to different source systems. For example, logs output by different service modules may differ in terms of time format, etc. Directly inputting them into a large model will lead to inconsistent input formats, affecting the stability and accuracy of semantic analysis. By formatting and converting the second log to be analyzed using a preset log template, log texts with different structures can be unified into standardized structured log data, providing a consistent input format for the large model. This allows the large model to focus on the semantic understanding task itself without needing to pay attention to the differences in the original format of the logs, thereby improving the accuracy and consistency of feature extraction. For example, in a financial scenario, the original content of a log from a payment service is "Payment service payment callback timeout order number 1234," while the original content of another log from a risk control service is "Risk control module risk detection abnormal transaction ID 5678." The original formats and field arrangements of the two logs are completely different. After formatting and conversion, both logs are unified into structured log data containing timestamps, service names, exception descriptions, and key parameters.
[0028] S22: Based on the structured log data, construct an input sequence containing log keywords, log levels, and anomaly codes, and send the input sequence to the pre-trained large model. Specifically, in this embodiment of the invention, the system extracts log keywords, log levels, and anomaly codes from the formatted structured log data. Log keywords are used to identify the core description of the anomaly, the log level is used to reflect the severity of the anomaly, and the anomaly code is used to provide a precise classification basis for the anomaly. Constructing the above three elements into an input sequence according to a preset order can significantly compress the input length while retaining key semantic information, thereby reducing the computational overhead of the large model. For example, in a financial scenario, extracting the keywords "risk detection anomaly," the log level "warning," and the anomaly code "risk control anomaly" from a risk control anomaly detection log, the constructed input sequence is "risk detection anomaly warning risk control anomaly."
[0029] S23: The pre-trained large model performs contextual understanding on the input sequence to identify the semantic intent and anomaly type of the second log to be analyzed, obtaining semantic labels and corresponding confidence scores. Specifically, in this embodiment of the invention, the pre-trained large model possesses deep semantic understanding capabilities, enabling it not only to identify keywords on the surface of the text but also to understand the true semantic intent of the log in conjunction with the context. When the input sequence is fed into the large model, the model, based on its language knowledge and pattern recognition capabilities learned during the pre-training phase, performs contextual reasoning on the sequence to determine what kind of business anomaly intent the log reflects, what type of anomaly it belongs to, and provides a quantified confidence score to indicate the reliability of the judgment. For example, in a financial scenario, after the large model performs contextual understanding on the input sequence "payment callback timeout error payment timeout," it identifies the semantic intent of the log as an anomaly in the response of a third-party payment channel, the anomaly type as "payment service timeout," and provides a confidence score of 0.97, indicating a high degree of reliability in the judgment.
[0030] S24: Embedding and encoding the semantic intent, the anomaly type, and the confidence score to generate a log semantic feature vector for the second log to be analyzed. Specifically, in this embodiment of the invention, the semantic intent, anomaly type, and confidence score output by the large model are discrete text labels and numerical values, which cannot be directly used for subsequent vectorization calculations and similarity comparisons. Through embedding and encoding techniques, these discrete semantic analysis results are mapped into a continuous high-dimensional vector space to generate a log semantic feature vector, providing a foundation for subsequent fingerprint construction and log clustering.
[0031] S30: The log features of the second log to be analyzed are fused with the log metadata, and a log fingerprint is constructed based on the fused information. Specifically, such as... Figure 5 The above, Figure 5 yes Figure 2 A schematic flowchart of a specific implementation of step S30 includes the following steps S31-S34: S31: Perform one-hot encoding on the categorical fields in the log metadata and normalize the encoding on the numerical fields to generate a metadata encoding vector. Specifically, in this embodiment of the invention, for categorical fields, one-hot encoding is used to convert them into a sparse vector composed of 0s and 1s, with each category corresponding to an independent dimension in the vector; for numerical fields, normalization encoding is used to map their original values to a unified range between 0 and 1, eliminating the influence of different units on subsequent calculations. The above two encoding results are combined to form a unified metadata encoding vector, providing a standardized metadata representation for subsequent vector fusion. For example, in a financial scenario, the metadata of a payment service log includes categorical fields such as system name (payment system), deployment environment (production environment), and log level (error level), as well as a numerical field (interface response time 3500ms). The system performs one-hot encoding on the categorical fields and normalizes the 3500ms value to 0.35, ultimately generating a metadata encoding vector that integrates categorical and numerical information.
[0032] S32: Concatenate the log semantic feature vector with the metadata encoding vector to generate a log fusion vector. Specifically, in this embodiment of the invention, the log semantic feature vector carries the business anomaly semantics extracted from the log text by the large model, while the metadata encoding vector carries the technical context information at the time the log was generated. The two have different information sources and representation dimensions, but together they constitute a complete feature profile of a log. Through vector concatenation, the log semantic feature vector and the metadata encoding vector are connected end-to-end in terms of dimension to generate a log fusion vector that simultaneously contains business semantic information and technical context information. For example, in a medical scenario, the semantic feature vector of insufficient registration resources is concatenated with the metadata encoding vector of registration service, production environment, error level, and time consumption of 0.20; the semantic feature vector of prescription review failure is concatenated with the metadata encoding vector of prescription service, production environment, and error level. The fused vector simultaneously contains information on the type of medical business anomaly and the technical operating environment, providing a sufficient information foundation for subsequently distinguishing anomalies in different departments and different business processes.
[0033] S33: The log fusion vector is hashed using the Locality Sensitive Hashing (LSH) algorithm to generate the hash fingerprint of the second log to be analyzed. Specifically, in this embodiment of the invention, the LSH algorithm can map a high-dimensional vector to a fixed-length hash code while maintaining the similarity relationship between vectors. That is, similar vectors generate the same or similar hash codes after hashing, while dissimilar vectors generate significantly different hash codes. The system hashes the log fusion vector using the LSH algorithm to generate a fixed-length hash value, which serves as the hash fingerprint of the second log to be analyzed. This significantly reduces computational complexity, enabling high fingerprint generation and matching efficiency even in scenarios with massive log volumes. For example, in a financial scenario, two payment timeout logs from a payment service may have different order numbers and occurrence times, but their generated log fusion vectors are close in distance in the vector space. After processing with the LSH algorithm, these two fusion vectors are mapped to the same hash fingerprint, indicating that they belong to the same type of anomaly. The fusion vector of the risk control trigger interception log and the fusion vector of the payment timeout log are far apart in space. After processing by the Locality Sensitive Hash algorithm, different hash fingerprints are generated, which clearly distinguishes the two different types of financial anomalies.
[0034] S331: Based on the log fusion vector, set the number of hash tables m and the hash encoding length k of each hash table. Specifically, in this embodiment of the invention, the number of hash tables m determines the robustness of the hash encoding. The larger the value of m, the stronger the ability of multiple hash tables to complement each other, the higher the tolerance for subtle differences between vectors, and the higher the probability that similar vectors are assigned to the same bucket. The hash encoding length k of each hash table determines the distinguishing granularity of a single hash. The larger the value of k, the stronger the distinguishing ability of a single hash. By reasonably setting the two parameters m and k, a balance can be achieved between the distinguishing ability of hash fingerprints and the clustering tolerance, so that different instances of the same type of anomaly can be classified into the same hash bucket, while different types of anomalies are assigned to different hash buckets, providing a parameterized and controllable technical means for subsequent clustering deduplication. For example, in a financial scenario, the system sets the number of hash tables m to 4 and the hash encoding length k of each hash table to 16 based on the data characteristics and historical experience of financial logs. This parameter combination enables the generated hash fingerprint to distinguish between different types of financial anomalies such as payment timeouts, risk control interceptions, and transaction failures, while also tolerating minor differences within the same type of anomaly due to slight fluctuations in response time or different order numbers, ensuring that similar anomalies are correctly aggregated.
[0035] S332: Construct m sets of random projection matrices, each containing k random projection vectors of the same dimension as the log fusion vector. Specifically, in this embodiment of the invention, the system constructs m sets of random projection matrices, each containing k random projection vectors of the same dimension as the log fusion vector. These random projection vectors define k different projection directions in the vector space, and the projection result in each direction will determine one bit in the final binary encoding. Since the projection matrices are randomly generated, different hash tables use different random projection matrices, ensuring the independence between multiple hash tables, thereby capturing the similarity relationship of vectors from different angles. For example, in a financial scenario, the system constructs 4 sets of random projection matrices for 4 hash tables, each containing 16 random projection vectors of the same dimension as the log fusion vector. Taking the fusion vector of the payment timeout log as an example, this vector will be projected onto 4 different random projection matrices respectively, capturing the spatial position features of the vector from 4 different directional combinations.
[0036] S333: Project the log fusion vector using each set of random projection matrices to obtain m sets of projection results for the log fusion vector, where each set of projection results contains k projection values. Specifically, in this embodiment of the invention, the corresponding projection value is obtained by performing an inner product calculation between the log fusion vector and each random projection vector in each set of random projection matrices. Since each set of random projection matrices contains k random projection vectors, the high-dimensional vector is compressed into a small number of projection values through the inner product operation, which significantly reduces the data dimensionality while preserving the relative similarity between vectors, providing a numerical basis for subsequent binarization and hash encoding.
[0037] S334: The projection values of the m groups of log fusion vectors are binarized, converting the projection values into binary bits, and concatenating them into k-bit binary sequences according to the order of their arrangement, generating m k-bit binary sequences. Specifically, in this embodiment of the invention, the projection values are continuous real numbers, which need to be converted into discrete binary bits to form a hash code. The binarization process uses zero as the dividing threshold; when the projection value is greater than or equal to zero, it is converted into a binary bit 1, and when the projection value is less than zero, it is converted into a binary bit 0. Each projection value becomes a binary bit, and the k projection values in a group are concatenated into a k-bit binary sequence according to the projection order. Since m groups of projections produce m groups of projection values, m k-bit binary sequences are ultimately generated, and each sequence is a hash code in a hash table. This invention compresses continuous real numbers into a single bit through binarization, greatly reducing storage space and comparison computation, allowing fingerprint storage and matching to be completed in constant time. For example, in a financial scenario, the 16 projected values obtained from the payment timeout log in the first projection group are binarized, with values greater than or equal to zero converted to 1 and values less than zero converted to 0, resulting in a 16-bit binary sequence, such as 1011001101011001. The projected values of this log in the other three projection groups are also binarized, generating three more 16-bit binary sequences. Because the fusion vector of the risk control interception log is located in a different position in the vector space, the four 16-bit binary sequences generated after binarizing the projected values are significantly different from the sequences in the payment timeout log.
[0038] S335: The m k-bit binary sequences are used as hash fingerprints of the second log to be analyzed in each hash bucket, and each hash fingerprint is associated and stored in the corresponding fingerprint index table. Specifically, in this embodiment of the invention, the k-bit binary sequence generated in each hash table is used as the independent hash fingerprint of the log in that hash table, and multiple hash tables together constitute the composite fingerprint identifier of the log. As long as the log has the same fingerprint as the historical log in any hash table, it can be determined that the two are similar, thereby ensuring the clustering recall rate while avoiding omissions that may be caused by a single hash table. The system associates each hash fingerprint with its hash table number and stores it in the fingerprint index table, so that subsequent queries can be retrieved and compared separately by hash table. For example, in a financial scenario, a payment timeout log generates four 16-bit binary sequences in four hash tables, and these four sequences are used as hash fingerprints of the log in four hash buckets. The system associates these four hash fingerprints with hash table numbers 1 to 4 one by one and stores them in the fingerprint index table. When another payment timeout log enters the system, it also generates 4 hash fingerprints. The system compares them table by table in the fingerprint index table. As long as the same fingerprint is found in any hash table, it can be determined that the two logs belong to the same type of anomaly.
[0039] S34: Store the hash fingerprint in the fingerprint index table and associate the log identifier of the second log to be analyzed with the hash fingerprint. Specifically, in this embodiment of the invention, after the hash fingerprint is generated, a mapping relationship needs to be established with the original log so that the original log can be quickly located during subsequent alarm push and problem tracing. By associating the log identifier of the second log to be analyzed with the hash fingerprint, the fingerprint index table provides a complete tracing link for the noise-reduced alarm. When a certain type of alarm is pushed, the operation and maintenance personnel can quickly retrieve all the original logs corresponding to that type of alarm in the index table through the hash fingerprint, which is convenient for root cause analysis and problem investigation. At the same time, the index table structure supports efficient CRUD operations and can adapt to the real-time continuous inflow of log data. For example, in the financial scenario, the system stores the hash fingerprint of the payment timeout category in the fingerprint index table and associates it with the log identifier of all the second logs to be analyzed under this category. When the operation and maintenance personnel receive the payment service timeout alarm, they can query all the order logs that have experienced payment timeouts in the index table through the hash fingerprint, and quickly count the scope of affected orders and the time period in which the timeouts occurred.
[0040] S40: Based on the log fingerprint, cluster the second logs to be analyzed generated within a preset time window, and deduplicate the second logs to be analyzed within the same cluster. The retained log fingerprint after deduplication is then used as an alarm message for the corresponding cluster and pushed to the log security management platform. Specifically, as follows... Figure 6 The above, Figure 6 yes Figure 2 A schematic flowchart of a specific implementation of step S40 includes the following steps S41-S45: S41: Based on the preset time window, obtain each of the second logs to be analyzed and their corresponding log fingerprints generated within the current time window. Specifically, in this embodiment of the invention, the clustering and deduplication of log fingerprints require a clear time boundary to ensure the timeliness and manageability of alarms. The preset time window defines a fixed time interval, such as 1 minute or 5 minutes, and the system processes logs in batches using this window as the unit. At the end of each time window, the system obtains the second logs to be analyzed and their corresponding log fingerprints generated within that window, as the processing objects for the current batch. This invention uses time windows for batch processing, which ensures the real-time nature of alarms, avoids the huge computational overhead caused by global clustering, and makes the number of alarms pushed in each batch controllable, facilitating periodic review and handling by maintenance personnel.
[0041] S42: Use the log fingerprint of the first second log to be analyzed within the current time window as the initial cluster center. Specifically, in this embodiment of the invention, the clustering algorithm needs a starting reference point to gradually build clusters. The system uses the log fingerprint of the first second log to be analyzed within the current time window as the first cluster center. The first log does not need to be compared with other logs and directly becomes the representative of the first cluster. For example, in a financial scenario, the 5000 logs within the current time window are sorted by generation time. The first log is a payment callback timeout log, and its log fingerprint is four 16-bit binary sequences generated from hash tables 1 to 4 respectively. The system uses the fingerprint of this payment timeout log as the first cluster center to create the first cluster, which represents the payment timeout exception.
[0042] S43: Calculate the similarity between the log fingerprints of each of the second logs to be analyzed that are not initially clustered within the current time window and each clustered center. If the similarity is greater than or equal to a preset threshold, the log is assigned to the corresponding cluster; if the similarity is less than the preset threshold, a new cluster is created. Specifically, in this embodiment of the invention, the similarity between log fingerprints is calculated by comparing hash fingerprints. Due to the use of the Locality Sensitive Hash algorithm and a multi-hash table structure, the hash fingerprints of two similar logs will be completely identical in at least one hash table. The system sequentially takes each unprocessed second log to be analyzed within the current time window and compares its log fingerprint with the fingerprints of existing clustered centers through a hash table-by-hash table. If the log fingerprint is the same as the fingerprint of a certain clustered center in any hash table, the similarity is determined to be greater than or equal to the preset threshold, and the log is assigned to the corresponding cluster; if they are different in all hash tables, the similarity is determined to be less than the preset threshold, and a new cluster is created using the fingerprint of the log as the new clustered center. For example, in a financial scenario, the system sequentially processes the remaining 4999 logs within the current time window. The second log entry is also a payment timeout log. Its hash fingerprint in hash table 3 is completely identical to the fingerprint of the first cluster center. The system determines that the similarity is greater than or equal to the preset threshold and assigns it to the first cluster. When processing the 3201st log entry, which is a risk control interception log, its hash fingerprint is different from the fingerprint of the first cluster center in all four hash tables. The system determines that the similarity is less than the preset threshold and uses this log entry's fingerprint as the second cluster center, creating a second cluster. The subsequent 1500 risk control interception log entries are all assigned to the second cluster because they match the second cluster center. The 4701st log entry is a database connection exception log. It does not match either of the existing two cluster centers in any of the hash tables, so the system creates a third cluster.
[0043] S44: Deduplicate the second log to be analyzed within each cluster, retaining the log fingerprint that appears for the first time in each cluster and deleting duplicate log fingerprints within the cluster. Specifically, in this embodiment of the invention, multiple logs within the same cluster share the same hash fingerprint, which are essentially records of repeated occurrences of the same type of anomaly within a short period of time. From the perspective of alarm push, only one representative fingerprint needs to be retained to fully convey the fact of the anomaly's occurrence; the remaining duplicate fingerprints are redundant information. The system performs deduplication processing on the second log to be analyzed within each cluster, retaining the log fingerprint that appears for the first time in that cluster and deleting all subsequent duplicate log fingerprints within that cluster that are identical to the retained fingerprint. The deduplication operation compresses hundreds or thousands of logs within a cluster into a single fingerprint record, achieving a second layer of noise reduction. For example, in a financial scenario, the first cluster contains 3200 payment timeout logs and their fingerprints; after deduplication, only the log fingerprint that appears for the first time in that cluster is retained, and the remaining 3199 duplicate fingerprints are deleted. The second cluster contains 1500 risk control interception logs; after deduplication, only one fingerprint is retained. The third cluster contains 300 database connection exception logs; after deduplication, only one fingerprint is retained. After clustering and deduplication of the 5000 logs within the entire time window, only 3 log fingerprints are retained.
[0044] S45: The log fingerprints retained in each cluster are each pushed as an alarm message to the log security management platform. Specifically, in this embodiment of the invention, each log fingerprint retained after deduplication represents an independent abnormal event occurring within the current time window. The system uses these retained log fingerprints as alarm identifiers for the corresponding clusters, and each fingerprint is encapsulated into an independent alarm message for push. The alarm message may include key fields such as log fingerprint, abnormality type, number of logs in the cluster, and first occurrence time, helping operations and maintenance personnel quickly understand the overall situation of the abnormality. The target of the push is the log security management platform, which is responsible for the display, distribution, tracking, and closed-loop management of alarms. For example, in a financial scenario, the system encapsulates the three deduplicated log fingerprints into three alarm messages. The first alarm corresponds to a payment service timeout anomaly, carrying information including that the anomaly occurred 3200 times within 1 minute, with the first occurrence at 10:30:05; the second alarm corresponds to a risk control trigger interception anomaly, occurring 1500 times; and the third alarm corresponds to a database connection anomaly, occurring 300 times. These three alarms are pushed to the log security management platform. Operations personnel can determine the scope of the payment service timeout impact based on the number of alarm log entries and prioritize handling it.
[0045] In this embodiment of the invention, after clustering the second logs to be analyzed generated within a preset time window based on the log fingerprint, deduplicating the second logs to be analyzed under the same cluster, and pushing the deduplicated and retained log fingerprints as an alarm message for the corresponding cluster to the log security management platform, the method further includes steps S51-S54: S51: Receive alarm confirmation information returned by the log security management platform. The alarm confirmation information includes an alarm identifier and a confirmation result label. Specifically, in this embodiment of the invention, after the log security management platform pushes the clustered and deduplicated alarm information to the operation and maintenance personnel, the personnel will manually confirm and handle the alarm, and mark the confirmation result on the platform. For example, confirming that the alarm is a real anomaly, determining that the alarm is a false alarm, or marking the alarm as needing further observation. The platform encapsulates the confirmation result of the operation and maintenance personnel into alarm confirmation information, including the alarm identifier and the confirmation result label, and returns it to the log noise reduction system. This step establishes a closed-loop path from automatic alarm to manual feedback, enabling the system to obtain high-quality feedback signals marked by humans. For example, in a financial scenario, if a payment service timeout alarm is confirmed by the operation and maintenance personnel as a real anomaly, and the confirmation result label is "real anomaly," the platform returns alarm confirmation information containing the alarm identifier and the "real anomaly" label to the system. After investigation by operations and maintenance personnel, it was found that the risk control trigger alarm was caused by a merchant initiating a batch of transactions, which triggered the risk control threshold. This was a normal business fluctuation rather than a system anomaly. The operations and maintenance personnel marked it as a false alarm, and the platform returned the alarm confirmation information to the system.
[0046] S52: Based on the alarm identifier, query the corresponding log identifier from the fingerprint index table. Specifically, in this embodiment of the invention, the alarm confirmation information carries the alarm identifier, i.e., the log fingerprint retained after deduplication, while model fine-tuning requires the original log data corresponding to the alarm. The system utilizes the mapping relationship between hash fingerprints and log identifiers in the established fingerprint index table, using the alarm identifier as the query key, to search for all log identifiers associated with the hash fingerprint in the fingerprint index table, thereby tracing back to all second logs to be analyzed corresponding to the alarm. For example, in a financial scenario, the system uses the fingerprint identifier of a payment service timeout alarm as the query key, and retrieves 3200 log identifiers associated with that fingerprint in the fingerprint index table. These log identifiers correspond to all original logs of payment timeout type within the current time window. For risk control trigger interception alarms, the system also retrieves 1500 associated log identifiers in the fingerprint index table, corresponding to all falsely reported risk control interception type logs.
[0047] S53: Based on the confirmation result label, construct feedback training samples using the log features and log metadata of the second log to be analyzed corresponding to the queried log identifier. Specifically, in this embodiment of the invention, the manual confirmation result is the most direct evaluation of the model's alarm quality. Combining the confirmation result with the semantic features and metadata of the original log can construct training samples with high-quality labels to correct model biases. For example, in a financial scenario, a payment service timeout alarm is labeled as a genuine anomaly. The system extracts 100 logs proportionally from the 3200 associated logs, and pairs the log features such as semantic intent, anomaly type, and confidence score generated in step S23, as well as the log metadata such as the system name (payment system) and deployment environment (production environment) generated in step S31, with the genuine anomaly label to construct 100 positive example feedback training samples. A risk control triggered interception alarm is labeled as a false alarm. The system extracts 100 logs from the 1500 associated logs, and pairs their log features and log metadata with the false alarm label to construct 100 negative example feedback training samples.
[0048] S54: The feedback training samples are sent to the pre-trained large model for incremental fine-tuning. Specifically, in this embodiment of the invention, although the pre-trained large model has strong capabilities in general semantic understanding, it may have biases in anomaly identification in specific business scenarios. For example, it may misjudge normal fluctuations specific to certain businesses as anomalies, or it may inconsistently identify the same type of anomaly in different expressions. Incremental fine-tuning involves using feedback training samples to update parameters on a small scale based on the pre-trained model, allowing the model to gradually adapt to the anomaly identification rules of specific business scenarios while retaining the model's original general semantic understanding capabilities. For example, in a financial scenario, the system merges 100 positive examples of payment service timeouts and 100 negative examples of risk control trigger interceptions into 200 feedback training samples, which are then sent to the large model for incremental fine-tuning. The model strengthens its ability to identify payment timeout-type anomaly expressions through positive examples, and learns to distinguish between normal risk control triggers caused by concentrated merchant transactions and genuine risk control system anomalies through negative examples, thus reducing false positives for similar normal business behaviors during subsequent operation.
[0049] As can be seen, in the above-described solution, specifically in this invention, by performing contextual semantic analysis and extracting log features using a pre-trained large model, key business information can be accurately identified from semantically complex logs, improving the accuracy of feature extraction. Simultaneously, by integrating the extracted log features with log metadata such as deployment environment data, a log fingerprint is constructed that includes both semantic anomaly descriptions and technical context. This fingerprint can accurately distinguish similar anomalies in different environments, providing a reliable basis for clustering and noise reduction, and effectively reducing the number of alarms from massive log data.
[0050] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0051] In one embodiment, a log denoising device based on a large model is provided, which corresponds one-to-one with the log denoising method based on a large model in the above embodiments. For example... Figure 7 As shown, Figure 7 This is a schematic diagram of a log denoising device based on a large model according to an embodiment of the present invention. The log denoising device based on a large model includes an acquisition module 71, an analysis module 72, a construction module 73, and a clustering module 74. Detailed descriptions of each functional module are as follows: The acquisition module 71 is used to acquire the first log to be analyzed and the log metadata corresponding to the first log to be analyzed, and to perform rule detection on the first log to be analyzed through the real-time stream processing engine to obtain the second log to be analyzed that matches the rule. Analysis module 72 is used to send the second log to be analyzed to a pre-trained large model for contextual semantic analysis and to extract log features of the second log to be analyzed; The construction module 73 is used to fuse the log features of the second log to be analyzed with the log metadata, and construct a log fingerprint based on the fused information; The clustering module 74 is used to cluster the second logs to be analyzed generated within a preset time window based on the log fingerprint, and to deduplicate the second logs to be analyzed under the same cluster. The log fingerprints retained after deduplication are used as an alarm message for the corresponding cluster and pushed to the log security management platform.
[0052] In one embodiment, the acquisition module 71 is specifically used for: Obtain the raw log data stream generated in real time, take each log in the raw log data stream as the first log to be analyzed, and synchronously collect the log metadata associated with the first log to be analyzed; Load a preset rule base, call at least one detection rule stored in the rule base, and extract the matching conditions and action strategies corresponding to the detection rule; The real-time stream processing engine parses the first log to be analyzed, extracts the log fields of the first log to be analyzed, and compares the log fields with the matching conditions in the rule base; If the log field satisfies the matching condition of any of the detection rules, then the corresponding first log to be analyzed is determined to have hit the rule, and the first log to be analyzed that has hit the rule is marked as the second log to be analyzed.
[0053] In one embodiment, the analysis module 72 is specifically used for: The second log to be analyzed is formatted and converted according to a preset log template to generate structured log data; Based on the structured log data, an input sequence containing log keywords, log levels, and exception codes is constructed, and the input sequence is sent to the pre-trained large model; The pre-trained large model performs contextual understanding on the input sequence to identify the semantic intent and anomaly type of the second log to be analyzed, and obtains semantic labels and corresponding confidence scores. The semantic intent, the anomaly type, and the confidence score are embedded and encoded to generate the log semantic feature vector of the second log to be analyzed.
[0054] In one embodiment, the construction module 73 is specifically used for: The categorical fields in the log metadata are one-hot encoded, and the numerical fields are normalized encoded to generate a metadata encoding vector. The log semantic feature vector is concatenated with the metadata encoding vector to generate a log fusion vector; The log fusion vector is hash-encoded using the Locality Sensitive Hash (LSH) algorithm to generate the hash fingerprint of the second log to be analyzed. The hash fingerprint is stored in the fingerprint index table, and the log identifier of the second log to be analyzed is associated with the hash fingerprint.
[0055] In one embodiment, the construction module 73 is further configured to: Based on the log fusion vector, the number of hash tables m and the hash encoding length k of each hash table are set; Construct m sets of random projection matrices, each set of random projection matrices containing k random projection vectors of the same dimension as the log fusion vector; The log fusion vector is projected using each set of random projection matrices to obtain m sets of projection results for the log fusion vector, where each set of projection results contains k projection values. The projection values of the m groups of log fusion vectors are binarized, converted into binary bits, and concatenated into k-bit binary sequences according to the order of the projection values to generate m k-bit binary sequences. The m k-bit binary sequences are used as hash fingerprints of the second log to be analyzed in each hash bucket, and each hash fingerprint is associated and stored in the corresponding fingerprint index table.
[0056] In one embodiment, the clustering module 74 is specifically used for: Based on the preset time window, obtain each of the second logs to be analyzed generated within the current time window and its corresponding log fingerprint; Use the log fingerprint of the first second log to be analyzed within the current time window as the initial cluster center; The similarity between the log fingerprint of each second log to be analyzed that is not the initial cluster center within the current time window and each cluster center is calculated sequentially. If the similarity is greater than or equal to the preset threshold, it is assigned to the corresponding cluster. If the similarity is less than the preset threshold, a new cluster is created. The second log to be analyzed within each cluster is deduplicated, retaining the log fingerprint that appears for the first time in each cluster and deleting duplicate log fingerprints within the cluster. The log fingerprints retained in each cluster are treated as alarm messages and pushed to the log security management platform.
[0057] In one embodiment, the large model-based log denoising device is further used for: Receive alarm confirmation information returned by the log security management platform, wherein the alarm confirmation information includes an alarm identifier and a confirmation result tag; Based on the alarm identifier, query the corresponding log identifier from the fingerprint index table; Based on the confirmation result label, a feedback training sample is constructed for the log features and log metadata of the second log to be analyzed corresponding to the queried log identifier; The feedback training samples are sent to the pre-trained large model, and the pre-trained large model is incrementally fine-tuned.
[0058] This invention provides a log denoising device based on a large model. By using a pre-trained large model to perform contextual semantic analysis on logs and extract log features, it can accurately identify key business information from semantically complex logs, improving the accuracy of feature extraction. Simultaneously, the extracted log features are integrated with log metadata such as deployment environment data to construct a log fingerprint that includes both semantic anomaly descriptions and technical context. This fingerprint can accurately distinguish similar anomalies in different environments, providing a reliable basis for clustering denoising and effectively reducing the number of alarms from massive log data.
[0059] Specific limitations regarding the large-model-based log denoising device can be found in the limitations of the large-model-based log denoising method described above, and will not be repeated here. Each module in the aforementioned large-model-based log denoising device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.
[0060] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, Figure 8 This is a schematic diagram of a computer device according to an embodiment of the present invention. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a server-side log denoising method based on a large model.
[0061] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 9 As shown, Figure 9 This is another structural schematic diagram of a computer device according to an embodiment of the present invention. The computer device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements client-side functions or steps of a log denoising method based on a large model.
[0062] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain the first log to be analyzed and the corresponding log metadata. Perform rule detection on the first log to be analyzed through the real-time stream processing engine to obtain the second log to be analyzed that matches the rule. The second log to be analyzed is sent to a pre-trained large model for contextual semantic analysis, and log features of the second log to be analyzed are extracted. The log features of the second log to be analyzed are fused with the log metadata, and a log fingerprint is constructed based on the fused information. Based on the log fingerprint, the second log to be analyzed generated within the preset time window is clustered, and the second log to be analyzed under the same cluster is deduplicated. The log fingerprint retained after deduplication is used as an alarm message for the corresponding cluster and pushed to the log security management platform.
[0063] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain the first log to be analyzed and the corresponding log metadata. Perform rule detection on the first log to be analyzed through the real-time stream processing engine to obtain the second log to be analyzed that matches the rule. The second log to be analyzed is sent to a pre-trained large model for contextual semantic analysis, and log features of the second log to be analyzed are extracted. The log features of the second log to be analyzed are fused with the log metadata, and a log fingerprint is constructed based on the fused information. Based on the log fingerprint, the second log to be analyzed generated within the preset time window is clustered, and the second log to be analyzed under the same cluster is deduplicated. The log fingerprint retained after deduplication is used as an alarm message for the corresponding cluster and pushed to the log security management platform.
[0064] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.
[0065] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0066] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0067] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.
[0068] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.
Claims
1. A log denoising method based on a large model, characterized in that, include: Obtain the first log to be analyzed and the corresponding log metadata. Perform rule detection on the first log to be analyzed through the real-time stream processing engine to obtain the second log to be analyzed that matches the rule. The second log to be analyzed is sent to a pre-trained large model for contextual semantic analysis, and log features of the second log to be analyzed are extracted. The log features of the second log to be analyzed are fused with the log metadata, and a log fingerprint is constructed based on the fused information; Based on the log fingerprint, the second log to be analyzed generated within the preset time window is clustered, and the second log to be analyzed under the same cluster is deduplicated. The log fingerprint retained after deduplication is used as an alarm message for the corresponding cluster and pushed to the log security management platform.
2. The log denoising method based on a large model according to claim 1, characterized in that, The process of obtaining the first log to be analyzed and the corresponding log metadata, and then performing rule detection on the first log to be analyzed through a real-time stream processing engine to obtain the second log to be analyzed that matches the rules, includes: Obtain the raw log data stream generated in real time, take each log in the raw log data stream as the first log to be analyzed, and synchronously collect the log metadata associated with the first log to be analyzed; Load a preset rule base, call at least one detection rule stored in the rule base, and extract the matching conditions and action strategies corresponding to the detection rule; The real-time stream processing engine parses the first log to be analyzed, extracts the log fields of the first log to be analyzed, and compares the log fields with the matching conditions in the rule base; If the log field satisfies the matching condition of any of the detection rules, then the corresponding first log to be analyzed is determined to have hit the rule, and the first log to be analyzed that has hit the rule is marked as the second log to be analyzed.
3. The log denoising method based on a large model according to claim 1, characterized in that, The step of sending the second log to be analyzed to a pre-trained large model for contextual semantic analysis and extracting log features from the second log to be analyzed includes: The second log to be analyzed is formatted and converted according to a preset log template to generate structured log data; Based on the structured log data, an input sequence containing log keywords, log levels, and exception codes is constructed, and the input sequence is sent to the pre-trained large model; The pre-trained large model performs contextual understanding on the input sequence to identify the semantic intent and anomaly type of the second log to be analyzed, and obtains semantic labels and corresponding confidence scores. The semantic intent, the anomaly type, and the confidence score are embedded and encoded to generate the log semantic feature vector of the second log to be analyzed.
4. The log denoising method based on a large model according to claim 3, characterized in that, The step of fusing the log features of the second log to be analyzed with the log metadata, and constructing a log fingerprint based on the fused information, includes: The categorical fields in the log metadata are one-hot encoded, and the numerical fields are normalized encoded to generate a metadata encoding vector. The log semantic feature vector is concatenated with the metadata encoding vector to generate a log fusion vector; The log fusion vector is hash-encoded using the Locality Sensitive Hash (LSH) algorithm to generate the hash fingerprint of the second log to be analyzed. The hash fingerprint is stored in the fingerprint index table, and the log identifier of the second log to be analyzed is associated with the hash fingerprint.
5. The log denoising method based on a large model according to claim 4, characterized in that, The step of hashing the log fusion vector using the locality-sensitive hashing algorithm to generate the hash fingerprint of the second log to be analyzed includes: Based on the log fusion vector, the number of hash tables m and the hash encoding length k of each hash table are set; Construct m sets of random projection matrices, each set of random projection matrices containing k random projection vectors of the same dimension as the log fusion vector; The log fusion vector is projected using each set of random projection matrices to obtain m sets of projection results for the log fusion vector, where each set of projection results contains k projection values. The projection values of the m groups of log fusion vectors are binarized, converted into binary bits, and concatenated into k-bit binary sequences according to the order of the projection values to generate m k-bit binary sequences. The m k-bit binary sequences are used as hash fingerprints of the second log to be analyzed in each hash bucket, and each hash fingerprint is associated and stored in the corresponding fingerprint index table.
6. The log denoising method based on a large model according to claim 1, characterized in that, The process of clustering the second logs to be analyzed generated within a preset time window based on the log fingerprint, deduplicating the second logs to be analyzed within the same cluster, and pushing the retained log fingerprints after deduplication as an alarm message for the corresponding cluster to the log security management platform includes: Based on the preset time window, obtain each of the second logs to be analyzed generated within the current time window and its corresponding log fingerprint; Use the log fingerprint of the first second log to be analyzed within the current time window as the initial cluster center; The similarity between the log fingerprint of each second log to be analyzed that is not the initial cluster center within the current time window and each cluster center is calculated sequentially. If the similarity is greater than or equal to the preset threshold, it is assigned to the corresponding cluster. If the similarity is less than the preset threshold, a new cluster is created. The second log to be analyzed within each cluster is deduplicated, retaining the log fingerprint that appears for the first time in each cluster and deleting duplicate log fingerprints within the cluster. The log fingerprints retained in each cluster are treated as alarm messages and pushed to the log security management platform.
7. The log denoising method based on a large model according to claim 1, characterized in that, The method further includes: clustering the second logs to be analyzed generated within a preset time window based on the log fingerprint, deduplicating the second logs to be analyzed within the same cluster, and pushing the retained log fingerprints after deduplication as an alarm message for the corresponding cluster to the log security management platform; Receive alarm confirmation information returned by the log security management platform, wherein the alarm confirmation information includes an alarm identifier and a confirmation result tag; Based on the alarm identifier, query the corresponding log identifier from the fingerprint index table; Based on the confirmation result label, a feedback training sample is constructed for the log features and log metadata of the second log to be analyzed corresponding to the queried log identifier; The feedback training samples are sent to the pre-trained large model, and the pre-trained large model is incrementally fine-tuned.
8. A log denoising device based on a large model, characterized in that, include: The acquisition module is used to acquire the first log to be analyzed and the log metadata corresponding to the first log to be analyzed, and to perform rule detection on the first log to be analyzed through the real-time stream processing engine to obtain the second log to be analyzed that matches the rule. The analysis module is used to send the second log to be analyzed to a pre-trained large model for contextual semantic analysis and to extract the log features of the second log to be analyzed. A construction module is used to fuse the log features of the second log to be analyzed with the log metadata, and construct a log fingerprint based on the fused information; The clustering module is used to cluster the second logs to be analyzed generated within a preset time window based on the log fingerprint, and to deduplicate the second logs to be analyzed under the same cluster. The log fingerprints retained after deduplication are used as an alarm message for the corresponding cluster and pushed to the log security management platform.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the log denoising method based on a large model as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the log denoising method based on a large model as described in any one of claims 1 to 7.
