A log false alarm whitelist filtering method, device, equipment and storage medium

Abstract

The application relates to the technical field of data processing, and discloses a log false alarm whitelist filtering method, device, equipment and storage medium, which comprises the following steps: acquiring log data; checking whether the log data hits a global whitelist in a preset whitelist library; if the log data hits the global whitelist, filtering the log data; if the log data does not hit the global whitelist, judging whether the log data contains sensitive information by using a preset large model; if the log data contains sensitive information, performing alarm pushing on the log data; after receiving false alarm information fed back by a user, taking the log data as false alarm log data, generating a latest global whitelist according to the false alarm log data, and configuring the latest global whitelist into the preset whitelist library. The application can be applied to the log sensitive information detection scene of financial technology and medical health, and saves the time cost and system management cost of developers.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, device, and medium for filtering false alarms in logs using a whitelist. Background Technology

[0002] In the fintech and healthcare sectors, applications generate large amounts of log data during operation, some of which may contain sensitive information. When third-party or malicious applications read log data containing sensitive information, it can lead to the leakage of sensitive information and pose information security risks. Current log sensitive information detection systems generally suffer from high false positive rates, inadequate feedback mechanisms, and inflexible whitelist mechanisms. Existing systems typically rely on static rules or keyword matching for sensitive information identification. When developers report a log entry as a false positive, the system often fails to incorporate this information into a rule, causing similar log entries to trigger alerts again in subsequent detections, increasing developers' time and system governance costs. Summary of the Invention

[0003] This invention provides a method, apparatus, computer equipment, and medium for filtering log false alarms using a whitelist, in order to solve the technical problem of high time costs for developers and system governance costs caused by duplicate log false alarms in the prior art.

[0004] Firstly, a method for filtering false alarms in logs using a whitelist is provided, including: Retrieve log data; Check whether the log data matches the global whitelist in the preset whitelist library; If the log data matches the global whitelist, then the log data is filtered. If the log data does not match the global whitelist, a preset large model is used to determine whether the log data contains sensitive information. If the log data contains sensitive information, an alarm will be pushed to the log data. After receiving false alarm information from users, the log data is used as false alarm log data, a new global whitelist is generated based on the false alarm log data, and the new global whitelist is configured into the preset whitelist library.

[0005] Secondly, a log false alarm whitelist filtering device is provided, including: The data acquisition module is used to acquire log data; The log checking module is used to check whether the log data matches the global whitelist in the preset whitelist library; A filtering module is used to filter the log data if the log data matches the global whitelist. The sensitive information judgment module is used to determine whether the log data contains sensitive information if the log data does not match the global whitelist. The alarm push module is used to push alarms to the log data if the log data contains sensitive information. The whitelist database update module is used to, upon receiving false alarm information from a user, use the log data as false alarm log data, generate a latest global whitelist based on the false alarm log data, and configure the latest global whitelist into the preset whitelist database.

[0006] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described log false alarm whitelist filtering method.

[0007] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described log false alarm whitelist filtering method.

[0008] In the aforementioned solution implemented by the log false alarm whitelist filtering method, apparatus, computer equipment, and storage medium, log data can be obtained through a client; the log data is checked to see if it matches the global whitelist in the preset whitelist library; if the log data matches the global whitelist, the log data is filtered; if the log data does not match the global whitelist, a preset large model is used to determine whether the log data contains sensitive information; if the log data contains sensitive information, an alarm is pushed to the log data; after receiving false alarm information from the user, the log data is used as false alarm log data, a latest global whitelist is generated based on the false alarm log data, and the latest global whitelist is configured into the preset whitelist library. This invention is specifically for the fields of financial technology and healthcare. Log data from fields such as healthcare is checked against a global whitelist in a pre-defined whitelist database. If the log data matches the global whitelist, it is filtered. If the log data does not match the global whitelist, a pre-defined large model is used to determine if the log data contains sensitive information. If sensitive information is found, an alert is pushed to the log data. Upon receiving false alarm feedback from users, the log data is used as the false alarm log data, and a new global whitelist is generated based on the false alarm log data. This new global whitelist is then configured into the pre-defined whitelist database. This allows the pre-defined whitelist database to be adjusted after a user reports a false alarm, enabling subsequent filtering of similar log data to prevent further alerts. This effectively reduces the possibility of false alarms, thereby saving developers' time and system governance costs. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a schematic diagram of an application environment for a log false alarm whitelist filtering method in one embodiment of the present invention.

[0011] Figure 2 This is a flowchart illustrating a log false alarm whitelist filtering method in one embodiment of the present invention.

[0012] Figure 3 This is a schematic diagram of a log false alarm whitelist filtering device in one embodiment of the present invention.

[0013] Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention.

[0014] Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] The log false alarm whitelist filtering method provided in this embodiment of the invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can receive and obtain log data from the client and check whether the log data matches the global whitelist in the preset whitelist library. If the log data does not match the global whitelist, a preset large model is used to determine whether the log data contains sensitive information. If the log data contains sensitive information, an alarm is pushed for the log data. After receiving false alarm information from the user, the log data is used as false alarm log data, and the structural features of the false alarm log data are extracted. A regular expression is generated based on the structural features of the false alarm log data, and a new global whitelist is formed based on the regular expression. The new global whitelist is then configured into the preset whitelist library. In this invention, log data is checked for log data in fields such as financial technology and healthcare. The system checks whether the log data matches the global whitelist in the preset whitelist library. If the log data matches the global whitelist, it is filtered. If the log data does not match the global whitelist, a preset large model is used to determine if the log data contains sensitive information. If the log data contains sensitive information, an alarm is pushed to the log data. After receiving false alarm feedback from users, the system uses the log data as false alarm log data, generates a new global whitelist based on the false alarm log data, and configures the new global whitelist into the preset whitelist library. This allows the preset whitelist library to be adjusted after users report that the log data is a false alarm, so that subsequent log data of the same type can be filtered, thereby preventing alarms from being triggered again, effectively reducing the possibility of false alarms, and thus saving developers' time and system governance costs. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0017] Please see Figure 2 As shown, Figure 2A flowchart illustrating the log false alarm whitelist filtering method provided in this embodiment of the invention includes the following steps: S10: Get log data.

[0018] The log false alarm whitelist filtering method provided by this invention can be applied to log sensitive information detection systems in various application scenarios. These systems are typically implemented through a server that can receive log data in real time. For example, in healthcare or insurance applications, there is a large amount of log data, and log sensitive information detection systems filter sensitive information to improve data security.

[0019] One implementation method is to consume log data from a log cloud Kafka cluster. In Apache Kafka, a cluster refers to a system composed of multiple Kafka servers (brokers) that jointly manage functions such as data replication, load balancing, and failover. In a Kafka cluster, each broker runs on an independent server, and they are interconnected and communicate with each other via a network.

[0020] After obtaining the log data, and before checking whether the log data matches the global whitelist in the preset whitelist library, the method further includes: standardizing the log data to parse the log data into a standard Log object.

[0021] A Log object typically refers to a Logger instance used in programming to record application runtime status, debugging information, or error messages, or a logging utility class provided by a specific language / framework. Standardizing the log data allows it to be parsed into standard Log objects. Specifically, key fields of the log data can be extracted, such as application name (appName), system name (systemName), message (message), and time (logTime). When checking whether log data matches the global whitelist in a preset whitelist library, matching can be performed using these key fields to improve the efficiency of this check.

[0022] In this embodiment, after acquiring log data, before checking whether the log data matches the global whitelist in the preset whitelist library, the log data is standardized to parse the log data into a standard Log object. Using a standard Log object makes it easier to check whether the log data matches the global whitelist in the preset whitelist library, which can improve the checking efficiency.

[0023] S20: Check whether the log data matches the global whitelist in the preset whitelist library.

[0024] It should be noted that the global whitelist (rules) applies to all sensitive types and can filter general false alarm logs.

[0025] As one implementation, step S20, checking whether the log data matches the global whitelist in the preset whitelist library, may include: converting the log data (standard Log object) into string data, matching the string data with any global whitelist in the preset whitelist library, and if at least a part of the string data matches the global whitelist, then it is determined that the log data matches the global whitelist in the preset whitelist library; otherwise, the log data does not match the global whitelist in the preset whitelist library.

[0026] As an example, before sensitivity detection, a global whitelist pre-filter is performed, using regular expressions to match the input Log objects. These regular expressions can be loaded from a database (a pre-defined whitelist library) and pre-compiled during initialization to avoid performance overhead from repeated parsing.

[0027] S30: If the log data matches the global whitelist, then the log data is filtered.

[0028] As an example, log data is matched against a global whitelist in a preset whitelist library. If the log data matches any entry in the global whitelist using regular expressions, false (error) is immediately returned, indicating that the information is non-sensitive. The log data is then filtered, and subsequent detection logic is skipped. This mechanism can be used to quickly allow security logs to pass through, prevent false alarms, and improve detection efficiency.

[0029] S40: If the log data does not match the global whitelist, a preset large model is used to determine whether the log data contains sensitive information.

[0030] In step S40, determining whether the log data contains sensitive information using a preset large model includes: Determine whether the log data conforms to a preset sensitive type, whether the log data hits a local whitelist of sensitive types, and whether the log data is marked as sensitive log data; If the log data does not conform to the preset sensitive type, the log data hits the sensitive type local whitelist, or the log data is marked as sensitive log data, then the preset large model is used to determine whether the log data contains sensitive information.

[0031] As one implementation method, log data is sent to an AI agent (a pre-defined large model) via Flink broadcasting for secondary verification without blocking the processing thread of the current Flink task. The AI ​​agent determines whether the log data contains sensitive information. After the AI ​​agent returns the verification result, its authenticity can be verified by iterating through all sensitive types and their specific values ​​(such as ID card numbers and mobile phone numbers) identified by the AI ​​agent and checking whether these values ​​are actually contained in the original log. If any sensitive value is found to be absent from the original log, it is determined that the AI ​​agent has made a mistake in identification, and false is returned for filtering, ensuring that only sensitive information confirmed by the AI ​​agent and actually present in the log data is retained.

[0032] In this embodiment, the system determines whether the log data conforms to a preset sensitive type, whether the log data hits a local whitelist of sensitive types, and whether the log data is marked as sensitive log data. If the log data does not conform to the preset sensitive type, hits a local whitelist of sensitive types, or is marked as sensitive log data, that is, if the log data meets any of the above conditions, the system needs to use a preset large model to determine again whether the log data contains sensitive information (the system has already performed this once by matching the log data using a global whitelist) in order to ensure the accuracy of the determination of sensitive information and avoid filtering log data containing sensitive information.

[0033] In step S40, it is determined whether the log data conforms to a preset sensitive type, whether the log data hits a local whitelist of sensitive types, and whether the log data is marked as sensitive log data, including: The log data is subjected to sensitivity type detection to determine whether the log data conforms to a preset sensitivity type; If the log data matches a preset sensitive type, then determine whether the log data matches the local whitelist of sensitive types; If the log data does not match the sensitive type local whitelist, then the log data will be marked as sensitive log data.

[0034] As one implementation method, the log data is subjected to sensitive type detection to determine whether the log data conforms to a preset sensitive type; if the log data does not conform to the preset sensitive type, a preset large model is used to determine whether the log data contains sensitive information. Here, the log data not conforming to the preset sensitive type may mean that the log data does not conform to all sensitive types in the local whitelist (rules), or the log data has completed the matching judgment of all sensitive rules (sensitive types) in the local whitelist.

[0035] If the log data matches a preset sensitive type, it is determined whether the log data matches the local whitelist of sensitive types. If the log data matches the local whitelist of sensitive types, a preset large model is used to determine whether the log data contains sensitive information. Here, the log data matching the local whitelist of sensitive types can mean that the log data has completed the matching and judgment of all sensitive rules in the local whitelist.

[0036] If the log data does not match the local whitelist of sensitive types, the log data will be marked as sensitive log data, and the pre-set large model will be used to reconfirm whether there is sensitive information in the log data. The log data not matching the local whitelist of sensitive types may mean that the log data has completed the matching and judgment of all sensitive rules in the local whitelist.

[0037] It should be noted that regardless of whether the log data matches the sensitive type local whitelist, it is necessary to use the preset large model to determine again whether the log data contains sensitive information.

[0038] As one implementation method, a global whitelist includes more sensitive types than a local whitelist, while a local whitelist has more granular sensitive types. For log data that does not match the global whitelist, regular expression matching detection is required for each preset sensitive type. For example, preset sensitive types may include ID card, bank card, mobile phone number, etc. For ID card, regular expression matching is used to detect logs that do not match the global whitelist. If the regular expression matches, the detected sensitive information is recorded. Regular expression matching detection can be performed for preset sensitive types such as bank card and mobile phone number to achieve multi-type sensitive information identification.

[0039] In this embodiment, the global whitelist is matched before the detection rules are executed, and it is applicable to all sensitive types, filtering general false alarm logs; the local whitelist is matched when entering the detection rules for each sensitive type, and it is applicable to specific types of false alarm logs, achieving more refined filtering control, improving the flexibility and accuracy of whitelist filtering, and avoiding the risk of missed detection that may be caused by the global whitelist.

[0040] In step S40, determining whether the log data matches the sensitive type local whitelist includes: The log data is verified by a signature. If the log data passes the signature verification, it is determined whether the log data matches the sensitive type local whitelist.

[0041] As one implementation method, log data is determined to conform to a preset sensitive type, meaning the log data is suspected of containing sensitive information and requires further verification. Specifically, the log data is subjected to feature code verification. For example, if the preset sensitive type is a mobile phone number, it can be verified through variable name keywords; if the preset sensitive type is an ID card, it can be verified through the first six digits of the area code and the verification code; if the preset sensitive type is a bank card, it can be verified through the bank card suffix and the Luhn algorithm, where the Luhn algorithm is a checksum algorithm used to verify identity identification codes.

[0042] S50: If the log data contains sensitive information, an alarm will be pushed to the log data; As one implementation method, if the log data contains sensitive information, the log data can be written to the database and pushed to the security management platform. The security management platform receives the log data, and the developers (users) confirm whether the log data contains sensitive information. If the log data does contain sensitive information, rectification can be scheduled. If it is confirmed that the log data does not contain sensitive information, a false alarm message is reported.

[0043] S60: After receiving false alarm information from the user, the log data is used as false alarm log data, a new global whitelist is generated based on the false alarm log data, and the new global whitelist is configured into the preset whitelist library.

[0044] In step S60, the latest global whitelist is generated based on the false alarm log data, including: Extract the structural features of the false alarm log data, generate a regular expression based on the structural features of the false alarm log data, and form the latest global whitelist based on the regular expression.

[0045] As one implementation method, once developers confirm that an alarm push is a false alarm, they analyze the structural characteristics of the false alarm log, generate a corresponding regular expression, and then create a new global whitelist based on this regular expression and store it in the database (a preset whitelist database). This new global whitelist includes the regular expression generated from the false alarm feedback. This regular expression can accurately match similar false alarm logs, preventing false alarm logs from repeatedly triggering alarms. For example, if a log entry is "[INFO] User loginfailed due to invalid password", its structural characteristics can be extracted to generate a regular expression such as "^\[INFO\]User loginfailed due to invalid password$", which can be used for subsequent matching.

[0046] In this embodiment, the structural characteristics of the false alarm logs confirmed by the developers are analyzed, and corresponding regular expressions are generated. Based on these regular expressions, similar false alarm logs can be accurately matched, avoiding repeated alarms triggered by false alarm logs and significantly reducing the false alarm rate.

[0047] In step S20, checking whether the log data matches the global whitelist in the preset whitelist library includes checking whether the log data matches the global whitelist in the preset whitelist library through a detection engine. After step S60 configures the latest global whitelist into the preset whitelist library, step 70 is further included: loading the latest whitelist from the preset whitelist library into the detection engine through a scheduled task.

[0048] One implementation approach is to use the Flink stream processing framework to periodically (e.g., every hour) synchronize global whitelist information from a pre-defined whitelist library and load the updated regular expressions (the latest whitelist) into the detection engine. This mechanism supports real-time updates, ensuring that the whitelist rules are always consistent with the latest feedback, without requiring manual system restarts or redeployment, thus improving system response speed and governance efficiency.

[0049] In this embodiment, after confirming a false alarm, developers can report the false alarm log through the system interface. Upon receiving the feedback, the log is added to the whitelist library, and a regular expression is generated based on its structural characteristics to realize the feedback of false alarm information and rule updates. This mechanism constructs a complete closed loop from alarm identification, false alarm confirmation to rule updates, improving the intelligence level and governance capabilities of whitelist filtering.

[0050] As can be seen, in the above solution, for log data in fields such as fintech and healthcare, the system checks whether the log data matches the global whitelist in the preset whitelist library. If the log data matches the global whitelist, it is filtered. If the log data does not match the global whitelist, a preset large model is used to determine whether the log data contains sensitive information. If the log data contains sensitive information, an alarm is pushed to the log data. After receiving false alarm information from users, the log data is used as the false alarm log data, and a new global whitelist is generated based on the false alarm log data. The new global whitelist is configured into the preset whitelist library. This allows the preset whitelist library to be adjusted after users report that the log data is a false alarm, so that subsequent log data of the same type can be filtered, thereby preventing alarms from being triggered again, effectively reducing the possibility of false alarms, and thus saving developers' time and system governance costs.

[0051] The log false alarm whitelist filtering method provided in this invention supports two types of whitelists: a global whitelist and a local whitelist. The global whitelist is matched before the detection rules are executed, applicable to all sensitive types, and can filter general false alarm logs. The local whitelist, on the other hand, is matched when entering the detection rules for each sensitive type, applicable to specific types of false alarm logs, enabling more granular filtering control. This classification mechanism improves the flexibility and accuracy of whitelist filtering, avoiding the risk of missed detections that may arise from a global whitelist.

[0052] The log false alarm whitelist filtering method provided in this invention significantly reduces the false alarm rate of log detection systems by introducing a structured whitelist filtering mechanism, thereby reducing repetitive troubleshooting work for developers and improving system governance efficiency. The Flink dynamic loading mechanism ensures real-time updates of the whitelist rules, improving system response speed and stability. The false alarm feedback closed-loop mechanism enables the system to continuously learn and optimize, enhancing its intelligence and governance capabilities, forming a self-evolving governance model, and providing a more efficient and intelligent solution for detecting sensitive log information.

[0053] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0054] In one embodiment, a log false alarm whitelist filtering device is provided, which corresponds one-to-one with the log false alarm whitelist filtering method in the above embodiments. For example... Figure 3 As shown, the log false alarm whitelist filtering device 30 includes a data acquisition module 31, a log inspection module 32, a filtering module 33, a sensitive information judgment module 34, an alarm push module 35, and a whitelist database update module 36. Data acquisition module 31 is used to acquire log data; Log checking module 32 is used to check whether the log data matches the global whitelist in the preset whitelist library; The filtering module 33 is used to filter the log data if the log data matches the global whitelist. The sensitive information judgment module 34 is used to determine whether the log data contains sensitive information if the log data does not match the global whitelist; The alarm push module 35 is used to push an alarm to the log data if the log data contains sensitive information. The whitelist database update module 36 is used to, after receiving false alarm information from the user, use the log data as false alarm log data, generate the latest global whitelist based on the false alarm log data, and configure the latest global whitelist into the preset whitelist database.

[0055] In one embodiment, the sensitive information determination module 34 is further configured to: Determine whether the log data conforms to a preset sensitive type, whether the log data hits a local whitelist of sensitive types, and whether the log data is marked as sensitive log data; If the log data does not conform to the preset sensitive type, the log data hits the sensitive type local whitelist, or the log data is marked as sensitive log data, then the preset large model is used to determine whether the log data contains sensitive information.

[0056] In one embodiment, the sensitive information determination module 34 is further configured to: The log data is subjected to sensitivity type detection to determine whether the log data conforms to a preset sensitivity type; If the log data matches a preset sensitive type, then determine whether the log data matches the local whitelist of sensitive types; If the log data does not match the sensitive type local whitelist, then the log data will be marked as sensitive log data.

[0057] In one embodiment, the sensitive information determination module 34 is further configured to: The log data is verified by a signature. If the log data passes the signature verification, it is determined whether the log data matches the sensitive type local whitelist.

[0058] In one embodiment, the log checking module 32 checks whether the log data matches the global whitelist in the preset whitelist library, including checking whether the log data matches the global whitelist in the preset whitelist library through a detection engine; In one embodiment, the log false alarm whitelist filtering device 30 further includes a loading module, which is used to load the latest whitelist from the preset whitelist library into the detection engine through a scheduled task after configuring the latest global whitelist into the preset whitelist library.

[0059] In one embodiment, the whitelist update module 36 is further configured to: Extract the structural features of the false alarm log data, generate a regular expression based on the structural features of the false alarm log data, and form the latest global whitelist based on the regular expression.

[0060] In one embodiment, the log false alarm whitelist filtering device 30 further includes a standardization module. The standardization module is used to standardize the log data after the data acquisition module 31 acquires the log data and before the log inspection module 32 checks whether the log data hits the global whitelist in the preset whitelist library, so as to parse the log data into a standard Log object.

[0061] This invention provides a log false alarm whitelist filtering device, which allows the preset whitelist database to be adjusted after a user reports that the log data is a false alarm. This enables subsequent log data of the same type to be filtered out, thereby preventing alarms from being triggered again, effectively reducing the possibility of false alarms, and thus saving developers' time and system governance costs.

[0062] For specific limitations regarding the log false alarm whitelist filtering device, please refer to the limitations of the intelligent question-answering method mentioned above, which will not be repeated here. Each module in the aforementioned log false alarm whitelist filtering device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0063] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the server-side functions or steps of a log false alarm whitelist filtering method.

[0064] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the client-side functions or steps of a log false alarm whitelist filtering method.

[0065] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Retrieve log data; Check whether the log data matches the global whitelist in the preset whitelist library; If the log data matches the global whitelist, then the log data is filtered. If the log data does not match the global whitelist, a preset large model is used to determine whether the log data contains sensitive information. If the log data contains sensitive information, an alarm will be pushed to the log data. After receiving false alarm information from users, the log data is used as false alarm log data, a new global whitelist is generated based on the false alarm log data, and the new global whitelist is configured into the preset whitelist library.

[0066] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Retrieve log data; Check whether the log data matches the global whitelist in the preset whitelist library; If the log data matches the global whitelist, then the log data is filtered. If the log data does not match the global whitelist, a preset large model is used to determine whether the log data contains sensitive information. If the log data contains sensitive information, an alarm will be pushed to the log data. After receiving false alarm information from users, the log data is used as false alarm log data, a new global whitelist is generated based on the false alarm log data, and the new global whitelist is configured into the preset whitelist library.

[0067] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0068] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0069] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0070] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0071] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for filtering false alarms in logs using a whitelist, characterized in that, include: Retrieve log data; Check whether the log data matches the global whitelist in the preset whitelist library; If the log data matches the global whitelist, then the log data is filtered. If the log data does not match the global whitelist, a preset large model is used to determine whether the log data contains sensitive information. If the log data contains sensitive information, an alarm will be pushed to the log data. After receiving false alarm information from users, the log data is used as false alarm log data, a new global whitelist is generated based on the false alarm log data, and the new global whitelist is configured into the preset whitelist library.

2. The log false alarm whitelist filtering method according to claim 1, characterized in that, The step of using a preset large model to determine whether the log data contains sensitive information includes: Determine whether the log data conforms to a preset sensitive type, whether the log data hits a local whitelist of sensitive types, and whether the log data is marked as sensitive log data; If the log data does not conform to the preset sensitive type, the log data hits the sensitive type local whitelist, or the log data is marked as sensitive log data, then the preset large model is used to determine whether the log data contains sensitive information.

3. The log false alarm whitelist filtering method according to claim 2, characterized in that, Determining whether the log data conforms to a preset sensitive type, whether the log data hits a partial whitelist of sensitive types, and whether the log data is marked as sensitive log data includes: The log data is subjected to sensitivity type detection to determine whether the log data conforms to a preset sensitivity type; If the log data matches a preset sensitive type, then determine whether the log data matches the local whitelist of sensitive types; If the log data does not match the sensitive type local whitelist, then the log data will be marked as sensitive log data.

4. The log false alarm whitelist filtering method according to claim 3, characterized in that, Determining whether the log data matches the sensitive type local whitelist includes: The log data is verified by a signature. If the log data passes the signature verification, it is determined whether the log data matches the sensitive type local whitelist.

5. The log false alarm whitelist filtering method according to claim 1, characterized in that, Check whether the log data matches the global whitelist in the preset whitelist database. This includes checking whether the log data matches the global whitelist in the preset whitelist library using a detection engine; After configuring the latest global whitelist into the preset whitelist library, the method further includes loading the latest whitelist from the preset whitelist library into the detection engine via a scheduled task.

6. The log false alarm whitelist filtering method according to claim 1, characterized in that, A new global whitelist is generated based on the false alarm log data, including: Extract the structural features of the false alarm log data, generate a regular expression based on the structural features of the false alarm log data, and form the latest global whitelist based on the regular expression.

7. The log false alarm whitelist filtering method according to claim 1, characterized in that, After obtaining the log data, and before checking whether the log data matches the global whitelist in the preset whitelist library, the method further includes: standardizing the log data to parse the log data into a standard Log object.

8. A log false alarm whitelist filtering device, characterized in that, include: The data acquisition module is used to acquire log data; The log checking module is used to check whether the log data matches the global whitelist in the preset whitelist library; A filtering module is used to filter the log data if the log data matches the global whitelist. The sensitive information judgment module is used to determine whether the log data contains sensitive information if the log data does not match the global whitelist; The alarm push module is used to push alarms to the log data if the log data contains sensitive information. The whitelist database update module is used to, upon receiving false alarm information from a user, use the log data as false alarm log data, generate a latest global whitelist based on the false alarm log data, and configure the latest global whitelist into the preset whitelist database.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the log false alarm whitelist filtering method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the log false alarm whitelist filtering method as described in any one of claims 1 to 7.

Images