Fusion risk analysis method, system and device for multi-source security logs and medium

Abstract

The invention provides a fusion risk analysis method, system and device for multi-source security logs and a medium, and belongs to the technical field of data security. The method comprises the steps that security logs are obtained from multiple data sources, structured analysis, machine learning labeling and standardization processing are conducted on the security logs, and standardized logs are generated; carrying out cleaning, duplicate removal, screening and preprocessing on the standardized logs; the standardized log is stored in a distributed database, asset identification is performed based on the standardized log, risk identification is performed by using a deep reinforcement learning model according to an asset identification result, and a risk log is identified and processed; and feeding back a processing result of the risk log to the deep reinforcement learning model for continuous training, and adjusting a judgment condition of the risk log. Security logs from various sources can be automatically processed, formatting, cleaning, asset identification and risk identification of the logs are achieved through technologies such as machine learning, and therefore the automation degree and defense capacity of a data security platform are improved.

Description

Technical Field

[0001] The present invention relates to the field of data security technology, and more specifically to a fusion risk analysis method, system, device and medium for multi-source security logs. Background Art

[0002] Data has become a key production factor in modern society. With the rapid development of information technology, the risks of data leakage, abuse and loss are increasing. Data security has become an important challenge that enterprises and institutions must face. Therefore, the data security platform has emerged as a comprehensive system to protect and manage the data assets of enterprises or institutions and prevent risks such as data leakage, abuse and loss.

[0003] Most existing data security platforms adopt complete solutions with fully self-deployed deployment. Such systems usually include complete hardware and software probe devices, analysis components, display components, etc., covering the entire process from data collection, analysis to display. Its basic process mainly includes: first, by deploying network probes or log collection devices at each network node, the traffic and logs from network traffic, databases, bastion hosts, VPNs, and application system servers are captured. Then the collected data is cleaned according to a certain format and sent to the analysis system. The analysis system receives and parses the logs collected by the probe, and analyzes assets and threat behaviors. Among them, most analysis technologies rely on rule engines and detection algorithms based on known threat models. Finally, the analysis results are presented through a visualization platform to show network threats to administrators, or to notify them through an alarm mechanism.

[0004] Although the existing system can provide full-process data security services, it also has some problems: 1. Compatibility issues: Most existing systems require the deployment of complete hardware and software equipment. Already deployed single analysis systems, such as network traffic statistics and database audit systems, cannot be directly used, resulting in a waste of resources. If the system is not compatible with previous analysis products, the enterprise may have to re-purchase new equipment and software.

[0005] 2. Manual configuration is required: Administrators need to manually configure the log import method from different sources and set analysis rules, baselines, and risk models. This is not only time-consuming and labor-intensive, but also prone to errors when faced with large amounts of log data. If the system does not support customization, it cannot be used when encountering a special log format.

[0006] 3. Lack of automated rule updates: Traditional rule engines lack the ability to respond to new types of threats, and administrators need to frequently manually update the rule base to discover new risks.

[0007] 4. Limitations of single log source analysis: Most existing systems rely on the analysis of a single log source (such as network traffic or database logs) and cannot perform correlation analysis on multi-source data. This limits the system's ability to capture multi-step complex attack behaviors. Summary of the invention

[0008] In view of the above problems, the purpose of the present invention is to provide a fusion risk analysis method, system, device and medium for multi-source security logs, which can automatically process security logs from multiple sources, and realize log formatting, cleaning, asset identification and risk identification through advanced technologies such as machine learning, thereby improving the automation level and defense capability of the data security platform.

[0009] In order to achieve the above object, the present invention is implemented through the following technical solutions: In a first aspect, the present invention discloses a fusion risk analysis method for multi-source security logs, comprising: Obtain security logs from multiple data sources, perform structured analysis, machine learning annotation, and standardization on the security logs to generate standardized logs; Clean, remove duplicates, filter, and preprocess standardized logs; Store standardized logs in a distributed database, identify assets based on standardized logs, use deep reinforcement learning models to identify risks based on asset identification results, identify risk logs, and process them; The processing results of the risk log are fed back to the deep reinforcement learning model for continuous training, and the judgment conditions of the risk log are adjusted.

[0010] Furthermore, the security logs are obtained from multiple data sources, and structured parsing, machine learning annotation and standardization processing are performed on the security logs to generate standardized logs, including: Receive security logs from multiple audit software or probes; Determine the log type; If the security log is marked as structured text or text with existing identification rules, the corresponding parser or rule is used to parse it and generate the corresponding key-value pairs; If the log type is not labeled, the built-in neural network annotation model is used to automatically annotate the text of the security log, identify the key-value pairs, and transcribe them into GROK expressions and save them in the rule library; Automatically annotate key-value pairs, standardize synonymous key fields, and form standardized logs. Furthermore, the synonymous key fields include but are not limited to: Source and target IP, port, log time, URL address, API information, request and response content, and SQL expression.

[0011] Furthermore, the cleaning, deduplication, screening and preprocessing of the standardized logs include: Clean the standardized logs to remove redundant information; Use the preset filtering strategy to deduplicate the standardized logs and filter out the standardized logs with specific feature key values; Preprocess the standardized logs with binary content and decode and convert the identified binary content.

[0012] Furthermore, the asset identification based on the standardized log includes: Extract key fields from the logs and identify assets; Collect statistics on the number of asset visits, status, time trends, requested parameter key values, access relationships between assets, and user relationships to generate asset portraits as a baseline for risk detection. Through the distributed processing framework, standardized logs from different sources are correlated in real time to build a behavior chain; Use graph databases or graph neural networks and other models to build an interaction path diagram between users and resources based on the behavior chain; Use a pre-trained neural network annotation model to annotate the key-value pairs of asset content and determine the sensitivity type and level of the asset.

[0013] Furthermore, the risk identification is performed using a deep reinforcement learning model according to the asset identification result to identify the risk log, including: Obtain real-time standardized logs, use baseline comparison methods, time series analysis models, interactive path map checks, and deep reinforcement learning models to identify risks in real-time standardized logs, determine risk logs by judging whether the behavior is abnormal, and define the corresponding risk level; The risk log is further judged according to the customized extended risk strategy. If it is still identified as a risk log, the corresponding behavior chain is queried and the risk log is processed.

[0014] Furthermore, the processing result of the risk log is fed back to the deep reinforcement learning model for continuous training, and the judgment condition of the risk log is adjusted, including: Feed the processing results of risk logs to the deep reinforcement learning model for continuous training to increase the attention paid to relevant behaviors or log features; Adjust the judgment conditions of risk logs by creating a new baseline; Use the newly established baseline to perform retrospective risk identification on standardized data stored over a specific period of time to identify missed risk logs.

[0015] In a second aspect, the present invention further discloses a fusion risk analysis system for multi-source security logs, including: The log standardization module is used to obtain security logs from various data sources, perform structured analysis, machine learning annotation and standardization on the security logs, and generate standardized logs; Log cleaning module, used to clean, remove duplicates, filter and pre-process standardized logs; The asset identification and risk identification module is used to store standardized logs in a distributed database, identify assets based on standardized logs, identify risks using a deep reinforcement learning model based on asset identification results, identify risk logs, and process them; The feedback adjustment module is used to feed back the processing results of the risk log to the deep reinforcement learning model for continuous training and adjust the judgment conditions of the risk log.

[0016] In a third aspect, the present invention further discloses a fusion risk analysis device for multi-source security logs, comprising: A memory for storing a fusion risk analysis program of multi-source security logs; A processor is used to implement the steps of the multi-source security log fusion risk analysis method as described in any one of the above items when executing the multi-source security log fusion risk analysis program.

[0017] In a fourth aspect, the present invention further discloses a readable storage medium, on which is stored a fusion risk analysis program for multi-source security logs. When the fusion risk analysis program for multi-source security logs is executed by a processor, the steps of the fusion risk analysis method for multi-source security logs as described in any of the above items are implemented.

[0018] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention automatically parses unknown format texts through machine learning technologies such as pre-trained neural networks, and automatically standardizes similar logs in different formats, saving a lot of manual configuration time.

[0019] 2. The present invention supports security logs from multiple sources and can flexibly cope with complex and changing network environments.

[0020] 3. The present invention utilizes a variety of models and technical means to perform asset identification and risk identification, thereby improving the accuracy and reliability of identification.

[0021] 4. The present invention realizes adaptive updating through user feedback and model training, enabling the system to continuously evolve and effectively improving the defense capability of the data security platform.

[0022] 5. The present invention uses a neural network annotation model to automatically annotate unannotated text, thereby improving the degree of automation in log processing; it uses a variety of technical means such as time series analysis models, graph neural network models, and deep reinforcement learning models to identify risks, thereby enhancing the accuracy and reliability of identification. At the same time, the system supports user feedback and model adaptive updates, and can continuously evolve to improve defense capabilities. The present invention effectively solves the problems of low automation and poor scalability of traditional data security platforms when processing multi-source security logs, thereby improving the overall performance of the data security platform.

[0023] It can be seen that compared with the prior art, the present invention has outstanding substantive features and significant progress, and the beneficial effects of its implementation are also obvious. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0025] Figure 1 It is a method flow chart of a specific implementation mode of the present invention.

[0026] Figure 2 It is a system structure diagram of a specific implementation mode of the present invention.

[0027] In the figure, 1. Log standardization module; 2. Log cleaning module; 3. Asset identification and risk identification module; 4. Feedback adjustment module. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0029] See also Figure 1 As shown, this embodiment provides a fusion risk analysis method for multi-source security logs, including the following steps: S1: Obtain security logs from multiple data sources, perform structured analysis, machine learning annotation and standardization on the security logs, and generate standardized logs.

[0030] In a specific implementation, first, security logs from multiple audit software or probes are received. Then, the log type is determined; if the security log is marked as structured text or text with existing identification rules, the corresponding parser or rule is used to parse and generate corresponding key-value pairs; if the log type is not marked, the built-in neural network annotation model is used to automatically annotate the text of the security log, identify the key-value pairs, and transcribe them into GROK expressions and save them in the rule library. Finally, the key-value pairs are automatically annotated, the synonymous key fields are standardized, and a standardized log is formed. Among them, the parser can use a JSON / XML parser; the rules can use regular expressions or GROK expressions; synonymous key fields include but are not limited to: source and target IP, port, log time, URL address, API information, request and response content, SQL expression.

[0031] As an example, security logs from multiple audit software or probes are received, and first determine whether they are: clear structured text or unstructured text with recognition rules. If the text of this type from this source is text that has been annotated with rules, it is parsed using an existing structured text parser (such as a JSON / XML parser) or existing rules (such as regular expressions, GROK expressions). If it is not annotated, the text content is automatically annotated using machine learning methods such as the built-in neural network annotation model, and the annotated text is transcribed into a format such as a GROK expression and saved in the rule library. The parsed key-value pairs are automatically annotated using methods such as a neural network annotation model that has been fine-tuned and optimized for network traffic and audit logs, and the synonymous key fields obtained are standardized.

[0032] Among them, key fields include but are not limited to source and target IP, port, log time. Depending on the type of audit software, there may be URL address, API information, request and response content, SQL expression, system user name and other information; fields such as log time, MAC address, IP address, etc. that may have multiple expressions are standardized into one format. After standardizing this information, a standard format log is formed.

[0033] It can be seen that compared with the traditional method of manually configuring various log formats, this step uses machine learning methods such as pre-trained neural networks to automatically generate rules for unknown format text parsing and automatically standardize similar logs in different formats. It also supports manual fine-tuning, has a higher degree of automation and stronger scalability, and saves time in configuring log formats.

[0034] S2: Clean, deduplicate, filter, and preprocess standardized logs.

[0035] In a specific implementation, the standardized logs are first cleaned to remove redundant information; then, the standardized logs are deduplicated using a preset screening strategy, and standardized logs with specific feature key values ​​are screened out; finally, the standardized logs with binary content are preprocessed, and the identified binary content is decoded and converted.

[0036] As an example, first clean the standardized logs generated in step S1, and deduplicate the logs with the same key content; then use the preset or configured strategy to filter the logs with certain characteristic key values; and then preprocess some logs with binary content for analysis. The goal of this step is to improve the signal-to-noise ratio of the data and ensure the accuracy of subsequent analysis and the efficiency of the data.

[0037] S3: Store standardized logs in a distributed database, identify assets based on standardized logs, use deep reinforcement learning models to identify risks based on asset identification results, identify risk logs, and process them.

[0038] In a specific implementation, the purpose of this step is to perform asset identification and risk identification simultaneously when storing the cleaned standardized logs in a distributed database.

[0039] When performing asset identification, we first extract the key fields in the log to identify the assets. Then, we collect statistics on the asset access volume, status, time trend, parameter key values ​​of requests, access relationships between assets, and user relationships to generate an asset portrait as a baseline for risk detection.

[0040] At this time, the standardized logs from different sources are associated in real time through a distributed processing framework to build a behavior chain. At the same time, models such as graph databases or graph neural networks are used to build an interaction path diagram between users and resources based on the behavior chain.

[0041] Finally, a pre-trained neural network annotation model is used to annotate the key-value pairs of the asset content to determine the sensitivity type and level of the asset.

[0042] When performing risk identification, first obtain real-time standardized logs, use baseline comparison methods, time series analysis models, interactive path map checks, and deep reinforcement learning models to identify risks in real-time standardized logs, determine whether the behavior is abnormal, and define the corresponding risk level. Then, further judge the risk log based on the customized extended risk strategy. If it is still identified as a risk log, query the corresponding behavior chain and process the risk log.

[0043] As an example, this step discloses an asset identification process, and the specific steps are as follows: S311: Extract key fields from standardized logs to identify assets including but not limited to servers, application systems, APIs, databases, VPNs, bastion hosts, user entities, etc. Count the number of visits, status, time trends, and parameter key values ​​of the identified assets; collect statistics on the access relationships and user relationships between assets, and create portraits. These will be used as short-term and long-term baselines for statistical risks, for subsequent risk detection, and the behavioral baseline will be continuously updated based on the input.

[0044] At this time, using daily asset profiles and operation statistics as input, a time series analysis model for user operation patterns of the asset or asset class is trained and enabled after a certain period of time for risk identification. The module also uses a distributed processing framework to correlate logs from different sources in real time.

[0045] At the same time, based on key fields such as user ID, IP address, and timestamp, operations from multiple sources are linked together to build a complete behavior chain. Using models such as graph databases or graph neural networks, behavior chains are used to build a general interaction path diagram between users and resources.

[0046] S312: Use the pre-trained neural network annotation model to annotate the key-value pairs of asset content that can obtain specific content, annotate the sensitive type and sensitivity level of the asset, and use it for data classification and grading. It should be noted that during the actual operation, the customer can modify the classification and grading results of the asset.

[0047] As an example, this step also discloses a risk identification process, the specific steps are as follows: S321: Whenever new standardized log data arrives (such as new operation behavior), find out whether there is a corresponding baseline generated in step S311, and use the comparison method applicable to the baseline for comparison. If the actual behavior deviates from the model prediction, mark the behavior as abnormal, and define the risk level based on the results given by the network.

[0048] S322: Whenever new standardized log data arrives (such as new operation behavior), the time series analysis model pre-trained during asset identification is input for prediction. If the actual behavior deviates from the model prediction, the behavior is marked as abnormal behavior, and the risk level is defined based on the results given by the network.

[0049] S323: Check the behavior chain according to the interaction path diagram created in asset identification. If the behavior deviates from the general data flow and data characteristics of the asset in the interaction path diagram, the system will mark it as abnormal and define the risk level based on the results given by the network.

[0050] S324: Input the standardized log and behavior chain into the deep reinforcement learning model, which has learned the log and behavior chain features marked as risks by the user. If the current standardized log is consistent with the risk log / call chain previously judged by the user, the behavior is marked as abnormal, and the risk level is defined according to the result given by the network.

[0051] S325: Determine whether the standardized log is abnormal and the degree of abnormality according to the user-defined extended risk policy.

[0052] S326: If it is an abnormal log, query the corresponding behavior chain and submit the log of the behavior chain to the downstream risk processing module for operation.

[0053] S327: The log sent to the downstream for processing includes the currently used baseline data, behavior chain and time sequence identification data, etc.

[0054] S4: Feedback the processing results of the risk log to the deep reinforcement learning model for continuous training, and adjust the judgment conditions of the risk log.

[0055] In a specific implementation, first, the processing results of the risk log are fed back to the deep reinforcement learning model for continuous training to increase the attention paid to related behaviors or log features. At the same time, the judgment conditions of the risk log can be adjusted by creating a new baseline, and the newly established baseline can be used to perform retrospective risk identification on the standardized data stored within a specific time to identify missed risk logs.

[0056] For example, when a user processes a risk log or behavior chain (such as marking it as risky, ignoring it, etc.), the processing result is fed back to the deep reinforcement learning model. The model uses this feedback data for continuous training and learns which behaviors or log features need to be focused on.

[0057] When the user believes that there is a problem with the current judgment condition (for example, the baseline / behavior chain is being attacked when it is created), the current judgment condition (for example, the baseline) can be reset, and the judgment condition can be re-established using the current data. The newly established baseline can be used to backtrack the data stored in the distributed database within a specific time period to capture previous attacks. The user's adjustment results for the baseline, etc. will be fed back to the corresponding machine learning model.

[0058] By using the above method to implement basic functions and adding user interface, data storage and other modules to realize productization, a fusion risk analysis system based on multi-source security logs can be realized.

[0059] The present invention provides a fusion risk analysis method for multi-source security logs, which can automatically process security logs from multiple sources and realize log formatting, cleaning, asset identification and risk identification through advanced technologies such as machine learning, thereby improving the automation level and defense capability of the data security platform.

[0060] See also Figure 2 As shown, the present invention also discloses a fusion risk analysis system of multi-source security logs, including: a log standardization module 1, a log cleaning module 2, an asset identification and risk identification module 3 and a feedback adjustment module 4. The log standardization module 1 is used to obtain security logs from various data sources, perform structured analysis, machine learning annotation and standardization on the security logs, and generate standardized logs.

[0061] The log cleaning module 2 is used to clean, remove duplicates, filter and pre-process the standardized logs.

[0062] The asset identification and risk identification module 3 is used to store standardized logs in a distributed database, perform asset identification based on standardized logs, perform risk identification using a deep reinforcement learning model based on the asset identification results, identify risk logs, and process them.

[0063] Feedback adjustment module 4 is used to feed back the processing results of the risk log to the deep reinforcement learning model for continuous training and adjust the judgment conditions of the risk log.

[0064] The specific implementation of the fusion risk analysis system of multi-source security logs in this embodiment is basically the same as the specific implementation of the fusion risk analysis method of multi-source security logs described above, and will not be repeated here.

[0065] The present invention also discloses a fusion risk analysis device for multi-source security logs, comprising a processor and a memory; wherein, when the processor executes the fusion risk analysis program for multi-source security logs stored in the memory, the steps of the fusion risk analysis method for multi-source security logs as described in any one of the above items are implemented.

[0066] Furthermore, the fusion risk analysis device for multi-source security logs in this embodiment may also include: The input interface is used to obtain the fusion risk analysis program of the multi-source security logs imported from the outside, and save the obtained fusion risk analysis program of the multi-source security logs to the memory, and can also be used to obtain various instructions and parameters transmitted by the external terminal device, and transmit them to the processor, so that the processor can use the above various instructions and parameters to carry out corresponding processing. In this embodiment, the input interface can specifically include but is not limited to a USB interface, a serial interface, a voice input interface, a fingerprint input interface, a hard disk reading interface, etc.

[0067] The output interface is used to output various data generated by the processor to the terminal device connected thereto, so that other terminal devices connected to the output interface can obtain various data generated by the processor. In this embodiment, the output interface may specifically include but is not limited to a USB interface, a serial interface, etc.

[0068] The communication unit is used to establish a remote communication connection between the fusion risk analysis device of the multi-source security log and the external server, so that the fusion risk analysis device of the multi-source security log can mount the image file to the external server. In this embodiment, the communication unit can specifically include but is not limited to a remote communication unit based on wireless communication technology or wired communication technology.

[0069] The keyboard is used to obtain various parameter data or instructions input by the user by tapping the keycaps in real time.

[0070] A display is used to display relevant information of the fusion risk analysis process of running multi-source security logs in real time.

[0071] The mouse can be used to assist users in inputting data and simplify user operations.

[0072] The present invention also discloses a readable storage medium, wherein the readable storage medium includes a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable hard disk, a CD-ROM or any other form of storage medium known in the technical field. The readable storage medium stores a fusion risk analysis program for multi-source security logs, and when the fusion risk analysis program for multi-source security logs is executed by a processor, the steps of the fusion risk analysis method for multi-source security logs as described in any one of the above items are implemented.

[0073] In summary, the present invention can automatically process security logs from various sources, and realize log formatting, cleaning, asset identification, and risk identification through advanced technologies such as machine learning, thereby improving the automation level and defense capability of the data security platform.

[0074] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. As for the method disclosed in the embodiment, since it corresponds to the system disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0075] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0076] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0077] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0078] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, or each module may exist physically separately, or two or more modules may be integrated into one unit.

[0079] Similarly, each processing unit in each embodiment of the present invention may be integrated into one functional module, or each processing unit may exist physically, or two or more processing units may be integrated into one functional module.

[0080] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0081] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0082] The above is a detailed introduction to the fusion risk analysis method, system, device and readable storage medium of the multi-source security log provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A fusion risk analysis method for multi-source security logs, characterized in that: include: Obtain security logs from multiple data sources, perform structured analysis, machine learning annotation, and standardization on the security logs to generate standardized logs; Clean, remove duplicates, filter, and preprocess standardized logs; Store standardized logs in a distributed database, identify assets based on standardized logs, use deep reinforcement learning models to identify risks based on asset identification results, identify risk logs, and process them; The processing results of the risk log are fed back to the deep reinforcement learning model for continuous training, and the judgment conditions of the risk log are adjusted.

2. The fusion risk analysis method of multi-source security logs according to claim 1 is characterized in that: The security logs are obtained from multiple data sources, and structured parsed, machine-learned labeled, and standardized to generate standardized logs, including: Receive security logs from multiple audit software or probes; Determine the log type; If the security log is marked as structured text or text with existing identification rules, the corresponding parser or rule is used to parse it and generate the corresponding key-value pairs; If the log type is not labeled, the built-in neural network annotation model is used to automatically annotate the text of the security log, identify the key-value pairs, and transcribe them into GROK expressions and save them in the rule library; Automatically annotate key-value pairs, standardize synonymous key fields, and form standardized logs.

3. The fusion risk analysis method of multi-source security logs according to claim 2 is characterized in that: The synonymous key fields include but are not limited to: Source and target IP, port, log time, URL address, API information, request and response content, and SQL expression.

4. The fusion risk analysis method of multi-source security logs according to claim 3 is characterized in that: The cleaning, deduplication, screening and preprocessing of the standardized logs include: Clean the standardized logs to remove redundant information; Use the preset filtering strategy to deduplicate the standardized logs and filter out the standardized logs with specific feature key values; Preprocess the standardized logs with binary content and decode and convert the identified binary content.

5. The fusion risk analysis method of multi-source security logs according to claim 4 is characterized in that: The asset identification based on the standardized log includes: Extract key fields from the logs and identify assets; Collect statistics on the number of asset visits, status, time trends, requested parameter key values, access relationships between assets, and user relationships to generate asset portraits as a baseline for risk detection. Through the distributed processing framework, standardized logs from different sources are correlated in real time to build a behavior chain; Use graph database or graph neural network model to build interaction path diagram between users and resources based on behavior chain; Use a pre-trained neural network annotation model to annotate the key-value pairs of asset content and determine the sensitivity type and level of the asset.

6. The fusion risk analysis method of multi-source security logs according to claim 5 is characterized in that: The risk identification is performed using a deep reinforcement learning model based on the asset identification results to identify risk logs, including: Obtain real-time standardized logs, use baseline comparison methods, time series analysis models, interactive path map checks, and deep reinforcement learning models to identify risks in real-time standardized logs, determine risk logs by judging whether the behavior is abnormal, and define the corresponding risk level; The risk log is further judged according to the customized extended risk strategy. If it is still identified as a risk log, the corresponding behavior chain is queried and the risk log is processed.

7. The fusion risk analysis method of multi-source security logs according to claim 6 is characterized in that: Feeding back the processing results of the risk log to the deep reinforcement learning model for continuous training and adjusting the judgment conditions of the risk log include: Feedback the processing results of risk logs to the deep reinforcement learning model for continuous training to increase the attention paid to relevant behaviors or log features; Adjust the judgment conditions of risk logs by creating a new baseline; Use the newly established baseline to perform retrospective risk identification on standardized data stored over a specific period of time to identify missed risk logs.

8. A fusion risk analysis system for multi-source security logs, characterized in that: The system adopts the fusion risk analysis method of multi-source security logs as described in any one of claims 1 to 7; The system comprises: The log standardization module is used to obtain security logs from various data sources, perform structured analysis, machine learning annotation and standardization on the security logs, and generate standardized logs; Log cleaning module, used to clean, remove duplicates, filter and pre-process standardized logs; The asset identification and risk identification module is used to store standardized logs in a distributed database, identify assets based on standardized logs, identify risks using a deep reinforcement learning model based on asset identification results, identify risk logs, and process them; The feedback adjustment module is used to feed back the processing results of the risk log to the deep reinforcement learning model for continuous training and adjust the judgment conditions of the risk log.

9. A fusion risk analysis device for multi-source security logs, characterized in that: include: A memory for storing a fusion risk analysis program of multi-source security logs; A processor, configured to implement the steps of the multi-source security log fusion risk analysis method as described in any one of claims 1 to 7 when executing the multi-source security log fusion risk analysis program.

10. A readable storage medium, characterized in that: The readable storage medium stores a fusion risk analysis program for multi-source security logs, and when the fusion risk analysis program for multi-source security logs is executed by a processor, the steps of the fusion risk analysis method for multi-source security logs as described in any one of claims 1 to 7 are implemented.

Images