A log processing method, apparatus, device, and storage medium

Abstract

The application discloses a log processing method, device and equipment and a storage medium. The method comprises the following steps: if a government affair server detects a request for performing correlation analysis on target logs of multiple types of security devices, the target logs are converted and extracted to obtain a target log data set; attack data set and victim data set corresponding to the target log data set are determined, the attack data set is subjected to vectorization processing to obtain an attack log vector, and the victim data set is subjected to vectorization processing to obtain a victim log vector; the attack data set is subjected to clustering processing according to the attack log vector, the victim data set is subjected to clustering processing according to the victim log vector, and attack correlation analysis and victim correlation analysis are performed on the target logs according to the clustering results. The technical scheme of the application can comprehensively analyze the target logs of multiple types of security devices, improve the efficiency of log processing, and thus assist relevant personnel in formulating more effective threat response schemes.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of network and information technology, and in particular to a log processing method, device, equipment and storage medium. BACKGROUND

[0002] With the continuous development of network and information technology, information security problems are also paid more and more attention by people. Especially for government servers, different types of information security devices are associated, and these information security devices will output alarms and security threat logs when detecting security threats. When the types and quantities of security devices become more and more, the number of various alarms and logs generated every day will also increase sharply, which increases the difficulty of formulating threat response schemes.

[0003] Therefore, how to more comprehensively analyze the logs of multiple types of security devices, improve the efficiency of log processing, and assist relevant personnel to formulate more effective threat response schemes is a problem to be solved at present. SUMMARY

[0004] The present application provides a log processing method, device, equipment and storage medium to more comprehensively analyze the target logs of multiple types of security devices, improve the efficiency of log processing, and assist relevant personnel to formulate more effective threat response schemes.

[0005] According to an aspect of the present application, a log processing method is provided, which is executed by a government server and includes:

[0006] If a request for correlation analysis of target logs of multiple types of security devices is detected, the target logs are converted and extracted for processing to obtain a target log dataset;

[0007] An attack dataset and a victim dataset corresponding to the target log dataset are determined, and the attack dataset is vectorized for processing to obtain an attack log vector, and the victim dataset is vectorized for processing to obtain a victim log vector;

[0008] The attack dataset is clustered according to the attack log vector, the victim dataset is clustered according to the victim log vector, and the target logs are analyzed for attack correlation and victim correlation according to the clustering results.

[0009] According to another aspect of the present application, a log processing device is provided, which includes:

[0010] The obtaining module is configured to, if a request for correlation analysis of target logs of multiple types of security devices is detected, convert and extract the target logs for processing to obtain a target log dataset;

[0011] The vectorization module is configured to determine an attack data set and a victim data set corresponding to the target log data set, perform vectorization processing on the attack data set to obtain an attack log vector, and perform vectorization processing on the victim data set to obtain a victim log vector.

[0012] The analysis module is configured to perform clustering processing on the attack data set according to the attack log vector, perform clustering processing on the victim data set according to the victim log vector, and perform attack correlation analysis and victim correlation analysis on the target log according to the clustering results.

[0013] According to another aspect of the present application, an electronic device is provided, which comprises:

[0014] at least one processor; and

[0015] a memory connected to the at least one processor in communication; wherein

[0016] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the log processing method according to any one of the embodiments of the present application.

[0017] According to another aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to implement the log processing method according to any one of the embodiments of the present application when executed by the processor.

[0018] According to another aspect of the present application, a computer program product is also provided, which comprises a computer program for implementing the log processing method according to any one of the embodiments of the present application when executed by a processor.

[0019] The technical solution of the embodiments of the present application is that if the government affair server detects a request for performing correlation analysis on the target log of the multiple types of security devices, the target log is converted and extracted to obtain a target log data set; an attack data set and a victim data set corresponding to the target log data set are determined, the attack data set is vectorized to obtain an attack log vector, and the victim data set is vectorized to obtain a victim log vector; the attack data set is clustered according to the attack log vector, the victim data set is clustered according to the victim log vector, and the target log is subjected to attack correlation analysis and victim correlation analysis according to the clustering results. By dividing the log data set into the attack data set and the victim data set and processing in combination with the clustering algorithm, the target log of the multiple types of security devices can be more comprehensively analyzed, the efficiency of log processing is improved, and thus a more effective threat response scheme can be formulated by relevant personnel.

[0020] It is to be understood that the details set forth herein do not limit the scope of the embodiments of the application to the specific embodiments described. Rather, the scope of the embodiments of the application is to be defined by the appended claims. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments description. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0022] Figure 1 is a flow chart of a log processing method provided by the application;

[0023] Figure 2 is a structural block diagram of a log processing device provided by the application;

[0024] Figure 3 is a structural schematic diagram of an electronic device provided by the application. DETAILED DESCRIPTION

[0025] In order to make the technical personnel in the art better understand the application scheme, the following will combine the drawings in the embodiments of the application to clearly and completely describe the technical solutions in the embodiments of the application. Obviously, the described is only some of the embodiments of the application, not all. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the application.

[0026] It should be noted that the terms "first", "second", "target", "candidate", "alternative" and the like in the specification and claims of the application and the above-mentioned drawings are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices. The acquisition, storage, use, processing and other data in the technical solutions of the application comply with the relevant provisions of national laws and regulations.

[0027] It should be noted that in the related art, the log of a single type of security device is generally analyzed, and the associated information existing in the logs of multiple types of security devices cannot be mined. When a security engineer processes a security threat according to the information obtained by analyzing the log of a single type of security device, it is difficult to associate and foresee other related security threats that appear with the security threat, and when a processing scheme is formulated, there will be a tendency of "treating the headache with the headache and the foot pain with the foot pain". When the types and quantities of security devices become more and more, the quantities of various alarms and logs generated every day will also increase sharply, and if the security engineer still only responds to the current single threat without considering other related threats, he or she will be in a passive situation of being tired of responding to various threats every day. Based on the above problems, according to the attack IP address and the victim IP address in the alarm log, the present application uniformly divides the logs of multiple types of security devices into attack logs and victim logs, and based on a clustering algorithm, clustering is performed to obtain a clustering set, so that according to the clustering set, the attack situation that each log may suffer can be associated and analyzed, more comprehensive analysis of the logs can be realized, and thus a more effective threat response scheme can be formulated for the relevant personnel, and the specific implementation steps will be described in detail later.

[0028] Figure 1 It is a flowchart of a log processing method provided by the present application; the embodiment can be applicable to the case that a government affair server performs associated analysis on target logs of multiple types of security devices based on a clustering algorithm to assist relevant personnel in formulating a more effective threat response scheme, the method can be executed by a log processing device, the log processing device can be realized in the form of hardware and / or software, and the log processing device can be configured in an electronic device, such as a government affair server, as shown in Figure 1 The log processing method includes the following steps.

[0029] S101, if a request for associated analysis of target logs of multiple types of security devices is detected, the target logs are converted and extracted for processing to obtain a target log data set.

[0030] The security device refers to a device configured on the government server for detecting and identifying security threats. The security device can be, for example, a firewall (Web Application Firewall, WAF), an intrusion-prevention system (Intrusion-prevention system, IPS), and a situation awareness device. The situation awareness device can specifically refer to a situation awareness software integrated into hardware. The target log refers to a security threat log sent by each security device to the government server. The target log can include at least one of the following information: log recording time, attack address, victim address, security device type, threat hit rule, and threat hit data. The attack address can be, for example, an IP (Internet Protocol) address of an attacker who performs an attack operation. The victim address can be, for example, an IP address of a victim who suffers an attack.

[0031] The target log dataset refers to a collection of target logs after conversion and extraction processing. It should be noted that after conversion and extraction processing, the target log can increase information such as attack alias and victim alias. Specifically, the field content and requirements of each target log in the target dataset can be: log recording time: the time when the target log is recorded, which cannot be empty; attack address: IP address of the attacker, which can be empty, but cannot be empty at the same time as the victim IP; attack alias: alias of the IP address of the attacker, from the IP preprocessing process; victim address: IP address of the victim, which can be empty, but cannot be empty at the same time as the attack IP; victim alias: alias of the IP address of the victim, from the IP preprocessing process; security device type: type of security device outputting the target log; hit rule: security threat rule hit when recording the target log, which cannot be empty; hit data: request / response data when recording the target log, which cannot be empty.

[0032] Exemplarily, the information of a target log record in the target log data set can be as follows: attack IP: 10.0.0.1; attack IP alias: baoaoaob; victim IP: 10.0.0.2; victim IP alias: baoaoaoc; security device type: Tianyan; hit rule: general command execution vulnerability; hit data: “name[#this.getClass().forName(“java.lang.Runtime”).getRuntime().exec(“whoami”)]=&username[#this.getClass().forName(“java.lang.Runtime”).getRuntime().exec(“whoami”)]=&password[#this.getClass().forName(“java.lang.Runtime”).getRuntime().exec(“whoami”)]=”.

[0033] Optionally, the government affair server can automatically generate an instruction for performing correlation analysis on the target logs generated by the multiple types of security devices in a preset historical time period based on a preset period, and in this case, it is considered that the request for performing correlation analysis on the target logs of the multiple types of security devices is detected; or the government affair server can consider that the request for performing correlation analysis on the target logs of the multiple types of security devices is detected when it detects a correlation analysis instruction issued by a relevant staff such as a security engineer.

[0034] Optionally, the target log is converted and extracted to obtain the target log data set, including: converting the attack address in each target log based on a preset conversion rule to obtain an attack alias, and converting the victim address in each target log to obtain a victim alias; extracting the log record time, attack address, victim address, security device type, hit rule and hit data corresponding to each target log, and combining the attack alias and the victim alias to generate the target log data set.

[0035] The attack alias refers to the information obtained by converting the attack address, the victim alias refers to the information obtained by converting the victim address, and the conversion rule refers to a rule for pre-processing the attack address and the victim address so that they can meet the coding requirements of the coding model. Exemplarily, the conversion rule can be to replace 0, 1, 2, …, 9 in the attack address and the victim address with lowercase letters a, b, c, …, j respectively, and replace the punctuation “.” in the attack address and the victim address with lowercase letter o. For example, the IP alias of the address 10.0.0.1 is “baoaoaob”. The security device type refers to the type of the security device that sends the log to the government affair server.

[0036] It should be noted that when the encoding model processes the IP address in the log, the IP address is decomposed into a single number for encoding. In order to enable the encoding model to encode the IP address as a whole, the IP address in the log needs to be converted. Specifically, all attack addresses and victim addresses in the target log can be converted into corresponding address aliases based on a preset conversion rule.

[0037] Optionally, the log record time, attack address, victim address, security device type, hit rule and hit data corresponding to each target log can be obtained by matching in each target log according to a preset field name.

[0038] S102, determine the attack data set and the victim data set corresponding to the target log data set, and perform vectorization processing on the attack data set to obtain an attack log vector, and perform vectorization processing on the victim data set to obtain a victim log vector.

[0039] The attack data set refers to a collection of target logs whose attack address field information is not empty, and the victim data set refers to a collection of target logs whose victim address field information is not empty. The attack log vector and the victim log vector are vectors generated by integrating and vectorizing the attack data set and the victim data set.

[0040] Optionally, the target log data set can be divided into an attack data set and a victim data set according to the attack address and victim address information of each target log in the target log data set.

[0041] Optionally, determining the attack data set and the victim data set corresponding to the target log data set comprises: filtering the target log data set according to the record of the attack address of each target log in the target log data set to obtain the attack data set; and filtering the target log data set according to the record of the victim address of each target log in the target log data set to obtain the victim data set.

[0042] Optionally, according to the record, if the attack address corresponding to the target log is not empty, the target log can be added to the attack data set, that is, the attack data set is generated according to the target log whose attack address is not empty in the target log data set. Similarly, the victim data set can be generated according to the target log whose victim address is not empty in the target log data set.

[0043] Optionally, the attack data set is vectorized to obtain an attack log vector, including: generating an attack statement according to the attack alias, the security device type and the hit rule corresponding to each attack address in the attack data set; processing the attack statement to obtain a code vector, and processing the hit data corresponding to each attack address to obtain a matching vector; and splicing the code vector and the matching vector to obtain an attack log vector corresponding to the attack data set.

[0044] The attack statement is a statement obtained by splicing the attack alias, the security device type and the hit rule based on a preset splicing rule, and the format of the attack statement can be as follows: "[CLS] IP alias [SEP] security device type [SEP] hit rule 1 [SEP] hit rule 2 … [SEP] hit rule m [SEP]", wherein [CLS] is a mark of a coding model (such as a BERT (Bidirectional Encoder Representations from Transformers) model) and is used to indicate the start of an input sequence. [SEP] is also a mark of BERT and is used to separate two sentences and indicate the end of a sentence.

[0045] Optionally, the hit rules in the log records with the same attack address and the same security device type can be put into an attack statement according to a preset format, and the attack statement is generated, that is, if the data of the security device type is K, then K attack statements corresponding to one attack address are generated.

[0046] Optionally, according to the format of the attack statement, the attack alias, the security device type and the hit rule corresponding to each attack address can be spliced and combined based on a preset splicing rule to generate the attack statement. Further, a preset coding model can be used to vectorize the attack statement to obtain a code vector, and a preset regular expression dictionary can be used to match the hit data to obtain a matching vector. Finally, the matching vector can be spliced to the end of the code vector to obtain an attack log vector.

[0047] It should be noted that the same method as that for determining the attack log vector can be used to determine a victim log vector corresponding to the victim data set, that is, the attack log vector and the victim log vector are determined in the same way, and the present application will not be described in detail.

[0048] Optionally, the attack statement is processed to obtain an encoding vector, and the hit data corresponding to each attack address is processed to obtain a matching vector, including: based on a preset insertion order, inserting the hit rule corresponding to each attack address into the attack statement, and using a preset encoding model to process the attack statement after inserting the hit rule to obtain an encoding vector; using a preset regular expression dictionary to match the hit data corresponding to each attack address to obtain a matching vector.

[0049] The preset insertion order can be the order of occurrence time of each hit rule. The preset encoding model can be a BERT model. The preset regular expression dictionary refers to a dictionary generated by storing all words and sentences related to security threats in historical logs in the form of regular expressions. Each regular expression in the preset regular expression dictionary can be numbered from 1 in the order of insertion of regular expressions.

[0050] For example, a regular expression dictionary Dictionary containing P regular expressions can be represented as:

[0051] Dictionary={[idx,Expression idx ]},

[0052] idx=1,2,…,P,Expression idx is the idx-th regular expression;

[0053] Optionally, after inputting the hit data corresponding to each attack address into the regular expression dictionary Dictionary, a P-dimensional regular matching vector match=[count1,count2,…,count P ] can be output, where count p p=1,2,…,P represents the number of times that the word or sentence satisfying the p-th regular expression Expression p in Dictionary appears in the hit data. If the input hit data is empty, the output matching vector is a zero vector.

[0054] Optionally, the word and sentence patterns related to security threats in the historical logs can be analyzed and converted into the form of regular expressions. If the regular expression to be inserted already exists in the dictionary, the regular expression is not inserted.

[0055] Optionally, for each attack address, the hit rule containing the attack address in the log data set can be inserted into the attack statement of the security device in the order of occurrence time, and the attack statement after inserting the hit rule can be input into the pre-trained encoding model to output an encoding vector.

[0056] It should be noted that since the regular expression dictionary can only extract the number of occurrences of specific pattern sentences in the text content, but cannot understand the sequence relationship between the sentences, the coding model is used to determine the coding vector to extract the sequence relationship between the sentences in the text content.

[0057] Exemplarily, assuming that the number of security device types is K, and the dimension of the coding vector output by the coding model is Q, each log record of the attack data set is composed of an attack address, K attack statements and K corresponding log vectors, and further, the attack statement can be input into the preset coding model to obtain a Q-dimensional coding vector. If the attack statement is empty, the coding output is a Q-dimensional all-zero vector. Further, the hit data corresponding to each attack address can be merged into a piece of text data and input into the preset regular expression dictionary Dictionary to obtain a P-dimensional matching vector, and finally the P-dimensional matching vector can be spliced to the end of the Q-dimensional coding vector to obtain a D-dimensional log vector, that is: D = Q + P.

[0058] S103, according to the attack log vector, the attack data set is clustered, and the victim data set is clustered according to the victim log vector, and according to the clustering result, the target log is attacked and associated with the victim.

[0059] Optionally, the attack data set can be clustered according to the attack log vector, the class weight corresponding to each security device category and the initial attack clustering center, and the victim data set can be clustered according to the victim log vector, the class weight corresponding to each security device category and the initial victim clustering center. Wherein, the preset clustering algorithm can be, for example, a clustering algorithm based on a clustering objective function; the clustering objective function refers to an objective function based on the class weight of different types of security devices.

[0060] Optionally, the clustering objective function can be constructed according to the class weight corresponding to each security device category and the initial attack clustering center, and the attack log vector is substituted into the clustering objective function, and the value of the clustering objective function is updated by iteration until the end condition is met, so that the attack data set is clustered to obtain the clustering label corresponding to each attack address.

[0061] Exemplarily, assuming that the number of target logs of the input data set Indata (i.e. the attack log vector corresponding to the attack data set or the victim log vector corresponding to the victim data set) is N, the number of security device types is K, the dimension of the log vector output by the coding model is D, and the number of clustering labels is C. Then the corresponding clustering objective function Obj can be expressed as:

[0062]

[0063] where w i is the category weight of the i-th security device, satisfying the condition: for all i = 1, 2, …, K, w i ∈ [0, 1] and The weight is initially set to the average value at init x ij is the i-th log vector in the j-th record of the input data set Indata, with vector dimension D.a ik is the cluster center vector of the k-th cluster in the i-th security device type, with vector dimension D.‖x ij -a ik ‖ is the Euclidean distance between two D-dimensional vectors x ij and a ik . The number of records in the input data set Indata is N.b i is the coefficient of the Gaussian kernel function. The number of clusters is C.u jk is the indicator value of whether it belongs to cluster k, taking values 0 or 1. That is, for all j = 1, 2, …, N and k = 1, 2, …, C, when u jk is 0, the j-th record of the input data set Indata does not belong to the k-th cluster.u jk is 1, the j-th record of the input data set Indata belongs to the k-th cluster.

[0064] Optionally, the coefficient b i of the Gaussian kernel function can be defined as follows:

[0065]

[0066] where MAX is the maximum value function, and MIN is the minimum value function; MIN k＝1,…,C () means taking the minimum value of the expression in () when k takes values from 1 to C.‖x ij -a ik ‖ is the Euclidean distance between two D-dimensional vectors x ij and a ik . The number of records of the target log in the input data set Indata is N.

[0067] Optionally, the update function of the indicator value u jk can be defined as follows:

[0068]

[0069] where the condition U is defined as:

[0070]

[0071] where‖x ij -aik ‖ is the Euclidean distance between two D-dimensional vectors x ij and a ik . b i is the coefficient of the Gaussian kernel function. w i is the class weight of the i-th type of security device, and the number of security device types is K.

[0072] When the equation of condition U is met, it is determined that condition U is met, otherwise it is not met.

[0073] Optionally, the update function of the class weight w i is defined as follows:

[0074]

[0075] where ‖x ij -a ik ‖ is the Euclidean distance between two D-dimensional vectors x ij and a ik . b i is the coefficient of the Gaussian kernel function. w i is the class weight of the i-th type of security device, and the number of security device types is K. u jk is the index value of whether it belongs to cluster k; the number of target logs of the input data set Indata is N.

[0076] Optionally, let x ij be the i-th log vector in the j-th record of the input data set Indata, and the vector dimension is D, then define the d-th component in the vector as x ij[d] , d = 1, 2, …, D. a ik is the cluster center vector of the k-th cluster in the i-th type of security device, and the d-th component in the vector is defined as a ik[d] , d = 1, 2, …, D; the update function of a ik[d] may be defined as follows:

[0077]

[0078] where ‖x ij -a ik ‖ is the Euclidean distance between two D-dimensional vectors x ij and a ik . b i is the coefficient of the Gaussian kernel function. u jk is the index value of whether it belongs to cluster k; xij[d] is the d-th component of the i-th log vector.

[0079] Optionally, based on the above formula, the clustering process of the attack data set or the victim data set can be as follows: (1) initialization process: for all i = 1, 2, …, K, the initial setting for all j = 1, 2, …, N and k = 1, 2, …, C, u jk randomly assign 0 or 1; for all i = 1, 2, …, K, k = 1, 2, …, C and d = 1, 2, …, D, use calculate the initial a ik[d] ; the initial value of the objective function Obj is set to 0; the convergence condition parameter E is set; (2) let i iterate from 1 to K, update b i according to (formula 2); update w i according to (formula 4); update a ik[d] for all k = 1, 2, …, C and d = 1, 2, …, D; calculate the new value of the objective function Obj new after each update according to (formula 1); and judge whether the convergence condition is met based on ‖Obj new - Obj‖ < E, if the convergence condition is met, the iteration is determined to be ended. And update u jk for all j = 1, 2, …, N and k = 1, 2, …, C according to (formula 3); (3) when the convergence condition is met, that is, the iteration is ended, the clustering label of each log record of Indata can be output according to the current u jk (j = 1, 2, …, N and k = 1, 2, …, C).

[0080] Optionally, according to the clustering result, the target log is subjected to attack correlation analysis and victim correlation analysis, including: generating an attack cluster set according to the attack statements and attack log vectors corresponding to each attack address, combining the clustering labels in the clustering result, and generating a victim cluster set according to the victim statements and victim log vectors corresponding to each victim address, combining the clustering labels in the clustering result; matching the target attack cluster set and the target victim cluster set according to the target attack addresses and target victim addresses recorded in each target log, to obtain a matching result; and determining the attack correlation analysis result and the victim correlation analysis result of each target log according to the matching result.

[0081] Wherein each record of the attack cluster set is composed of an attack address, K attack statements, K attack log vectors corresponding to the K attack statements, and a clustering label. The clustering label comes from the clustering calculation result of the attack data set by the clustering algorithm. Similarly, each record of the victim cluster set is composed of a victim address, K victim statements, K victim log vectors corresponding to the K victim statements, and a clustering label. The clustering label comes from the clustering calculation result of the victim data set by the clustering algorithm.

[0082] Optionally, for each target log in the target log data set, if the attack address is not empty, attack correlation analysis can be performed, and if the victim address is not empty, victim correlation analysis can be performed, that is, each target log in the target log data set can be sequentially taken as a log to be analyzed to perform attack correlation analysis and victim correlation analysis, and if only the attack address is recorded in the target log, only the attack clustering set is matched and attack correlation analysis is performed.

[0083] Optionally, according to the matching result, the attack correlation analysis result of each target log is determined, including: determining the target clustering label corresponding to the target log according to the matching result, and determining the target attack statement corresponding to the target clustering label in the target attack clustering set according to the target clustering label; if the target attack statement corresponds to the same attack address as the target attack address, the historical attack situation of the target attack address is analyzed according to the target attack statement; if the attack address to which the target attack statement belongs is different from the target attack address, the potential attack situation with high correlation with the target attack address is determined according to the target attack statement.

[0084] Among them, the historical attack situation refers to the analysis of the possible historical attacks of the attack address corresponding to the attack party, and the potential attack situation refers to the analysis of the possible future attacks of the attack address corresponding to the attack party.

[0085] Optionally, the attack address of the target log can be used to search for the clustering label corresponding to the attack address in the attack clustering set to determine the target clustering label, and all attack statements corresponding to the attack IP under the clustering label are determined as the target attack statement.

[0086] Optionally, the hit rule in the attack statement corresponding to all attack addresses under the target clustering label can be used to obtain the possible attack, that is, to determine the historical attack situation.

[0087] Exemplarily, the attack correlation analysis method can be as follows: (1) according to the attack address of the target log, searching for the clustering label corresponding to the attack address in the attack clustering set, outputting all attack statements of the clustering label, that is, the target attack statement; (2) if the attack address of the target attack statement output in step (1) is the same as the attack address of the target log, the attack address of the target log can be traced back to the attack once performed; (3) if the attack address of the output attack statement in step (1) is different from the attack address of the target log, it can be inferred that the attack address of the target log may perform other attacks with high relevance.

[0088] Exemplarily, the victim correlation analysis method can be as follows: (1) input the victim address of the target log, find the cluster label corresponding to the victim address in the victim cluster set, and output all victim statements of the cluster label; (2) if the victim address of the victim statement output in step (1) is the same as the victim address of the target log, the victim address of the target log can be traced to have suffered attacks; (3) if the victim address of the victim statement output in step (1) is different from the victim address of the target log, it can be inferred that the victim address of the target log may suffer from other attacks with high relevance.

[0089] Optionally, after determining the results of the attack correlation analysis and the victim correlation analysis, the analysis results can be sent to a relevant security engineer to instruct the security engineer to determine a threat response scheme.

[0090] It should be noted that the present scheme can comprehensively analyze logs output by different types of security devices, and can comprehensively analyze a security threat log to multiple different attacks possibly initiated by the attack IP of the security threat log and multiple different attacks possibly suffered by the victim IP. With the aid of these analysis results, the security engineer can more efficiently and comprehensively formulate a response scheme for preventing the attack IP and reinforcing the victim IP, prevent the occurrence of related other security threats while processing the current security threat, and improve the overall security of the network information system.

[0091] It should be noted that the present application designs a unified log vector generation algorithm for logs of different types of security devices, and converts logs in text form into log vectors that can be used for numerical calculation. The log vector algorithm can extract both the sequence information of the text statements and the frequency information of the text statements. When designing a clustering algorithm for the unified log vector for correlation analysis, it is also considered that the actual semantics corresponding to the log vectors of different types of security devices may have large differences. The clustering algorithm designs an iteratively updated type weight and type center point for the log vectors of different types of security devices, which can distinguish the differences between different types and correlate different types when iterating.

[0092] The technical scheme of the embodiment of the present application, if the government affair server detects a request of performing correlation analysis on the target log of the multiple types of security devices, the target log is converted and extracted to obtain a target log data set; the attack data set and the victim data set corresponding to the target log data set are determined, the attack data set is vectorized to obtain an attack log vector, and the victim data set is vectorized to obtain a victim log vector; the attack data set is clustered according to the attack log vector, the victim data set is clustered according to the victim log vector, and the target log is analyzed in correlation with attacks and victims according to the clustering results. By dividing the log data set into the attack data set and the victim data set and processing in combination with the clustering algorithm, the target log of the multiple types of security devices can be more comprehensively analyzed, the efficiency of log processing is improved, and thus a more effective threat response scheme can be formulated by relevant personnel.

[0093] Figure 2 is a structural block diagram of a log processing device provided by the present application; the embodiment can be applicable to the case that the government affair server performs correlation analysis on the target log of the multiple types of security devices based on a clustering algorithm to assist relevant personnel in formulating a more effective threat response scheme, the log processing device provided by the present application can execute the log processing method provided by the present application, has the function modules and beneficial effects corresponding to the execution method; the log processing device can be realized in the form of hardware and / or software and configured in an electronic device, such as a government affair server. As shown in the figure, the log processing device specifically includes: Figure 2

[0094] The obtaining module 201 is configured to, if a request of performing correlation analysis on the target log of the multiple types of security devices is detected, convert and extract the target log to obtain a target log data set;

[0095] The vectorization module 202 is configured to determine the attack data set and the victim data set corresponding to the target log data set, vectorize the attack data set to obtain an attack log vector, and vectorize the victim data set to obtain a victim log vector;

[0096] The analysis module 203 is configured to cluster the attack data set according to the attack log vector, cluster the victim data set according to the victim log vector, and analyze the target log in correlation with attacks and victims according to the clustering results.

[0097] ​The technical scheme of the embodiment of the present application, if the government affair server detects a request for associating and analyzing the target log of the multiple types of security devices, the target log is converted and extracted to obtain a target log data set; the attack data set and the victim data set corresponding to the target log data set are determined, the attack data set is vectorized to obtain an attack log vector, and the victim data set is vectorized to obtain a victim log vector; the attack data set is clustered according to the attack log vector, the victim data set is clustered according to the victim log vector, and the target log is analyzed for attack association and victim association according to the clustering results. By dividing the log data set into the attack data set and the victim data set and processing by combining the clustering algorithm, the target log of the multiple types of security devices can be more comprehensively analyzed, the efficiency of log processing is improved, and thus a more effective threat response scheme can be formulated by relevant personnel.

[0098] Further, the vectorization module 202 is specifically configured to:

[0099] According to the record of the attack address in each target log in the target log data set, the target log data set is filtered to obtain an attack data set;

[0100] According to the record of the victim address in each target log in the target log data set, the target log data set is filtered to obtain a victim data set.

[0101] Further, the vectorization module 202 can include:

[0102] The statement generation unit is configured to generate an attack statement according to the attack alias corresponding to each attack address in the attack data set, the security device type, and the hit rule;

[0103] The processing unit is configured to process the attack statement to obtain an encoding vector, and process the hit data corresponding to each attack address to obtain a matching vector;

[0104] The splicing unit is configured to splice the encoding vector and the matching vector to obtain an attack log vector corresponding to the attack data set.

[0105] Further, the processing unit is specifically configured to:

[0106] According to the preset insertion order, the hit rule corresponding to each attack address is inserted into the attack statement, and a preset encoding model is used to process the attack statement after the hit rule is inserted to obtain an encoding vector;

[0107] A preset regular expression dictionary is used to match the hit data corresponding to each attack address to obtain a matching vector.

[0108] Further, the analysis module 203 is specifically configured to:

[0109] According to the attack log vector, the category weight corresponding to each security device category, and the initial attack clustering center, the attack data set is clustered;

[0110] According to the victim log vector, the category weight corresponding to each security device category, and the initial victim clustering center, the victim data set is clustered.

[0111] Further, the analysis module 203 can include:

[0112] The clustering set generation unit is configured to generate an attack clustering set according to the attack statement corresponding to each attack address and the attack log vector, in combination with the clustering label in the clustering result, and generate a victim clustering set according to the victim statement corresponding to each victim address and the victim log vector, in combination with the clustering label in the clustering result;

[0113] The matching unit is configured to match the target attack clustering set and the target victim clustering set according to the target attack address and the target victim address recorded in each target log, to obtain a matching result;

[0114] The analysis unit is configured to determine the attack correlation analysis result and the victim correlation analysis result of each target log according to the matching result.

[0115] Further, the analysis unit is specifically configured to:

[0116] According to the matching result, determine the target clustering label corresponding to the target log, and according to the target clustering label, determine the target attack statement corresponding to the target clustering label in the target attack clustering set;

[0117] If the attack address corresponding to the target attack statement and the target attack address are the same, then according to the target attack statement, analyze the historical attack situation of the target attack address;

[0118] If the attack address to which the target attack statement belongs and the target attack address are different, then according to the target attack statement, determine the potential attack situation having high correlation with the target attack address.

[0119] Further, the obtaining module 201 is specifically configured to:

[0120] According to the preset conversion rule, convert the attack addresses in each target log to obtain attack aliases, and convert the victim addresses in each target log to obtain victim aliases;

[0121] Extract the log record time, attack address, victim address, security device type, hit rule, and hit data corresponding to each target log, combine the attack aliases and the victim aliases, and generate a target log data set.

[0122] Figure 3 is a structural schematic diagram of an electronic device provided by the present application. Figure 3 A structural schematic diagram of an electronic device 10 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application described and / or claimed in this document.

[0123] As shown in Figure 3 The electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., connected in communication with the at least one processor 11, where the memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer programs stored in the read-only memory (ROM) 12 or loaded into the random access memory (RAM) 13 from the storage unit 18. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0124] A plurality of components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc., an output unit 17, such as various types of displays, speakers, etc., a storage unit 18, such as a magnetic disk, an optical disk, etc., and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0125] The processor 11 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the log processing method.

[0126] In some embodiments, the log processing method can be implemented as a computer program tangibly embodied in a computer readable storage medium, e.g., storage unit 18. In some embodiments, parts or all of the computer program can be loaded and / or installed onto electronic device 10 via, e.g., ROM 12 and / or communication unit 19. When the computer program is loaded onto RAM 13 and executed by processor 11, one or more steps of the log processing method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the log processing method by way of other means, e.g., by way of firmware.

[0127] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0128] Computer programs used to implement the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed by the processor of the machine, implements the functions / acts specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0129] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0130] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0131] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0132] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. Servers can be cloud servers, also known as cloud computing servers or cloud hosts, which are a host product in the cloud computing service system to solve the defects of large management difficulty and weak business scalability in traditional physical hosts and VPS services.

[0133] In an embodiment, the present embodiment further includes a computer program product, which includes a computer program, the computer program, when executed by a processor, implements the log processing method of any embodiment of the present application.

[0134] The computer program code implementing the operations of the present application can be written in one or more programming languages or combinations of languages including object oriented languages such as Java, Smalltalk, C++ or conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0135] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the spirit and scope of the present application. For example, the steps recited in the present application can be performed in parallel, in series, or in a different order, and the present application is not limited in this regard.

[0136] The above detailed description does not limit the scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modification, equivalent replacement, and improvement within the spirit and principles of the present application should be included in the scope of the present application.

Claims

1. A log processing method, characterized in that, Executed by the government server, including: If a request to perform correlation analysis on target logs of multiple types of security devices is detected, the target logs are transformed and extracted to obtain the target log dataset. Identify the attack dataset and victim dataset corresponding to the target log dataset, and vectorize the attack dataset to obtain attack log vectors, and vectorize the victim dataset to obtain victim log vectors. The attack dataset is clustered based on the attack log vectors, the victim dataset is clustered based on the victim log vectors, and attack correlation analysis and victim correlation analysis are performed on the target logs based on the clustering results. Based on the clustering results, attack correlation analysis and victim correlation analysis are performed on the target logs, including: Based on the attack statements and attack log vectors corresponding to each attack address, and combined with the clustering labels in the clustering results, an attack cluster set is generated. Based on the victim statements and victim log vectors corresponding to each victim address, and combined with the clustering labels in the clustering results, a victim cluster set is generated. Based on the target attack address and target victim address recorded in each target log, a match is made in the target attack cluster set and the target victim cluster set to obtain the matching result; Based on the matching results, determine the attack correlation analysis results and victim correlation analysis results for each target log; Based on the matching results, the attack correlation analysis results for each target log are determined, including: Based on the matching results, determine the target cluster label corresponding to the target log, and based on the target cluster label, determine the target attack statement corresponding to the target cluster label in the target attack cluster set; If the attack address corresponding to the target attack statement is the same as the target attack address, then analyze the historical attack situation of the target attack address based on the target attack statement; If the attack address to which the target attack statement belongs is different from the target attack address, then based on the target attack statement, determine potential attack situations that are highly correlated with the target attack address. Among them, historical attack situation refers to the analysis of the historical attacks of the attacker corresponding to the attack address, and potential attack situation refers to the analysis of the attacks that the attacker corresponding to the attack address may launch in the future. Based on the matching results, the victim association analysis results for each target log are determined, including: Based on the matching results, determine the target cluster label corresponding to the target log, and based on the target cluster label, determine the target victim statement corresponding to the target cluster label in the target victim cluster set; If the victim address corresponding to the target victim statement is the same as the target victim address, then analyze the historical attacks suffered by the target victim address based on the target victim statement. If the victim address to which the target victim statement belongs is different from the target victim address, then based on the target victim statement, determine potential attack scenarios that are highly correlated with the target victim address.

2. The method according to claim 1, characterized in that, Identify the attack dataset and victim dataset corresponding to the target log dataset, including: Based on the records of attack addresses in each target log in the target log dataset, the target log dataset is filtered to obtain the attack dataset; Based on the records of victim addresses in each target log in the target log dataset, the target log dataset is filtered to obtain the victim dataset.

3. The method according to claim 1, characterized in that, The attack dataset is vectorized to obtain attack log vectors, including: Based on the attack aliases, security device types, and hit rules corresponding to each attack address in the attack dataset, generate attack statements; The attack statements are processed to obtain the encoding vector, and the hit data corresponding to each attack address is processed to obtain the matching vector; The encoded vector and the matching vector are concatenated to obtain the attack log vector corresponding to the attack dataset.

4. The method according to claim 3, characterized in that, The attack statements are processed to obtain encoded vectors, and the hit data corresponding to each attack address is processed to obtain matching vectors, including: Based on the preset insertion order, the hit rules corresponding to each attack address are inserted into the attack statement, and the attack statement after the insertion of the hit rules is processed by the preset encoding model to obtain the encoding vector. A pre-defined regular expression dictionary is used to match the hit data corresponding to each attack address to obtain a matching vector.

5. The method according to claim 1, characterized in that, Clustering of the attack dataset based on attack log vectors and clustering of the victim dataset based on victim log vectors include: The attack dataset is clustered based on the attack log vectors, the category weights corresponding to each security device category, and the initial attack cluster centers. The victim dataset is clustered based on the victim log vector, the category weights corresponding to each security device category, and the initial victim cluster centers.

6. The method according to claim 1, characterized in that, The target logs are transformed and extracted to obtain the target log dataset, including: Based on preset conversion rules, the attack addresses in each target log are converted to obtain attack aliases, and the victim addresses in each target log are converted to obtain victim aliases. Extract the log recording time, attack address, victim address, security device type, hit rules, and hit data corresponding to each target log. Combine these with the attack alias and victim alias to generate a target log dataset.

7. A log processing device, characterized in that, include: The module is used to transform and extract the target logs to obtain the target log dataset if a request for correlation analysis of target logs of multiple types of security devices is detected. The vectorization module is used to determine the attack dataset and victim dataset corresponding to the target log dataset, and to vectorize the attack dataset to obtain attack log vectors and the victim dataset to obtain victim log vectors. The analysis module is used to cluster the attack dataset based on the attack log vector, cluster the victim dataset based on the victim log vector, and perform attack correlation analysis and victim correlation analysis on the target logs based on the clustering results. The analysis module includes: The clustering set generation unit is used to generate an attack clustering set based on the attack statements and attack log vectors corresponding to each attack address, combined with the clustering labels in the clustering results; and to generate a victim clustering set based on the victim statements and victim log vectors corresponding to each victim address, combined with the clustering labels in the clustering results. The matching unit is used to match the target attack address and the target victim address recorded in each target log in the target attack cluster set and the target victim cluster set to obtain the matching result; The analysis unit is used to determine the attack correlation analysis results and victim correlation analysis results for each target log based on the matching results; Specifically, the analysis unit is used for: Based on the matching results, determine the target cluster label corresponding to the target log, and based on the target cluster label, determine the target attack statement corresponding to the target cluster label in the target attack cluster set; If the attack address corresponding to the target attack statement is the same as the target attack address, then analyze the historical attack situation of the target attack address based on the target attack statement; If the attack address to which the target attack statement belongs is different from the target attack address, then based on the target attack statement, determine potential attack situations that are highly correlated with the target attack address. Among them, historical attack situation refers to the analysis of the historical attacks of the attacker corresponding to the attack address, and potential attack situation refers to the analysis of the attacks that the attacker corresponding to the attack address may launch in the future. The analysis unit is also specifically used for: Based on the matching results, determine the target cluster label corresponding to the target log, and based on the target cluster label, determine the target victim statement corresponding to the target cluster label in the target victim cluster set; If the victim address corresponding to the target victim statement is the same as the target victim address, then analyze the historical attacks suffered by the target victim address based on the target victim statement. If the victim address to which the target victim statement belongs is different from the target victim address, then based on the target victim statement, determine potential attack scenarios that are highly correlated with the target victim address.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that is executed by the at least one processor to enable the at least one processor to perform the log processing method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the log processing method according to any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the log processing method according to any one of claims 1-6.

Images