Log processing method and device, equipment and storage medium

Through government servers, the logs of multiple types of security devices are converted, vectorized and clustered, and the problem of low log processing efficiency in the existing technology is solved, more effective threat response solutions are achieved, and the security of network information systems is improved. sex.

CN118939518AActive Publication Date: 2024-11-12DIGITAL GUANGDONG NETWORK CONSTR CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411007188.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-12
Estimated Expiration
2044-07-25

AI Technical Summary

Technical Problem

The prior art is difficult to effectively analyze logs of multiple types of security devices, which leads to increased difficulty in formulating threat response plans and low log processing efficiency.

Method used

The target log is transformed and extracted through the government server, divided into attack data sets and victim data sets, and vectorized processing and clustering analysis are carried out to realize attack correlation analysis and victim correlation analysis.

Benefits of technology

It improves the efficiency of log processing, assists relevant personnel in formulating more effective threat response plans, comprehensively analyzes the logs of multiple types of security equipment, and improves the overall security of network information systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939518A_ABST
    Figure CN118939518A_ABST
Patent Text Reader

Abstract

The invention discloses a log processing method and device, equipment and a storage medium. The method comprises the following steps: if a government affair server detects a request for carrying out association analysis on target logs of multi-type security equipment, carrying out conversion and extraction processing on the target logs to obtain a target log data set; determining an attack data set and a victim data set corresponding to the target log data set, performing vectorization processing on the attack data set to obtain an attack log vector, and performing vectorization processing on the victim data set to obtain a victim log vector; and performing clustering processing on the attack data set according to the attack log vector, performing clustering processing on the victim data set according to the victim log vector, and performing attack association analysis and victim association analysis on the target log according to a clustering result. According to the technical scheme, the target logs of the multiple types of safety devices can be analyzed more comprehensively, the log processing efficiency is improved, and therefore related personnel are assisted in making a more effective threat coping scheme.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of network and information technology, and in particular to a log processing method, device, equipment and storage medium. Background Art

[0002] With the continuous development of network and information technology, information security issues are receiving more and more attention. Especially for government servers, they are associated with different types of information security devices. When these information security devices detect security threats, they will output alarms and security threat logs. When the types and number of security devices increase, the number of various alarms and logs generated every day will also increase sharply, which increases the difficulty of formulating threat response plans.

[0003] Therefore, how to conduct a more comprehensive analysis of the logs of multiple types of security devices, improve the efficiency of log processing, and assist relevant personnel in developing more effective threat response plans is an urgent problem to be solved. Summary of the invention

[0004] The present invention provides a log processing method, apparatus, device and storage medium to perform a more comprehensive analysis of target logs of multiple types of security devices, improve the efficiency of log processing, and thus assist relevant personnel in formulating more effective threat response plans.

[0005] According to one aspect of the present invention, a log processing method is provided, which is executed by a government affairs server and includes:

[0006] If a request for correlation analysis of target logs of multiple types of security devices is detected, the target logs are converted and extracted to obtain a target log data set;

[0007] Determine the attack data set and victim data set corresponding to the target log data set, and vectorize the attack data set to obtain an attack log vector, and vectorize the victim data set to obtain a victim log vector;

[0008] The attack data set is clustered according to the attack log vector, the victim data set is clustered according to the victim log vector, and based on the clustering results, attack correlation analysis and victim correlation analysis are performed on the target log.

[0009] According to another aspect of the present invention, there is provided a log processing device, comprising:

[0010] The obtaining module is used to convert and extract the target logs to obtain a target log data set if a request for correlation analysis of target logs of multiple types of security devices is detected;

[0011] A vectorization module is used to determine the attack data set and the victim data set corresponding to the target log data set, and to vectorize the attack data set to obtain the attack log vector, and to vectorize the victim data set to obtain the victim log vector;

[0012] The analysis module is used to cluster the attack data set according to the attack log vector, cluster the victim data set according to the victim log vector, and perform attack correlation analysis and victim correlation analysis on the target log according to the clustering result.

[0013] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log processing method described in any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the log processing method described in any embodiment of the present invention when executed.

[0018] According to another aspect of the present invention, a computer program product is further provided. The computer program product includes a computer program. When the computer program is executed by a processor, the log processing method of any embodiment of the present invention is implemented.

[0019] According to the technical solution of the embodiment of the present invention, if the government server detects a request for correlation analysis of target logs of multiple types of security devices, the target logs are converted and extracted to obtain a target log data set; the attack data set and the victim data set corresponding to the target log data set are determined, and the attack data set is vectorized to obtain an attack log vector, and the victim data set is vectorized to obtain a victim log vector; the attack data set is clustered according to the attack log vector, and the victim data set is clustered according to the victim log vector, and based on the clustering results, attack correlation analysis and victim correlation analysis are performed on the target log. By dividing the log data set into an attack data set and a victim data set, and processing them in combination with a clustering algorithm, a more comprehensive analysis of the target logs of multiple types of security devices can be performed, and the efficiency of log processing can be improved, thereby assisting relevant personnel in formulating more effective threat response plans.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 It is a flow chart of a log processing method provided by the present invention;

[0023] Figure 2 It is a structural block diagram of a log processing device provided by the present invention;

[0024] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, what is described is only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first", "second", "target", "candidate", "alternative", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. The acquisition, storage, use, processing, etc. of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0027] It should be noted that in the related art, the logs of a single type of security device are generally analyzed, and the associated information existing in the logs of multiple types of security devices cannot be mined. When security engineers deal with security threats based on the information analyzed in the logs of a single type of security device, it is difficult to associate and foresee other related security threats that accompany the security threat, and there will be a tendency to "treat the head when the headache is and the foot when the foot is" when formulating a treatment plan. When the types and number of security devices become more and more, the number of various alarms and logs generated every day will also increase sharply. If the security engineer still only responds to the current single threat and does not consider other related threats together, he may fall into a passive situation of being exhausted by dealing with various threats every day. Based on the above problems, the present invention divides the logs of multiple types of security devices into attack logs and victim logs according to the attack IP address and victim IP address in the alarm log, and clusters them respectively based on the clustering algorithm to obtain cluster sets, so that the possible attacks on each log can be analyzed according to the cluster set, and a more comprehensive analysis of the log can be achieved, so that relevant personnel can be assisted in formulating a more effective threat response plan. The specific implementation steps will be described in detail later.

[0028] Figure 1 is a flow chart of a log processing method provided by the present invention; this embodiment can be applied to the situation where a government server performs correlation analysis on target logs of multiple types of security devices based on a clustering algorithm to assist relevant personnel in formulating a more effective threat response plan. The method can be executed by a log processing device, which can be implemented in the form of hardware and / or software. The log processing device can be configured in an electronic device, such as a government server, such as Figure 1 As shown, the log processing method includes:

[0029] S101: If a request for correlation analysis of target logs of multiple types of security devices is detected, the target logs are converted and extracted to obtain a target log data set.

[0030] Among them, security equipment refers to equipment configured on government servers to detect and identify security threats. Types of security equipment may include, for example, firewalls (Web Application Firewall, WAF), intrusion prevention systems (Intrusion-prevention system, IPS) and situational awareness equipment. Situational awareness equipment may specifically refer to situational awareness software integrated into hardware. Target log refers to the security threat log sent by each security device to the government server. The target log may include at least one of the following information: log recording time, attack address, victim address, security device type, threat hit rules, and threat hit data. The attack address may be, for example, the IP (Internet Protocol) address that performed the attack operation. The victim address may be, for example, the IP address that was attacked.

[0031] The target log data set refers to the collection of target logs after conversion and extraction. It should be noted that after conversion and extraction, the target log can add information such as attack alias and victim alias. Specifically, the field content and requirements of each target log in the target data set can be: Log recording time: the time when the target log is recorded, which cannot be empty; Attack address: the IP address of the attacker, which can be empty, but cannot be empty at the same time as the victim IP; Attack alias: the alias of the IP address of the attacker, which comes from the IP preprocessing process; Victim address: the IP address of the victim, which can be empty, but cannot be empty at the same time as the attack IP; Victim alias: the alias of the IP address of the victim, which comes from the IP preprocessing process; Security device type: the type of security device that outputs the target log; Hit rule: the security threat rule hit when recording the target log, which cannot be empty; Hit data: the request / response data when recording the target log, which cannot be empty.

[0032] Exemplarily, the information of a target log record in the target log data set may be as follows: attacking IP: 10.0.0.1; attacking IP alias: baoaoaob; victim IP: 10.0.0.2; victim IP alias: baoaoaoc; security device type: Sky Eye; hit rule: general command execution vulnerability; hit data: "name[#this.getClass().forName("java.lang.Runtime").getRuntime().exec("whoam i")]=&username[#this.getClass().forName("java.lang.Runtime").getRuntime().exe c("whoami")]=&password[#this.getClass().forName("java.lang.Runtime").getRun time().exec("whoami")]=".

[0033] Optionally, the government affairs server may automatically generate instructions for performing correlation analysis on target logs generated within a preset historical time period of multiple types of security devices based on a preset cycle, and in this case, it is considered that a request for performing correlation analysis on target logs of multiple types of security devices has been detected; the government affairs server may also consider that a request for performing correlation analysis on target logs of multiple types of security devices has been detected when it detects correlation analysis instructions issued by relevant staff such as security engineers.

[0034] Optionally, the target log is converted and extracted to obtain a target log data set, including: based on a preset conversion rule, converting the attack address in each target log to obtain an attack alias, and converting the victim address in each target log to obtain a victim alias; extracting the log record time, attack address, victim address, security device type, hit rule and hit data corresponding to each target log, combining the attack alias and victim alias to generate a target log data set.

[0035] Among them, the attack alias refers to the information obtained by converting the attack address, the victim alias refers to the information obtained by converting the victim address, and the conversion rule refers to the preset rule for preprocessing the attack address and the victim address so that they can meet the encoding requirements of the encoding model. Exemplarily, the conversion rule can be to replace 0, 1, 2, ..., 9 in the attack address and the victim address with lowercase letters a, b, c, ..., j, respectively, and replace the punctuation mark "." in the attack address and the victim address with lowercase letter o. For example: the IP alias corresponding to the address 10.0.0.1 is "baoaoaob". The security device type refers to the type of security device that sends logs to the government server.

[0036] It should be noted that when the encoding model processes the IP addresses in the logs, it will decompose the IP addresses into single numbers for encoding. In order for the encoding model to encode the IP addresses as a whole, the IP addresses in the logs need to be converted. Specifically, all attack addresses and victim addresses in the target logs can be converted into corresponding address aliases based on preset conversion rules.

[0037] Optionally, matching can be performed in each target log according to preset field names to obtain the log recording time, attack address, victim address, security device type, hit rule and hit data corresponding to each target log.

[0038] S102: Determine an attack data set and a victim data set corresponding to the target log data set, and perform vectorization processing on the attack data set to obtain an attack log vector, and perform vectorization processing on the victim data set to obtain a victim log vector.

[0039] The attack data set refers to the collection of target logs whose attack address field information is not empty, and the victim data set refers to the collection of target logs whose victim address field information is not empty. The attack log vector and victim log vector are vectors generated by integrating and vectorizing the attack data set and the victim data set.

[0040] Optionally, the target log data set may be divided into an attack data set and a victim data set according to the attack address and victim address information of each target log in the target log data set.

[0041] Optionally, determining the attack data set and the victim data set corresponding to the target log data set includes: filtering the target log data set according to the records of the attack addresses in each target log in the target log data set to obtain the attack data set; filtering the target log data set according to the records of the victim addresses in each target log in the target log data set to obtain the victim data set.

[0042] Optionally, according to the record situation, if the attack address corresponding to the target log is not empty, the target log can be added to the attack data set, that is, the attack data set is generated according to the target log whose attack address is not empty in the target log data set. Similarly, the victim data set can be generated according to the target log whose victim address is not empty in the target log data set.

[0043] Optionally, the attack data set is vectorized to obtain an attack log vector, including: generating an attack statement according to the attack alias, security device type and hit rule corresponding to each attack address in the attack data set; processing the attack statement to obtain an encoding vector, and processing the hit data corresponding to each attack address to obtain a matching vector; concatenating the encoding vector and the matching vector to obtain an attack log vector corresponding to the attack data set.

[0044] The attack statement refers to the statement obtained by concatenating the attack alias, security device type, and hit rule based on the preset concatenation rules. The format of the attack statement can be as follows: "[CLS] IP alias [SEP] security device type [SEP] hit rule 1 [SEP] hit rule 2 ... [SEP] hit rule m [SEP]", where [CLS] is a marker of the encoding model (such as the BERT (Bidirectional Encoder Representations from Transformers) model) to indicate the beginning of the input sequence. [SEP] is also a marker of BERT, used to separate two sentences and indicate the end of a sentence.

[0045] Optionally, all hit rules in log records with the same attack address and the same security device type may be put into an attack statement in a preset format to generate an attack statement, that is, if the data of the security device type is K, then K attack statements are generated corresponding to 1 attack address.

[0046] Optionally, according to the format of the attack statement and based on preset splicing rules, the attack aliases, security device types and hit rules corresponding to each attack address can be spliced ​​and combined to generate an attack statement. The preset encoding model can be further used to vectorize the attack statement to obtain an encoding vector. At the same time, a preset regular expression dictionary can be used to match the hit data process to obtain a matching vector. Finally, the matching vector can be spliced ​​to the end of the encoding vector to obtain an attack log vector.

[0047] It should be noted that the victim log vector corresponding to the victim data set can be determined by the same method as that for determining the attack log vector. That is, the attack log vector and the victim log vector are determined in the same manner, which will not be elaborated in the present invention.

[0048] Optionally, the attack statement is processed to obtain a coding vector, and the hit data corresponding to each attack address is processed to obtain a matching vector, including: based on a preset insertion order, the hit rules corresponding to each attack address are inserted into the attack statement, and a preset coding model is used to process the attack statement after the hit rules are inserted to obtain the coding vector; a preset regular expression dictionary is used to match the hit data corresponding to each attack address to obtain a matching vector.

[0049] The preset insertion order may be the order of occurrence of each hit rule. The preset encoding model may be a BERT model. The preset regular expression dictionary refers to a preset dictionary generated by storing all words and sentences related to security threats in historical logs in the form of regular expressions. In the preset regular expression dictionary, each regular expression may be numbered incrementally starting from 1 according to the insertion order of the regular expression.

[0050] Exemplarily, a regular expression dictionary Dictionary containing P regular expressions can be expressed as:

[0051] Dictionary = {[idx,Expression idx ]},

[0052] idx=1,2,…,P,Expression idx is the idxth regular expression;

[0053] Optionally, after inputting the hit data corresponding to each attack address into the regular expression dictionary, a P-dimensional regular expression matching vector match=[count 1 ,count 2 ,…,count P ], where count p ,p=1,2,…,P means that the hit data contains the regular expression Expression that satisfies the pth regular expression in the Dictionary. p If the input hit data is empty, the output matching vector is a vector of all zeros.

[0054] Optionally, the word and sentence patterns related to security threats in the historical logs may be analyzed and converted into regular expressions. If the regular expression to be inserted already exists in the dictionary, the regular expression is not inserted.

[0055] Optionally, for each attack address, the hit rule containing the attack address in the log data set can be inserted into the attack statement of this type of security device in order of appearance time, and the attack statement after the hit rule is inserted is input into a pre-trained encoding model to output an encoding vector.

[0056] It should be noted that since the regular expression dictionary can only extract the number of occurrences of specific pattern words and sentences in the text content, but cannot understand the order of words and sentences, the present invention uses a coding model to determine the coding vector to extract the order of words and sentences in the text content.

[0057] Exemplarily, assuming that the number of security device types is K, and the dimension of the encoding vector output by the encoding model is Q, then each log record of the attack data set consists of an attack address, K attack statements and their corresponding K log vectors. Furthermore, the attack statement can be input into the preset encoding model to obtain a Q-dimensional encoding vector. If the attack statement is empty, the encoding output is a Q-dimensional all-zero vector. Furthermore, the hit data corresponding to each attack address can be merged into a text data and input into the preset regular expression dictionary Dictionary to obtain a P-dimensional matching vector. Finally, the P-dimensional matching vector can be spliced ​​to the end of the Q-dimensional encoding vector to obtain a D-dimensional log vector, that is: D = Q + P.

[0058] S103: clustering the attack data set according to the attack log vector, clustering the victim data set according to the victim log vector, and performing attack correlation analysis and victim correlation analysis on the target log according to the clustering result.

[0059] Optionally, based on a preset clustering algorithm, the attack data set can be clustered according to the attack log vector, and the victim data set can be clustered according to the victim log vector, including: clustering the attack data set according to the attack log vector, the category weights corresponding to each security device category, and the initial attack clustering center; clustering the victim data set according to the victim log vector, the category weights corresponding to each security device category, and the initial victim clustering center. The preset clustering algorithm can be, for example, an algorithm for clustering based on a clustering objective function; the clustering objective function refers to an objective function based on the category weights of different types of security devices.

[0060] Optionally, a clustering objective function can be constructed based on the category weights corresponding to each security device category and the initial attack clustering center, and the attack log vector can be substituted into the clustering objective function. The value of the clustering objective function is made to meet the termination condition through iterative updating, so that the attack data set is clustered to obtain the clustering label corresponding to each attack address.

[0061] For example, let the number of target logs of the input data set Indata (i.e., the attack log vector corresponding to the attack data set or the victim log vector corresponding to the victim data set) be N, the number of security device types be K, the dimension of the output log vector of the encoding model be D, and the number of cluster labels be C. Then the corresponding clustering objective function Obj can be expressed as:

[0062]

[0063] Among them, w i is the category weight of the i-th category safety device, satisfying the condition: for all i=1,2,…,K,w i ∈[0,1] and The weights are initially set to the mean at init x ij is the i-th log vector in the j-th record of the input data set Indata, and the vector dimension is D. a ik is the cluster center vector of the kth cluster in the i-th security device type, and the vector dimension is D. ij -a ik ‖ is to calculate two D-dimensional vectors x ij with a ik The Euclidean distance between them. i is the coefficient of the Gaussian kernel function. jk is the index value of whether it belongs to cluster k, and its value is 0 or 1. That is to say, for all j=1,2,…,N and k=1,2,…,C, when u jk When the value is 0, the jth record of the input data set Indata does not belong to the kth cluster. jk When the value is 1, the j-th record of the input data set Indata belongs to the k-th cluster.

[0064] Optional, coefficient b of the Gaussian kernel function i It can be defined as follows:

[0065]

[0066] Among them, MAX is the maximum value function, and MIN is the minimum value function; MIN k=1,…,C () means when k takes the value from 1 to C, it takes the minimum value of the expression in (). ‖x ij -a ik ‖ is to calculate two D-dimensional vectors x ij with a ik The number of target logs in the input data set Indata is N.

[0067] Optional, indicator value u jk The update function can be defined as follows:

[0068]

[0069] Among them, the condition U is defined as:

[0070]

[0071] Among them, ‖x ij-a ik ‖ is to calculate two D-dimensional vectors x ij with a ik The Euclidean distance between them. i is the coefficient of the Gaussian kernel function. i is the category weight of the i-th category safety device, and the number of safety device types is K.

[0072] When the equation of the above condition U holds true, it is determined that the condition U is satisfied, otherwise the condition U is not satisfied.

[0073] Optional, class weight w i The update function is defined as follows:

[0074]

[0075] Among them, ‖x ij -a ik ‖ is to calculate two D-dimensional vectors x ij with a ik The Euclidean distance between them. i is the coefficient of the Gaussian kernel function. i is the category weight of the i-th category safety device, and the number of safety device types is K. jk is the indicator value of whether it belongs to cluster k; the number of target logs in the input data set Indata is N.

[0076] Optionally, let x ij is the i-th log vector in the j-th record of the input data set Indata, and the vector dimension is D. Then the d-th component in the vector is defined as x ij[d] , d=1,2,…,D. a ik is the cluster center vector of the kth cluster in the i-th security device type, and the dth component in this vector is defined as a ik[d] , d=1,2,…,D; a ik[d] The update function can be defined as follows:

[0077]

[0078] Among them, ‖x ij -a ik ‖ is to calculate two D-dimensional vectors x ij with a ik The Euclidean distance between them. i is the coefficient of the Gaussian kernel function. jk is the indicator value of whether it belongs to cluster k; xij[d] is the dth component of the ith log vector.

[0079] Optionally, based on the above formula, the clustering process for the attack dataset or the victim dataset can be as follows: (1) Initialization process: For all i = 1, 2, …, K, initially set For all j = 1, 2, …, N and k = 1, 2, …, C, for u jk Randomly assign 0 or 1; For all i = 1, 2, …, K, k = 1, 2, …, C and d = 1, 2, …, D, use Calculate the initial a ik[d] ; Initially set the value of the objective function Obj to 0; Set the convergence condition parameter E; (2) Let i iterate from 1 to K, update b according to (Equation 2) i ; Update w according to (Equation 4) i ; Update a according to (Equation 5) ik[d] , for all k = 1, 2, …, C and d = 1, 2, …, D; After each update, calculate the new value of the objective function Obj according to Equation (1) new ; And based on ‖Obj new - Obj‖ < E to determine whether the convergence condition is satisfied. If the convergence condition is satisfied, determine to end the iteration. And update u according to (Equation 3) jk , for all j = 1, 2, …, N and k = 1, 2, …, C; (3) When the convergence condition is satisfied, that is, when the iteration ends, the clustering label of each log record in Indata can be output according to the current u jk (j = 1, 2, …, N and k = 1, 2, …, C).

[0080] Optionally, according to the clustering results, perform attack correlation analysis and victim correlation analysis on the target logs, including: According to the attack statements and attack log vectors corresponding to each attack address, combined with the clustering labels in the clustering results, generate an attack clustering set, and according to the victim statements and victim log vectors corresponding to each victim address, combined with the clustering labels in the clustering results, generate a victim clustering set; According to the target attack addresses and target victim addresses recorded in each target log, perform matching in the target attack clustering set and the target victim clustering set to obtain a matching result; According to the matching result, determine the attack correlation analysis result and victim correlation analysis result for each target log.

[0081] Among them, each record in the attack clustering set consists of an attack address, K attack statements and their corresponding K attack log vectors, and a clustering label. The clustering label comes from the clustering calculation result of the attack dataset by the clustering algorithm. Similarly, each record in the victim clustering set consists of a victim address, K victim statements and their corresponding K victim log vectors, and a clustering label. The clustering label comes from the clustering calculation result of the victim dataset by the clustering algorithm.

[0082] Optionally, for each target log in the target log data set, if its attack address is not empty, attack correlation analysis can be performed; if its victim address is not empty, victim correlation analysis can be performed. In other words, each target log in the target log data set can be taken as a log to be analyzed in turn, and attack correlation analysis and victim correlation analysis can be performed. If only the attack address is recorded in the target log, it is only matched in the attack cluster set and attack correlation analysis is performed.

[0083] Optionally, based on the matching results, the attack correlation analysis results for each target log are determined, including: based on the matching results, determining the target cluster label corresponding to the target log, and based on the target cluster label, determining the target attack statement corresponding to the target cluster label in the target attack cluster set; if the attack address corresponding to the target attack statement is the same as the target attack address, then analyzing the historical attacks on the target attack address based on the target attack statement; if the attack address to which the target attack statement belongs is different from the target attack address, then determining potential attacks that are highly correlated with the target attack address based on the target attack statement.

[0084] Among them, historical attack situations refer to the analysis of possible historical attacks of the attacker corresponding to the attack address, and potential attack situations refer to the analysis of possible future attacks of the attacker corresponding to the attack address.

[0085] Optionally, according to the attack address of the target log, a cluster label corresponding to the attack address can be searched in the attack cluster set to determine the target cluster label, and the attack statements corresponding to all attack IPs under the cluster label can be determined as the target attack statements.

[0086] Optionally, possible attacks may be obtained according to the hit rules in the attack statements corresponding to all attack addresses under the target cluster label, that is, historical attack situations may be determined.

[0087] Exemplarily, the attack correlation analysis method can be as follows: (1) According to the attack address of the target log, the cluster label corresponding to the attack address is searched in the attack cluster set, and all attack statements with the cluster label are output, namely, the target attack statement; (2) If the attack address of the target attack statement output in step (1) is the same as the attack address of the target log, then the attack that has been carried out by the attack address of the target log can be traced; (3) If the attack address of the attack statement output in step (1) is different from the attack address of the target log, then it can be inferred that the attack address of the target log may carry out other attacks with high correlation.

[0088] Exemplarily, the victim association analysis method can be as follows: (1) Input the victim address of the target log, search for the cluster label corresponding to the victim address in the victim cluster set, and output all victim statements with the cluster label; (2) If the victim address of the victim statement output in step (1) is the same as the victim address of the target log, then the attack that the victim address of the target log has suffered can be traced; (3) If the victim address of the victim statement output in step 1 is different from the victim address of the target log, then it can be inferred that the victim address of the target log may suffer other attacks with high correlation.

[0089] Optionally, after determining the results of the attack correlation analysis and the victim correlation analysis, the analysis results may be sent to relevant security engineers to instruct the security engineers to determine a threat response plan.

[0090] It should be noted that this solution can comprehensively analyze the logs output by different types of security devices. For a security threat log, it can comprehensively analyze the various different attacks that the attacker's IP may launch, and the various different attacks that the victim's IP may suffer. With the help of these analysis results, security engineers can more efficiently and comprehensively formulate response plans to prevent attackers and strengthen victims. While dealing with current security threats, they can also prevent the occurrence of other related security threats and improve the overall security of network information systems.

[0091] It should be noted that the present invention designs a unified log vector generation algorithm for logs of different types of security devices, converting logs in text form into log vectors that can be used for numerical calculations. The log vector algorithm can extract both the order relationship information of text sentences and the frequency of occurrence information of text sentences. When designing a clustering algorithm for association analysis of unified log vectors, it is also considered that the actual semantics corresponding to the log vectors of different types of security devices may be quite different. The clustering algorithm designs iteratively updateable category weights and category center points for the log vectors of different types of security devices, which can both distinguish the differences between different categories and associate different categories when iterating.

[0092] According to the technical solution of the embodiment of the present invention, if the government server detects a request for correlation analysis of target logs of multiple types of security devices, the target logs are converted and extracted to obtain a target log data set; the attack data set and the victim data set corresponding to the target log data set are determined, and the attack data set is vectorized to obtain an attack log vector, and the victim data set is vectorized to obtain a victim log vector; the attack data set is clustered according to the attack log vector, and the victim data set is clustered according to the victim log vector, and based on the clustering results, attack correlation analysis and victim correlation analysis are performed on the target log. By dividing the log data set into an attack data set and a victim data set, and processing them in combination with a clustering algorithm, a more comprehensive analysis of the target logs of multiple types of security devices can be performed, and the efficiency of log processing can be improved, thereby assisting relevant personnel in formulating more effective threat response plans.

[0093] Figure 2 This is a structural block diagram of a log processing device provided by the present invention; this embodiment can be applied to the situation where the government server performs correlation analysis on the target logs of multiple types of security devices based on the clustering algorithm to assist relevant personnel in formulating more effective threat response plans. The log processing device provided by the present invention can execute the log processing method provided by the present invention, and has the corresponding functional modules and beneficial effects of the execution method; the log processing device can be implemented in the form of hardware and / or software, and configured in an electronic device, such as a government server. Figure 2 As shown, the log processing device specifically includes:

[0094] The obtaining module 201 is used to convert and extract the target logs to obtain a target log data set if a request for correlation analysis of target logs of multiple types of security devices is detected;

[0095] A vectorization module 202 is used to determine the attack data set and the victim data set corresponding to the target log data set, and perform vectorization processing on the attack data set to obtain an attack log vector, and perform vectorization processing on the victim data set to obtain a victim log vector;

[0096] The analysis module 203 is used to perform clustering processing on the attack data set according to the attack log vector, perform clustering processing on the victim data set according to the victim log vector, and perform attack correlation analysis and victim correlation analysis on the target log according to the clustering result.

[0097] According to the technical solution of the embodiment of the present invention, if the government server detects a request for correlation analysis of target logs of multiple types of security devices, the target logs are converted and extracted to obtain a target log data set; the attack data set and the victim data set corresponding to the target log data set are determined, and the attack data set is vectorized to obtain an attack log vector, and the victim data set is vectorized to obtain a victim log vector; the attack data set is clustered according to the attack log vector, and the victim data set is clustered according to the victim log vector, and based on the clustering results, attack correlation analysis and victim correlation analysis are performed on the target log. By dividing the log data set into an attack data set and a victim data set, and processing them in combination with a clustering algorithm, a more comprehensive analysis of the target logs of multiple types of security devices can be performed, and the efficiency of log processing can be improved, thereby assisting relevant personnel in formulating more effective threat response plans.

[0098] Furthermore, the vectorization module 202 is specifically used for:

[0099] According to the record of the attack address in each target log in the target log data set, the target log data set is screened to obtain the attack data set;

[0100] According to the record status of the victim address in each target log in the target log data set, the target log data set is screened to obtain the victim data set.

[0101] Furthermore, the vectorization module 202 may include:

[0102] A statement generating unit, used for generating an attack statement according to the attack alias, the security device type and the hit rule corresponding to each attack address in the attack data set;

[0103] A processing unit, used to process the attack statement to obtain a coding vector, and to process the hit data corresponding to each attack address to obtain a matching vector;

[0104] The concatenation unit is used to concatenate the encoding vector and the matching vector to obtain an attack log vector corresponding to the attack data set.

[0105] Furthermore, the processing unit is specifically used for:

[0106] Based on a preset insertion order, the hit rules corresponding to each attack address are inserted into the attack statement, and the attack statement after the hit rules are inserted is processed using a preset encoding model to obtain an encoding vector;

[0107] The preset regular expression dictionary is used to match the hit data corresponding to each attack address to obtain a matching vector.

[0108] Furthermore, the analysis module 203 is specifically used for:

[0109] Cluster the attack data set according to the attack log vector, the category weight corresponding to each security device category, and the initial attack cluster center;

[0110] The victim data set is clustered according to the victim log vector, the category weight corresponding to each security device category and the initial victim cluster center.

[0111] Furthermore, the analysis module 203 may include:

[0112] A clustering set generating unit, used to generate an attack clustering set according to the attack statements and attack log vectors corresponding to each attack address, combined with the clustering labels in the clustering results, and to generate a victim clustering set according to the victim statements and victim log vectors corresponding to each victim address, combined with the clustering labels in the clustering results;

[0113] A matching unit, used for matching the target attack address and the target victim address recorded in each target log in the target attack cluster set and the target victim cluster set to obtain a matching result;

[0114] The analysis unit is used to determine the attack correlation analysis result and the victim correlation analysis result of each target log according to the matching result.

[0115] Furthermore, the analysis unit is specifically used for:

[0116] According to the matching result, determine the target cluster label corresponding to the target log, and according to the target cluster label, determine the target attack statement corresponding to the target cluster label in the target attack cluster set;

[0117] If the attack address corresponding to the target attack statement is the same as the target attack address, then analyze the historical attacks on the target attack address based on the target attack statement;

[0118] If the attack address to which the target attack statement belongs is different from the target attack address, a potential attack situation having a high correlation with the target attack address is determined according to the target attack statement.

[0119] Further, the module 201 is specifically used for:

[0120] Based on the preset conversion rules, the attack address in each target log is converted to obtain an attack alias, and the victim address in each target log is converted to obtain a victim alias;

[0121] The log recording time, attack address, victim address, security device type, hit rule and hit data corresponding to each target log are extracted, and the target log data set is generated by combining the attack alias and victim alias.

[0122] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. Figure 3 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0123] like Figure 3 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0124] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0125] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a log processing method.

[0126] In some embodiments, the log processing method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the log processing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the log processing method in any other appropriate manner (e.g., by means of firmware).

[0127] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0128] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0129] In the context of the present invention, a computer readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer readable storage medium may be a machine readable signal medium. A more specific example of a machine readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0130] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0131] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0132] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0133] In one embodiment, the embodiment of the present invention further includes a computer program product, the computer program product includes a computer program, and the computer program implements the log processing method of any embodiment of the present invention when executed by a processor.

[0134] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0135] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0136] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A log processing method, It is characterized in that Executed by the government server, including: If a request for correlation analysis of target logs of multiple types of security devices is detected, the target logs are converted and extracted to obtain a target log data set; Determine the attack data set and victim data set corresponding to the target log data set, and vectorize the attack data set to obtain an attack log vector, and vectorize the victim data set to obtain a victim log vector; The attack data set is clustered according to the attack log vector, the victim data set is clustered according to the victim log vector, and based on the clustering results, attack correlation analysis and victim correlation analysis are performed on the target log.

2. The method according to claim 1, It is characterized in that Determine the attack dataset and victim dataset corresponding to the target log dataset, including: According to the record of the attack address in each target log in the target log data set, the target log data set is screened to obtain the attack data set; According to the record status of the victim address in each target log in the target log data set, the target log data set is screened to obtain the victim data set.

3. The method according to claim 1, It is characterized in that The attack data set is vectorized to obtain the attack log vector, including: Generate attack statements according to the attack alias, security device type and hit rule corresponding to each attack address in the attack data set; The attack statement is processed to obtain a coding vector, and the hit data corresponding to each attack address is processed to obtain a matching vector; The encoding vector and the matching vector are concatenated to obtain the attack log vector corresponding to the attack data set.

4. The method according to claim 3, It is characterized in that The attack statement is processed to obtain the encoding vector, and the hit data corresponding to each attack address is processed to obtain the matching vector, including: Based on a preset insertion order, the hit rules corresponding to each attack address are inserted into the attack statement, and the attack statement after the hit rules are inserted is processed using a preset encoding model to obtain an encoding vector; The preset regular expression dictionary is used to match the hit data corresponding to each attack address to obtain a matching vector.

5. The method according to claim 1, It is characterized in that Clustering the attack data set according to the attack log vector, and clustering the victim data set according to the victim log vector, including: Cluster the attack data set according to the attack log vector, the category weight corresponding to each security device category, and the initial attack cluster center; The victim data set is clustered according to the victim log vector, the category weight corresponding to each security device category and the initial victim cluster center.

6. The method according to claim 1, It is characterized in that Based on the clustering results, attack correlation analysis and victim correlation analysis are performed on the target logs, including: Generate an attack cluster set based on the attack statements and attack log vectors corresponding to each attack address, combined with the cluster labels in the cluster results, and generate a victim cluster set based on the victim statements and victim log vectors corresponding to each victim address, combined with the cluster labels in the cluster results; According to the target attack address and target victim address recorded in each target log, matching is performed in the target attack cluster set and the target victim cluster set to obtain a matching result; According to the matching results, the attack correlation analysis results and victim correlation analysis results of each target log are determined.

7. The method according to claim 6, It is characterized in that Based on the matching results, determine the attack correlation analysis results for each target log, including: According to the matching result, determine the target cluster label corresponding to the target log, and according to the target cluster label, determine the target attack statement corresponding to the target cluster label in the target attack cluster set; If the attack address corresponding to the target attack statement is the same as the target attack address, then analyze the historical attacks on the target attack address based on the target attack statement; If the attack address to which the target attack statement belongs is different from the target attack address, a potential attack situation having a high correlation with the target attack address is determined according to the target attack statement.

8. The method according to claim 1, It is characterized in that The target log is converted and extracted to obtain the target log data set, including: Based on the preset conversion rules, the attack address in each target log is converted to obtain an attack alias, and the victim address in each target log is converted to obtain a victim alias; The log recording time, attack address, victim address, security device type, hit rule and hit data corresponding to each target log are extracted, and the target log data set is generated by combining the attack alias and victim alias.

9. A log processing device, It is characterized in that include: The obtaining module is used to convert and extract the target logs to obtain a target log data set if a request for correlation analysis of target logs of multiple types of security devices is detected; A vectorization module is used to determine the attack data set and the victim data set corresponding to the target log data set, and to vectorize the attack data set to obtain the attack log vector, and to vectorize the victim data set to obtain the victim log vector; The analysis module is used to cluster the attack data set according to the attack log vector, cluster the victim data set according to the victim log vector, and perform attack correlation analysis and victim correlation analysis on the target log according to the clustering result.

10. An electronic device, It is characterized in that The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the log processing method according to any one of claims 1 to 8.

11. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the log processing method according to any one of claims 1 to 8 when executed.

12. A computer program product, It is characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program implements the log processing method according to any one of claims 1 to 8.

Citation Information

Patent Citations

  • Mining method and device of attacking organization

    CN106375331A

  • Log clustering method, device and equipment and storage medium

    CN111159413A

  • Security event mining method and device, storage medium and electronic equipment

    CN112559595A

  • Alarm information generation method and device, equipment and storage medium

    CN115242608A

  • Web hacking responses through real time web log collection

    KR1020070072835A