A three-layer association method based on event template similarity recommendation

CN114357439BActive Publication Date: 2024-11-22QUANZHI TECH (HANGZHOU) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210015416.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-07
Publication Date
2024-11-22
Estimated Expiration
2042-01-07

AI Technical Summary

Technical Problem

在现实场景下,由于事件总数非常巨大,实时分析应用事件和数据库事件的关联计算量非常大,而且传统的关联算法准确率并不高

Benefits of technology

[0021] 1. There are existing relevant patents, inventions and their key algorithms that mainly implement three-layer correlation around the event perspective. The method proposed by the present invention performs three-layer correlation between database logs and application logs around templates, greatly improving timeliness and accuracy;

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114357439B_ABST
    Figure CN114357439B_ABST
Patent Text Reader

Abstract

The present invention discloses a three-layer association method based on event template similarity recommendation, and specifically relates to the field of big data security technology. The present invention performs three-layer association of database logs and application logs around templates through the proposed method, greatly improving timeliness and accuracy. The present invention first proposes to apply the DBSCAN clustering algorithm to URL request and response parameter aggregation, which helps to improve the accuracy of the entire algorithm. The recommended collaborative filtering algorithm is applied to the field of three-layer association technology, and word2vec similarity matrix matching is proposed to realize the three-layer association of application log events and database log templates, greatly improving the association accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of big data security technology, and in particular to a three-layer association method based on event template similarity recommendation. Background Art

[0002] With the development of network technology, information security issues have received more and more attention. At present, in audit products, the general demand is to detect which applications the data in the tables in the database flows to, so as to support risk discovery, log tracing, sensitive leakage flow detection and other security needs.

[0003] Currently, audit products usually perform three-layer correlation through the request or response parameters of database log events and application log events. In real-time analysis of the correlation between application events and database events requires a lot of calculations, and the accuracy of traditional correlation algorithms is not high. Summary of the invention

[0004] To this end, an embodiment of the present invention provides a three-layer association method based on event template similarity recommendation. The proposed method performs three-layer association of database logs and application logs around the template, greatly improving timeliness and accuracy. It is the first to propose the application of the DBSCAN clustering algorithm in the aggregation of URL request and response parameters, which helps to improve the accuracy of the entire algorithm. The recommended collaborative filtering algorithm is applied in the field of three-layer association technology, and word2vec similarity matrix matching is proposed to realize the three-layer association of application log events and database log templates, greatly improving the association accuracy.

[0005] In order to achieve the above-mentioned purpose, the embodiment of the present invention provides the following technical solution: a three-layer association method based on event template similarity recommendation, characterized in that: the specific flow is as follows:

[0006] Step 1: Filter the SQL templates that need to be analyzed;

[0007] Step 2: Filter all SQL events within a period of time for a SQL template;

[0008] Step 3: Aggregate the request and response parameters of all sql events of the sql template into the collections sqlTemplateReqs and sqlTemplateRsps and remove duplicates respectively;

[0009] Step 4: For all SQL events of the SQL template, filter all URL events within a period of time before each SQL event and remove duplicates;

[0010] Step 5: All URL events are clustered into apiUrlReqs and apiUrlRsps using the DBSCAN clustering algorithm according to the request and response parameters of apiUrl, and duplicates are removed;

[0011] Step 6: Based on the idea of ​​recommendation system, apiUrlReqs and sqlTemplateReqs, apiUrlRsps and sqlTemplateRsps are converted into numerical vectors respectively based on the results obtained in the previous steps, and the correlation is calculated using collaborative filtering algorithm, and the appropriate apiUrl is recommended to the corresponding sql template;

[0012] Step 7: Store the association results in the database and update them every month;

[0013] Step 8: Repeat steps 1-7 to loop the SQL template.

[0014] Furthermore, the sql template in step one is an abstraction of the audit product to the sql event, and the user can input the sql template to be analyzed. These sql templates are generally the ones that users are more concerned about, have a greater impact on the business, and are of higher importance.

[0015] Furthermore, in the step 2, one sql template corresponds to multiple sql events. The number of sql events varies in different scenarios. In actual implementation, sql events of 1 hour, 1 day or 1 week are screened according to the situation.

[0016] Furthermore, in the step 3, sqlTemplateReqs and sqlTemplateRsps respectively store the request and response parameters of all sql events of the sql template. Since there are a large number of duplicate response parameters of many sql events, they must be deduplicated first to avoid affecting the subsequent algorithm flow.

[0017] Furthermore, the apiUrl in step 5 is an abstraction of the url. The DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm is a density-based clustering algorithm. DBSCAN defines a cluster as the largest set of density-connected points. It can divide areas with sufficiently high density into clusters and can find clusters of arbitrary shapes in the spatial database of noise. Among all the url events we extracted, there may be multiple apiUrls. We need to use DBSCAN to cluster the request and response parameters according to different apiUrls.

[0018] Furthermore, the core of the collaborative filtering in step six is ​​the similarity calculation between targets. The matrix concept can be used to calculate the similarity. The sqlTemplateRsps is constructed into a behavior Rsps matrix. An element of the matrix represents the Rsp parameter value of a certain sqlTemplate, wherein the row vector represents the relevant vector of a certain sqlTemplate for all apiUrls, and the column vector represents the parameter value of sqlTemplateRsps. The word2vec algorithm can be used to convert apiUrlRsps and sqlTemplateRsps into row vectors and column vectors. With the row vectors and column vectors, the similarity between apiUrlRsps and sqlTemplateRsps can be calculated. Specifically, the similarity between the row vectors is the similarity between apiUrlRsps and sqlTemplateRsps, and the similarity between the column vectors is the similarity between apiUrlRsps. Similarly, the Reqs behavior matrix is ​​constructed, and the similarity between apiUrlReqs and sqlTemplateReqs is calculated. The two similarities are added to recommend the most suitable apiUrl to the corresponding sql template.

[0019] Furthermore, the number of URL events in different scenarios in step 4 is different. In actual implementation, all URL events within 0.5s or 1s are screened according to the situation.

[0020] The embodiments of the present invention have the following advantages:

[0021] 1. Existing related patents, inventions and their key algorithms mainly implement three-layer association from the perspective of events. The method proposed in this invention performs three-layer association of database logs and application logs based on templates, which greatly improves timeliness and accuracy.

[0022] 2. The first to propose the application of DBSCAN clustering algorithm in URL request and response parameter aggregation, which helps to improve the accuracy of the entire algorithm.

[0023] 3. The present invention recommends the application of collaborative filtering algorithm in the field of three-layer association technology, and proposes word2vec similarity matrix matching to realize three-layer association of application log events and database log templates, greatly improving the association accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A dbscan algorithm diagram is provided for the present invention.

[0025] Figure 2 The behavior matrix diagram provided by the present invention.

[0026] Figure 3 This is a schematic diagram of the word2vec algorithm provided by the present invention.

[0027] Figure 4 This is a key flow chart of the algorithm provided by the present invention. DETAILED DESCRIPTION

[0028] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] A three-layer association method based on event template similarity recommendation, the specific flow is as follows:

[0030] Step 1: Filter the SQL templates that need to be analyzed;

[0031] Step 2: Filter all SQL events within a period of time for a SQL template;

[0032] Step 3: Aggregate the request and response parameters of all sql events of the sql template into the collections sqlTemplateReqs and sqlTemplateRsps and remove duplicates respectively;

[0033] Step 4: For all SQL events of the SQL template, filter all URL events within a period of time before each SQL event and remove duplicates. It should be noted that the number of URL events varies in different scenarios. In actual implementation, we can filter all URL events within 0.5s or 1s according to the situation. Because many URLs are repeated, deduplication is required to improve computing efficiency.

[0034] Step 5: All URL events are clustered into apiUrlReqs and apiUrlRsps using the DBSCAN clustering algorithm according to the request and response parameters of apiUrl, and duplicates are removed;

[0035] Step 6: Based on the idea of ​​recommendation system, apiUrlReqs and sqlTemplateReqs, apiUrlRsps and sqlTemplateRsps are converted into numerical vectors respectively based on the results obtained in the previous steps, and the correlation is calculated using collaborative filtering algorithm, and the appropriate apiUrl is recommended to the corresponding sql template;

[0036] Step 7: Store the association results in the database and update them every month;

[0037] Step 8: Repeat steps 1-7 to loop the SQL template.

[0038] Furthermore, the sql template in step 1 is an abstraction of the audit product to the sql event. For example, select name from appName where nameId=1 is a sql event, and the corresponding sql template is selectname from appName where nameId=?. The user can enter the sql template to be analyzed. These sql templates are generally the ones that users are more concerned about, have a greater impact on the business, and are of higher importance.

[0039] Furthermore, in the step 2, one SQL template corresponds to multiple SQL events. Capturing multiple SQL events of the template helps us detect the flow of sensitive data. It should be noted that the number of SQL events is different in different scenarios. In actual implementation, we can filter SQL events of 1 hour, 1 day or 1 week according to the situation.

[0040] Furthermore, in the step 3, sqlTemplateReqs and sqlTemplateRsps respectively store the request and response parameters of all sql events of the sql template. Since there are a large number of duplicate response parameters of many sql events, they must be deduplicated first to avoid affecting the subsequent algorithm flow.

[0041] Furthermore, in step 5, apiUrl is an abstraction of url. For example, for url: http: / / 192.168.0.1 / getName? name=1, http: / / 192.168.0.1 / getName is an apiUrl. DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm is a density-based clustering algorithm. DBSCAN defines a cluster as the largest set of density-connected points. It can divide areas with sufficiently high density into clusters and find clusters of any shape in the spatial database of noise. In all the url events we extract, there may be multiple apiUrls. We need to use DBSCAN to cluster the request and response parameters according to different apiUrls.

[0042] Furthermore, the core of collaborative filtering in step 6 is the calculation of similarity between targets. We can use matrix thinking to calculate similarity. We construct the behavior Rsps matrix (as shown below) using sqlTemplateRsps Figure 2), an element of the matrix represents the Rsp parameter value of a sqlTemplate, where the row vector represents the relevant vector of a sqlTemplate for all apiUrls, and the column vector represents the parameter value of sqlTemplateRsps. We can use the word2vec algorithm (see below) Figure 3 )Convert apiUrlRsps and sqlTemplateRsps into row vectors and column vectors. With row vectors and column vectors, we can calculate the similarity between apiUrlRsps and sqlTemplateRsps. Specifically, the similarity between row vectors is the similarity between apiUrlRsps and sqlTemplateRsps, and the similarity between column vectors is the similarity between apiUrlRsps. Similarly, construct the Reqs behavior matrix, calculate the similarity between apiUrlReqs and sqlTemplateReqs, add the two similarities, and recommend the most suitable apiUrl to the corresponding sql template.

[0043] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A three-layer association method based on event template similarity recommendation, characterized in that: Step 1: Filter the SQL templates that need to be analyzed; Step 2: Filter all SQL events within a period of time for a SQL template; Step 3: Aggregate the request and response parameters of all sql events of the sql template into the collections sqlTemplateReqs and sqlTemplateRsps and remove duplicates respectively; Step 4: For all SQL events of the SQL template, filter all URL events within a period of time before each SQL event and remove duplicates; Step 5: All URL events are clustered into apiUrlReqs and apiUrlRsps using the DBSCAN clustering algorithm according to the request and response parameters of apiUrl, and then duplicates are removed; Step 6: Drawing on the idea of ​​the recommendation system, based on the results obtained in the previous steps, convert apiUrlReqs and sqlTemplateReqs, apiUrlRsps and sqlTemplateRsps into numerical vectors respectively, and use the collaborative filtering algorithm to calculate the correlation, and recommend the appropriate apiUrl to the corresponding sql template; the core of collaborative filtering is the similarity calculation between targets, and the matrix idea is used to calculate the similarity, and sqlTemplateRsps is constructed into a behavior Rsps matrix, where an element of the Rsps matrix represents the Rsp parameter value of a certain sqlTemplate, where the row vector represents the correlation vector of a certain sqlTemplate for all apiUrls, and the column vector represents the correlation vector of sqlTemplate for all apiUrls. teRsps parameter value, use word2vec algorithm to convert apiUrlRsps and sqlTemplateRsps into row vector and column vector, with row vector and column vector, calculate the similarity between apiUrlRsps and sqlTemplateRsps, specifically, the similarity between row vectors is the similarity between apiUrlRsps and sqlTemplateRsps, the similarity between column vectors is the similarity between apiUrlRsps, similarly build Reqs behavior matrix, and calculate the similarity between apiUrlReqs and sqlTemplateReqs, add the two similarities, and recommend the most suitable apiUrl to the corresponding sql template; Step 7: Store the association results in the database and update them every month; Step 8: Repeat steps 1-7 to loop the SQL template.

2. The three-layer association method based on event template similarity recommendation according to claim 1 is characterized in that: The sql template in step 1 is an abstraction of sql events by the audit product.

3. The three-layer association method based on event template similarity recommendation according to claim 1 is characterized in that: In the step 2, one SQL template corresponds to multiple SQL events. The number of SQL events varies in different scenarios. In actual implementation, SQL events of 1 hour, 1 day or 1 week are screened according to the situation.

4. The three-layer association method based on event template similarity recommendation according to claim 1 is characterized in that: In the step 3, sqlTemplateReqs and sqlTemplateRsps respectively store the request and response parameters of all sql events of the sql template.

5. The three-layer association method based on event template similarity recommendation according to claim 1 is characterized in that: The apiUrl in step 5 is an abstraction of the url. The DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm is a density-based clustering algorithm. DBSCAN defines a cluster as the largest set of density-connected points. It can divide areas with sufficiently high density into clusters and can find clusters of any shape in the spatial database of noise. Among all the url events we extracted, there may be multiple apiUrls. We need to use DBSCAN to cluster the request and response parameters according to different apiUrls.

6. The three-layer association method based on event template similarity recommendation according to claim 1 is characterized in that: In step 4, the number of URL events in different scenarios is different. In actual implementation, all URL events within 0.5s or 1s are filtered according to the situation.