A knowledge graph-based method for identifying abnormal behavior in web logs

Abstract

To enhance the ability to analyze network DNS server logs, this invention innovatively proposes constructing a knowledge graph for the domain name resolution system, integrating multiple technologies. First, an automated crawler model combining aiohttp and dig technologies was designed based on principles of domain name resolution, authoritative name servers, alias resolution, and autonomous systems, and a corresponding domain knowledge base was constructed. Second, based on this knowledge base, a prototype knowledge graph for the domain name resolution system was designed and the knowledge graph was completed, with nearly 5 million nodes. Finally, this knowledge graph was used to assist in the construction of a web log abnormal behavior recognition model. Experimental and practical application feedback shows that this knowledge graph plays a crucial role in the detection and analysis of abnormal behavior in network server logs, improving the recognition rate of the web log abnormal behavior model.

Description

Technical Field

[0001] This invention relates to a method for identifying abnormal behavior in WebI logs based on knowledge graphs, belonging to the field of knowledge graph information utilization technology. Background Technology

[0002] On May 26, 2021, the National Computer Network Emergency Response Technical Team / Coordination Center of China (CNCERT / CC) released the "Overview of my country's Internet Cybersecurity Situation in 2020," stating that the key aspects of my country's internet cybersecurity in 2020 were APT attacks, data security, security vulnerabilities, malicious programs, online fraud prevention, and industrial control system security. Therefore, cybersecurity is a common challenge faced in the internet age.

[0003] Generally, addressing the root cause is the best approach. To detect abnormal internet behavior, one can start with the domain name resolution process. Domain name resolution is the process of converting a domain name into an IP address through a dedicated Domain Name System (DNS) server. Each resolution generates four main types of log data, including DNS, URL, IP, and SSL logs. These logs may be filled with a large amount of machine activity, such as web crawlers, port scans, brute-force domain name attacks, and connectivity tests—in other words, abnormal behavior.

[0004] The applicant's research revealed that current methods for identifying abnormal behavior largely rely on researchers' limited experience and single-function log analysis tools. The lack of a comprehensive and effective knowledge system hinders the efficient log analysis capabilities of individual researchers, resulting in anomaly identification failing to meet actual business needs. Furthermore, log analysis suffers from challenges such as the omission of POST request data, unreliable status code responses, and attackers using multiple IP proxies. Simply relying on existing log fields is insufficient to identify more abnormal behavior log entries. Therefore, constructing a comprehensive and effective knowledge system to improve the anomaly identification rate is a pressing issue that needs to be addressed.

[0005] Network DNS server log analysis is currently a crucial aspect of network information security. Regarding log analysis, the main methods used domestically and internationally for network server log analysis are as follows: Ji Xing et al., in "DNS Query Anomaly Detection Algorithm Based on Log Information," proposed a DNS query anomaly detection algorithm based on log information. This algorithm utilizes cluster analysis and calculates the credibility of each source IP to detect anomalous source IPs. However, this method uses an unsupervised algorithm, and the computational cost increases as the dataset grows. Nadler A et al., in "Detection of malicious and low throughput data exfiltration over the DNS protocol," proposed a method based on DNS tunneling and low throughput data leakage. This method extracts and analyzes DNS log features to detect anomalous behavior of malware leaking data through covert channels. Niu W et al., in "Identifying APT malware domain based on mobile DNSlogging," proposed an effective method for high-precision detection of APT malware C&C domains by analyzing DNS logs. This method uses the Global Anomaly Forest (GAF) anomaly detection algorithm to identify malware C&C domains. Chen et al., in "Detection of DNSDDoS attacks with random forest algorithm on...", proposed a method based on DNS logs for high-precision detection of APT malware C&C domains. The paper "Spark" proposes a random forest algorithm model based on the Spark platform to detect Distributed Denial-of-Service (DDoS) attacks in DNS logs; Song Jiaming et al., in "Network Anomaly Behavior Analysis Based on Artificial Intelligence," propose a data feature modeling method based on Web logs. This method constructs single HTTP request features and attack category statistical features based on the implicit statistical features of the HTTP request fields in Web logs, significantly improving the ability to identify abnormal behavior in Web logs; Wang Qi et al., in "DNS Tunnel Detection Based on Log Statistical Features," propose a DNS tunnel detection method based on log statistical features, incorporating cache hit rate, etc. Multidimensional statistical features were used to train a DNS tunnel detection model based on the random forest algorithm. Zhang Wenqi et al., in "Visual Analysis of Abnormal Behavior Based on Multidimensional Time-Series Logs," proposed a log visualization system based on the multidimensionality and temporal nature of enterprise logs. By designing multiple visualization views, an interactive visual analysis system was implemented, improving decision-makers' ability to detect abnormal behavior. Qin Huidong et al., in "Research on DNS Anomaly Detection Technology Based on Multi-Features," proposed a local anomaly factor detection algorithm based on multidimensional time-series features for DNS source IP anomaly detection. Based on this, they proposed a domain name anomaly analysis method based on multi-features to achieve more accurate DNS anomaly identification.

[0006] Analysis of existing technical literature reveals that current identification methods primarily rely on statistics, clustering, and random forests to effectively identify single or category-specific abnormal behaviors, mainly referring to malicious software and other offensive behaviors. However, research on identifying machine access behavior in web logs is limited. Furthermore, the applicant's research indicates that relying on existing feature attributes in web logs can only identify a small range of machine access behaviors.

[0007] Knowledge graphs are essentially large-scale semantic networks that can structure heterogeneous knowledge within a domain and connect this structured knowledge, thereby reducing the analytical cost of tasks. Therefore, since Google proposed the concept of knowledge graphs in 2012, their construction has flourished across various fields. It can be observed that the application of knowledge graphs simplifies tasks; however, in the subfield of log analysis within network information security, there is currently no complete and effective knowledge graph to support researchers in completing log analysis tasks. Although the literature by Yan Zhihao et al., "CDN Domain Name Recognition Technology Based on Domain Name System Knowledge Graph," constructed a Domain Name System knowledge graph based on the top 1 million domains ranked by Alexa, the role of their constructed domain name system knowledge graph is limited to the identification of CDN domains, and its contribution to overall log analysis is minimal. Summary of the Invention

[0008] To address the problems existing in the prior art, this invention provides a method for identifying abnormal behavior in web logs based on knowledge graphs. By utilizing the characteristic of knowledge graphs to connect different types of information, the feature attributes of web logs are increased.

[0009] To achieve the above-mentioned technical objectives, the present invention adopts the following technical solution.

[0010] A method for identifying abnormal behavior in web logs based on knowledge graphs includes the following steps:

[0011] S1 Multi-Source Data Acquisition and Construction of Corresponding Domain Knowledge Base

[0012] An automated crawler model based on aiohttp and dig technologies is designed using methods such as domain name resolution, authoritative domain name servers, alias resolution, and autonomous systems, and a corresponding domain knowledge base is constructed.

[0013] S2 Knowledge Graph Prototype Design and Knowledge Graph Construction

[0014] Based on S1, a domain knowledge base was designed to create a prototype knowledge graph for the domain name resolution system, and the knowledge graph was constructed.

[0015] Construction of an S3 web log abnormal behavior identification model

[0016] The knowledge graph built on S2 helps in the construction of a web log abnormal behavior recognition model.

[0017] Furthermore, in the knowledge graph-based web log abnormal behavior identification method of the present invention, in S1 multi-source data collection and corresponding domain knowledge base construction, firstly, the data types of various knowledge are established based on domain name resolution, authoritative domain name servers, alias resolution, and autonomous system methods; then, the data sources of various types of data are determined, and the table structure of various types of knowledge is determined according to the data sources, while a specific collection scheme is designed for each type of knowledge; finally, the collected web page data is processed and filtered according to the table rules designed in the knowledge base, unstructured data is converted into structured data, and stored in batches in the corresponding knowledge base.

[0018] 1. Furthermore, in the knowledge graph-based web log abnormal behavior identification method of the present invention, S1 includes S11 website information collection and website classification system construction;

[0019] S12 domain name information collection and domain name information knowledge base construction;

[0020] S13 IP Information Data Collection and IP Information Knowledge Base Construction;

[0021] S14 Authoritative Domain Name Server Information Collection and Authoritative Domain Name Server Knowledge Base Construction;

[0022] S15 ASN Information Collection and ASN Information Knowledge Base Construction;

[0023] S11 includes the collection of basic attribute information of the S111 website.

[0024] The website's title, description, and keywords were collected from the StuffGate website. The collection process was as follows:

[0025] S1111 sets the number of executable threads based on the number of domain names;

[0026] S1112 sets the request URL based on the domain name;

[0027] S1113 configures the IP proxy pool and sets a random sleep time;

[0028] S1114 randomly extracts UA information from the UserAgent data pool to simulate a browser requesting a webpage;

[0029] S1115 analyzes webpage data and formulates resolution strategies for various domain attribute parameters;

[0030] S1116 encapsulates the obtained domain name attribute information;

[0031] S1117 loops through S1112-S1116, using the executemany method to process 1k domain names and then insert them into the database;

[0032] For the Alexa ranking acquisition process, an asynchronous crawler framework based on aiohttp is used to introduce an IP proxy pool, forge login passwords to obtain cookie information, and set random sleep time.

[0033] The process for collecting information about the country to which the website belongs is as follows:

[0034] S111A sets the length of the domain name processed by the thread (number of domain names / number of threads) based on the number of domain names and the size of the number of executable threads, thereby reducing the data crawling time;

[0035] S111B uses web crawling technology to obtain WHOIS information from Chinaz.com and Alibaba.

[0036] S111C compares the country information of the two websites. If they match, the information is stored in the result set and the next website is processed. Otherwise, S111D is executed.

[0037] S111D retrieves the main geographical location of websites accessed from alexa.com, corrects the results, stores them in the result set, and then processes the next website.

[0038] It also includes the construction of the S112 website classification system, which includes,

[0039] S1121 Data Preprocessing: The Baidu Translate API interface is used to batch convert the website title, description, and keywords in the website's basic information attributes;

[0040] S1122 Data Labeling: Website type labeling was performed on a batch of websites using manual methods;

[0041] S1123 Word segmentation processing:

[0042] Jieba word segmentation technology is used to segment the website's title, description information, and keyword-based sentences to obtain segmented text;

[0043] S1124 Model Training: The segmented and labeled text is used as the training set for training. First, the text is represented in vector space to obtain the vector space representation of each connected text. Then, a logistic regression model is used to classify the text vectors based on the feature representation.

[0044] S1125 Website Tag Prediction: Unlabeled text classification using a trained model;

[0045] Based on the processing of S111 and S112, the construction of the website information knowledge base is completed;

[0046] Among them, S12 domain name information collection and domain name information knowledge base construction includes,

[0047] S121 sets the number of executable threads based on the number of domain names;

[0048] S122 sets the request URL based on the domain name;

[0049] S123 sets up an IP proxy pool and sets a random sleep time;

[0050] S124 randomly extracts UA information from the UserAgent data pool and simulates a browser requesting a webpage;

[0051] S125 analyzes webpage data and formulates resolution strategies for various domain attribute parameters;

[0052] S126 encapsulates the obtained domain name attribute information;

[0053] S127 loops through S122-S126, using the executemany method to insert 1k domain names into the database after processing them;

[0054] At the same time, based on this batch of domain name data and according to the principle of domain name resolution, a table relating domain names to IP addresses was constructed; finally, the domain name information knowledge base was completed.

[0055] S13 IP information data collection and IP information knowledge base construction are underway.

[0056] Based on the domain name information knowledge base and in accordance with the principle of domain name resolution, the dig command is used to obtain the set of IP addresses mapped to the domain name. The TCP protocol is introduced during the dig query process to de-DNS pollution and obtain the set of IP addresses.

[0057] The process for obtaining IP attribute information is as follows:

[0058] S131 sets the number of executable threads based on the number of IPs;

[0059] S132 sets the request URL based on the IP address;

[0060] S133 configures the IP proxy pool and sets a random sleep time;

[0061] S134 randomly extracts UA information from the UserAgent data pool and simulates a browser requesting a webpage;

[0062] S135 analyzes webpage data and formulates resolution strategies for various domain attribute parameters;

[0063] S136 encapsulates the obtained IP attribute information;

[0064] S137 loops through S132-S136, using the executemany method to process 1k IP sets and then insert them into the database.

[0065] Finally, the IP information knowledge base was completed;

[0066] S14 authoritative domain name server information collection and authoritative domain name server knowledge base construction are underway.

[0067] The authoritative domain name servers obtained from the domain name information knowledge base are collected, and the relevant IP information of the authoritative domain name servers is obtained by using the dig command.

[0068] S15 ASN information collection and ASN information knowledge base construction are underway.

[0069] ASN information collection includes,

[0070] ASN Collection

[0071] Step 1: Set the request URL using 239 region names as units;

[0072] Step 2: Design a data collection strategy based on the website layout of whois.ipip.net;

[0073] Step 3: Batch input the collected ASN information from various regions into the database.

[0074] ASN Attribute Information Collection

[0075] Step 1: Set the request URL based on the ASN;

[0076] Step 2: Analyze the page structure and design a data collection strategy;

[0077] Step 3: Encapsulate the acquired ASN information;

[0078] Step 4 uses the executemany method to complete the data insertion operation;

[0079] CIDR attribute information collection

[0080] Step1 performs drill-down analysis based on the collected ASNs to obtain the CIDRs corresponding to the ASNs;

[0081] Step2 sets the request URL based on the ASN;

[0082] Step 3: Analyze the style of the page where the corresponding CIDR is located and design a data collection strategy.

[0083] Furthermore, in the present invention's method for identifying abnormal behavior in web logs based on knowledge graphs, the S2 knowledge graph prototype design and knowledge graph construction process includes, in the knowledge graph prototype design...

[0084] Based on the principle of domain name resolution, two major entities, domain name and IP address, are constructed, and the parent-child relationship between domain name and subdomain and the carrying relationship between domain name and IP address are established using these entities.

[0085] The Name Server principle is used to establish the NS entity, and to establish the service relationship between the domain name and the NS, and the address relationship between the IP and the NS;

[0086] Based on WHOIS query technology and the principle of autonomous systems, two major entities, ASN and country, are established, and two relationships are established: the affiliation relationship between IP and ASN and the belonging relationship between ASN and country.

[0087] The mapping relationship between domain names and websites was established based on the principles of website composition.

[0088] Furthermore, in the knowledge graph-based web log abnormal behavior identification method of the present invention, in the S2 knowledge graph prototype design and knowledge graph construction, the entities in the knowledge graph prototype design include domain name, website, IP, NS, CNAME, ASN, and country.

[0089] Relationships include: parent-child relationship, carrier relationship, service relationship, address relationship, membership relationship, belonging relationship, alias relationship, and mapping relationship;

[0090] The ternary types include: domain1, parent-child relationship; domain2, domain, carrier relationship; IP, domain, service relationship; NS, IP, address relationship; NS, IP, membership relationship; ASN, ASN, belonging relationship; country, domain, alias relationship; CNAME, domain, mapping relationship; website.

[0091] Furthermore, the present invention provides a knowledge graph-based method for identifying abnormal behavior in web logs. The construction of the S3 web log abnormal behavior identification model includes,

[0092] For DNS logs, IP logs, URL logs, and SSL logs generated by network servers, non-human-caused abnormal logs are extracted using machine learning and manual analysis methods. The characteristics of abnormal logs are summarized, and abnormal behaviors are defined and identified based on these behavioral characteristics.

[0093] Furthermore, in the knowledge graph-based web log abnormal behavior identification method of the present invention, the construction of the web log abnormal behavior identification model (S3) is assisted by the knowledge graph constructed in (S2) to complete the construction of the web log abnormal behavior identification model.

[0094] During data preprocessing, the source and destination IP addresses in the log records are supplemented by the country and city attribute information in the IP nodes of the knowledge graph. When the geographical location is not one of the two opposing internal and external networks, the ASN information corresponding to the IP is queried according to the affiliation relationship in the graph to obtain the country geographic information.

[0095] Furthermore, in the knowledge graph-based web log abnormal behavior identification method of the present invention, the construction of the web log abnormal behavior identification model (S3) is assisted by the knowledge graph constructed in (S2) to complete the construction of the web log abnormal behavior identification model.

[0096] When segmenting users, user groups are based on the IP geographic location, IP segmentation attributes, and website category information of the website node in the knowledge graph.

[0097] Furthermore, in the knowledge graph-based web log abnormal behavior identification method of the present invention, the construction of the web log abnormal behavior identification model (S3) is assisted by the knowledge graph constructed in (S2) to complete the construction of the web log abnormal behavior identification model.

[0098] We introduce a strategy of using whitelists of crawler IPs from major internet companies in the IP nodes of the graph for data crawling.

[0099] Furthermore, in the knowledge graph-based web log abnormal behavior identification method of the present invention, the construction of the web log abnormal behavior identification model (S3) is assisted by the knowledge graph constructed in (S2) to complete the construction of the web log abnormal behavior identification model.

[0100] When performing brute-force DNS resolution, the analysis formula for using a domain name for brute-force resolution is as follows:

[0101]

[0102] Cnt(domain2)-k>0

[0103] Where i represents the domain level, Count() represents the number of times a certain level of domain appears, domain2 represents the second-level domain of the domain, and k represents the threshold;

[0104] The process involves first using a crawler behavior recognition strategy and feature rules to filter the data, then introducing parent-child relationships from a knowledge graph, querying all subdomain information by domain name, setting the threshold k directly based on the number of subdomains, and directly performing domain name matching to determine whether the subdomain exists in the graph.

[0105] Furthermore, in the knowledge graph-based web log abnormal behavior identification method of the present invention, the construction of the web log abnormal behavior identification model (S3) is assisted by the knowledge graph constructed in (S2) to complete the construction of the web log abnormal behavior identification model.

[0106] By introducing address relationships from a knowledge graph, we can query authoritative name server information through IP addresses to determine whether a target IP belongs to a root name server.

[0107] The present invention adopts the above-described solution and achieves the following technical effects.

[0108] This invention aims to improve the analysis and identification capabilities of abnormal behavior in logs by constructing a complete and effective knowledge graph for domain name resolution systems. This knowledge graph uses the top 100w domain data from alexa.com, a website with relatively complete global website ranking data, as its research object. Through an automated crawler model combining aiohttp and dig technologies, data was collected, establishing knowledge bases for domain names, IP addresses, websites, authoritative name servers, and ASNs. Based on this, a knowledge graph for domain name resolution systems was designed and constructed, improving the identification rate of abnormal behavior in web logs.

[0109] This invention designs an automated web crawler model based on aiohttp and dig technologies, which solves the problem of difficult multi-source data collection.

[0110] This invention constructs a knowledge graph of nearly 5 million characters for the domain name resolution system, providing web log researchers with a convenient knowledge retrieval system for domain names, IP addresses, etc.

[0111] This invention further constructs a web log abnormal behavior recognition model based on knowledge graph, which improves the recognition rate of abnormal behavior in web logs.

[0112] Experimental results show that the knowledge graph for the domain name resolution system adds feature attributes to web logs, reduces the difficulty of web log analysis, and improves the ability of the abnormal behavior identification model to analyze and identify web logs.

[0113] Furthermore, the construction of this knowledge graph is not only significant for detecting abnormal behaviors such as machine access in logs, but also for other work in the field of network information security, such as detecting malicious domains and APT attacks. Attached Figure Description

[0114] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0115] Figure 1 This is a schematic diagram illustrating the multi-source data acquisition and corresponding domain knowledge base construction, knowledge graph prototype design, and knowledge graph construction method of the present invention.

[0116] Figure 2 This is a prototype diagram of the knowledge graph of the present invention;

[0117] Figure 3 A diagram illustrating an abnormal behavior recognition model using the method of this invention and existing techniques;

[0118] Figure 4 This is a comparison chart of web crawler behavior recognition experiments using the method of this invention and existing technologies;

[0119] Figure 5 This is a comparative experimental diagram showing the brute-force domain name resolution behavior performed using the method of this invention and existing technologies.

[0120] Figure 6 This is a comparison chart of experiments using the method of this invention and existing methods for identifying DNS duplicate resolution behavior. Detailed Implementation

[0121] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0122] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0123] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0124] In the description of the embodiments of the present invention, it should be noted that the terms "first," "second," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0125] Specifically, this invention provides a method for identifying abnormal behavior in web logs based on knowledge graphs, including the following:

[0126] S1 Multi-Source Data Acquisition and Construction of Corresponding Domain Knowledge Base

[0127] An automated crawler model based on aiohttp and dig technologies was designed using methods such as domain name resolution, authoritative domain name servers, alias resolution, and autonomous systems, and a corresponding domain knowledge base was constructed.

[0128] S2 Knowledge Graph Prototype Design and Knowledge Graph Construction

[0129] Based on S1, a domain knowledge base was designed to create a prototype knowledge graph for the domain name resolution system, and the knowledge graph was constructed.

[0130] Construction of an S3 web log abnormal behavior identification model

[0131] The knowledge graph built on S2 helps in the construction of a web log abnormal behavior recognition model.

[0132] Specifically, this invention provides a method for identifying abnormal behavior in web logs based on knowledge graphs. In the S1 multi-source data collection and corresponding domain knowledge base construction process, firstly, the data types of various knowledge categories are established based on methods such as domain name resolution, authoritative domain name servers, alias resolution, and autonomous systems. Then, relevant knowledge research is conducted to determine the data sources for each type of data, and the table structure for each type of knowledge is determined based on the data sources. Simultaneously, a specific collection scheme is designed for each type of knowledge. Finally, the collected web page data is processed and filtered according to the table rules designed for the knowledge base, converting unstructured data into structured data, and then stored in batches in the corresponding knowledge base.

[0133] The multi-source data collection mentioned above is mainly based on entities in the knowledge graph prototype, which in this invention includes knowledge such as domain names, IP addresses, and websites. Multi-source data collection and knowledge base construction include the collection and construction of five knowledge bases: website information knowledge base, domain name information knowledge base, IP information knowledge base, authoritative domain name server knowledge base, and ASN information knowledge base.

[0134] S11 Website Information Collection and Website Classification System Construction

[0135] The website information collection method of this invention is based on the top 100w domains provided on the alexa.com website, and also analyzes the websites of Chinaz.com, StuffGate, Alibaba Cloud domain name service, and the Alexa official website to collect basic website attribute information and construct a website classification system.

[0136] The collection of basic attribute information for the S111 website includes the following:

[0137] The task involves collecting basic attribute information about a website, including but not limited to the website's title, description, keywords, Alexa ranking, and country of origin. Among these, obtaining the Alexa ranking and country of origin are the most challenging tasks.

[0138] The website's title, description, and keywords were collected from the StuffGate website. The specific collection process is as follows:

[0139] S1111 sets the number of executable threads based on the number of domain names;

[0140] S1112 sets the request URL based on the domain name;

[0141] S1113 configures the IP proxy pool and sets a random sleep time;

[0142] S1114 randomly extracts UA information from the UserAgent data pool to simulate a browser requesting a webpage;

[0143] S1115 analyzes webpage data and formulates resolution strategies for various domain attribute parameters;

[0144] S1116 encapsulates the obtained domain name attribute information;

[0145] S1117 loops through S1112-S1116, using the executemany method to process 1k domain names and then insert them into the database.

[0146] The Alexa ranking acquisition process presents challenges due to its robust verification mechanism, slow crawling speed, and susceptibility to abnormal interruptions. This invention addresses these issues using the aiohttp asynchronous crawler framework by introducing an IP proxy pool, forging login passwords to obtain cookie information, and setting random sleep times.

[0147] The main issue with determining a website's country of origin is accuracy. The initial design relied on the country code top-level domain (ccTLD) and the geographic location data provided by Alexa to determine the website's country of origin. However, conflicts arose with ccTLDs; for example, Tuvalu's ccTLD is .tv, but this typically indicates video websites. Furthermore, the geographic location data provided by alexa.com only indicates the top-ranked countries for website visits and cannot directly indicate the website's country of origin, serving as a supplementary factor. Therefore, this invention uses the WHOIS results from Chinaz.com and Alibaba.com as a benchmark, employing Python web scraping to obtain the website's country of origin information. When inconsistencies exist between the two query results, the country of origin information provided by alexa.com is used to correct them. The specific process for collecting website country of origin information is as follows:

[0148] S111A sets the length of the domain name processed by the thread (number of domain names / number of threads) based on the number of domain names and the size of the number of executable threads, thereby reducing the data crawling time;

[0149] S111B uses web crawling technology to obtain WHOIS information from Chinaz.com and Alibaba.

[0150] S111C compares the country information of the two websites. If they match, the information is stored in the result set and the next website is processed. Otherwise, S111D is executed.

[0151] S111D retrieves the main geographical location of the website provided by alexa.com, corrects the results, stores them in the result set, and then processes the next website.

[0152] In this embodiment, after all domain names have been processed, they are compared with 10,000 websites randomly selected from the top 100,000 and manually labeled. The accuracy rate is 98.6%, and the result meets the needs of subsequent business.

[0153] S112 Website Classification System Construction

[0154] Currently, major classified websites have their own classification systems, such as Chinaz.com and 360. However, these systems have too much overlap and are not suitable for direct use. Furthermore, subsequent log analysis primarily focuses on websites in categories such as government and news media. Therefore, this invention aims to build a 30-category website tagging system for information security services, based on existing online classification tagging systems and the actual needs of log analysis. Examples of such categories include government organizations, news media, and social forums.

[0155] This invention employs a solution that primarily uses machine classification, supplemented by human classification, to design a website classification model based on a combination of TF-IDF and logistic regression algorithms. The specific implementation method is as follows:

[0156] S1121 Data Preprocessing: Using the Baidu Translate API, the website title, description, and keywords in the website's basic information attributes are converted in batches.

[0157] Since the website titles, descriptions, and keywords in the collected basic website attribute information are expressed in multiple languages, this invention uses the Baidu Translate API interface for batch conversion.

[0158] S1122 Data Labeling: A batch of websites were labeled with website types using manual methods.

[0159] Specifically, in this embodiment, a batch of websites were manually labeled to construct a training set of approximately 30,000 entries.

[0160] S1123 Word Segmentation Processing: Using jieba word segmentation technology, the website's title, description information, and keyword-based sentences are segmented to obtain segmented text;

[0161] S1124 Model Training: The labeled segmented text after word segmentation is used as the training set for training. First, the text is represented by a vector space to obtain the vector space representation of each connected text. Then, a logistic regression model is used to classify the text vectors based on the feature representation.

[0162] S1125 Website Tag Prediction: Using a trained model for unlabeled text classification.

[0163] Based on the processing of S111 and S112, the construction of the website information knowledge base is completed.

[0164] The table structure of the website's information knowledge base is as follows:

[0165] Field name: domainName; title; description; keywords; type; country; alexa_rank;

[0166] The corresponding data types are: Varchar; Text; Text; Text; Varchar; Varchar; int;

[0167] The corresponding fields have the following meanings: website; title; description; keywords; website type; and Alexa ranking by country.

[0168] The embodiments of this invention have collected approximately 2.1 million pieces of website information. The specific design of the table structure of the website information knowledge base is shown in Table 1:

[0169] Table 1. Structure of the Website Information Knowledge Base

[0170] Serial Number field name Data types Field meaning 1 domainName Varchar website 2 title Text title 3 description Text describe 4 keywords Text Keywords 5 type Varchar Website Types 6 country Varchar Country of origin 7 alexa_rank int Alexa ranking

[0171] S12 Domain Name Information Collection and Domain Name Information Knowledge Base Construction

[0172] For domain name data collection, the domain name data source used is the official website of alexa.com, mainly including two aspects: first, the basic attribute information of the domain name, including the domain name, domain status, authoritative DNS server, domain registration time, domain expiration time, etc.; second, the IP information obtained after domain name resolution. The data collection process is as follows: first, obtain the top 1 million popular website domain names of alexa.com; then, use these domain names as the data source to obtain the basic attribute information of the domain names.

[0173] For collecting basic domain attribute information, the large number of domains leads to an excessively heavy collection task. This invention addresses this issue by using the aiohttp asynchronous crawling framework based on asyncio, combined with multi-threading to batch acquire domain attribute information, significantly improving crawling efficiency. Asyncio supports single-threaded concurrent I / O operations. A comprehensive and accurate identification and parsing strategy needs to be developed based on the website's page structure.

[0174] The specific process for obtaining domain name attribute information is designed as follows:

[0175] S121 sets the number of executable threads based on the number of domain names;

[0176] S122 sets the request URL based on the domain name;

[0177] S123 sets up an IP proxy pool and sets a random sleep time;

[0178] S124 randomly extracts UA information from the UserAgent data pool and simulates a browser requesting a webpage;

[0179] S125 analyzes webpage data and formulates resolution strategies for various domain attribute parameters;

[0180] S126 encapsulates the obtained domain name attribute information;

[0181] S127 loops through S122-S126, using the executemany method to process 1k domain names and then insert them into the database.

[0182] This embodiment uses the domain names and subdomains of the top 1 million popular websites provided by alexa.com as the basic collection unit, and supplements them with the domain names provided by Chinaz.com to complete the task of collecting domain name attribute information.

[0183] Based on this batch of domain name data and the principles of domain name resolution, a table associating domain names with IP addresses was constructed. In the crawling task, strategies such as setting IP proxy pools, random sleep times, and UserAgent (user agent) data pools were used to simulate normal user access behavior, preventing website anti-crawler mechanisms and improving the stability of the crawler program.

[0184] Ultimately, the domain name information knowledge base was completed, and the collection frequency is expected to be updated once a week. This invention has currently collected approximately 2.3 million domain names.

[0185] The table structure of the domain name information knowledge base is as follows:

[0186] Field name domainName; registryDomainId; domainStatus; nameServers; registrationDate; expirationDate; IANAID; domainMX; domainCNAME; domainText;

[0187] The corresponding data types are: Varchar; varchar; longtext; longtext; Varchar; Varchar; Varchar; longtext; Varchar; longtext;

[0188] The corresponding fields have the following meanings: Domain name; Registered domain ID; Domain status; Authoritative name server; Domain registration time; Domain expiration time; Number allocation authority ID; Email address; Alias; Domain information remarks.

[0189] Specifically, the data structure design of the domain name information knowledge base is shown in Table 2:

[0190] Table 2. Domain Name Information Knowledge Base Structure

[0191] Serial Number field name Data types Field meaning 1 domainName Varchar domain name 2 registryDomainId varchar Registered domain ID 3 domainStatus longtext Domain Status 4 nameServers longtext Authoritative Domain Name Server 5 registrationDate Varchar Domain registration time 6 expirationDate Varchar Domain expiration time 7 IANAID Varchar Digital Allocation Authority ID 8 domainMX longtext Mail 9 domainCNAME Varchar Alias 10 domainText longtext Domain Information Notes

[0192] S13 IP Information Data Collection and IP Information Knowledge Base Construction

[0193] The collection of IP data is primarily based on a domain name information knowledge base. Following the principles of domain name resolution, the `dig` command is used to obtain the set of IP addresses mapped to a domain name. Currently, domain name resolution faces two main problems: first, due to the existence of the Great Firewall in China, most overseas domain name resolutions suffer from DNS poisoning, leading to incorrect domain name resolution; second, due to the existence of load balancing technology, the set of IP addresses after domain name resolution is incomplete.

[0194] To address the issues with domain name resolution, the applicant initially employed two methods: first, introducing DNS encryption technology when querying the corresponding IP information using the `dig` command to resolve DNS pollution; and second, directing the `dig` command to public DNS servers in multiple countries, including the United States, Japan, Germany, and Singapore, to resolve the issue of a single IP address after domain name resolution. However, even without using foreign servers, directly querying A records related to a domain name using DNS encryption technology still yielded incorrect results, failing to effectively resolve the DNS pollution problem. Since the `dig` query command defaults to the UDP protocol, and UDP DNS query traffic is highly susceptible to tampering, this invention introduces the TCP protocol into the `dig` query process. This strategy resolves the DNS pollution problem. Ultimately, a total of approximately 1.2 million IP addresses were collected.

[0195] For collecting IP attribute information, the data source is the website pro.ip-api.com, which provides IP attribute information including country, city, latitude and longitude. The acquisition process is basically the same as that for domain name attribute information.

[0196] The specific process for obtaining IP attribute information is designed as follows:

[0197] S131 sets the number of executable threads based on the number of IPs;

[0198] S132 sets the request URL based on the IP address;

[0199] S133 configures the IP proxy pool and sets a random sleep time;

[0200] S134 randomly extracts UA information from the UserAgent data pool and simulates a browser requesting a webpage;

[0201] S135 analyzes webpage data and formulates resolution strategies for various domain attribute parameters;

[0202] S136 encapsulates the obtained IP attribute information;

[0203] S137 loops through S132-S136, using the executemany method to process 1k IP sets and then insert them into the database.

[0204] Finally, the IP information knowledge base was completed. The table structure of the domain name information knowledge base is as follows:

[0205] Field names are: Ip; region; regionName; city; district; latitude; longitude; isp; asnNum; reverse; mobile; proxy;

[0206] The corresponding data types are: Char; Varchar; Varchar; Varchar; Varchar; Double; Double; Varchar; Varchar; Varchar; Varchar; Varchar; varchar;

[0207] The corresponding fields have the following meanings: IP address; region; region name; city; town; latitude; longitude; Internet service provider; autonomous system number; DNS of reverse IP; whether mobile phone connection is possible; whether a proxy is used.

[0208] Its data structure design is shown in Table 3:

[0209] Table 3. Structure of the IP Information Knowledge Base

[0210] Serial Number field name Data types Field meaning 1 Ip Char IP address 2 region Varchar area 3 regionName Varchar Region Name 4 city Varchar City 5 district Varchar Townships 6 latitude Double latitude 7 longitude Double longitude 8 isp Varchar Internet service providers 9 asnNum Varchar Autonomous System Number 10 reverse Varchar DNS of reverse IP 11 mobile Varchar Can it be connected by mobile phone? 12 proxy varchar Is there an agent?

[0211] S14 Authoritative Domain Name Server Information Collection and Authoritative Domain Name Server Knowledge Base Construction

[0212] For collecting authoritative domain name server information, this invention mainly relies on authoritative domain name servers obtained from the domain name information knowledge base, and uses the dig command to obtain the relevant IP information of authoritative domain name servers.

[0213] S15 ASN Information Collection and ASN Information Knowledge Base Construction

[0214] For the collection of ASN information, this invention uses the data source whois.ipip.net, which provides ASN information for 239 regions, including 75 in the Asia-Pacific region, 31 in North America, 53 in Europe, 24 in South America, and 56 in Africa. ASN information collection mainly includes two aspects: First, the collection of basic ASN information, such as the Autonomous System Number (AS), the country code to which the AS belongs, the full name of the AS, the number of IPv4 IPs owned by the AS, and the number of IPv6 IPs owned by the AS. Second, the collection of basic Classless Inter-Domain Routing (CIDR) information, such as the number of IPs in the CIDR IP range, the AS number, the organization name, and the country of registration. Classless Inter-Domain Routing is a method for classifying IP addresses to assign IP addresses to users and to effectively route IP packets on the Internet.

[0215] The ASN information collection task is based on the aiohttp crawler framework and mainly consists of three steps: first, ASN collection; second, ASN attribute information collection; and finally, CIDR attribute information collection. The specific collection process is as follows:

[0216] ASN Collection

[0217] Step 1: Set the request URL using 239 region names as units;

[0218] Step 2: Design a data collection strategy based on the website layout of whois.ipip.net;

[0219] Step 3: Batch input the collected ASN information from various regions into the database.

[0220] ASN Attribute Information Collection

[0221] Step 1: Set the request URL based on the ASN;

[0222] Step 2: Analyze the page structure and design a data collection strategy;

[0223] Step 3: Encapsulate the acquired ASN information;

[0224] Step 4 uses the executemany method to complete the data insertion operation;

[0225] CIDR attribute information collection

[0226] Step1 performs drill-down analysis based on the collected ASNs to obtain the CIDRs corresponding to the ASNs;

[0227] Step2 sets the request URL based on the ASN;

[0228] Step 3: Analyze the style of the page where the corresponding CIDR is located and design a data collection strategy;

[0229] This invention employs a three-step strategy—ASN collection, ASN attribute information collection, and CIDR attribute information collection—to complete the ASN information collection task. In this embodiment, 70,656 ASN records and 980,704 classless inter-domain routing records were collected.

[0230] The table structure of the ASN information knowledge base is as follows:

[0231] The field names are asnNum, asnNumHref, countryCode, asnName, ipv4NumIps, ipv6NumIps, and cirrIdSet;

[0232] The corresponding data types are all Varchar;

[0233] The corresponding fields have the following meanings: Autonomous System (AS) ID, AS link, country code, full name of AS ID, number of IPv4 IPs owned by the AS, number of IPv6 IPs owned by the AS, and set of CIDR IDs.

[0234] The CIDR information table structure associated with the ASN information knowledge base is as follows, with field names including: cider, ciderDescription, ipNum, asnNum, asName, orgName, registryRegion, ipv4Prefixes, ipv6Prefixes, ipv4NumIps, and id;

[0235] The corresponding data types are: Varchar, Varchar, Int, Varchar, Varchar, Varchar, Varchar, Varchar, Varchar, Varchar, Varchar, varchar;

[0236] The corresponding fields have the following meanings: Classless Inter-Domain Routing, Classless Inter-Domain Routing Description, Number of IPs in the IP Segment, Autonomous System (AS) Number, Autonomous System (AS) Abbreviation, Organization Name, Registered Country, IPv4 Prefix, IPv6 Prefix, Number of IPv4s, and CIDR ID.

[0237] Table 4. Structure of the ASN Information Knowledge Base

[0238]

[0239]

[0240] The CIDR information associated with the ASN information knowledge base is shown in Table 5:

[0241] Table 5 CIDR Information Table Structure

[0242] Serial Number field name Data types Field meaning 1 cidr Varchar Classless Inter-Domain Routing 2 cidrDescription Varchar Classless Inter-Domain Routing Description 3 ipNum Int Number of IPs in an IP range 4 asnNum Varchar Autonomous System (AS) Number 5 asName Varchar Autonomous System (AS) 6 orgName Varchar Organization Name 7 registryRegion Varchar Registered country 8 IPv4 Prefixes Varchar IPv4 prefix 9 IPv6 Prefixes Varchar IPv6 prefix 10 IPv4NumIPs varchar Number of IPv4 11 id Int CIDR ID

[0243] As mentioned above, this invention aims to improve the domain name resolution knowledge system. It primarily uses the aiohttp crawler and secondarily uses the dig script technology. It adopts an automated crawler model that combines aiohttp and dig technologies. Based on the table rules designed in the knowledge base, it processes and filters the data, extracting unstructured data such as domain names, IPs, websites, and authoritative domain name servers in batches into the corresponding knowledge base.

[0244] Furthermore, the knowledge graph-based method for identifying daily abnormal behaviors in web logs of the present invention specifically includes the S2 knowledge graph prototype design and knowledge graph construction, wherein the knowledge graph prototype design...

[0245] Entities, relationships between entities, and entity attributes are key elements in constructing a knowledge graph. This invention aims to construct a knowledge graph for a domain name resolution system, and designs a prototype knowledge graph for this system based on a constructed domain knowledge base. A combination of top-down and bottom-up approaches is used to construct the knowledge graph. The design of the knowledge graph prototype is primarily based on the principles of domain name resolution. The process of establishing relationships between entities is as follows:

[0246] 1. Domain name resolution is the process of converting a domain name into an IP address through a dedicated Domain Name System (DNS) server. This invention constructs two main entities—domain name and IP address—based on the principle of domain name resolution, and uses these entities to establish the parent-child relationship between domain names and subdomains, as well as the carrying relationship between domain names and IP addresses.

[0247] 2. The authoritative name server (Name Server) is responsible for the authoritative name servers in this DNS zone. It is generally used in the domain name resolution process. For example, if a user visits www.baidu.com, the IP information corresponding to www.baidu.com will be obtained iteratively or recursively from the local name server, root name server, top-level name server, and authoritative name server. Based on the principles of the Name Server, the NS entity is established, and the service relationship between the domain name and the NS, and the address relationship between the IP address and the NS are established.

[0248] 3. An Autonomous System Number (ASN) consists of 16 bits, with a total of 65,536 possible values. Each country is assigned a unique ASN, and the ASN reveals the country to which an IP address belongs. DNS reverse lookup is the process of querying domain name information from an IP address. To query the ASN of an IP address, a WHOIS query is typically used. If the query result contains an "origin" field, it represents the ASN. Therefore, based on WHOIS query technology and the principles of autonomous systems, two main entities are established: the ASN and the country, and two relationships are established: the affiliation relationship between the IP address and the ASN, and the relationship between the ASN and the country.

[0249] 4. Alias ​​resolution typically involves creating a CNAME record for a domain name and pointing it to a domain name rather than an IP address. This invention establishes two main entities—the website and the CNAME—based on the mapping relationship between domain names and websites and the principle of alias resolution, and establishes the alias relationship between the website and the CNAME.

[0250] 5. Most websites consist of domain names, DNS domain name resolution, web hosting servers, and website programs. This invention establishes the mapping relationship between domain names and websites based on the principles of website composition.

[0251] Ultimately, seven entities and eight relationships were established. The seven entities are domain name, website, IP address, NS (Authoritative Name Server), CNAME (Alias), ASN (Autonomous System Number), and country. The eight relationships are: parent-child, carrier, service, address, membership, belonging, alias, and mapping. The triple types are: domain name 1 (parent-child relationship), domain name 2 (domain name, carrier relationship), IP address (domain name, service relationship), NS address (NS address, membership), ASN belonging, country, domain name (alias relationship), CNAME (domain name, mapping relationship), and website. The specific relationships are shown in Table 6 below.

[0252] Table 6. Knowledge Graph Prototype Relationship Table

[0253]

[0254]

[0255] In summary, and incorporating feedback from log analysis experts, the final prototype of the knowledge graph is shown in the attached figure. Figure 2 As shown.

[0256] This invention is based on a knowledge base that integrates multi-source data and is supported by a domain name resolution knowledge graph prototype. It has completed the task of constructing a knowledge graph for the domain name resolution system. The final data scale is nearly 5 million, as shown in Table 7.

[0257] Table 7. Statistics on the Node Size of Knowledge Graph

[0258] Serial Number node Data volume Remark 1 domain 2162995 Domain Node 2 NS 322175 authoritative domain node 3 IP 1273717 IP Node 4 ASN 70656 Autonomous System Nodes 5 country 239 National and regional nodes 6 Website 1035622 Website Nodes 7 CNAME 69897 Domain alias node

[0259] The knowledge graph-based web log abnormal behavior identification method of the present invention includes the following specific contents in the construction of the S3 web log abnormal behavior identification model.

[0260] The abnormal behavior identification model primarily uses machine learning and manual analysis to extract non-human-generated abnormal logs from DNS, IP, URL, and SSL logs generated by network servers. It summarizes the characteristics of these abnormal logs and defines and identifies abnormal behaviors based on these characteristics. This invention constructs an abnormal behavior identification model based on the knowledge graph of the application resolution system. In this embodiment, nearly 40 abnormal behavior models are constructed. The abnormal behavior models are attached. Figure 3 As shown.

[0261] This includes using knowledge graphs for dataset preprocessing.

[0262] By leveraging the country and city attributes in the IP nodes of the knowledge graph, the source and destination IP addresses in the log records can be quickly completed. When the geographical location is not one of the two opposing internal and external networks, the ASN information corresponding to the IP can be queried based on the affiliation in the graph to obtain the country geographic information and correct errors in a timely manner.

[0263] This also includes using knowledge graphs for daily behavior analysis.

[0264] Knowledge graphs also play a crucial role in the identification of abnormal behavior, for example:

[0265] 1. User Group Segmentation. The original segmentation strategy primarily relied on attributes such as IP geographic location and IP range distribution in logs to divide users into groups. However, this invention, by introducing website category information from the website nodes in a knowledge graph, enables more detailed segmentation based on the types of websites visited by users.

[0266] Specifically, in this embodiment, the user group is based on the IP geographic location, IP segmentation attributes, and website category information of the website node in the knowledge graph of the log data.

[0267] 2. Web crawling refers to the behavior of search engines or individual users periodically retrieving page data from various domains to meet the pre-caching needs during searches. The initial behavior analysis process primarily involves matching the User Agent (UA) field information in log data with the local user_agent table (containing approximately 40 million UA ​​records collected from the web) to determine if the UA belongs to a web crawler, or judging whether the URL field contains the characters "robots.txt" based on summarized behavioral characteristics. However, this analysis method cannot identify records with forged UA information or empty fields.

[0268] To address this problem, this invention improves the recognition rate of crawler behavior by introducing a strategy of whitelisting crawler IPs from major internet companies in the IP nodes of the graph.

[0269] 3. Brute-force DNS resolution is a behavior that uses brute-force DNS resolution or dictionary manipulation to concatenate domain names to check if a website is accessible. The original behavioral analysis calculation formula is as follows:

[0270]

[0271] Cnt(domain2)-k>0 (2)

[0272] Where i represents the domain level, Count() represents the number of times a certain level of domain appears, domain2 represents the second-level domain of the domain, and k represents the threshold (derived from analytical experience, generally set to 100). Formula (1) means using the second-level domain doamin2 of the domain as the key and the number of times its third-level and above domains appear Cnt as the value; Formula (2) means judging whether the number of times Cnt(domain2) of different subdomains of the second-level domain appears within a certain period of time is greater than the set threshold k; if so, the record is considered to be a brute-force domain resolution behavior.

[0273] The thresholds used in this analysis method were always set based on log analysis experience, but the identification results were inconsistent. The identified data contained a large amount of crawling and VPN proxy communication activities. For example, the logs showed approximately 87,000 records of crawling personal blog websites such as blogspot.com and approximately 25,000 records of VPN proxy connections to vpn.rockstargames.com.

[0274] The improved solution of this invention first filters the data using a crawler behavior recognition strategy and feature rules. Then, it introduces parent-child relationships from a knowledge graph, queries all subdomains of a domain name, and calculates the total number of subdomains. The threshold k can be directly set equal to the total number of subdomains of that domain name, and domain name matching can be performed directly to determine whether the subdomain exists in the knowledge graph. This improved solution solves the problems of filtering behavioral feature data such as crawlers and VPNs, and the problem of selecting the threshold k, thus improving the accuracy of identifying brute-force domain name resolution behavior.

[0275] 4. DNS duplicate resolution behavior is caused by DNS pollution leading to incorrect resolution result verification, resulting in duplicate DNS requests being sent to 13 root name servers. The original behavior analysis calculation formula is as follows:

[0276] F (Ftime,Cip,qname) ={Sip1,Sip2,...,Sip n} (3)

[0277] Len(F)-13>>0 (4)

[0278] Where Ftime represents the log discovery time, Cip represents the source IP, qname represents the domain name requested by Cip, Sip represents the destination IP, F represents the set, and Len() represents the length of the set. Formula (3) means constructing set F with time, source IP, and qname as labels, and using the destination IP as the set element; Formula (4) determines whether the length of the set is greater than or equal to 13. If it is, the record is considered to be a DNS duplicate resolution behavior.

[0279] Although this process can identify DNS duplicate resolution behavior, there is a problem that the target IP does not belong to the root name server, resulting in low accuracy of identification.

[0280] The improved solution of this invention introduces address relationships from a knowledge graph, using IP addresses to query authoritative name server information and determine whether the target IP belongs to a root name server. This improved solution solves the problem of whether the target belongs to a root name server, thus improving the accuracy of identifying DNS duplicate resolution behavior.

[0281] Experimental Example

[0282] 1. Experimental Data

[0283] This invention uses network server logs provided by a research institution as experimental data, and the server blocks harmful information hundreds of millions of times per day.

[0284] 2. Experimental Environment

[0285] The experimental environment for this invention is as follows: the compilation environment is Scala 2.12.13 based on Spark 3.1.2 and Hadoop 2.7.1, the database used is Neo4j and MySQL, and the development tool is IntelliJ IDEA 2021.1.3 x64.

[0286] 3. Evaluation Standards

[0287] The accuracy P of abnormal behavior identification t Recall rate R e The knowledge graph of a domain name resolution system is evaluated using three metrics: F1 score, F2 score, and F1 score. The specific calculation formula is shown in Table 8.

[0288] Table 8 Evaluation Criteria

[0289]

[0290] Among them, the accuracy rate P of abnormal behavior identification t The recall rate (R) indicates the precision of abnormal behavior identification. e This represents the prediction coverage of the abnormal behavior recognition model based on the knowledge graph of the domain name-oriented resolution system; the F1 score for abnormal behavior recognition indicates the prediction coverage of P. t and R e By merging entities of equal importance, a comprehensive assessment of the identified anomalous behaviors is obtained.

[0291] 4. Analysis of Experimental Results

[0292] This invention conducts comparative experiments on crawler behavior, domain name brute-force resolution behavior, and DNS duplicate resolution behavior in the abnormal behavior identification model under two conditions: one based on a knowledge graph for the domain name resolution system and the other without a knowledge graph, to verify the role of knowledge graph in the abnormal behavior identification model.

[0293] Experiment 1: Analysis of the Results of the Crawler Behavior Recognition Experiment

[0294] pass Figure 4 It can be seen that the precision of the crawler behavior recognition model based on the knowledge graph is 100% for both models and the non-knowledge graph-based model, indicating that the abnormal behaviors identified by both methods were correctly identified. However, the recall and F1 score increased by 25.79% and 14.88% respectively, indicating that the knowledge graph-based model correctly identified more abnormal behaviors than the non-knowledge graph-based model, and the recognition rate was significantly improved. This proves that knowledge graphs play a facilitating role in crawler behavior recognition.

[0295] Experiment 2: Analysis of Results of the Experiment on Identifying Brutal Domain Name Resolution Behavior

[0296] The results of the comparative analysis of the experiment on identifying brute-force domain name resolution behavior are as follows: Figure 5 As shown:

[0297] pass Figure 5 The results show that the precision of the crawler behavior recognition model based on knowledge graphs is 38.47% higher than that of the non-knowledge graph-based model, while the recall rates are the same. This indicates that the number of correctly identified abnormal behaviors is consistent between the two approaches, but the noise level of the knowledge graph-based model is significantly lower than that of the non-knowledge graph-based model. Furthermore, the F1 score of this experiment is also improved by 47.23%, demonstrating that knowledge graphs can enhance the noise reduction capability of brute-force DNS resolution behavior recognition models.

[0298] Experiment 3: Analysis of Results of DNS Repeated Resolution Behavior Identification Experiment

[0299] pass Figure 6 It can be seen that the accuracy of the DNS duplicate resolution behavior identification model based on the knowledge graph is improved by 95.3% compared with that without the knowledge graph. However, the recall rates of the two models are the same, indicating that the number of abnormal behaviors correctly identified is the same. However, the model without the knowledge graph identifies too much noisy data in the log data, resulting in a particularly large difference in accuracy between the two models, which in turn leads to a 91.63% improvement in the F1 score of this experiment. This experiment can demonstrate that the knowledge graph improves the ability of the DNS duplicate resolution behavior identification model to filter out non-abnormal behavior data.

[0300] This invention proposes a knowledge graph-based method for identifying abnormal behavior in web logs, aiming to improve the analysis and recognition capabilities of abnormal behavior identification models. It also provides researchers with a convenient and large-scale domain knowledge retrieval tool for log analysis. The experimental results of Experiments 1, 2, and 3 demonstrate that the knowledge graph for domain name resolution systems constructed using the method of this invention increases the feature attributes of web logs, reduces the difficulty of web log analysis, and improves the analysis and recognition capabilities of abnormal behavior identification models.

[0301] Furthermore, the method of this invention for constructing knowledge graphs is not only significant for detecting abnormal behaviors such as machine access in logs, but also has important implications for other work in the field of network information security, such as detecting malicious domain names and APT attacks.

[0302] The technical solutions provided by this invention are not limited to the above embodiments. Any technical solutions formed by using the structure and method of this invention through transformation and substitution are within the protection scope of this invention.

Claims

1. A method for identifying abnormal behavior in web logs based on knowledge graphs, characterized in that: Includes the following steps, S1 Multi-Source Data Acquisition and Construction of Corresponding Domain Knowledge Base An automated crawler model based on aiohttp and dig technologies is designed using methods such as domain name resolution, authoritative domain name servers, alias resolution, and autonomous systems, and a corresponding domain knowledge base is constructed. S2 Knowledge Graph Prototype Design and Knowledge Graph Construction Based on S1, a domain knowledge base was designed to create a prototype knowledge graph for the domain name resolution system, and the knowledge graph was constructed. Construction of an S3 web log abnormal behavior identification model The knowledge graph built on S2 assists in the construction of a web log abnormal behavior recognition model; In the S1 multi-source data collection and corresponding domain knowledge base construction, firstly, the data types of various knowledge are established based on domain name resolution, authoritative domain name servers, alias resolution, and autonomous system methods; then, the data sources of various types of data are determined, and the table structure of various types of knowledge is determined according to the data sources, while a specific collection scheme is designed for each type of knowledge; finally, the collected web page data is processed and filtered according to the table rules designed for the knowledge base, unstructured data is converted into structured data, and stored in batches in the corresponding knowledge base; S1 includes S11 website information collection and website classification system construction; S12 domain name information collection and domain name information knowledge base construction; S13 IP Information Data Collection and IP Information Knowledge Base Construction; S14 Authoritative Domain Name Server Information Collection and Authoritative Domain Name Server Knowledge Base Construction; S15 ASN Information Collection and ASN Information Knowledge Base Construction; S11 includes the collection of basic attribute information of the S111 website. The website's title, description, and keywords were collected from the StuffGate website. The collection process was as follows: S1111 sets the number of executable threads based on the number of domain names; S1112 sets the request URL based on the domain name; S1113 configures the IP proxy pool and sets a random sleep time; S1114 randomly extracts UA information from the UserAgent data pool to simulate a browser requesting a webpage; S1115 analyzes webpage data and formulates resolution strategies for various domain attribute parameters; S1116 encapsulates the obtained domain name attribute information; S1117 loops through S1112-S1116, using the executemany method to process 1k domain names and then insert them into the database; For the Alexa ranking acquisition process, an asynchronous crawler framework based on aiohttp is used to introduce an IP proxy pool, forge login passwords to obtain cookie information, and set random sleep time. The process for collecting information about the country to which the website belongs is S111A. Based on the number of domain names and the size of the number of executable threads, the length of the domain name processed by the thread is set to reduce the data crawling time. S111B uses web crawling technology to obtain WHOIS information from Chinaz.com and Alibaba. S111C compares the country information of the two websites. If they match, the information is stored in the result set and the next website is processed. Otherwise, S111D is executed. S111D retrieves the main geographical location of websites accessed from alexa.com, corrects the results, stores them in the result set, and then processes the next website. It also includes the construction of the S112 website classification system, which includes, S1121 Data Preprocessing: The Baidu Translate API interface is used to batch convert the website title, description, and keywords in the website's basic information attributes; S1122 Data Labeling: Website type labeling was performed on a batch of websites using manual methods; S1123 Word segmentation processing: Jieba word segmentation technology is used to segment the website's title, description information, and keyword-based sentences to obtain segmented text; S1124 Model Training: The segmented and labeled text is used as the training set for training. First, the text is represented in vector space to obtain the vector space representation of each connected text. Then, a logistic regression model is used to classify the text vectors based on the feature representation. S1125 Website Tag Prediction: Using a trained model for unlabeled text classification; Based on the processing of S111 and S112, the construction of the website information knowledge base is completed; Among them, S12 domain name information collection and domain name information knowledge base construction includes, S121 sets the number of executable threads based on the number of domain names; S122 sets the request URL based on the domain name; S123 sets up an IP proxy pool and sets a random sleep time; S124 randomly extracts UA information from the UserAgent data pool and simulates a browser requesting a webpage; S125 analyzes webpage data and formulates resolution strategies for various domain attribute parameters; S126 encapsulates the obtained domain name attribute information; S127 loops through S122-S126, using the executemany method to insert 1k domain names into the database after processing them; At the same time, based on this batch of domain name data and according to the principle of domain name resolution, a table relating domain names to IP addresses was constructed; finally, the domain name information knowledge base was completed. S13 IP information data collection and IP information knowledge base construction are underway. Based on the domain name information knowledge base and in accordance with the principle of domain name resolution, the dig command is used to obtain the set of IP addresses mapped to the domain name. The TCP protocol is introduced during the dig query process to de-DNS pollution and obtain the set of IP addresses. The process for obtaining IP attribute information is as follows: S131 sets the number of executable threads based on the number of IPs; S132 sets the request URL based on the IP address; S133 configures the IP proxy pool and sets a random sleep time; S134 randomly extracts UA information from the UserAgent data pool and simulates a browser requesting a webpage; S135 analyzes webpage data and formulates resolution strategies for various domain attribute parameters; S136 encapsulates the obtained IP attribute information; S137 loops through S132-S136, using the executemany method to process 1k IP sets and then insert them into the database. Finally, the IP information knowledge base was completed; S14 authoritative domain name server information collection and authoritative domain name server knowledge base construction are underway. The authoritative domain name servers obtained from the domain name information knowledge base are collected, and the relevant IP information of the authoritative domain name servers is obtained by using the dig command. S15 ASN information collection and ASN information knowledge base construction are underway. ASN information collection includes, ASN Collection Step 1: Set the request URL using 239 region names as units; Step 2: Design a data collection strategy based on the website layout of whois.ipip.net; Step 3: Collect ASN information from various regions and import it into the database in batches; ASN Attribute Information Collection Step 1: Set the request URL based on the ASN; Step 2: Analyze the page structure and design a data collection strategy; Step 3: Encapsulate the acquired ASN information; Step 4 uses the executemany method to complete the data insertion operation; CIDR attribute information collection Step1 performs drill-down analysis based on the collected ASNs to obtain the CIDRs corresponding to the ASNs; Step2 sets the request URL based on the ASN; Step 3: Analyze the style of the page where the corresponding CIDR is located and design a data collection strategy.

2. The method for identifying abnormal behavior in web logs based on knowledge graphs according to claim 1, characterized in that: In the S2 knowledge graph prototype design and construction, specifically in the knowledge graph prototype design... Based on the principle of domain name resolution, two major entities, domain name and IP address, are constructed, and the parent-child relationship between domain name and subdomain and the carrying relationship between domain name and IP address are established using these entities. The Name Server principle is used to establish the NS entity, and to establish the service relationship between the domain name and the NS, and the address relationship between the IP and the NS; Based on WHOIS query technology and the principle of autonomous systems, two major entities, ASN and country, are established, and two relationships are established: the affiliation relationship between IP and ASN and the belonging relationship between ASN and country. Establish the mapping relationship between domain names and websites based on the principles of website composition.

3. The method for identifying abnormal behavior in web logs based on knowledge graphs according to claim 2, characterized in that: In the S2 knowledge graph prototype design and knowledge graph construction, the entities in the knowledge graph prototype design include domain name, website, IP, NS, CNAME, ASN, and country. Relationships include: parent-child relationship, carrier relationship, service relationship, address relationship, membership relationship, belonging relationship, alias relationship, and mapping relationship; The ternary types include: domain1, parent-child relationship; domain2, domain, carrier relationship; IP, domain, service relationship; NS, IP, address relationship; NS, IP, membership relationship; ASN, ASN, belonging relationship; country, domain, alias relationship; CNAME, domain, mapping relationship; website.

4. The method for identifying abnormal behavior in web logs based on knowledge graphs according to claim 3, characterized in that: The construction of the S3 web log abnormal behavior recognition model includes, For DNS logs, IP logs, URL logs, and SSL logs generated by network servers, non-human-caused abnormal logs are extracted using machine learning and manual analysis methods. The characteristics of abnormal logs are summarized, and abnormal behaviors are defined and identified based on these behavioral characteristics.

5. The method for identifying abnormal behavior in web logs based on knowledge graphs according to claim 4, characterized in that: The construction of the S3 web log abnormal behavior recognition model is assisted by the knowledge graph built on S2. During data preprocessing, the source IP address and target IP address information of the log records are supplemented by the country and city attribute information in the IP nodes of the knowledge graph; When a geographic location is not one of two opposing internal and external networks, the ASN information corresponding to the IP address is queried based on the affiliation in the geographic map to obtain the country geographic information.

6. The method for identifying abnormal behavior in web logs based on knowledge graphs according to claim 5, characterized in that: The construction of the S3 web log abnormal behavior recognition model is assisted by the knowledge graph built on S2. When segmenting users, user groups are based on the IP geographic location, IP segmentation attributes, and website category information of the website node in the knowledge graph.

7. The method for identifying abnormal behavior in web logs based on knowledge graphs according to claim 6, characterized in that: The construction of the S3 web log abnormal behavior recognition model is assisted by the knowledge graph built on S2. We introduce a strategy of using whitelists of crawler IPs from major internet companies in the IP nodes of the graph for data crawling.

8. The method for identifying abnormal behavior in web logs based on knowledge graphs according to claim 7, characterized in that: The construction of the S3 web log abnormal behavior recognition model is assisted by the knowledge graph built on S2. When performing brute-force DNS resolution, the analysis formula for using a domain name for brute-force resolution is as follows: Cnt(domain2)-k>0 Where i represents the domain level, Count() represents the number of times a certain level of domain appears, domain2 represents the second-level domain of the domain, and k represents the threshold; The process involves first using a crawler behavior recognition strategy and feature rules to filter the data, then introducing parent-child relationships from a knowledge graph, querying all subdomain information by domain name, setting the threshold k directly based on the number of subdomains, and directly performing domain name matching to determine whether the subdomain exists in the graph.

9. The method for identifying abnormal behavior in web logs based on knowledge graphs according to claim 7, characterized in that: The construction of the S3 web log abnormal behavior recognition model is assisted by the knowledge graph built on S2. By introducing address relationships from a knowledge graph, we can query authoritative name server information through IP addresses to determine whether a target IP belongs to a root name server.

Images