A method, device, computer equipment and readable storage medium for knowledge mining and application of network access log data

Abstract

The present invention discloses a knowledge mining and application method, apparatus, computer equipment, and readable storage medium for network access log data. The method comprises: first, dividing the original network access log data into subsets, constructing a probabilistic grammar model based on a knowledge graph, repeating this step and completing graph merging, and performing node clustering to obtain a target probabilistic knowledge graph. The method then obtains the knowledge graph query language corresponding to the user's query request, executes the query in the target graph to obtain node-related results, constructs context information, constructs prompt words, invokes a large language model for processing, and obtains the query answer. This method effectively mines network access log knowledge and meets user query requirements.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method, apparatus, computer equipment and readable storage medium for knowledge mining and application of network access log data. Background Art

[0002] With the development of network technology, the volume of network access log data has increased dramatically. Traditional methods for analyzing network access log data rely primarily on simple statistics, making it difficult to deeply explore the underlying knowledge and connections. Existing technologies lack the means to effectively integrate temporal and spatial information and leverage knowledge graphs for deep mining. Consequently, they are unable to accurately and intelligently extract valuable information from massive log data when faced with complex query requirements. Summary of the Invention

[0003] The object of the present invention is to provide a method, apparatus, computer equipment and readable storage medium for knowledge mining and application of network access log data.

[0004] In a first aspect, an embodiment of the present invention provides a knowledge mining and application method for network access log data, comprising:

[0005] Divide the original network access log data according to preset rules to obtain multiple network access log data subsets;

[0006] Constructing a probabilistic grammar model based on a knowledge graph for each subset of the network access log data, wherein the knowledge graph is composed of triples including a source IP node, a target node, and an initiated connection as a relationship, wherein the source IP node and the target node include a unique identifier, a probability parameter, a raw count, a start time, and an end time;

[0007] Repeating the step of constructing a probabilistic grammar model based on a knowledge graph for each subset of the network access log data, and completing graph merging based on the original count, the start time, and the end time to obtain a knowledge graph merged based on time and space;

[0008] Performing node clustering on the merged knowledge graph to obtain a target probability knowledge graph corresponding to each subset of the network access log data;

[0009] Obtain the knowledge graph query language corresponding to the query request entered by the user;

[0010] Performing a query operation based on the knowledge graph query language in each of the target probabilistic knowledge graphs to obtain node-related results corresponding to each of the target probabilistic knowledge graphs;

[0011] Context is constructed based on the knowledge graph query language and the node-related results to obtain context information for large language model query;

[0012] A prompt word is constructed according to the context information and the query request, and a preset large language model is called to perform processing to obtain an answer content for the query request.

[0013] In a second aspect, an embodiment of the present invention provides a knowledge mining and application device for network access log data, comprising:

[0014] A mining module is configured to partition the original network access log data according to preset rules to obtain a plurality of network access log data subsets; construct a probabilistic grammatical model based on a knowledge graph for each of the network access log data subsets, wherein the knowledge graph is composed of a triple including a source IP node, a target node, and an initiated connection as a relationship, wherein the source IP node and the target node include a unique identifier, a probability parameter, an original count, a start time, and an end time; repeatedly execute the steps of constructing a probabilistic grammatical model based on a knowledge graph for each of the network access log data subsets, and complete graph merging based on the original count, the start time, and the end time to obtain a knowledge graph merged based on time and space; perform node clustering on the merged knowledge graph to obtain a target probabilistic knowledge graph corresponding to each of the network access log data subsets;

[0015] The application module is used to obtain the knowledge graph query language corresponding to the query request input by the user; perform a query operation based on the knowledge graph query language in each target probabilistic knowledge graph to obtain node-related results corresponding to each target probabilistic knowledge graph; construct a context based on the knowledge graph query language and the node-related results to obtain context information for large language model query; construct a prompt word based on the context information and the query request, and call a preset large language model to perform processing to obtain the answer content for the query request.

[0016] In a third aspect, an embodiment of the present invention provides a computer device, comprising a processor and a non-volatile memory storing computer instructions, wherein when the computer instructions are executed by the processor, the computer device executes the method described in the first aspect.

[0017] In a fourth aspect, an embodiment of the present invention provides a readable storage medium, wherein the readable storage medium includes a computer program, and when the computer program is executed, the computer device where the readable storage medium is located is controlled to execute the method described in the first aspect.

[0018] Compared with the existing technology, the beneficial effects provided by the present invention include: using a knowledge mining and application method, device, computer equipment and readable storage medium for network access log data disclosed by the present invention, including: first dividing the original network access log data into subsets, building a probabilistic grammar model based on the knowledge graph, repeating this step and completing the graph merging, and performing node clustering to obtain the target probabilistic knowledge graph. Obtaining the knowledge graph query language corresponding to the user query request, executing the query in the target graph to obtain node-related results, constructing context information, constructing prompt words to call the large language model for processing, and obtaining the query answer. This method effectively mines network access log knowledge and meets user query needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly describes the drawings required for use in the embodiments. It should be understood that the following drawings illustrate only certain embodiments of the present invention and should not be construed as limiting the scope of the present invention. Those skilled in the art can, without inventive effort, derive other relevant drawings from these drawings.

[0020] Figure 1 A schematic flow chart of the steps of a method for knowledge mining and application of network access log data provided by an embodiment of the present invention;

[0021] Figure 2 A schematic block diagram of the structure of a device for knowledge mining and application of network access log data provided by an embodiment of the present invention;

[0022] Figure 3 A schematic block diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the embodiments of the present invention. It should be understood that the described embodiments are only a portion of the embodiments of the present invention, not all of them. Generally, the components of the embodiments of the present invention described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations.

[0024] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0025] In order to solve the technical problems in the above background technology, Figure 1 This is a flow chart of a method for knowledge mining and application of network access log data provided by an embodiment of the present disclosure. The method for knowledge mining and application of network access log data is introduced in detail below.

[0026] Step S201, dividing the original network access log data according to a preset rule to obtain multiple network access log data subsets;

[0027] Step S202: constructing a probabilistic grammar model based on a knowledge graph for each subset of the network access log data, wherein the knowledge graph is composed of triples including a source IP node, a target node, and an initiated connection as a relationship, wherein the source IP node and the target node include a unique identifier, a probability parameter, a raw count, a start time, and an end time;

[0028] Step S203, repeatedly executing the step of constructing a probabilistic grammar model based on a knowledge graph for each subset of the network access log data, and completing graph merging based on the original count, the start time, and the end time to obtain a knowledge graph merged based on time and space;

[0029] Step S204: performing node clustering on the merged knowledge graph to obtain a target probability knowledge graph corresponding to each subset of the network access log data;

[0030] Step S205: Obtain the knowledge graph query language corresponding to the query request input by the user;

[0031] Step S206: performing a query operation in each of the target probabilistic knowledge graphs based on the knowledge graph query language to obtain node-related results corresponding to each of the target probabilistic knowledge graphs;

[0032] Step S207: constructing a context based on the knowledge graph query language and the node-related results to obtain context information for large language model query;

[0033] Step S208 : constructing prompt words according to the context information and the query request, and calling a preset large language model to perform processing to obtain answer content for the query request.

[0034] In an embodiment of the present invention, for example, assume that there is one month of network access log data. To facilitate processing and analysis, the data needs to be divided according to preset rules. First, according to the node splitting method, the source IP address and the destination IP address are used as the basis. For example, it is found that there are a large number of access records from the source IP address "192.168.1.100" and many access records directed to the destination IP address "10.0.0.50". Then, the log data related to these two IP addresses is divided separately to form two subsets.

[0035] At the same time, the data is also split according to the time window. Assuming that every 24 hours is set as a time window, the log data from 0:00 to 24:00 on the first day will be divided into one subset, the log data from 0:00 to 24:00 on the second day will be divided into another subset, and so on.

[0036] For example, if "192.168.1.100" frequently accesses "10.0.0.50" between 12:00 and 13:00 on a certain day, this data will be divided into the time window subset corresponding to that day.

[0037] Through this combination of node splitting and time window splitting, we obtain numerous subsets of network access log data, each of which has specific characteristics, which facilitates subsequent processing.

[0038] For each data subset, we begin to build a probabilistic grammar model based on the knowledge graph. For example, a specific subset contains a series of access records with a source IP address of "192.168.1.100" and a destination IP address of "10.0.0.50."

[0039] First, we create nodes for the knowledge graph. The source IP node "192.168.1.100" and the destination IP node "10.0.0.50" are created. Each node has attributes such as a unique identifier, probability parameter, raw count, start time, and end time.

[0040] Then, relationships are constructed by scanning the log records line by line. For example, if a record shows "192.168.1.100 initiated a connection to 10.0.0.50 using protocol type 80 and a packet size of 500 bytes," a new triple "<192.168.1.100, 10.0.0.50, initiated connection>" is added, and the relationship strength between the two nodes is increased by 1.

[0041] Next, we model the probability. For the conditional probability distribution of "protocol type - packet size," we find that when the source IP node "192.168.1.100" uses protocol type 80, the packet size is usually between 400 and 600 bytes during multiple visits. Therefore, we adjust and record this probability distribution accordingly.

[0042] For the joint probability distribution of "source node port-destination node port", assuming that the port used by the source IP node "192.168.1.100" is usually 8080, and the port received by the destination IP node "10.0.0.50" is usually 80, then this two-dimensional probability distribution will also be recorded.

[0043] Similarly, conditional probability distributions such as "source node-protocol type", "target node-protocol type", "source node-target node", "target node-source node", and the prior probability distribution of the source node and target node are constructed by analyzing and counting a large number of access records.

[0044] In this way, a rich and accurate probabilistic grammar model is constructed for each data subset.

[0045] After constructing probabilistic grammar models for multiple data subsets, we begin graph merging. Suppose we first partition the data into 1-hour time intervals, resulting in a 24-hour probabilistic grammar model.

[0046] First, the nodes are grouped according to their start and end times. For example, all nodes from 0:00 to 1:00 are grouped together, all nodes from 1:00 to 2:00 are grouped together, and so on.

[0047] Then, nodes within the same time period are spatially merged. For example, if there are multiple source and target IP nodes between midnight and 1:00, these nodes are merged into the same knowledge graph to ensure that the node relationships and probability distribution within this time period are fully presented.

[0048] Next, we use the raw count information to perform temporal merging. For example, suppose the source IP node "192.168.1.100" appears 100 times between 0:00 and 1:00 and 80 times between 1:00 and 2:00. Using the raw counts, we can calculate its probability distribution over the longer time period from 0:00 to 2:00, thus completing the temporal merging and obtaining a probability model for the node at different time scales.

[0049] Through such graph merging, a more comprehensive and accurate knowledge graph can be obtained, providing a more valuable foundation for subsequent node clustering and application.

[0050] For the merged knowledge graph, node clustering is performed. Taking a knowledge graph containing many nodes as an example, each node has a series of probability distribution characteristics.

[0051] First, calculate the eigenvector of each node. Assume that the probability distribution of node A includes the probability distribution of "protocol type-packet size" as P1, the probability distribution of "source node port-destination node port" as P2, and so on.

[0052] Then, we use unsupervised clustering methods such as k-NN to perform clustering by setting a distance threshold. For example, if the distance threshold is set to 0.5, if the distance between node A and node B is less than 0.5, then they are classified into the same category.

[0053] After clustering is complete, the clustering information is set for each node. For example, node A is classified as class 1, its distance from the cluster center is 0.2, and the maximum distance between clusters is 0.8.

[0054] Through node clustering, nodes with similar characteristics can be grouped into one category, which is convenient for subsequent analysis and application.

[0055] When a user enters a query request, such as "Query the target nodes accessed between 10:00 AM and 11:00 AM yesterday with a source IP address of 192.168.1.100 and the protocol type used," the server uses natural language processing technology to parse the request.

[0056] First, the server identifies the key entity information, namely the source IP address "192.168.1.100" and the time range "yesterday morning 10:00 to 11:00".

[0057] The server then converts this entity and relationship information into a knowledge graph query language, such as the SPARQL query statement: "SELECT DISTINCT?destination_ip?protocol_type WHERE{<192.168.1.100><Initiate connection>?destination_ip.?destination_ip<Use protocol>?protocol_type.FILTER(TIMESTAMP>='Yesterday 10 am'ANDTIMESTAMP<='Yesterday 11 am')}".

[0058] Through such semantic understanding and conversion, the server can accurately understand the user's query intention and convert it into a query language that can be executed in the knowledge graph.

[0059] Based on the query language generated by semantic understanding, the server performs query operations in each target probabilistic knowledge graph.

[0060] Assume there are multiple target probabilistic knowledge graphs, each corresponding to a different time period and node set. The server will search for information that meets the query criteria in these graphs in turn.

[0061] Taking the query just mentioned as an example, the server will filter out access records with a source IP address of "192.168.1.100" and a timestamp between 10:00 and 11:00 yesterday morning from all knowledge graphs. It will then extract information such as the corresponding target node and the protocol type used.

[0062] For example, the server finds a record such as "192.168.1.100 accessed 10.0.0.50 at 10:15 yesterday morning, using protocol type 80" and returns it as a query result.

[0063] Based on the results returned by the knowledge query module and the user's query statement, the server generates a context that can be used for large language model queries.

[0064] Suppose the result returned by the knowledge query is "192.168.1.100 accessed 10.0.0.50 at 10:15 yesterday morning, and the protocol type used is 80", and the user's query statement is "Query the access target nodes and the protocol type used by the source IP 192.168.1.100 between 10:00 and 11:00 yesterday morning".

[0065] The server converts this information into a string, for example: "(192.168.1.100, 10.0.0.50, initiate connection), protocol type: 80".

[0066] At the same time, the server will also convert the node's related attribute list into a string, such as "node name: 192.168.1.100, - conditional probability distribution of "source node-protocol type": the value range is [0,1000], represented by a Gaussian mixture model, with parameters (k=1, =100, =1), - category ID: 1".

[0067] In this way, the server builds complete and clear context information to prepare for the processing of large language models.

[0068] The server uses the constructed context information as a character string STR1 and the user query content as a character string STR2 to construct a prompt word.

[0069] Assume that STR1 is "(192.168.1.100, 10.0.0.50, initiate connection), protocol type: 80, node name: 192.168.1.100, - conditional probability distribution of "source node-protocol type": the value range is [0, 1000], represented by Gaussian mixture model, parameters are (k=1, =100, =1), - category ID: 1", STR2 is "query the access target node and the protocol type used by the source IP 192.168.1.100 between 10 and 11 am yesterday."

[0070] The prompt words constructed by the server might be: "Given: (192.168.1.100, 10.0.0.50, initiating connection), protocol type: 80, node name: 192.168.1.100, - conditional probability distribution of "source node-protocol type": the value range is [0, 1000], represented by a Gaussian mixture model with parameters (k=1, k=100, k=1), - category ID: 1; the user's query is to find the target node accessed between 10:00 and 11:00 yesterday morning with a source IP address of 192.168.1.100 and the protocol type used; as a professional knowledge question-answering system, what is your answer?"

[0071] The server then calls a pre-set large language model, such as Tongyi Qianwen or Wenxin Yiyan, and inputs the prompt word. The large language model processes and analyzes the input information and ultimately generates an answer to the user's query request, such as "Between 10:00 and 11:00 yesterday morning, the source IP address 192.168.1.100 accessed the target node 10.0.0.50, using protocol type 80."

[0072] Through this series of steps and processing, the server can effectively mine knowledge from network access log data and provide users with accurate and useful answers.

[0073] In the embodiment of the present invention, dividing the original network access log data according to a preset rule to obtain multiple network access log data subsets can be implemented through the following examples.

[0074] Divide the original network access log data by node according to the source IP address and the destination IP address to obtain the plurality of network access log data subsets; or

[0075] Dividing the original network access log data according to a preset time interval to obtain the plurality of network access log data subsets; or

[0076] The original network access log data is divided according to the source IP address, the destination IP address, and the preset time interval to obtain the multiple network access log data subsets.

[0077] In an embodiment of the present invention, for example, the server receives a large amount of raw network access log data and first begins to divide the nodes according to the source IP address and the destination IP address. Assume that the raw data contains the following records:

[0078] Record 1: The source IP address is 10.10.10.1, and the destination IP address is 20.20.20.2.

[0079] Record 2: The source IP address is "10.10.10.1" and the destination IP address is "30.30.30.3".

[0080] Record 3: The source IP address is "20.20.20.2" and the destination IP address is "10.10.10.1".

[0081] The server groups records with the same source and destination IP address combination together. For example, records 1 and 2 are grouped together because they both have the same source IP address, "10.10.10.1." Record 3, on the other hand, is grouped together because it has a different source and destination IP combination, "20.20.20.2" and "10.10.10.1."

[0082] Assume that the preset time interval is every 2 hours. The original network access log data contains the following records, each with an accurate timestamp:

[0083] Record 4: timestamp is 08:00:00;

[0084] Record 5: timestamp is 08:30:00;

[0085] Record 6: timestamp is 09:50:00;

[0086] Record 7: timestamp is 10:10:00;

[0087] The server divides the data by timestamp. Records 4, 5, and 6 between 08:00:00 and 09:59:59 are divided into one subset, while record 7 between 10:00:00 and 11:59:59 is divided into another subset.

[0088] For example, the preset time interval is still every 2 hours, and both the source IP address and the destination IP address are considered.

[0089] The original data contains the following records:

[0090] Record 8: The source IP address is "30.30.30.3", the destination IP address is "40.40.40.4", and the timestamp is 12:00:00;

[0091] Record 9: Source IP address is 30.30.30.3, destination IP address is 40.40.40.4, and timestamp is 13:30:00.

[0092] Record 10: Source IP address is 50.50.50.5, destination IP address is 60.60.60.6, and timestamp is 13:00:00.

[0093] The server first performs a preliminary classification based on source and destination IP addresses. Records 8 and 9, because they have the same source and destination IP addresses, are grouped together. Then, the server further divides the data based on time intervals. Records 8, 9, and 10 between 12:00:00 and 13:59:59 are grouped together.

[0094] Through these different data partitioning methods, the server can effectively decompose the original network access log data into multiple subsets with specific characteristics and patterns, providing a clear and organized data foundation for subsequent knowledge mining and application work, and helping to analyze and process data more accurately and efficiently.

[0095] In an embodiment of the present invention, the construction of a probabilistic grammar model based on a knowledge graph for each subset of the network access log data may be performed through the following examples.

[0096] Obtaining an initial knowledge image corresponding to a target network access log data subset; the target network access log data subset is any data subset among multiple network access log data subsets;

[0097] Configuring parameterized probabilistic attributes for the nodes and edges of the initial knowledge graph, and limiting the nodes of the initial knowledge graph to the source IP node and the target node, where the relationship between the source IP node and the target node is limited to initiating a connection;

[0098] The target network access log data subset is scanned line by line to obtain multiple triples with the source IP node, the target node, and the initiated connection as relationships, until a probabilistic grammar model based on the knowledge graph is completed. For each newly added triple, the relationship strength between the corresponding nodes increases by one unit strength.

[0099] In the embodiment of the present invention, illustratively, the server first obtains a subset of target network access log data. Assume that this subset includes the following records:

[0100] Record 1: Source IP address is "192.168.1.10", destination IP address is "10.0.0.50", protocol type is "TCP", and packet size is "500 bytes";

[0101] Record 2: Source IP address is "192.168.1.10", destination IP address is "10.0.0.50", protocol type is "UDP", and packet size is "800 bytes";

[0102] Record 3: Source IP address is "192.168.1.20", destination IP address is "10.0.0.50", protocol type is "TCP", and packet size is "600 bytes";

[0103] The server obtains the initial knowledge graph corresponding to the target network access log data subset. The initial knowledge graph does not yet have probability attributes configured.

[0104] Next, the server configures parameterized probabilistic attributes for the nodes and edges of the initial knowledge graph. In this example, the source IP nodes "192.168.1.10" and "192.168.1.20" and the target node "10.0.0.50" are defined as nodes in the knowledge graph, and the relationship between them is defined as "initiating a connection."

[0105] The server then scans this target network access log data subset line by line. When it encounters record 1, it adds the triple "<192.168.1.10, 10.0.0.50, initiates connection>," increasing the strength of the relationship between the two nodes by one unit. It then scans record 2. Because the source and destination IP addresses are identical to the previous triple, the strength of the relationship between the two nodes is increased again.

[0106] When record 3 is scanned, a new triplet “<192.168.1.20,10.0.0.50,initiate connection>” is added, and the strength of the relationship between the two nodes is increased accordingly.

[0107] During this process, the server not only records the connection relationships and strengths between nodes but also models probabilities. For example, for the conditional probability distribution of "protocol type - packet size," the server discovered that when the source IP node "192.168.1.10" connects to "10.0.0.50" multiple times, it is more likely to use the "TCP" protocol and have a packet size between 400 and 600 bytes. For the joint probability distribution of "source node port - destination node port," the server counts the common ports used by the source IP node "192.168.1.10" and the common ports received by the destination node "10.0.0.50."

[0108] By scanning and analyzing this subset of target network access log data line by line, the server continuously refines the probabilistic grammar model. Once all records have been scanned, the probabilistic grammar model based on the knowledge graph is constructed. This model contains rich information such as the connection relationships between nodes, the strength of relationships, and various probability distributions, providing strong support for subsequent analysis and application.

[0109] For example, this model can be used to infer information such as the protocol type, packet size, and port that "192.168.1.10" may use when it initiates a connection to "10.0.0.50" again, which is helpful for network optimization, security monitoring, and other tasks.

[0110] In an embodiment of the present invention, the probabilistic grammar model includes:

[0111] The conditional probability distribution of protocol type-packet size is: P1=P(c|S i ), is about the source node S i A one-dimensional probability distribution, where S i represents the i-th source node, c represents the size of the data packet;

[0112] The joint probability distribution of source node port and target node port is: P2=P(Dp|S i .p), is about the source node S i A two-dimensional probability distribution, where S i .p represents node S i All ports of the target node, Dp represents all ports of the target node;

[0113] The conditional probability distribution of source node-protocol type is: P3=P(r|S i ), is the source node S i A one-dimensional probability distribution of , where r represents the protocol type;

[0114] The conditional probability distribution of target node-protocol type is: P4=P(r|D i ), is the target node D i A one-dimensional probability distribution, where r represents the protocol type, and D i represents the i-th target node;

[0115] The conditional probability distribution of source node-destination node is: P5=P(D|S i ), is the source node S i A one-dimensional probability distribution of ;

[0116] The conditional probability distribution of the target node-source node is: P6=P(S|D i ), is the target node D i A one-dimensional probability distribution of ;

[0117] The prior probability distribution of source node i is:

[0118] In an embodiment of the present invention, the step of repeatedly executing the step of constructing a probabilistic grammar model based on the knowledge graph for each subset of the network access log data, and completing the graph merging based on the original count, the starting time and the ending time to obtain a knowledge graph based on time and space merging can be implemented through the following examples.

[0119] Repeat the step of constructing a probabilistic grammar model based on a knowledge graph for each subset of the network access log data to obtain a probabilistic knowledge graph for different nodes and different time periods corresponding to each subset of the network access log data;

[0120] Group the nodes by the start time and the end time to obtain multiple candidate knowledge graphs, and place all nodes in the same time period in the same knowledge graph to complete spatial merging;

[0121] Based on the raw counts, the probability distribution of each node in a preset time range is obtained to complete time merging;

[0122] Based on the spatial merging and the temporal merging, a knowledge graph based on temporal and spatial merging is obtained.

[0123] In an embodiment of the present invention, the server first repeatedly executes the steps of constructing a probabilistic grammar model based on a knowledge graph on multiple subsets of network access log data. Assume there are three data subsets: subset A, subset B, and subset C.

[0124] Subset A covers access logs from 8:00 AM to 9:00 AM, including source IP nodes "10.10.10.1" and "10.10.10.2", and destination IP nodes "20.20.20.1" and "20.20.20.2".

[0125] Subset B covers access logs from 9:00 AM to 10:00 AM, including source IP nodes "10.10.10.1" and "10.10.10.3" and destination IP nodes "20.20.20.1" and "20.20.20.3".

[0126] Subset C covers access logs from 10:00 AM to 11:00 AM, including source IP nodes "10.10.10.2" and "10.10.10.3" and destination IP nodes "20.20.20.2" and "20.20.20.3".

[0127] By building a probabilistic grammar model, the server obtains the probabilistic knowledge graph corresponding to different nodes and different time periods of each subset.

[0128] Next, the server groups the nodes by start and end time. For the nodes in subset A, their start time is 8:00 and their end time is 9:00. Similarly, the nodes in subset B have a start time of 9:00 and an end time of 10:00, and the nodes in subset C have a start time of 10:00 and an end time of 11:00.

[0129] The server obtains multiple candidate knowledge graphs based on this time information. For example, the nodes from 8 to 9 o'clock are placed in one candidate knowledge graph, the nodes from 9 to 10 o'clock are placed in another candidate knowledge graph, and the nodes from 10 to 11 o'clock are placed in yet another candidate knowledge graph. Then, all nodes in the same time period are placed in the same knowledge graph to complete the spatial merging. For example, in the time period from 9 to 10 o'clock, the node "10.10.10.1" appears in both subset A and subset B. Then, during the spatial merging, its information in the two subsets is integrated into the same knowledge graph.

[0130] After completing spatial merging, the server uses the raw counts to determine the probability distribution of each node within a preset time range, completing temporal merging. For example, if the raw count for node "10.10.10.1" in subset A is 50 times and the raw count in subset B is 30 times, the server calculates the probability distribution of node "10.10.10.1" appearing within the longer time range of 8:00 to 10:00.

[0131] Ultimately, through spatial and temporal merging, the server obtains a knowledge graph based on both spatial and temporal merging. This knowledge graph integrates node information and probability distribution across different subsets within different time periods, more comprehensively and accurately reflecting the patterns and characteristics of network access. For example, this merged knowledge graph clearly shows which source and destination IP nodes frequently connect throughout the morning, as well as the probability trends of these connections. This provides valuable guidance for network monitoring, optimization, and security analysis.

[0132] In an embodiment of the present invention, node clustering is performed on the merged knowledge graph to obtain a target probability knowledge graph corresponding to each subset of the network access log data, which can be implemented through the following examples.

[0133] The merged knowledge graph is clustered by nodes using an unsupervised clustering method to obtain a target probability knowledge graph corresponding to each subset of the network access log data. After clustering is completed, the category, distance from the class center, and maximum distance between classes are set in the node information to record the clustering information of the node.

[0134] In an embodiment of the present invention, illustratively, the server obtains a merged knowledge graph, assuming that the knowledge graph contains a large number of nodes, and each node has a series of probability distribution features.

[0135] First, the server uses the k-NN algorithm, an unsupervised clustering method, to cluster nodes. The server calculates the feature vector for each node. These feature vectors contain various probability distribution information about the node, such as the probability distribution of "protocol type-packet size" and the joint probability distribution of "source node port-destination node port."

[0136] Assume that the feature vector of node A contains the following probability distribution: the probability distribution of "protocol type-packet size" shows that the packet size under a specific protocol is mainly concentrated in the range of 500-800 bytes; the joint probability distribution of "source node port-destination node port" shows that the commonly used source port is 8080 and the destination port is 80.

[0137] The server calculates the distance between node A and other nodes based on the configured distance calculation method. For example, when calculating the distance between node A and node B, the distance calculation takes into account the difference in probability distribution and the difference in the absolute value of the probability values.

[0138] If the distance threshold set by the server is 10, and after calculation it is found that the distance between node A and node B is 8, which is less than the distance threshold, then node A and node B are considered similar and may be classified into the same category.

[0139] By continuously calculating and comparing the distances between nodes, the server completes the clustering process. Assume that three cluster categories are finally obtained: category 1, category 2, and category 3.

[0140] For clustered nodes, the server sets the relevant clustering information in the node information. For example, if node A is classified as cluster 1, the server calculates the distance between node A and the center of cluster 1 and records it. The server also calculates the maximum distance between nodes in cluster 1 and records it in node A's information.

[0141] Assume that the distance between node A and the center of category 1 is 5, and the maximum distance between nodes in category 1 is 15, then the information of node A will be set as "category: 1", "distance from category center: 5", and "maximum distance between categories: 15".

[0142] In this way, the server clusters the nodes in the merged knowledge graph and records detailed clustering information for each node, thereby obtaining a target probabilistic knowledge graph corresponding to each subset of network access log data. The clustering information in this target probabilistic knowledge graph can help better understand and analyze the patterns and characteristics of network access. For example, it can be found that certain types of nodes have similar access behaviors, or that nodes in certain cluster categories differ significantly from those in other categories, providing valuable reference for further network optimization, security monitoring, and other tasks.

[0143] In an embodiment of the present invention, the knowledge graph query language package for obtaining the query request input by the user can be implemented through the following examples.

[0144] Get the query request entered by the user;

[0145] Converting the query request into the knowledge graph query language containing entity and relationship query information;

[0146] The context construction based on the knowledge graph query language and the node-related results to obtain context information for large language model query can be implemented through the following examples.

[0147] The knowledge graph query language and the node-related results are converted into character strings, and the character strings are used as the context information.

[0148] In the embodiment of the present invention, for example, the server first obtains a query request input by the user. Assume that the query request input by the user is: "Find the destination IP addresses and the protocol types accessed by the source IP address 192.168.0.10 between 9:00 AM and 10:00 AM yesterday."

[0149] After receiving this query, the server converts it into a knowledge graph query language containing entity and relationship query information. The server identifies the key entities, namely the source IP address "192.168.0.10" and the time range "Yesterday morning 9:00 to 10:00," as well as the query relationships "destination IP address" and "protocol type used."

[0150] The server converts this information into a knowledge graph query language. For example, a SPARQL query statement may be: "SELECT DISTINCT?destination_ip?protocol_type WHERE{<192.168.0.10><Initiate connection>?destination_ip.?destination_ip<Use protocol>?protocol_type.FILTER(TIMESTAMP>='Yesterday's 9 am' AND TIMESTAMP<='Yesterday's 10 am')}".

[0151] Next, the server executes a query operation in each target probabilistic knowledge graph based on this knowledge graph query language, obtaining node-related results related to the query request. Suppose the query result is "Source IP address 192.168.0.10 accessed destination IP address 10.0.0.50 at 9:30 yesterday morning, using TCP as the protocol."

[0152] The server then constructs a context based on the knowledge graph query language and the node-related results. The server converts the knowledge graph query language "SELECT DISTINCT?destination_ip?protocol_type WHERE {<192.168.0.10><Initiate connection>?destination_ip.?destination_ip<Used protocol>?protocol_type.FILTER(TIMESTAMP>='Yesterday's 9:00 AM' AND TIMESTAMP<='Yesterday's 10:00 AM')}" and the node-related result "Source IP address 192.168.0.10 accessed destination IP address 10.0.0.50 at 9:30 AM yesterday, using TCP as the protocol type" into a string.

[0153] The converted string may be: "Knowledge graph query language: SELECT DISTINCT?destination_ip?protocol_type WHERE{<192.168.0.10><Initiate connection>?destination_ip.?destination_ip<Use protocol>?protocol_type.FILTER(TIMESTAMP>='Yesterday's 9 am'ANDTIMESTAMP<='Yesterday's 10 am')}Node related results: The source IP address 192.168.0.10 accessed the destination IP address 10.0.0.50 at 9:30 am yesterday, and the protocol type used was TCP".

[0154] The server uses this string as context information for large language model query, so that it can subsequently construct prompt words based on this context information and the user's query request, and call the preset large language model for processing, thereby obtaining an accurate answer to the user's query request.

[0155] Through such detailed and precise processing flow, the server can effectively understand user needs, extract valuable information from complex network access log data, and respond to user queries in a clear and useful manner.

[0156] In order to more clearly describe the solution provided by the implementation of the present invention, a relatively complete implementation method is provided below.

[0157] This embodiment uses more than 30,000 access data records of a company's intranet node as input to demonstrate a specific implementation example of the present invention.

[0158] Knowledge mining and expression system

[0159] The data used in this embodiment are as follows Figure 1 As shown, the data collection lasts for one hour and involves no more than 200 intranet nodes. Extranet nodes are divided into approximately 200 countries and regions based on IP location.

[0160] Data partitioning: This module sets the slicing time to 1 hour, so data with a duration of 1 hour is not partitioned;

[0161] Probabilistic grammar model construction:

[0162] For each row of data, assuming its content is "node 1 initiates a connection to node 2", a triple of the form "<node 1, node 2, initiate connection>" is added to the knowledge graph.

[0163] For example, in the currently scanned row, the source node is "192.168.28.31" and the destination node is "219.151.145.151" (the location is China and the code is 86). Then, a new triplet <"192.168.28.31", "86", initiates a connection>, and the strength value of the relationship between the two nodes is increased by 1.

[0164] For each node, the following probabilities are calculated:

[0165] The conditional probability distribution of “source node-protocol type” P3=P(r|S i ), the current node is the source node S i is the one-dimensional probability distribution of the protocol type, where r represents the protocol type and its value range is between 0 and 255;

[0166] The conditional probability distribution of “target node-protocol type” P4=P(r|D i ), the current node is the target node D i The one-dimensional probability distribution of the protocol type is, where r represents the protocol type, and D i represents the i-th target node;

[0167] The conditional probability distribution of “source node - target node” P5=P(D|S i ), the current node is the source node S i , the one-dimensional probability distribution when selecting the target node D;

[0168] Prior probability distribution of the current node appearing as source node i

[0169] The prior probability distribution of the current node appearing as the target node i

[0170] Considering that the sample spaces of the above five probabilities are relatively small, this embodiment uses one-dimensional vectors to represent them. Specifically, for each node, probability distributions P3 and P4 are each represented by a 1x256-dimensional normalized (element sums to 1) vector. Since the total number of nodes does not exceed 512, probability distribution P5 is represented by a 1x512-dimensional normalized vector.

[0171] For all nodes, the probability distributions P7 and P8 are also represented by 1x512 dimensional normalized vectors.

[0172] Graph merging:

[0173] Since this embodiment doesn't partition the data, this module only performs spatial merging of the knowledge graph. Specifically, in step c (probabilistic grammar model construction), triples were obtained by scanning the log records line by line. Therefore, in this module, all triples need only be stored in the same database (this embodiment uses Triple Store to store triples).

[0174] Node clustering:

[0175] In this embodiment, for each node i, a 1x1026-dimensional (256+256+512+2=1026) feature vector about the node can be established. Where P7(i) and P8(i) are the specific values ​​of the node taken from the probability distribution P7 and P8. The distance between nodes i and j is defined as

[0176]

[0177] When the distance between two nodes is obtained, the K-NN algorithm is used to complete node clustering.

[0178] Knowledge application system:

[0179] Semantic understanding:

[0180] The user question types supported by this embodiment include but are not limited to

[0181] Ask for information about a node mentioned in the log, such as "Do you think node 192.168.10.23 is a server or a user terminal?", "Is node a an active node?", "Which regions' external networks does node b most frequently access?"

[0182] Ask about the behavioral similarity between any two nodes mentioned in the log record, for example, "Are nodes A and B similar in behavior?", "Find all nodes that behave similarly to node A"

[0183] Query the strength of the association between any two nodes mentioned in the log records, for example, "Which external network region does node a visit the most?", "Which internal network node visits node c the most?"

[0184] Ask for statistics in the log records, such as "What are the top five most active IP addresses in this hour?", "Which category contains the most nodes?"

[0185] Knowledge query:

[0186] This embodiment uses a large language model to implement this module. The main principle is to set up a semantic classifier and a problem solver. The semantic classifier decides which problem solver to call, while the problem solver sets solutions for typical problems. Specific categories of semantic classifiers include "node information query," "node attribute question and answer," "relationship information query," "relationship attribute question and answer," "statistical information query," "statistical information question and answer," etc.

[0187] Taking the "node information query" problem as an example, this module constructs a solver using the following method: First, it defines the string variables STR_TABLE (used to store the detailed description of the data table), STR_NODE (used to store the attribute descriptions and data table fields of the source and target nodes), and STR_QUERY (used to store the user's query text). Then, it uses the prompt words shown in Table 1 to invoke a large language model with strong command generation capabilities (for example, qwen2-72B).

[0188] Table 1

[0189]

[0190] Based on the query language generated by semantic understanding, the generated code is executed in all probabilistic knowledge graphs to extract the most relevant information fragments to solve this type of problem.

[0191] Context construction:

[0192] Define the return result string of the knowledge query module as STR_RES (for example, information about a certain node is queried), define the semantic classification of the user query (for example, "node information query") as the string variable STR_Q_TYPE, and convert the node attribute list and relationship list into the string STR_RES_NODE. The context string variable is

[0193] STR_Context = "[Known user intent is STR_Q_TYPE, knowledge graph query result is STR_RES, and information about related nodes and relationships is STR_RES_NODE]".

[0194] Note that string variables should be replaced with corresponding texts when actually used.

[0195] Content Generator:

[0196] This module is implemented with the content generation capability of the large language model qwen2-72B. The specific method is to call the large model using the following prompt words:

[0197] "Given the STR_Context, the user query is STR_QUERY. You are a knowledge application assistant for log data. Please provide an appropriate response. Only the response is required, no other content is required."

[0198] Please refer to Figure 2 , Figure 2 An embodiment of the present invention provides a knowledge mining and application device 110 for network access log data, including:

[0199] The mining module 1101 is configured to partition the original network access log data according to preset rules to obtain a plurality of network access log data subsets; construct a probabilistic grammatical model based on a knowledge graph for each of the network access log data subsets, wherein the knowledge graph is composed of a triple including a source IP node, a target node, and an initiated connection as a relationship, wherein the source IP node and the target node include a unique identifier, a probability parameter, an original count, a start time, and an end time; repeatedly execute the steps of constructing a probabilistic grammatical model based on a knowledge graph for each of the network access log data subsets, and complete graph merging based on the original count, the start time, and the end time to obtain a knowledge graph merged based on time and space; perform node clustering on the merged knowledge graph to obtain a target probabilistic knowledge graph corresponding to each of the network access log data subsets;

[0200] Application module 1102 is used to obtain the knowledge graph query language corresponding to the query request input by the user; perform a query operation based on the knowledge graph query language in each of the target probabilistic knowledge graphs to obtain node-related results corresponding to each of the target probabilistic knowledge graphs; construct a context based on the knowledge graph query language and the node-related results to obtain context information for large language model query; construct a prompt word based on the context information and the query request, and call a preset large language model to perform processing to obtain the answer content for the query request.

[0201] It should be noted that the implementation principles of the aforementioned knowledge mining and application device 110 for network access log data can be referenced with the implementation principles of the aforementioned knowledge mining and application method for network access log data, and will not be elaborated upon here. It should be understood that the division of the various modules of the above device is merely a division of logical functions. In actual implementation, they can be fully or partially integrated into a single physical entity, or physically separated. Furthermore, these modules can be implemented entirely in the form of software called by a processing element; or entirely in the form of hardware; or some modules can be implemented in the form of software called by a processing element, and some modules can be implemented in the form of hardware. For example, the knowledge mining and application device 110 for network access log data can be a separate processing element, or it can be integrated into a chip of the above device. Furthermore, it can be stored in the form of program code in the memory of the above device, and called by a processing element of the above device to execute the functions of the above knowledge mining and application device 110 for network access log data. The implementation of other modules is similar. Furthermore, these modules can be fully or partially integrated together, or implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each module above may be completed by an integrated logic circuit of hardware in a processor element or by instructions in the form of software.

[0202] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented by scheduling program code on a processing element, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0203] The embodiment of the present invention provides a computer device 100, which includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device 100 executes the aforementioned knowledge mining and application device 110 for network access log data. Figure 3 As shown, Figure 3This is a structural block diagram of a computer device 100 provided in an embodiment of the present invention. The computer device 100 includes a knowledge mining and application device 110 for network access log data, a memory 111 , a processor 112 , and a communication unit 113 .

[0204] In order to realize the transmission or interaction of data, the memory 111, the processor 112 and the communication unit 113 are electrically connected to each other directly or indirectly. For example, the electrical connection between these elements can be realized through one or more communication buses or signal lines. The knowledge mining and application device 110 for network access log data includes at least one software function module that can be stored in the memory 111 in the form of software or firmware or solidified in the operating system (OS) of the computer device 100. The processor 112 is used to execute the knowledge mining and application device 110 for network access log data stored in the memory 111, such as the software function modules and computer programs included in the knowledge mining and application device 110 for network access log data.

[0205] An embodiment of the present invention provides a readable storage medium, which includes a computer program. When the computer program is executed, the computer device where the readable storage medium is located is controlled to execute the aforementioned knowledge mining and application device 110 for network access log data.

[0206] For illustrative purposes, the foregoing description has been made with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the present disclosure to the precise forms disclosed. Numerous modifications and variations are possible in light of the above teachings. These embodiments have been selected and described in order to best illustrate the principles of the present disclosure and its practical application, thereby enabling those skilled in the art to best utilize the present disclosure and to utilize various embodiments with various modifications as appropriate for the specific application contemplated.

Claims

1. A knowledge mining and application method for network access log data, characterized in that: include: Divide the original network access log data according to preset rules to obtain multiple network access log data subsets; Constructing a probabilistic grammar model based on a knowledge graph for each subset of the network access log data, wherein the knowledge graph is composed of triples including a source IP node, a target node, and an initiated connection as a relationship, wherein the source IP node and the target node include a unique identifier, a probability parameter, a raw count, a start time, and an end time; Repeating the step of constructing a probabilistic grammar model based on a knowledge graph for each subset of the network access log data, and completing graph merging based on the original count, the start time, and the end time to obtain a knowledge graph merged based on time and space; Performing node clustering on the merged knowledge graph to obtain a target probability knowledge graph corresponding to each subset of the network access log data; Obtain the knowledge graph query language corresponding to the query request entered by the user; Performing a query operation based on the knowledge graph query language in each of the target probabilistic knowledge graphs to obtain node-related results corresponding to each of the target probabilistic knowledge graphs; Context is constructed based on the knowledge graph query language and the node-related results to obtain context information for large language model query; A prompt word is constructed according to the context information and the query request, and a preset large language model is called to perform processing to obtain an answer content for the query request.

2. The method according to claim 1, characterized in that The original network access log data is divided according to the preset rules to obtain multiple network access log data subsets, including: Divide the original network access log data by node according to the source IP address and the destination IP address to obtain the plurality of network access log data subsets; or Dividing the original network access log data according to a preset time interval to obtain the plurality of network access log data subsets; or The original network access log data is divided according to the source IP address, the destination IP address, and the preset time interval to obtain the multiple network access log data subsets.

3. The method according to claim 1, characterized in that The constructing of a probabilistic grammar model based on a knowledge graph for each subset of the network access log data includes: Obtaining an initial knowledge graph corresponding to a target network access log data subset; the target network access log data subset is any data subset among multiple network access log data subsets; Configuring parameterized probabilistic attributes for the nodes and edges of the initial knowledge graph, and limiting the nodes of the initial knowledge graph to the source IP node and the target node, where the relationship between the source IP node and the target node is limited to initiating a connection; The target network access log data subset is scanned line by line to obtain multiple triples with the source IP node, the target node, and the initiated connection as relationships, until a probabilistic grammar model based on the knowledge graph is completed. For each newly added triple, the relationship strength between the corresponding nodes increases by one unit strength.

4. The method according to claim 3, characterized in that The probabilistic grammar model includes: The conditional probability distribution of protocol type-packet size is: P1=P(c|S i ), is about the source node S i A one-dimensional probability distribution, where S i represents the i-th source node, c represents the size of the data packet; The joint probability distribution of source node port and target node port is: P2=P(Dp|S i .p), is about the source node S i A two-dimensional probability distribution, where S i .p represents node S i All ports of the target node, Dp represents all ports of the target node; The conditional probability distribution of source node-protocol type is: P3=P(r|S i ), is the source node S i A one-dimensional probability distribution of , where r represents the protocol type; The conditional probability distribution of target node-protocol type is: P4=P(r|D i ), is the target node D i A one-dimensional probability distribution, where r represents the protocol type, and D i represents the i-th target node; The conditional probability distribution of source node-destination node is: P5=P(D|S i ), is the source node S i A one-dimensional probability distribution of ; The conditional probability distribution of the target node-source node is: P6=P(S|D i ), is the target node D i A one-dimensional probability distribution of ; The prior probability distribution of source node i is: The prior probability distribution of target node i is:

5. The method according to claim 1, wherein The step of repeatedly performing the step of constructing a probabilistic grammar model based on a knowledge graph for each subset of the network access log data, and completing graph merging based on the original count, the start time, and the end time to obtain a knowledge graph merged based on time and space, includes: Repeat the step of constructing a probabilistic grammar model based on a knowledge graph for each subset of the network access log data to obtain a probabilistic knowledge graph for different nodes and different time periods corresponding to each subset of the network access log data; Group the nodes by the start time and the end time to obtain multiple candidate knowledge graphs, and place all nodes in the same time period in the same knowledge graph to complete spatial merging; Based on the raw counts, the probability distribution of each node in a preset time range is obtained to complete time merging; Based on the spatial merging and the temporal merging, a knowledge graph based on temporal and spatial merging is obtained.

6. The method according to claim 1, wherein The node clustering is performed on the merged knowledge graph to obtain a target probability knowledge graph corresponding to each subset of the network access log data, including: The merged knowledge graph is clustered by nodes using an unsupervised clustering method to obtain a target probability knowledge graph corresponding to each subset of the network access log data. After clustering is completed, the category, distance from the class center, and maximum distance between classes are set in the node information to record the clustering information of the node.

7. The method according to claim 1, characterized in that The step of obtaining the knowledge graph query language corresponding to the query request input by the user includes: Get the query request entered by the user; Converting the query request into the knowledge graph query language containing entity and relationship query information; The context construction is performed based on the knowledge graph query language and the node-related results to obtain context information for large language model query, including: The knowledge graph query language and the node-related results are converted into character strings, and the character strings are used as the context information.

8. A knowledge mining and application device for network access log data, characterized in that: include: A mining module is used to divide the original network access log data according to preset rules to obtain multiple network access log data subsets; For each subset of the network access log data, a probabilistic grammatical model based on a knowledge graph is constructed, where the knowledge graph is composed of a triple including a source IP node, a target node, and an initiated connection as a relationship, where the source IP node and the target node include a unique identifier, a probability parameter, an original count, a start time, and an end time; repeatedly executing the step of constructing a probabilistic grammatical model based on the knowledge graph for each subset of the network access log data, and completing graph merging based on the original count, the start time, and the end time to obtain a knowledge graph merged based on time and space; Performing node clustering on the merged knowledge graph to obtain a target probability knowledge graph corresponding to each subset of the network access log data; An application module is used to obtain the knowledge graph query language corresponding to the query request input by the user; perform a query operation based on the knowledge graph query language in each of the target probabilistic knowledge graphs to obtain node-related results corresponding to each of the target probabilistic knowledge graphs; Context is constructed based on the knowledge graph query language and the node-related results to obtain context information for large language model query; A prompt word is constructed according to the context information and the query request, and a preset large language model is called to perform processing to obtain an answer content for the query request.

9. A computer device, characterized in that: The computer device includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device executes the method according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium includes a computer program, and when the computer program is executed, the computer device where the readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

Images