Network unknown asset type identification method, system and device and storage medium

Abstract

The invention belongs to the technical field of aviation network asset identification, and discloses a network unknown asset type identification method, system and device and a storage medium, the method mainly comprises the processes of network node access relation extraction, network node representation, access degree and PageRank calculation, asset type identification and the like, node representation is efficient, and asset type identification precision is high. The problems that a traditional network asset maintenance method is low in accuracy and poor in portability, depends too much on experience of operation and maintenance engineers, cannot be reused and the like are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of aviation network asset classification, and specifically relates to a method, system, device and storage medium for identifying network unknown asset types based on the Word2vec network node characterization method. Background Art

[0002] As the scale of the network continues to expand, some unknown types of assets will appear in the network environment due to untimely filing. The existence of unknown types of assets, on the one hand, is inconvenient for the overall management of assets, and on the other hand, they may be illegal assets, threatening the overall security of the network space. At present, unknown assets are mainly classified by operation and maintenance engineers through the division of network segments or through log-related rules. On the one hand, this method has low accuracy and poor portability. It is too dependent on the experience of operation and maintenance engineers and cannot be reused. Summary of the invention

[0003] The present invention application proposes a method, system, device and storage medium for identifying network unknown asset types, which are used to solve the problems of low accuracy, poor portability, over-reliance on the experience of operation and maintenance engineers and inability to be reused of traditional network asset maintenance methods.

[0004] In order to achieve the above invention objectives, the technical solutions adopted in this application are as follows:

[0005] A method for identifying unknown network asset types comprises the following steps:

[0006] Extract access relationships through traffic logs;

[0007] Establish a Word2vec node representation model based on access relations;

[0008] Calculate node features of other dimensions;

[0009] Concatenate the node representation vector with features of other dimensions;

[0010] Node asset type classification model training;

[0011] Node asset type identification.

[0012] As a further solution of the present invention: the access relationship extraction is specifically: according to the traffic log, the access relationship between the nodes in the network is extracted, and an access relationship document in the format of "source node, destination node" is formed, and a network topology diagram is constructed.

[0013] As a further solution of the present invention: the establishment of the Word2vec node representation model: based on the access relationship document, the Word2vec method is used to generate a semantic representation vector of the node, which can map the node in a high-dimensional semantic space; based on the network topology graph, a deep adjacency relationship document of the node is formed by a random walk method, and the topological adjacency semantic representation of the node is calculated on this document using the Word2vec method.

[0014] As a further solution of the present invention: the extraction of node features of other dimensions mainly includes: calculating the out-degree, in-degree and page rank value of the node in the network topology diagram. These three features can represent the importance and structural similarity of the node in the network.

[0015] As a further solution of the present invention: the feature vector splicing mainly includes: performing dimensionless operation or normalization operation on features of other dimensions, and then splicing them with the semantic representation vector and the topological adjacent semantic representation vector to form a final feature vector.

[0016] As a further solution of the present invention: the classification model training: according to different performance requirements, a suitable classification model is selected for training. This method adopts a multi-classification logistic regression model.

[0017] As a further solution of the present invention: the network node characterization method based on Word2vec specifically includes the following steps:

[0018] Get traffic data;

[0019] Extract the source IP field and destination IP field in the traffic data and generate the topological structure of the network nodes;

[0020] Construct sample data based on the source IP field and the destination IP field;

[0021] Based on the network topology of nodes, a random walk method is used to generate deep adjacency relationship sample data;

[0022] Based on the two sample data, two different Word2vec models are input for training respectively to obtain the semantic vector and topological adjacency semantic vector corresponding to each IP;

[0023] Based on two semantic vectors, vector splicing is performed.

[0024] As a further solution of the present invention: after extracting the source IP field and the destination IP field in the traffic data, the method further includes: deleting the default values ​​in both the source IP field and the destination IP field.

[0025] As a further solution of the present invention: the construction of sample data based on the source IP field and the destination IP field specifically includes: writing the source IP field and the destination IP field in pairs into a txt file with spaces as separators, with each traffic record in a separate line, to construct the sample data.

[0026] As a further solution of the present invention: the node-based network topology structure generates deep adjacency relationship sample data, including: taking a node as the starting node, randomly walking in the topology map according to a certain step length, generating an IP sequence of a certain length, using spaces as separators, writing into a txt file, with each IP sequence as a line, to construct sample data.

[0027] A network unknown asset type identification system, comprising:

[0028] A feature extraction module is used to obtain traffic data, extract the source IP field and the destination IP field in the traffic data, and generate a network topology diagram of the node;

[0029] Data set construction module, based on the source IP field and the destination IP field, sample data construction; sample data construction based on the network topology diagram;

[0030] The calculation and model extraction module inputs the word2ve model for training based on sample data to obtain the semantic vector and adjacent semantic vector corresponding to each IP;

[0031] Feature concatenation module, used to de-dimensionalize and concatenate representation vectors and other dimensional vectors;

[0032] The unknown asset classification module is used for model training and model prediction. Model training is to use known type data as samples to train a classifier and evaluate the classifier; model prediction is to input unknown type data into the classifier and input the prediction type.

[0033] An electronic device comprising:

[0034] processor;

[0035] a memory for storing instructions executable by the processor;

[0036] The processor is used to read the executable instructions from the memory and execute the instructions to implement the steps of the above-mentioned method for identifying the type of unknown network assets.

[0037] A computer-readable storage medium, characterized in that the storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned method for identifying the type of unknown network assets are implemented.

[0038] Compared with the prior art, the beneficial effects of this application are:

[0039] 1. The unsupervised Word2vec method is used to semantically represent network nodes, solving the problem that traditional artificial features rely on the experience of operation and maintenance engineers.

[0040] 2. Based on the topological graph, a random walk method is used to form a deep adjacency relationship document of the nodes, which solves the problem that deep relationships cannot be learned in latent representation learning.

[0041] 3. Feature vector splicing can combine features extracted by different dimensions, different dimensions, and different methods, which can effectively improve the classification accuracy and solve the problems of low classification accuracy and difficulty in feature expansion.

[0042] 4. The semantic representation and page rank methods of network nodes are both unsupervised and applicable to entity representation and classification problems in various networks, solving the problems of portability and reusability of the methods.

[0043] 5. This method can be used in combination with multiple classifiers and has a wider range of application scenarios.

[0044] The present application is further described below in conjunction with the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0046] In the attached picture:

[0047] Figure 1 This is a schematic diagram of a method for identifying unknown network asset types based on Word2vec in this application;

[0048] Figure 2 This is a schematic diagram of the network node representation method based on Word2vec in this application;

[0049] Figure 3 This is a flow chart of the method for identifying unknown network asset types based on Word2vec in this application;

[0050] Figure 4 This is a schematic diagram of the structure of a network unknown asset type recognition device based on Word2vec in this application;

[0051] Figure 5 This is a performance test diagram for this application.

[0052] The reference numerals in the figure are: 31 - feature extraction module; 32 - data set construction module; 33 - calculation and model extraction module; 34 - feature splicing module, 35 - unknown asset classification module. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0054] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the methods and systems according to various embodiments of the present disclosure. It should be noted that each box in the flowchart or block diagram may represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code may include one or more executable instructions for implementing the logical functions specified in the various embodiments. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession may actually be executed substantially in parallel, or they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the flowchart and / or block diagram, and the combination of boxes in the flowchart and / or block diagram, may be implemented using a dedicated hardware-based system that performs a specified function or operation, or may be implemented using a combination of dedicated hardware and computer instructions.

[0055] Embodiment 1:

[0056] like Figure 1 As shown, this embodiment provides a method for identifying unknown network asset types based on Word2vec, which specifically includes the following steps:

[0057] 1) Access relationship extraction: Based on the traffic log, the access relationship between nodes in the network is extracted, and an access relationship document in the format of "source node, destination node" is formed, and a network topology diagram is constructed.

[0058] 2) Network node representation: Based on the access relationship documents, the Word2vec method is used to generate the semantic representation vector of the node, which can map the node in the high-dimensional semantic space; based on the network topology graph, the deep adjacency relationship document of the node is formed by the random walk method, and the topological adjacency semantic representation of the node is calculated on this document using the Word2vec method.

[0059] 3) Extraction of other dimensional features: Calculate the out-degree, in-degree and page rank values ​​of nodes in the network topology graph. These three features can represent the importance and structural similarity of nodes in the network.

[0060] 4) Feature vector concatenation: De-dimensionalize or normalize the features of other dimensions, and then concatenate them with the semantic representation vector and the topological adjacency semantic representation vector to form the final feature vector.

[0061] 5) Classification model training: According to different performance requirements, select a suitable classification model for training. This method uses a multi-classification logistic regression model.

[0062] 6) Identification of asset type.

[0063] like Figure 2 As shown, this embodiment provides a network node characterization method of Word2vec. This embodiment uses the method applied to a server as an example. It can be understood that the method can also be applied to a terminal, and can also be applied to a terminal, a server and a system, and is implemented through the interaction between the terminal and the server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communications, middleware services, domain name services, security services CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application is not limited here.

[0064] In this embodiment, Figure 2 Step 101, the method comprises the following steps:

[0065] Get traffic data;

[0066] Extract the source IP field and destination IP field in the traffic data and generate the network topology structure of the node;

[0067] Construct sample data based on the source IP field and the destination IP field;

[0068] Based on the network topology of nodes, a random walk method is used to construct deep adjacent sample data of nodes;

[0069] Based on the two sample data, two different Word2vec models are input for training to obtain the semantic vector and adjacent semantic vector corresponding to each IP;

[0070] Concatenate the vectors.

[0071] First of all, Word2vec is a very simple neural network, which was originally used in the field of natural language processing to map words to a high-dimensional semantic space based on the co-occurrence relationship between words and background words. The closer the relationship between two words, the smaller the cosine distance between the vectors corresponding to the two words.

[0072] This embodiment draws on the idea of ​​text word embedding and aims to solve the problem of network node representation.

[0073] 1. Algorithm Introduction

[0074] (1) Word2vec

[0075] Word2vec is a language model that learns semantic knowledge from a large amount of text corpus in an unsupervised manner and is widely used in natural language processing.

[0076] Word2vec is a lightweight neural network. Its model only includes input layer, hidden layer and output layer. The model framework mainly includes CBOW and Skip-gram models according to the input and output. The CBOW method predicts the current word when the context of the word is known, while Skip-gram predicts the context of the word when the word is known.

[0077] Generally, when a neural network language model predicts, it outputs the probability of predicting the target word, which means that each prediction must be calculated based on the entire data set, which undoubtedly brings a lot of time overhead. Unlike other neural networks, Word2vec proposes two ways to speed up training, one is Hierarchical softmax, and the other is Negative Sampling.

[0078] First, explain Hierarchical softmax:

[0079] Hierarchical softmax records the positive and negative classification of samples in the form of a Huffman tree, where the left subtree 0 is the positive class and the right subtree 1 is the negative class, as shown in the following formula.

[0080]

[0081] in, is the probability of being classified as the left and right subtrees of the Huffman tree, is the probability of being classified as positive or negative, and w is the word.

[0082] When performing binary classification, the Sigmoid function is selected here. Then, the probability of a node being classified as a positive class is:

[0083]

[0084] For any word w in the dictionary, there must be a path p in the Huffman tree from the root node to the leaf node corresponding to word w. w , and this path is unique. Consider each branch as a binary classification, and the objective function is p(w|Context(w)).

[0085]

[0086] in:

[0087]

[0088] in, is the probability of each branch of the Huffman tree, and the loss function L(w, j) is:

[0089]

[0090] Next, Negative Sampling is explained:

[0091] The essence of Negative Sampling is to use known probability density functions to estimate unknown probability density functions. Compared with Hierarchical Softmax, it no longer uses complex Huffman trees, but uses relatively simple random negative sampling, which can greatly improve performance.

[0092] For a given positive sample (Context(w), w), the objective function that we want to maximize is:

[0093]

[0094] Among them, L w (u) is the probability of classification into positive and negative classes.

[0095]

[0096] The loss function L(w,u) is:

[0097] L(w,u)=L w (u)·log(β)+[1-L w (u)]·log(1-β).

[0098] (2) Random Walk

[0099] The basic idea of "random walk" is to traverse a graph starting from one or a series of vertices. At any vertex, the traverser will move to a neighbor vertex of this vertex with probability 1 - a, and randomly jump to any vertex in the graph with probability a. Here, a is called the jump occurrence probability. After each walk, a probability distribution is obtained, which depicts the probability of each vertex in the graph being visited. Use this probability distribution as the input for the next walk and iterate this process repeatedly. When certain preconditions are met, this probability distribution will tend to converge. After convergence, a stable probability distribution can be obtained. It is a bit similar to DFS in search algorithms. Starting from a certain point, a sequence of nodes is obtained in a depth-first search manner. This sequence can be used to describe the nodes.

[0100] Let f(x) be a multivariate function with n variables, where x = (x1, x2,..., x n ) is an n-dimensional vector.

[0101] 1. Given an initial iteration point x, an initial walking step size λ, and a control precision ε (ε is a very small positive number used to control the end of the algorithm).

[0102] 2. Given an iteration control number N, let k be the current iteration number, and set k = 1.

[0103] 3. When k < N, randomly generate an n-dimensional vector within (-1, 1).

[0104] u = (u1, u2,..., u n ), (-1 < u i < 1, i = 1, 2,..., n), and normalize it to get Let x1 = x + λu′, and complete the first step of the walk.

[0105] 4. Calculate the function value. If f(x1) < f(x), that is, a point better than the initial value is found, then reset k to 1, change x1 to x, and go back to step 2; otherwise, k = k + 1 and go back to step 3.

[0106] 5. If no better value can be found for N consecutive times, it is considered that the optimal solution is within an N-dimensional sphere centered at the current optimal solution with the current step size as the radius (if it is three-dimensional, it is just a sphere in space). At this time, if λ < ε, end the algorithm; otherwise, let Go back to step 1 and start a new round of walks.

[0107] The algorithm flow is as follows:

[0108] Taking the traffic access relationship as an example, such as Figure 2 , the following content can be considered for implementation:

[0109] Step 1: As in step 101, first download part of the traffic data.

[0110] Step 2: As in step 102, the source IP field and the destination IP field in the traffic data are extracted, the default values ​​are deleted, and a network topology structure of the nodes is generated.

[0111] Step 3: As in step 103, write the source IP and destination IP into a txt file in pairs with spaces as separators, with each traffic record in a separate line, to construct training sample 1.

[0112] Step 4: As in step 104, on the network topology diagram of the node, a random walk method is used to construct the deep adjacency sample data 2 of the node.

[0113] Step 5: As in step 105, training samples 1 and 2 are respectively input into Word2vec for training. According to the training results, the semantic vectors and adjacent semantic vectors corresponding to each IP are extracted.

[0114] Step 6: As in step 106, concatenate the semantic vector and the adjacent semantic vector as the representation vector of the node.

[0115] After practical verification, this representation method is superior to artificial features in node classification and solves the problem of artificial dependence of node features.

[0116] Embodiment 2:

[0117] This embodiment provides a network unknown asset type recognition system based on Word2vec.

[0118] A network unknown asset type recognition system based on Word2vec, the implementation details are as follows Figure 2 Flowchart, where the modules are constructed as Figure 3 ,include:

[0119] The feature extraction module 31 is configured to: obtain traffic data, extract the source IP field and the destination IP field in the traffic data, and generate a network topology diagram of the node;

[0120] The data set construction module 32 is configured to: 1. construct sample data based on the source IP field and the destination IP field; 2. construct sample data based on the network topology diagram;

[0121] The calculation and model extraction module 33 is configured to: input the word2ve model for training based on the sample data to obtain the semantic vector and adjacent semantic vector corresponding to each IP;

[0122] A feature concatenation module 34 is configured to: de-dimensionalize and concatenate the representation vector and other dimensional vectors;

[0123] The unknown asset classification module 35 is configured as follows: model training and model prediction. Model training is to use known type data as samples to train a classifier and evaluate the classifier; model prediction is to input unknown type data into the classifier and input the prediction type.

[0124] It should be noted that the data feature extraction module, data set construction module, calculation and model extraction module, feature splicing module, and unknown asset classification module are the same as the examples and application scenarios implemented by the steps in Example 1, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0125] Embodiment 3:

[0126] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the method for identifying unknown network asset types based on Word2vec as described in the above embodiment 1 are implemented.

[0127] Embodiment 4:

[0128] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the method for identifying unknown network asset types based on Word2vec as described in the above-mentioned embodiment 1 are implemented.

[0129] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0130] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0131] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0133] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0134] So far, the purpose of the present invention has been achieved.

[0135] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for identifying unknown network asset types, characterized in that: The steps include: Extract access relationships through traffic logs; Establish a Word2vec node representation model based on access relations; Calculate node features of other dimensions; Concatenate the node representation vector with features of other dimensions; Node asset type classification model training; Node asset type identification.

2. The method for identifying the type of unknown network assets according to claim 1, characterized in that: The access relationship extraction is specifically as follows: according to the traffic log, the access relationship between each node in the network is extracted, and an access relationship document in the format of "source node, destination node" is formed, and a network topology diagram is constructed.

3. The method for identifying the type of unknown network assets according to claim 1, characterized in that: The establishment of the Word2vec node representation model: based on access relation documents, using the Word2vec method, a semantic representation vector of a node is generated, which can map the node in a high-dimensional semantic space; Based on the network topology graph, a random walk method is used to form a deep adjacency relationship document of the nodes. The Word2vec method is also used to calculate the topological adjacency semantic representation of the nodes on this document.

4. The method for identifying the type of unknown network assets according to claim 1, characterized in that: The extraction of node features in other dimensions mainly includes: calculating the out-degree, in-degree and page rank value of the node in the network topology graph. These three features can represent the importance and structural similarity of the node in the network.

5. The method for identifying the type of unknown network assets according to claim 1, characterized in that: The feature vector concatenation mainly includes: performing a dimensionless operation or a normalization operation on features of other dimensions, and then concatenating the features with the semantic representation vector and the topologically adjacent semantic representation vector to form a final feature vector.

6. The method for identifying the type of unknown network assets according to claim 1, characterized in that: The classification model training: according to different performance requirements, a suitable classification model is selected for training. This method adopts a multi-classification logistic regression model.

7. The method for identifying the type of unknown network assets according to claim 3, characterized in that: The network node characterization method based on Word2vec specifically includes the following steps: Get traffic data; Extract the source IP field and the destination IP field in the traffic data, delete the default values ​​in both the source IP field and the destination IP field, and generate a topological structure of the network nodes; Based on the source IP field and the destination IP field, write them into a txt file in pairs with spaces as delimiters, with each traffic record in a separate line to construct sample data; Based on the network topology of nodes, a random walk method is used to generate deep adjacency relationship sample data; Based on the two sample data, two different Word2vec models are input for training respectively to obtain the semantic vector and topological adjacency semantic vector corresponding to each IP; Based on two semantic vectors, vector splicing is performed. Among them, the node-based network topology structure generates deep adjacency relationship sample data, including: taking a node as the starting node, randomly walking in the topology map according to a certain step length, generating an IP sequence of a certain length, using spaces as separators, writing into a txt file, with each IP sequence as a line, to construct sample data.

8. A network unknown asset type identification system, characterized in that: include: A feature extraction module is used to obtain traffic data, extract the source IP field and the destination IP field in the traffic data, and generate a network topology diagram of the node; The dataset construction module builds sample data based on the source IP field and the destination IP field; Sample data construction based on network topology diagram; The calculation and model extraction module inputs the word2ve model for training based on sample data to obtain the semantic vector and adjacent semantic vector corresponding to each IP; Feature concatenation module, used to de-dimensionalize and concatenate representation vectors and other dimensional vectors; The unknown asset classification module is used for model training and model prediction. Model training is to use known type data as samples to train a classifier and evaluate the classifier; model prediction is to input unknown type data into the classifier and input the prediction type.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the steps of the method for identifying the type of unknown network assets described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for identifying the type of unknown network assets described in any one of claims 1 to 7 are implemented.

Images