Integrated community and role discovery in corporate networks

Abstract

A method for detecting anomalous communications, comprising: Simulating a network graph based on community and role labels of each node in the network graph based on one or more connection rules; Setting the community and role labels of each node based on differences between the simulated network graph and a real network graph; Repeating the simulation and adjustment until the simulated network graph converges to the real network graph to determine a final set of community and role labels; and Determine whether a network communication is anomalous based on the final set of community and role labels.

Description

INFORMATION ABOUT ASSOCIATED REGISTRATION

[0001] This application claims priority from 62 / 148,232, filed April 16, 2015. BACKGROUNDTechnical field

[0002] The present invention relates to computer and network security, and more particularly to integrated discovery of node community and role in such networks. Description of the related prior art

[0003] Corporate networks are key systems within companies, and they carry the vast majority of mission-critical information. As a result of their importance, these networks are often the target of an attack. Communications on corporate networks are therefore frequently monitored and analyzed to detect anomalous network communications, as a step toward detecting attacks.

[0004] However, accurate and effective mapping is difficult if the system lacks knowledge of a community and roles. A community represents the workgroup to which a machine belongs, while a role represents the machine's function (e.g., as an email server, as a file server, as a personal desktop, etc.). It is often impossible for users to provide an accurate picture of a community and a role for an entire network.

[0005] Existing approaches for community and role detection treat the issues separately, such as detecting roles without considering community structures and detecting a node's community while ignoring its role, when in fact communities and roles are tightly coupled and cannot be separated in real networks.

[0006] US 2015 / 0019762 A1 discloses an information management system. The information management system may be policy-based. Activity data may be organized as entries containing information about the user, application, machine, action, object or document, time, and location. When testing for patterns in the activity or historical data, techniques such as inference, frequency testing, location and distance testing, and relationship testing, or any combination of these, may be used. Analyzing the activity data may involve comparing similar information types or categories for two or more entries.

[0007] US 2005 / 0 055 573 A1 discloses a method and apparatus for providing network security using role-based access control. A network device implementing such a method may, for example, include an access control list. Such an access control list includes an access control list entry, which in turn includes a user group field. Alternatively, a network device implementing such a method may, for example, include a forwarding table containing a plurality of forwarding table entries. In such a case, at least one of the forwarding table entries includes a user group field. SUMMARY

[0008] A method for detecting anomalous communications includes simulating a network graph based on community and role labels of each node in the network graph based on one or more connection rules. The community and role labels of each node are adjusted based on differences between the simulated network graph and a real network graph. The simulation and adjustment are repeated until the simulated network graph converges to the real network graph to determine a final set of community and role labels. Whether a network communication is anomalous is determined based on the final set of community and role labels.

[0009] A system for detecting anomalous communications includes a community and role detection module with a processor configured to simulate a network graph based on community and role labels of each node in the network graph based on one or more connection rules, adjust the community and role labels of each node based on differences between the simulated network graph and a real network graph, and repeat the simulation and adjustment until the simulated network graph converges to the real network graph to determine a final set of community and role labels. An anomaly detection module is configured to determine whether a network communication is anomalous based on the final set of community and role labels.

[0010] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The disclosure will provide details in the following description of preferred embodiments with reference to the following figures, wherein: Fig. 1 is directed to a network graph that represents communities and roles of nodes according to the present principles. Fig. 2 is a block / flow diagram of a method for discovering community and role memberships and detecting anomalies according to the present principles. Fig. 3 is a block / flow diagram of a method for detecting anomalies according to the present principles. Fig. 4 is a block diagram of a system for discovering community and role memberships and detecting anomalies according to the present principles. Fig. 5 is a block diagram of a processing system according to the present principles. DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

[0012] According to the present principles, the present embodiments capture communities and roles in a network in an integrated manner. In particular, each node in a network is associated not only with a community membership, but also with a role membership, so that the system can capture both community and role structures simultaneously. When two nodes attempt to interact (e.g., when an edge is formed between two nodes on the graph representing the network), both community and role memberships are taken into account when determining how likely the connection is and thus whether the connection can be considered anomalous. The community and role of each node are determined, in one embodiment, according to Gibbs sampling-based learning.

[0013] Referring now in detail to the figures, in which like reference numerals represent the same or similar elements, and initially to Fig. 1, an exemplary computer network 100 is illustratively shown according to an embodiment of the present principles. The network 100 is formed from a group of nodes 101, each of which has a role and a community. In the embodiment of Fig. 1, the nodes labeled 102 have a community 108, while the nodes labeled 104 have a community 110. It should be noted that the network graph 100 does not represent a physical network, but instead represents communications between the nodes 101, with each edge of the graph representing a communication link. There is, in principle, nothing preventing a node 102 from community 108 from forming a connection with a node 104 in community 110. However, the present embodiments will consider the communities and roles of the nodes 101 when determining whether this connection is anomalous.The nodes 101 are described herein as representing individual devices, but it should be understood that in some embodiments, a single node 101 may contain multiple devices, and conversely, a single device may host multiple nodes 101. Likewise, a single node 101 may occupy multiple roles.

[0014] It should be understood that nodes 101 in different communities will have a low probability of interacting with each other (e.g., a low probability of forming a connection). However, an exception is the case of a node 106 that has a specific role, such as a router or a bridge. In this case, node 106 may belong to one, both, or neither of communities 108 and 110, and its role as an intermediary between these two communities will strongly influence its probability of forming connections with other nodes 101. This may be called role-based connection in the background.It should be noted, however, that communities do not have to be identified with physical network segments - instead, a community can simply represent, for example, a department or other organizational structure that communicates frequently within itself and relatively rarely with other departments.

[0015] Similarly, if two nodes are in the same community, they are more likely to interact, but roles are also a strict factor. For example, a file server 103 within community 108 may interact with user terminals 102 more frequently than these nodes 102 interact with each other. This may be called role-based connection within the community.

[0016] If we now refer to Fig. 2, a method for detecting anomalous connections is shown. A block 202 generates a structure matrix representation of a blueprint graph, which is a heterogeneous graph formed from a historical data set of communications in network 100, where nodes 101 represent physical devices on an enterprise network and edges represent the normal communication patterns among nodes 101. For each pair of nodes in the structure matrix, a block 204 generates community and role labels. The initial labels generated by block 204 may be random or may be generated according to available initial information (e.g., based on known software installed on respective nodes 101 or based on an existing network map).

[0017] A block 206 then simulates the interactions of node pairs between different communities and roles. The simulation is based on a set of rules for known interactions between community elements and according to roles. For example, the nodes 104 marked by the labels as being elements of community 110 will have a simulated connection between them. In another example, server / client role relationships can be represented as connections. This simulation is used to generate a simulated graph blueprint. A block 207 uses the simulated graph blueprint to form a synthetic structure matrix for the simulated graph.

[0018] If there are discrepancies between the structure matrix and the synthetic structure matrix, a block 208 adjusts the community and role labels to bring the simulated connections closer to the current connections in the blueprint graph. A block 210 then determines whether the synthetic matrix has converged to the real structure matrix, such that the connections in the simulated graph match those of the blueprint graph. Convergence may be satisfied if the synthetic structure matrix is identical to the real structure matrix, or alternatively, may be based on a similarity metric for the matrices, with convergence being achieved when the similarity measure is below a threshold. If so, a block 212 uses the detected community and role labels to determine whether there is an anomaly.If not, processing returns to block 206 until the synthetic matrix converges.

[0019] An example of anomaly detection considers a first node n1 having the role label "Database Server" and a community label "Systems Team." A second node n2 has the role label "Email Server" and the community label "Operations Team." If a new network connection is detected between n1 and n2, the system may determine that the database server of one team will rarely have a legitimate need to communicate with the email server of another team (such information being set by the domain user). Block 212 may then determine that an intrusion has occurred.

[0020] The assignment of labels in block 204 can be represented as a respective community membership vector π i and a respective role membership vector θi for each node i. When a pair of nodes (i,j) attempts to form a connection, their community and role membership assignments Zijc,Zjic,Zijr,Zjir drawn according to a multinomial distribution, parameterized by their membership distribution vectors, where Zijc is the community assignment of node i for the pair of nodes (i,j) and Zijr is the role assignment of node i for the pair of nodes (i,j). The question of whether a connection is formed is considered a Bernoulli event based on the community and role assignments of the two nodes and an interaction parameter B, which represents the interaction probability between two community and role assignment tuples, such as Zijc,Zijr marked, shown.

[0021] The parameters π, θ and B are treated as random variables, with beta prior and beta prior distributions at each entry of B. The expression B εpq is a Bernoulli distribution and π i and θ i have a multinomial distribution with Dirichlet priors. The present model can then be summarized as follows:

[0022] For each entry (δ, p, q) in B: Draw Bδpq~Beta(ξδpq1,ξδpq2).

[0023] For each node i: Drawing a community membership vector π i ∼Dirichlet(a c ) Drawing a role membership distribution vector θ i ∼Dirichlet(a r )

[0024] For each pair of nodes (i,j): Drawing the community of node i Zijc~Multinominal(πi) Drawing the community of node j Zjic~Multinominal(πj) Drawing the role of node i Zijr~Multinominal(θi) Drawing the role of node j Zjir~Multinominal(θj) Drawing the connection Eij~Bernoulli(Bδ(Zijc,Zjic),Zijr,Zjir)

[0025] Under the above generative model, if the structure matrix E ij observed, the posterior distribution of hidden variables, such as membership vectors, can be inferred. Given network communication data, the posterior distribution and, in particular, the posterior determination of the variables in the model are inferred. Due to the complicated integrals over hidden states in the posterior derivation, an accurate derivation is difficult to manage. The present embodiments therefore use Gibbs sampling inference, although it should be understood that other types of inference or derivation may be used instead.

[0026] In Gibbs sampling, a Markov chain is maintained. The chain sequentially reaches its next state by sampling a variable from its distribution when it is conditioned on current values of all the other variables. As the Markov chain approaches an equilibrium distribution, subsequent samples are generated from the target distribution. Using collapsed Gibbs sampling, direct samples of the Dirichlet membership variables π and θ are avoided by integrating out these variables. Thus, only the membership assignments of a pair of nodes (i,j) according to the pair's conditional distribution are sampled simultaneously. The conditional distribution P is therefore computed, which represents the community and role assignments of the pair of nodes (i,j), given the structure matrix E. ij and given the current assignments of the other node pairs. The conditional distribution P is defined as: P∝(nδ(a,b)pq+−ij+ξ1)Eij(nδ(a,b)pq−−ij+ξ2)1−Eijnδ(a,b)pq+−ij+nδ(a,b)pq−−ij+ξ1+ξ2(hia−ij+αc)(hjb−ij+αc)(mip−ij+αr) where a=Zijc, b=Zjic, p=Zijr, q=Zjir, hia is the number of node i assigned to a community a, m ip is the number of node i assigned to role b, nδ(a,b)pq+−ij is a number of connected node pairs with community assignments a and b and role assignments p and q, nδ(a,b)pq−−ij is a number of unconnected node pairs with community assignments a and b and role assignments p and q, ξ 1 and ξ 2 are scalar beta hyperparameters for (k, p, q) in the interaction tensor B.

[0027] It is worth noting that the conditional distribution P is proportional to two parts: the connection / disconnection rate assigned to the community and role assignments of the two nodes, and the ratio (after normalization) of community and role membership assignments of both nodes. Both parts are calculated by excluding their current assignments.

[0028] The Markov chain can then be initialized by a given community and role membership assignment for all pairs of nodes. The chain can be run by sequentially resampling assignments from each pair of nodes, conditioned on the remainder. Once the assignments of a pair of nodes are updated, the counters n, m, and h are also updated. After sufficient iterations, the Markov chain approaches the equilibrium distribution. Subsequent samples of the community and role assignments can be collected to estimate the posterior distribution of the variables.

[0029] The community membership of node i is Dirichlet distributed and its mean at a ten Dimension is: πia=(hia+αc)∑a=1Kchia+Kcαc where K c is the number of communities and α c the Dirichlet hyperparameter for π iThe role membership of node i is also Dirichlet distributed and its mean at the p ten Dimension is given by: θip=(mip+αr)∑p=1Krmip+Krαr where K r is the number of roles and α r the Dirichlet hyperparameter for θ i The interaction tensor B is beta-distributed, with the mean of each entry estimated by: Bkpq=nkpq++ξ1nkpq++nkpq−+ξ1+ξ2

[0030] Blocks 206 and 207 therefore calculate the conditional distribution for each pair of nodes (i, j) and block 208 determines π ia , θ ip and B kpq .

[0031] Embodiments described herein may be entirely hardware, entirely software, or include both hardware and software elements. In a preferred embodiment, the present invention is implemented in software, including, but not limited to, firmware, resident software, microcode, etc.

[0032] Embodiments may include a computer program product accessible from a computer-usable or computer-readable medium that provides program code for use by, or in connection with, a computer or any instruction execution system. A computer-usable or computer-readable medium may include any device that stores, communicates, distributes, or transports the program for use by, or in connection with, the instruction execution system, device, or apparatus. The medium may be magnetic, optical, electronic, electromagnetic, infrared, or a semiconductor system (or device or apparatus) or distribution medium.The medium may include a computer-readable storage medium such as a semiconductor or solid-state memory, a magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a fixed magnetic disk, and an optical disk, etc.

[0033] Each computer program may be tangibly stored in a machine-readable storage medium or device (e.g., program memory or magnetic disk) readable by a general or special-purpose programmable computer, for configuring and controlling an operation of a computer when the storage medium or device is read by the computer to perform the procedures described herein. The inventive system may also be considered to be embodied in a computer-readable storage medium configured with a computer program, the storage medium being configured to cause a computer to operate in a specific and predefined manner to perform the functions described herein.

[0034] A data processing system capable of storing and / or executing program code may include at least one processor coupled directly or indirectly to storage elements through a system bus. The storage elements may include local memory used during actual execution of the program code, mass storage, and cache memory that provides temporary storage of at least some of the program code to reduce the number of times code is read from mass storage during execution. Input / output, or I / O, devices (including, but not limited to, keyboards, displays, pointing devices, etc.) may be coupled to the system either directly or through intervening I / O controllers.

[0035] Network adapters can also be coupled to the system to allow the data processing system to connect to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modems, and Ethernet cards are just a few of the types of network adapters currently available.

[0036] If we now refer to Fig. 3, a method for performing intrusion detection based on integrated network-level analysis that includes both community and role information is shown. A block 302 collects data from agents installed on each of the nodes 101. The agents collect information related to each node's activity, including, for example, host-level activities (e.g., user-to-process events, process-to-file events, user-to-registry events, etc.) and network-level activities (e.g., TCP and UDP connections with other nodes 101 in the network 100).

[0037] A block 304 performs a network-level analysis using the collected information. The network-level analysis is described in more detail above and integrates both node community membership and node role membership to detect anomalous communications. A block 306 performs a host-level analysis based on the collected information to determine whether anomalous behavior has occurred locally within an individual node 101.

[0038] A block 308 integrates the network-level and host-level anomalies to provide intrusion detection events. This may further include context analysis to detect interactions between network-level and host-level anomalies, such as noting that certain host-level and network-level anomalies may be more significant when they occur together. A block 310 then presents the detected intrusion events to a user for review and further action. In some embodiments, a block 312 may automatically respond to the intrusion detection event. The response may include, for example, blocking certain network-level communications, restricting access at the individual host level, changing security strategies, and delivering alerts to interested parties, such as a system administrator.Block 312 may consider the specific intrusion information determined by block 308 to determine a best course of action.

[0039] If we now refer to Fig. 4, a network-level anomaly detection system 400 is shown. The detection system 400 includes a hardware processor 402 and a memory 404, as well as a network interface 405. The system 400 further includes certain functional modules, which in some embodiments may be implemented as software stored in the memory 404 and executed by the processor 402. In other embodiments, the functional modules may be implemented as one or more discrete hardware components, such as in the form of an application-specific integrated chip or a field-programmable gate array.

[0040] The system 400 collects historical data 406 relating to the network 100 via the network interface 405 and stores the historical data 406 in memory 404. This historical data 406 contains information reflecting communications between nodes 101 on the network 100 and is provided by agents at the individual node 101 that report what each respective node 101 is doing. The historical data 406 is used to form a blueprint graph 410 of the network 100, where the nodes 101 of the blueprint graph represent individual hosts on the network 100 and edges represent normal communications between the communications 101.

[0041] A community and role detection module 408 automatically discovers the community and role memberships of each node 101 in the network 100, as described in detail above. The community and role detection module 408 uses the processor 402 to analyze the blueprint graph 410 and provides membership vectors θ and π. An anomaly detection module 412 uses the membership vectors and the blueprint graph to examine incoming information about current network communications and to determine whether a given communication is anomalous. The anomaly detection module 412 further uses the incoming network communications to make adjustments to the blueprint graph 410, which may subsequently result in adjustments related to the community and role memberships.

[0042] If we now refer to Fig.5, an exemplary processing system 500 is shown that may represent the network-level anomaly detection system 400. The processing system 500 includes at least one processor (CPU) 504 operatively coupled to other components via a system bus 502. A cache 506, a read-only memory (ROM) 508, a random access memory (RAM) 510, an input / output (I / O) adapter 520, a sound adapter 530, a network adapter 540, a user interface adapter 550, and a display adapter 560 are operatively coupled to the system bus 502.

[0043] A first storage device 522 and a second storage device 524 are operatively coupled to the system bus 502 through the I / O adapter 520. The storage devices 522 and 524 may be any of a disk storage device (e.g., a magnetic or optical disk storage device), a solid-state magnetic device, and so on. The storage devices 522 and 524 may be the same type of storage device or different types of storage devices.

[0044] A speaker 532 is operatively coupled to the system bus 502 through the sound adapter 530. A transceiver 542 is operatively coupled to the system bus 502 through the network adapter 540. A display device 562 is operatively coupled to the system bus 502 through the display adapter 560.

[0045] A first user input device 552, a second user input device 554, and a third user input device 556 are operatively coupled to the system bus 502 through the user interface adapter 550. The user input devices 552, 554, and 556 may be any of a keyboard, a mouse, a membrane keypad, an image capture device, a motion capture device, a microphone, a device incorporating the functionality of at least two of the foregoing devices, and so on. Of course, other types of input devices may also be used while maintaining the spirit of the present principles. The user input devices 552, 554, and 556 may be the same type of user input device or different types of user input devices.The user input devices 552, 554 and 556 are used to input and output information to and from the system 500.

[0046] Of course, processing system 500 may also include other elements (not shown) as readily contemplated by those skilled in the art, as well as omit certain elements. For example, depending on the particular implementation thereof, various other input devices and / or output devices may be included in processing system 500, as readily understood by those skilled in the art. For example, various types of wireless and / or wired input and / or output devices may be used. Furthermore, additional processors, controllers, memory, and so forth may also be used in various configurations, as readily appreciated by those skilled in the art.These and other variations of the processing system 500 have already been contemplated by those skilled in the art, given the teachings of the present principles provided herein.

[0047] The foregoing is to be understood as illustrative and exemplary in all respects, but not restrictive, and the scope of the invention disclosed herein is to be determined not from the detailed description, but rather from the claims as interpreted to the full breadth permitted by the patent laws. It is to be understood that the embodiments shown and described herein are merely illustrative of the principles of the present invention, and that those skilled in the art may implement various modifications without departing from the scope and spirit of the invention. Those skilled in the art could implement various other combinations of features without departing from the scope and spirit of the invention.Thus, aspects of the invention have been described with the detail and particularity required by the patent laws, and what is claimed and desired to be protected by the letter of the patent law is set forth in the appended claims.

Claims

[1] A method for detecting anomalous communications, comprising: Simulating a network graph based on community and role labels of each node in the network graph based on one or more connection rules; Setting the community and role labels of each node based on differences between the simulated network graph and a real network graph; Repeating the simulation and adjustment until the simulated network graph converges to the real network graph to determine a final set of community and role labels; and Determine whether a network communication is anomalous based on the final set of community and role labels. [2] The method of claim 1, wherein setting the community and role labels of each node comprises determining a conditional distribution for each pair of nodes in a network graph based on a rate of connection for a community and role label of each node in the pair of nodes and a ratio of community and role labels of both nodes. [3] The method of claim 1, further comprising determining initial community and role labels for each of a plurality of nodes. [4] The method of claim 3, wherein determining initial community and role labels comprises randomly assigning a community and role label to each node. [5] The method of claim 1, wherein the real network graph is based on historical communications between the nodes. [6] The method of claim 1, wherein repeating the simulating and the adjusting comprises determining a true structure matrix based on the true network graph and a synthetic structure matrix based on the simulated network graph. [7] The method of claim 6, wherein repeating the simulating and the adjusting further comprises determining whether the simulated network graph has converged to the true network graph by determining a similarity of the synthetic structure matrix to the true structure matrix. [8] The method of claim 1, wherein determining whether a network communication is anomalous comprises determining a probability that the network communication occurs between an associated first node and a second node based on the community and role labels of the respective first and second nodes. [9] The method of claim 1, further comprising automatically responding to a detected intrusion event, wherein the response comprises one or more of blocking network communication, restricting access, changing security policies, and alerting a system administrator. [10] A system for detecting anomalous communications, comprising: a community and role detection module having a processor configured to simulate a network graph based on community and role labels of each node in the network graph based on one or more connection rules, to adjust the community and role labels of each node based on differences between the simulated network graph and a real network graph, and to repeat the simulation and adjustment until the simulated network graph converges to the real network graph to determine a final set of community and role labels; and an anomaly detection module configured to, based on the final set of community and role labels to determine whether a network communication is anomalous. [11] The system of claim 10, wherein the community and role detection module is further configured to determine a conditional distribution for each pair of nodes in a network graph based on a rate of connection for a community and role label from each node in the pair of nodes and a ratio of community and role labels from both nodes. [12] The system of claim 10, wherein the community and role detection module is further configured to determine initial community and role labels for each of a plurality of nodes. [13] The system of claim 12, wherein the community and role detection module is further configured to randomly assign a community and role label to each node. [14] The system of claim 10, wherein the real network graph is based on historical communications between the nodes. [15] The system of claim 10, wherein the community and role detection module is further configured to determine a real structure matrix based on the real network graph and a synthetic structure matrix based on the simulated network graph. [16] The system of claim 15, wherein the community and role detection module is further configured to determine whether the simulated network graph has converged to the true network graph by determining a similarity of the synthetic structure matrix to the true structure matrix. [17] The system of claim 10, wherein the anomaly detection module is further configured to determine a probability that the network communication occurs between an associated first node and a second node based on the community and role labels of the respective first and second nodes. [18] The system of claim 10, wherein the anomaly detection module is further configured to automatically respond to a detected intrusion event, the response comprising one or more of blocking network communication, restricting access, changing security policies, and alerting a system administrator.

Images