Asset search and discovery system using graph data structures

Abstract

In search and database systems, a graph generator constructs a graph of nodes and edges and stores the graph in a database or other data structure using a repeated process of expansion and pruning. From this storage, the graph can be generated for display to a user interested in understanding the graph and / or for executing queries on the graph data.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 025,900, filed on May 15, 2020, U.S. Provisional Patent Application No. 62 / 937,432, filed on November 19, 2019, and U.S. Non-Provisional Application No. 16 / 890,995, filed on February 2, 2020, the disclosures of which are hereby expressly incorporated herein by reference. Technical Field

[0003] The present disclosure generally relates to managing assets in a distributed computing environment. More particularly, the present disclosure relates to apparatus and techniques for performing searches of network-connected assets to identify assets under the control of an entity. Background Art

[0004] Securing, controlling, and accessing an organization's network-connected computer and digital assets often requires the ability to track and inventory those assets. Assets can include computer systems, databases, and logical assets such as domain names, hosts, addresses, and more. Often, organizations acquire another organization without a complete understanding of the assets they own and / or control.

[0005] Lack of information about assets owned or controlled can pose a security risk. For example, a server might respond to requests for a specific URL. If that server is insecure, it could provide hackers with an opportunity to infiltrate the rest of the owner's network infrastructure. If the owner is unaware of the asset they own, they may never invest the effort to secure it or remove it from their network resources.

[0006] Therefore, it is desirable to be able to identify assets under control in a network-connected environment. Summary of the Invention

[0007] In one embodiment of a search database system, a graph generator constructs a graph, including nodes and edges, and stores the graph in a database or other data structure using a repeated process of expansion and pruning. From the storage, the graph can be generated for display to a user interested in understanding the graph and / or for executing queries on the graph data.

[0008] In some embodiments, a graph represents an inventory of internet-connected assets, and building the graph is accomplished through a discovery process from an initial set of nodes to a larger graph. The discovery process may involve a computer system, such as software running on a computer or server, that can access data about the assets, evaluate the data, and add edges and nodes to the graph, and access data about those edges and nodes. Some uses of the graph include identifying, inventorying, displaying, managing, and the like the assets on the graph.

[0009] An asset may be owned by a person or entity, or it may be controlled by a person, entity, or organization. For the purposes of the methods and systems described herein, it may not be important who or what actually holds legal ownership of the asset, as long as the figure identifies the asset as controlled by a person, entity, or organization because they own the asset, they possess the asset, they are tasked with managing the asset, they have an interest in or responsibility for protecting the asset to maintain its value, they are required to maintain the asset, etc. Therefore, the terms "owner" and "controller" of an asset may be used in different contexts and may be used interchangeably unless otherwise indicated.

[0010] In certain embodiments, an initial node represents an asset of a known owner, where the owner is an individual, entity, or organization with the legal right to control the asset and / or the technical ability to control the asset for themselves or on behalf of others. In this embodiment, a graph discovery computer or computer process expands the graph from the initial node to identify a larger graph of assets owned by the known owner. The resulting larger graph may represent a list of the owner's assets. This graph may be represented as a graph, or as a list of assets.

[0011] Once an inventory is created, the owner can maintain these assets. Without a complete inventory, some assets may be overlooked. For example, the owner may own a hostname or domain name but not know it, so the hostname may no longer resolve to an IP address. If the owner is a large organization, perhaps after acquiring other entities, there may be no central record of which assets the owner owns.

[0012] In a specific embodiment, a computer-implemented method for constructing a graph data structure representing a prospective asset inventory graph is provided, the graph data structure comprising nodes and edges, each node representing an asset of a network, each edge representing a connection between nodes, and at least some edges having weights represented in the graph data structure and indicating joint control of the assets represented in the graph data structure. The method comprises: storing a representation of at least one seed asset in the graph data structure; extending the prospective asset inventory graph to include additional nodes based on matching criteria, the additional nodes based on matching criteria indicating that the additional nodes are assumed to have joint control of an asset based on the matching criteria, the matching criteria indicating a match between the additional nodes and existing nodes already present in the prospective asset inventory graph; recursively applying the matching criteria between the additional nodes and a third node representing a third asset not yet represented in the prospective asset inventory graph; pruning branches of the prospective asset inventory graph based on identifying edges having weights less than a threshold indicating joint control; and updating the graph data structure based on the expansion and pruning.

[0013] Recursion can be automatic or user-defined. For user-defined recursion, the user or system can choose to use only a subset of the available linking methods based on selection depth, confidence, settings, or other criteria. The selection can be a selection to include methods and / or exclude methods.

[0014] Extensions may include: reading the expected asset inventory graph to identify metadata for known assets, the known assets including a set of assets already present on the expected asset inventory graph; selecting predetermined metadata about the known assets; normalizing the predetermined metadata to form normalized metadata; searching an asset database for a match to the normalized metadata; filtering out nodes for assets where the metadata is a product of a registration process rather than an indication of ownership or control; and for each remaining match, adding the asset to the expected asset inventory graph.

[0015] The weight of the edges of the expected asset inventory graph can be the sum of the weights of the individual matches, whereby the weight increases as the number of matches increases. Nodes can be assigned a confidence level, where the confidence level of a node corresponds to the likelihood that the asset at the node is a jointly controlled asset. Additional nodes can come from a Domain Name System (DNS) database, a historical DNS database, and / or a WHOIS database, and expanding the graph includes searching the Domain Name System (DNS) database, the historical DNS database, and / or the WHOIS database to identify nodes that meet the matching criteria. Removal can include obtaining manual input reflecting manual input. Other nodes can come from IP addresses, DNS types (e.g., A, AAAA, MX, NS, SOA, etc.), DNS registration data (e.g., email address, postal address, phone number, company name, etc.), and / or ASN information of the IP address.

[0016] The method may include formatting the representation of the prospective asset inventory map into a display format and presenting the display format on a display.

[0017] Metadata associated with assets on the prospective asset inventory map may include one or more of a host name, a list of vulnerabilities, a list of open ports used, an estimated geographic location of the asset, the operating system used for the asset, the asset's service title, and / or the asset's TLS certificate details. Network assets may include domains, internet-connected assets, subdomains, IP addresses, virtual hosts, web servers, name servers, IoT devices, desktop computers, network printers, mail servers, or devices connected to the internet or an internal network. Other assets may include content delivery networks, proxies, web application firewalls, intrusion detection systems, firewalls, routers, switches, or any device that can accept network traffic.

[0018] In a specific embodiment, 16. The method of claim 1, wherein the first asset is a first domain and the second asset is a second domain, the first asset is on the expected asset inventory map, and the matching criteria includes one or more of the following: a first test regarding whether the second domain shares a common registered email address with the first domain; a second test regarding whether the second domain shares a common registered email address domain with the first domain; a third test regarding whether the second domain is registered using an email address whose email domain matches the email domain of the first domain; a fourth test regarding whether the second domain and the first domain share a common WHOIS field; a fifth test regarding whether the second domain and the first domain both include hosted content that refers back to a common host; a sixth test regarding whether the second domain and the first domain both use a common authentication authority; a seventh test regarding whether the second domain and the first domain share a common CIDR signature; whether both the second domain and the first domain include hosted content that references a common host; and a ninth test regarding whether both the second domain and the first domain use a common authentication authority. Certain tests may be performed using historical data. For example, certain tests may involve comparing data elements that existed at a common point in time, such as testing whether two domains shared a common IP address at a particular point in time, despite the fact that they no longer share the common IP address.

[0019] The following detailed description, taken together with the accompanying drawings, will provide a better understanding of the nature and advantages of the present embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A graph processing system for processing an asset graph according to an embodiment is illustrated.

[0021] Figure 2 A graph processor according to an embodiment is illustrated.

[0022] Figure 3 Illustrated are elements used as part of a graph processor according to an embodiment.

[0023] Figure 4 Illustrated are examples of weightings that may be applied to data points that may indicate joint control of an asset.

[0024] Figure 5 is a flowchart of the example graph construction process.

[0025] Figure 6 Illustrate graph search and graph restriction.

[0026] Figure 7 Additional embodiments are illustrated. DETAILED DESCRIPTION

[0027] In the following description, various embodiments are described. For illustrative purposes, specific configurations and details are set forth to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that the embodiments can be practiced without these specific details. Furthermore, well-known features may be omitted or simplified to avoid obscuring the described embodiments.

[0028] The techniques described and suggested in this article involve building asset graphs, sometimes using expansion and culling processes.

[0029] Overview

[0030] A search database system is described. In one embodiment, the system is a graph generator that constructs a graph including nodes and edges and stores the graph in a database or other data structure using a repetitive expansion-elimination process. From the storage, the graph can be generated for display to a user interested in understanding the graph and / or for executing queries on the graph data.

[0031] In some embodiments, a graph represents an inventory of internet-connected assets, and building the graph is accomplished through a discovery process from an initial set of nodes to a larger graph. The discovery process may involve a computer system, such as software running on a computer or server, that can access data about the assets, evaluate the data, and add edges and nodes to the graph, and access data about those edges and nodes. Some uses of the graph include identifying, inventorying, displaying, managing, and the like the assets on the graph.

[0032] An asset may be owned by a person or entity, or it may be controlled by a person, entity, or organization. For the purposes of the methods and systems described herein, it may not be important who or what actually holds legal ownership of the asset, as long as the figure identifies the asset as controlled by a person, entity, or organization because they own the asset, they possess the asset, they are tasked with managing the asset, they have an interest in or responsibility for protecting the asset to maintain its value, they are required to maintain the asset, etc. Therefore, the terms "owner" and "controller" of an asset may be used in different contexts and may be used interchangeably unless otherwise indicated.

[0033] In certain embodiments, an initial node represents an asset of a known owner, where the owner is an individual, entity, or organization with the legal right to control the asset and / or the technical ability to control the asset for themselves or on behalf of others. In this embodiment, a graph discovery computer or computer process expands the graph from the initial node to identify a larger graph of assets owned by the known owner. The resulting larger graph may represent a list of the owner's assets. This graph may be represented as a graph, or as a list of assets.

[0034] Once an inventory is created, the owner can maintain these assets. Without a complete inventory, some assets may be overlooked. For example, the owner may own a hostname or domain name but not know it, so the hostname may no longer resolve to an IP address. If the owner is a large organization, perhaps after acquiring other entities, there may be no central record of which assets the owner owns. In some cases, control and ownership are shared, but in other cases, they may not be. In general, processes involving determining ownership can be used for ownership, control, or both.

[0035] The common asset is the domain because there are often many Internet-connected assets that can be identified by a URL that references the domain, and often assets identified by a domain are under common ownership or control, which may not be readily apparent. However, assets need not be so limited. Internet-connected assets or Internet-related assets may include indicators such as domains (identifiable by domain names), subdomains (e.g., domain names with host names appended, sometimes more accurately described as fully qualified domain names or FQDNs), IP addresses, virtual hosts, and / or any combination thereof, and devices connected to the Internet or an intranet that use these indicators may also be assets of the owners of these indicator assets. Internet-connected assets may be located on a public network, a non-routable network, or an intranet, among others.

[0036] Assets may include web servers, name servers, IoT devices, desktop computers, network printers, mail servers, other servers, and host computers. An asset inventory may be represented by a data structure, such as a relational database, that identifies the assets and metadata for each asset. An asset management system can provide asset owners with control over the assets in the inventory. Managing assets may include transferring these assets to others, controlling them, configuring them, maintaining them, and setting up network security to protect them.

[0037] Metadata about assets in an asset inventory can include hostnames, details of vulnerabilities, open ports used, and more, and can be important when securing assets. Other metadata can include geolocation, operating system, service title, TLS certificate details, and more. Graph data structures can have metadata on a node-by-node basis, an edge-by-edge basis, some combination of node-by-node and edge-by-edge basis, or other basis.

[0038] Constructing a map of the expected asset inventory

[0039] In certain embodiments of a discovery process for building an asset inventory, the discovery process expands a graph from one or more initial nodes, possibly with some predetermined edges, into a larger graph that can be considered an inventory of expected assets. The inventory of expected assets can have nodes that are not actually co-owned with other nodes, but can serve as an initial guide for further human or computer review to assess whether these assets are actually owned by the asset's owner.

[0040] Figure 1 A graph processing system 100 for processing an asset graph according to an embodiment is illustrated. As shown, a graph processor 102 queries various public or private databases 104 to obtain data and records, and interacts with various assets 106 to obtain data from these assets. The data and records obtained may be as described herein. The graph processor 102 may be coupled to other systems 112 via a communication network 110, such as the Internet. As described herein, a user interface 114 may be provided and may be used to cull, adjust, and / or provide feedback for graph construction. A graph memory 120 may be provided for storing the constructed graph and making it available to other processes, not shown. The graph processor 102 may use a number of processing methods to construct a graph from data from the databases and the assets themselves to generate the graph.

[0041] Figure 2 The diagram illustrates further details of the graph processor 102, which may include one or more computer systems and one or more processors 202, which may be configured to communicate with and operatively couple to a plurality of peripheral subsystems via a bus subsystem 204. These peripheral subsystems may include a storage subsystem 206, including a memory subsystem 208 and a file storage subsystem 210, one or more user interface input devices 212, a user interface output device 214, and a network interface subsystem 216.

[0042] The bus subsystem 204 may provide a mechanism that enables the various components and subsystems of the computer system 200 to communicate with each other as intended. Although the bus subsystem 204 is shown schematically as a single bus, alternative embodiments of the bus subsystem may utilize multiple busses.

[0043] The network interface subsystem 216 may provide an interface 222 to other computer systems and networks. The network interface subsystem 216 may serve as an interface for receiving data from and transmitting data to other systems, for example, to obtain graphical data or user feedback.

[0044] The user interface input devices 212 may include keyboards, pointing devices, and other types of input devices. The user interface output devices 214 may include a display subsystem, a printer, a non-visual display (e.g., an audio and / or tactile output device), or other such display devices. Generally, the use of the term "output device" is intended to include all possible types of devices and mechanisms for outputting information. The user interface output devices 214 may be used, for example, to generate and / or present a user interface to facilitate user interaction with an application executing the processes described herein and variations therein, when such interaction is appropriate.

[0045] The storage subsystem 206 may provide computer-readable storage media for storing programming and data structures that provide graph processor functionality. Software (programs, code modules, instructions) that, when executed by one or more processors 202, can provide the functionality of the embodiments described herein may be stored in the storage subsystem 206. The storage subsystem 206 may also provide a repository for storing data used in graph processing. Example software may include program code for implementing culling, filtering, adjusting, searching, and other functionality described herein.

[0046] The memory subsystem 208 may include a plurality of memory devices, including, for example, random access memory (RAM) 218 ​​for storing instructions and data during program execution, and read-only memory (ROM) 220 in which fixed instructions may be stored. The file storage subsystem 210 may provide non-transitory persistent (non-volatile) storage for program and data files, and may include a hard drive and other storage media.

[0047] The graph processor 102 may include various types of computers and be implemented in a variety of operating environments, which in some cases may include one or more user computers, computing devices, or processing devices that can be used to operate any of a plurality of applications. The user or client device may include any of a number of general-purpose personal computers, such as a desktop, laptop, or tablet computer running a standard operating system, as well as cellular, wireless, and handheld devices that may run mobile software and support a variety of networking and messaging protocols depending on the interface selected by the user. Various embodiments may use at least one network familiar to those skilled in the art to support communications using any of a variety of commercially available protocols, such as the Transmission Control Protocol / Internet Protocol ("TCP / IP"), the User Datagram Protocol ("UDP"), protocols operating in the various layers of the Open Systems Interconnection ("OSI") model, the File Transfer Protocol ("FTP"), and the like. Storage media and computer-readable media used to contain code or portions of code may include appropriate media known or used in the art, including storage media and communication media, such as but not limited to volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing and / or transmitting information, such as computer-readable instructions, data structures, program modules or other data.

[0048] Figure 3 Illustrated are elements used as part of a graph processor according to an embodiment. Figure 3 Also illustrated are examples of memory elements that can be used by the processor to implement elements of the embodiments described herein. For example, where a functional block is referenced, it can be implemented as program code stored in the memory. Figure 3 3 is a simplified functional block diagram of a storage device 348 having an application that can be accessed and executed by a processor in a computer system, which can be part of a graph processor and / or a computer system that uses an asset graph to manage assets under control. The application can be one or more of the applications described herein that run on a server, client, or other platform or device that can represent one of the clients and / or servers shown elsewhere. Storage device 348 can be one or more memory devices accessible by a processor, and storage device 348 can store application code 350, which can be configured to store one or more processor-readable instructions. Application code 350 can include application logic 352, library functions 354, and file I / O functions 356 associated with the application.

[0049] The storage device 348 may also include application variables 362, which may include one or more storage locations configured to receive application variables 364. The application variables 362 may include variables generated by or local to the application. The application variables 362 may be generated, for example, from data retrieved from an external source, such as a user or an external device or application. The processor may execute the application code 350 to generate the application variables 362 that are provided to the storage device 348.

[0050] One or more memory locations may be configured to store device data 366. Device data 366 may include data from an external source, such as a user or an external device. Device data 366 may include, for example, records passed between servers before or after transmission or reception. Other data 368 may also be provided.

[0051] The storage device 348 may also include a log file 380 having one or more storage locations 384 configured to store results of the application or input provided to the application. For example, the log file 380 may be configured to store a history of actions.

[0052] Figure 3 The memory elements of the server or computer may be used to interact with users, generate graphs, and / or manage other aspects of the processes described herein.

[0053] The operations of the processes described herein may be performed in any appropriate order, unless otherwise noted herein or clearly contradictory to the context. The processes described herein (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as codes (e.g., executable instructions, one or more computer programs, or one or more applications) that are executed together on one or more processors by hardware or a combination thereof. The code may be stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions that may be executed by one or more processors. The computer-readable storage medium may be non-transitory.

[0054] Build a graph

[0055] In some embodiments, a map is created, while in other embodiments, the map is obtained from an external source. A process for constructing a map of the expected asset inventory will now be described. The process can be executed on a computer system having input, memory, output, and access to a network, such as the Internet, and having access to databases of interest, such as a Domain Name System (DNS) database, a historical DNS database, a WHOIS database, and the like.

[0056] In such a process for building an expected asset inventory graph, a number of steps, as described below, may be used to add nodes and / or edges to the expected asset inventory graph based on the nodes and edges in the graph and / or metadata about those edges and nodes.

[0057] In some processes, the graph starts with one node, and as other assets with some relationship to the existing node are discovered, additional nodes are added (and perhaps some weights are assigned to the graph edges between the existing and new nodes), but the process involves removing nodes that later appear to be less relevant to the graph. Removal can also involve manual input. For example, a graph might be constructed to include example.com, example1.com, example-inc.com, and example-inc-sucks.com, and if a human reviewer determines that the domain is not actually jointly owned or controlled with other assets, or that this is unlikely, the human reviewer would remove example-inc-sucks.com from the graph.

[0058] Using certain feedback, including user feedback after observing the display of the graph, the process may add additional nodes or cull branches of the graph.

[0059] Domain Addition

[0060] In some cases, domains are added to the expected asset inventory map if they share some metadata with domains already in the expected asset inventory map, such as the same email address used to register the domain, or the same phone number. Generally speaking, during the matching process, depending on the field, normalization of the data can be done before the matching step (e.g., making all strings lowercase, removing punctuation, dashes, and spaces from phone numbers, replacing common homographs - so that zero is treated the same as the letter "O", etc.).

[0061] The matching process may include the following steps: (a) reading the expected asset inventory graph to identify metadata for a set of domains that are already in the expected asset inventory graph (known domains), (b) selecting specific metadata about the known domains, (c) normalizing the metadata, (d) searching a domain database (e.g., a DNA database, a WHOIS database, etc.) for matches to the normalized metadata, (e) optionally filtering out metadata that may be a sign of a registration process rather than ownership (e.g., registrations using "whoisguard" type email addresses, IP addresses pointing to large cloud services that provide services to the owner and many other unrelated parties, domain names of email service providers rather than the owner's own domain names, etc.), (f) for each remaining match, adding the domain to the expected asset inventory graph, and (g) iterating as needed (e.g., until no new domain names are added, or until some predetermined number of iterations are performed).

[0062] In addition to or instead of normalization, wildcard expressions, such as regular expressions, can be used. For example, when searching the WHOIS database, the search term could be "example.*\.com" and all domains in the WHOIS database with whois / zone="example.*\.com" would be returned.

[0063] The database used for the search can be a current database or a historical database. For example, if a domain was registered using a company email address as the registration address or technical contact address, but later changed to private registration, where the registration address or technical contact address is the registrar's address, referencing a historical DNS database or a historical WHOIS database can provide more connections and better identify the specific owner's assets. As another example, when an owner switches to a cloud provider's subnet (e.g., Cloudflare, Incapsula, DoSArrest, Akamai, etc.), the matching process can scan the historical database to find a time when the owner's domain was not using that subnet, look at the IP address at that time, and identify which other domains were pointing to the same IP address at that time. It can then go back to today to see which ones are still pointing to the same IP address. However, due to shared hosting and other reasons, this can result in many false positives for the matching process. In addition to the database itself, other data structures can be searched, such as one or more binary trees, file shard sets, lookup tables, and / or mtbl data structures. More generally, data structures that allow for efficient searches can be used, either directly integrated, accessed via an API, iterated over flat files, using pointers or caches in memory, or some other method.

[0064] The databases accessed by a graph processor can be public databases, such as publicly available DNS servers, private databases maintained and provided by subscription, such as the historical WHOIS database, or private databases maintained internally by the graph processor operator.

[0065] culling

[0066] In some embodiments, a graph or database of assets is known but may not contain any indication of ownership or control. In other cases, this graph or database can be constructed or accessed for queries, eliminating the need for creating the graph or database. The graph or database can then be used to determine the graph of assets under common ownership by filtering out assets that appear not to be under common ownership and / or pruning branches that appear not to belong to the common ownership graph. Some data indicating common ownership or its opposite can be noisy. Using multiple metrics, a more robust common ownership graph can be constructed. The common ownership graph can be further improved by providing human feedback on where the graph can be split or multiple graphs can be merged. Other metrics can be processed to adjust the graph edge weights to improve the graph's fidelity. A graph may not accurately represent all jointly owned and non-jointly owned assets, but such a graph can still be useful in providing valuable information about asset ownership. An example is an entity attempting to protect its assets by implementing cyber controls, updating assets with patches, decommissioning unused assets, and so on. With a good graph, an entity can track and service the assets owned by the entity, even if the graph sometimes implies assets that are not actually owned by the entity and may miss some assets that are actually owned by the entity. Therefore, the graph does not need to be perfect, but should provide some filtering so that the task of asset review is manageable.

[0067] In an embodiment, the asset evaluation system may process or access a map or database of network-connected assets and run a series of evaluations to determine which assets may or may not be associated with a particular owning entity. Thus, some evaluations are positive indicators and some are negative indicators.

[0068] Other indicators can come from human input, perhaps after auditing the graph. In some embodiments, human input can be used to modify not only the data, but also the rules that the graph processor uses to process the data it finds. For example, a user can select an option for the graph processor to skip or not consider any URL similarities when evaluating asset co-ownership.

[0069] Weighted Edges

[0070] In some cases, weights can be assigned to edges. For example, if a first domain in the expected asset inventory graph is known to have something in common with a second domain that is not in the expected asset inventory graph, the second domain can be added as a node in the graph with an edge connecting the first and second domains and the edge having a weight of 1.0. In cases where adding a new node makes a particular search step more speculative, lower weights can be assigned to the edge. For example, if the domain<companyname1.com> is already a node in the expected asset inventory graph and has been normalized, then consider the domain<companynameI.com> Added as a new node in the expected asset inventory graph, edges can be added to<companyname1.com> The nodes and<companynameI.com> The expected asset inventory graph between the nodes of , and the metadata of the edge recorded indicates a weight of 0.6 to reflect the possible uncertainty about whether the two domains are jointly owned. Therefore, in addition to the connection weight, some search steps may also return domain or other node details.

[0071] When presenting a representation of the asset graph, some indication of weights may be presented, and may be presented as confidence ratings. If the confidence of a node for an asset owned by a possible owner is based on a series of nodes, the confidence rating may be adjusted accordingly. For example, if the expected asset inventory graph indicates with certainty that owner O owns the domain "owner-o.tld" and has a 40% probability of also owning the domain "owner-oh.tld", and the discovery process determines that the domain "owners-of-oh.tld" has a 50% probability of being co-owned with the domain "owner-oh.tld", then the discovery process may involve adding a node to the expected asset inventory graph for the domain "owner-oh.tld" and giving that node a confidence value or weight indicating these probabilities. This may be 40% * 50% = 20%, or some other combination of weights.

[0072] Weights can be additive. For example, if one search reveals that two domains have a common registered email address, then the edge between the two domains can be assigned a weight of W1. If another search reveals that the two domains share authentication authority and that sharing would itself carry a weight of W2, then the weight used in the expected asset inventory graph between the nodes of the two domains can be W1 + W2, or at least a weight greater than the higher of W1 and W2.

[0073] In addition to weighting edges, which may represent relative confidence, if a node represents an owner's asset, and another node may also be an owner's asset, the node itself can have metadata indicating a confidence level. This would be the confidence level of a given node owned by a given owner, independent of other nodes and edges that may exist. A node's confidence level can be a function of the confidence levels of the edges connected to it, but can be calculated in other ways.

[0074] Figure 4 An example of weighting that can be applied to data points that may indicate joint control of an asset is shown, which can be used to avoid introducing excessive noise into the added data. As shown there, some more relevant markers will be ranked higher, while noisy markers will be ranked lower. The graph processor can be programmed so that markers falling in region 402 are considered of sufficient quality to allow automatic acceptance of the data, while markers falling in region 404 are marked as manual intervention, and markers falling in region 406 are not considered or hidden from view.

[0075] Example of a domain matching rule

[0076] Some examples of domain matches that result in a domain being added to the prospective inventory map of assets with common control may include:

[0077] 1. Two (or more) domains share a common registered email address: For domains that exist in the graph, the WHOIS database can be queried to identify the email addresses of the contacts provided when the domain was registered, such as technical contacts, administrative contacts, etc. These email addresses can then be used as query input to find records for domains with registered email addresses that match the registered email addresses of domains that already exist in the prospective inventory graph. The logical expression for this sub-process can be:

[0078] (domain1.registration_email_address===domain2.registration_email_address).

[0079] 2. Two (or more) domains share a common registered email address domain: Similar to #1, but the usernames of the registered email addresses do not need to match, only the email domains, thereby optionally filtering out a large number of shared domains that do not indicate possible joint control of assets. (For example, registrations using username1@gmail.com and username2@gmail.com do not necessarily indicate joint control because gmail.com is shared between many unrelated users). The logical expression for this sub-process can be:

[0080] (hostname(domain1.registration_email_address)===hostname(domain2.registration_email_address).

[0081] 3. Domains registered with email addresses versus existing domains in the graph: The graph can be queried for the domain name of each domain already on the graph. The WHOIS database can then be searched for other domains with contact information email addresses, where these addresses include domains on the graph. For example, if example.com is on the graph, this matching step will identify other domains in the WHOIS database that have contact information email addresses of the form user@example.com. Filters can be set so as not to capture domains whose contact information email addresses are hosted by common email services. However, in cases where the graph is unlikely to contain domains that reference common email services as assets (for example, a graph built out of a controller for gmail.com is unlikely to include gmail.com as one of a user's assets), a filter may not be needed. The logical expression for this sub-process could be:

[0082] (domain1===domain2.registration_email_address.domain).

[0083] 4. Two (or more) domains share a common IP address: For domains present in the graph, a DNS lookup can be performed for each existing domain to identify the IP address hosting the domain. The DNS data can then be searched for other domains also hosted at the same IP address, as well as for additional domains added to the graph. A filter can be provided to prevent the addition of domains whose IP addresses are associated with cloud hosting services—otherwise, a large number of unrelated domains for customers of cloud hosting services not controlled by the common end user will be added to the graph. The logical expression for this sub-process could be:

[0084] (domain1.ip_address===domain2.ip_address).

[0085] 5. Two (or more) domains share other WHOIS fields: For domains present in the graph, the WHOIS database can be queried to identify WHOIS fields other than the contact email address. These fields can then be used as query input to find records for other domains that also have these fields in common, which can indicate shared control. Examples include:

[0086] a. Mobile phone number, as follows:

[0087] domain1.phone_number-domain2.phone_number

[0088] b.CIDR block

[0089] c. The IP address of the person or entity that registered the domain (which can be obtained from an abuse monitoring database service).

[0090] 6. Matching Content Cross-References: Some hosted content on a first host may reference content on another host, and / or hosted content on both hosts may reference the other host, and this may be a sign that the first and second hosts are assets controlled by the same entity. In this case, if the first host is on the graph and the second host is not, the second host may be added to the graph with an edge between the first and second hosts having a weight determined based on the cross-references. An example of such a cross-reference is backtracking used on blogging sites, which sends signals between blogs regarding updates and other events on one blog so that the software managing the other blogs can be updated accordingly. Backtracking can be represented by a backtracking URL embedded in the content.

[0091] A specific example of a callback URL is for a WordPress blog, where WordPress content may include callback URLs. The logical expression for this subprocess could be:

[0092] (domain1.html.wordpress_trackback===domain2.html.wordpress_trackback).

[0093] Another example involves callbacks. Program code can be triggered that issues a request, such as an HTTP request, to a first host on the graph. The machine issuing the HTTP request can receive a callback in response. This callback can originate from a second host not on the graph. The domain name, IP address, and / or URL returned by the callback can be added to the graph based on the likelihood that the recipient of the request and the issuer of the callback are jointly controlled assets.

[0094] 7. Two (or more) assets share a CIDR block: A Classless Inter-Domain Routing (CIDR) block can represent a range of IP addresses that are generally under common control. Some assets can be mapped to a specific CIDR block. For example, a search process can query the local DNS system to find a first IP address associated with a first asset as a first subdomain, and a second IP address associated with a second asset as a second subdomain. If the first IP address and the second IP address are in the same CIDR block, the first asset is on the map, and the second asset is not, the second asset can be added to the map based on the fact that they both point to the commonly allocated CIDR block.

[0095] Typically, if an entire CIDR block is allocated to an entity (this might be determined through an IP address registry or allocation database), then subdomains pointing to IP addresses in that CIDR block are typically under the common control of that entity. The decision to add assets to the graph can be filtered to exclude adding assets where the IP addresses are associated with cloud hosting services or software-as-a-service services - otherwise, a large number of unrelated assets of customers of services that are not under the common control of a common end user will be added to the graph.

[0096] When assets are added based on a common CIDR block with assets already existing on the graph, edges between the assets may be added to the graph with weights corresponding to an indication of the strength of the asset's connection to the CIDR block.

[0097] The suitability of adding such nodes or the weights used may depend on data stored from feedback obtained (perhaps from human users). For example, if other users indicate that assets in a common CIDR are generally not under common control, this may reduce the weight used for later matching.

[0098] The step of adding nodes based on a common CIDR block may have a threshold where more than one controlled asset, such as five or ten assets, needs to be present in the common CIDR block before the asset is added to the graph.

[0099] 8. Two (or more) assets share a common IP range defined by other assets: If a first asset and a second asset are listed in the graph and have different IP addresses, then those IP addresses define an IP address range. As other assets with associated IP addresses that are not on the graph are considered, they can be added to the graph on this basis. This can be based on the assumption that if assets at multiple IP addresses are controlled by a single entity, then that single entity likely also controls IP addresses between those known IP addresses. The decision to add assets to the graph can be filtered to exclude adding assets that have IP addresses associated with cloud hosting services or software-as-a-service services - otherwise, a large number of unrelated assets of customers of services that are not under the common control of a common end user will be added to the graph.

[0100] When assets are added based on their IP addresses being in a range between IP addresses of assets present on the graph, edges between the assets may be added to the graph with weights corresponding to an indication of the strength of association of the asset with the IP address range.

[0101] The suitability of adding such nodes or the weights used may depend on data stored from feedback obtained (perhaps from human users). For example, if other users indicate that assets in an IP address range are generally not under common control, this may reduce the weight used for later matching.

[0102] 9. Two (or more) assets share other CIDR block characteristics: If one asset is on the map, but another asset shares the same characteristics but is not, the second asset can be added. An example of a shared characteristic is domains sharing a common characteristic, such as "example.*\.com" domains. This encompasses the IP space of all domains targeting similar-looking domains. This is narrowed down by the least / most restrictive CIDR blocks obtained from the RIR zone files.

[0103] 10. Two (or more) assets have an ASN correlation: When the ASN information of a first domain on the graph matches the ASN information of a second host that is not on the graph, the second host can be added to the graph with an edge between the first and second hosts. Domains with common ASN information can all point to the same space - especially when the overlap between two or more domains pointing to the same IP space is large. An ASN can be an automated system number that uniquely and globally identifies an automated system. An automated system can include hosts that use connected IP routing prefixes and are controlled by a specific administrative entity or domain or by an Internet Service Provider (ISP). ASNs can be registered in a global registry.

[0104] 11. Certification Authority (CA) Affinity: This process extracts available metadata about CA domains in use from node metadata. The list of CAs in use is then used as search input to search the CA database or tree to identify other domains not yet represented in the graph that share a common CA with domains in the graph. This captures assets that share a CA, which may indicate the likelihood that the asset is commonly owned. This can be weighted based on the size of the CA—two domains sharing a larger CA are likely less likely to be commonly owned than two domains sharing a smaller CA.

[0105] 12. Analyze Historical DNS and WHOIS Entries: In addition to traversing the current DNS or WHOIS database, some of the above steps may be performed on historical or archived versions of the DNS or WHOIS database to identify additional nodes of the graph. The weights of the edges of the nodes added by the historical search may be weighted based on the age of the records. For example, if the first domain is in the graph and the second and third domains are not, but the second and third domains share a registered email address with the first domain, or were both associated with a common IP address in the past, then the second and third domains may be added to the graph. If the connection between the first and second domains in the historical data is closer than the connection between the first and third domains, then the edge between the first and second domains will be assigned a higher weight than the edge between the first and third domains.

[0106] Domain or page addition based on page content

[0107] In another process, temporary ownership of assets such as domain names and web pages is accomplished by processing the content of the web pages. By considering this content, an automated process can determine whether to add the asset to the expected asset inventory graph. Normalization can also be used in this context.

[0108] The matching process may include the following steps: reading the expected asset inventory graph to identify strings or other metadata that can be used as indicia of ownership. For example, a web page may include a copyright notice and a company name, or a company name embedded in its HTML code. These indicia are then searched for in other content, such as web pages, search engine indexes, etc. Matches found in the search can be added to the expected asset inventory graph. Known false positives can also be filtered out. Iteration can be performed as assets are added, iterating on newly added assets as needed (e.g., until no new domains are added, or until some predetermined number of iterations have been performed).

[0109] Examples of such signs may include:

[0110] 1. Advertising system keywords (for example, Google Analytics keys, or Google Adwords keys): Search content for specific keywords, identify domains or other assets that include the keywords, and add them to the graph.

[0111] 2. Link tags: Searches for link tags containing the rel type to identify when two or more sites claim to be the same site / page. Can be used with or without canonicalization.

[0112] 3. Unique strings on different HTML pages: Look for company names and copyright notices that match other HTML on other websites (for example: Copyright 20xx, Company Inc.) These names can be normalized so that slight spelling errors and variations are treated as the same.

[0113] Other Examples

[0114] 1) One test that can be performed is to check whether site S has pages that include links to specific other sites, thereby indicating possible relevance. Some overly common links to websites can be excluded to avoid inferring relevance to unrelated but common websites (such as large search engine sites and large social media sites). These relevance can be calculated and presented as a histogram, stored as a blacklist, or using user feedback to indicate possible false positives, such as machine learning output indicating that the object is not an asset. For example, the data in the graph can be presented to users familiar with network and internet protocols and service providers, who can then easily spot links marked as under common control but are known to be used by certain service providers for unrelated customer assets. Such links can be marked as false positives.

[0115] 2) Another test involves having the system perform a URL proximity test. For example, a URL containing "example.com" might be considered related to another URL containing "example-test.com." User feedback or computer processes can determine that certain URLs appear to be under common control, but the underlying assets are not. For example, a content delivery network or cloud service provider generates URLs for multiple unrelated customers, and objects may have similar and related URLs, such as asset1. <cdn-tld>.com and asset2. <cdn-tld>.com, such similarity does not necessarily arise from common control of these assets.

[0116] 3) In another test, if site Y is discovered when following a link from site X, and site Y includes a link to a page on site X, then X and Y are considered related.

[0117] 4) If site X and site Z are both stored in the site list and are found to both link to pages on site Y, and links on site Y pages point to either X or Z, this can be used as an indication of relevance.

[0118] 5) Filtering Internally Used Addresses: In some embodiments, it may be useful to flag IP addresses that appear related, but are internally used—for example, addresses 10.xxx and 192.168.xx—assuming relevance, which is known to be used by unrelated parties for their internal networks. This may also apply to address ranges known to be reserved for use outside of normal global usage.

[0119] 6) Filter out commonly used TLDs that are not necessarily related: For example, two or more IPs may both have a PTR record for a hostname with the .arpa TLD, which may not be due to any degree of relatedness, as this may be the case with many unrelated IPv4 and IPv6 addresses.

[0120] 7) Filtering for valid TLDs: In some embodiments, if the perceived relevance is based on "valid" TLDs, they are discounted. For example, a *.co.uk domain is not correlated with a *.example.com domain. Instead, they should be correlated on valid domains, e.g., "example.co.uk" and "example2.co.uk" are considered different domains, not different subdomains. In a variant, the system considers valid TLDs or TLDs as relevant but as a secondary or adjustable measure of relevance for valid TLDs. For example, <company> . <tld>or. <company>The association can be made knowing that they are all owned by a single entity.

[0121] Relevance Adjustment: The system can increase or decrease the weight of relevance based on factors such as whether the relevance was originally determined based on unusual usage of address space, such as internal RFC 1918-style addresses, RFC 4193-style addresses, RFC 6890-style addresses, RFC 3927-style addresses, loopback addresses, link-local addresses, broadcast addresses, carrier-grade NAT processes, unique local addresses, other non-routable Internet Protocol addresses, etc. This could indicate that two unrelated entities have different DNS references to the same non-routable IP. More generally, address space can be reused across multiple unrelated entities, so while two assets can be in the same address space, the address space is not necessarily associated with a single entity, and thus the relevance can lead to false positive associations of assets.

[0122] Another relevance adjustment the system can make is to remove or significantly reduce the weighting of the probability of connection between known providers that share IP space, such as shared hosting providers, cloud-based WAFs, CDNs, and virtual hosting providers.

[0123] Another relevance adjustment the system may make is by email address, phone number, and physical address, including but not limited to WHOIS or any known or derived company address, when the link found is known to be a privacy service, including but not limited to a domain proxy service, a domain privacy service, a blank or undefined WHOIS result, a placeholder result indicating it is unknown or hidden, a placeholder result requiring manual / non-automated steps to discover, or any type of privacy proxy company.

[0124] Figure 5 is a flow chart of an example graph building process. As shown therein, at step 501, the graph processor may obtain a list of sources. At step 502, the graph processor may begin a domain suggestion process based on new or known weights. At step 503, the graph processor may remove sources that are known to give false positives. At step 504, the graph processor may find links on the remaining sources, which may be limited by how long the paths may be. At step 505, the graph processor may remove known false positives. At step 506, the graph processor may test whether to automatically add the source to the list, which may be based on factors such as Figure 4 At step 507, the graph processor may request manual review, such as in Figure 4 The process then selects those tokens that fall within region 404 and, in step 508, hides or discards the low likelihood edges. In step 506, the process may return to step 502. In step 507, the process may proceed to step 509 to adjust the weights based on user input and then proceed to step 510 to add the user-approved sources to the list, and the process may return to step 502.

[0125] Example of implementation

[0126] In some embodiments, graph processing is performed to determine, based on input data, a graph or other data structure that may indicate, with varying degrees of certainty or probability, a set of assets that are considered to be under common control. This set of assets may include network-connected devices, services, logical objects, trademarks, names, references, and the like. In one approach, an expanded search is performed, and then, based on human user feedback, false positives in the results or other indicators are adjusted to improve the resulting data. In some cases, the graph is a connectivity graph indicating assets and their connections to other assets. In other embodiments, the resulting graph may simply be a data structure that does not necessarily connect all assets to other assets, or even some assets.

[0127] In some embodiments, different indicators may have different weights, and the commonality of control over an asset may be determined by the weighted sum of the individual indicators. The weights may be derived from manual input, machine learning, feedback, or other methods. The weights may be positive and / or negative, where, for example, a positive weight for an indicator may mean that the indicator implies or determines that the asset is under common control, while a negative weight for the indicator may mean that the indicator implies or determines that the asset is not under common control.

[0128] In some embodiments, machine rules and / or human input can be used to reduce logarithmic tail joins. For example, the graph processor may determine that assets A and B are jointly owned, and that assets B and C are jointly owned, thereby concluding that assets A and C are jointly owned. The graph processor can be programmed to limit the number of expansions. For example, if each of assets A, B, C, ..., J, K, L is found to be jointly owned with the next asset in the sequence, as described above, and there is an eight-step graph path, then assets A to H may be considered jointly owned, but not assets I to L simply because a path exists from asset A to asset L in each step.

[0129] In some embodiments, a user tool is used to upload an initial graph, such as a comma-delimited file or spreadsheet maintained by an operations team, listing assets known to be controlled by that team's organization. From there, the graph processor can infer assets not in the list.

[0130] Figure 6 Graph search and graph restriction are illustrated. As shown there, graph processor 602 reads asset inventory 604, which contains asset A as one of its entries. Graph processor 602 queries asset A to identify more data about asset A, to check whether it still exists and is responsive, etc. Graph processor 602 can do this for other assets already in asset inventory 604. Noting that asset A is related to asset B and asset C according to one or more flags, graph processor 602 can similarly query asset B and asset C to identify more data about them and to check whether they still exist and are responsive. Graph processor 602 can access asset C based on its link to asset A and find a correlation between asset C and asset D. In this example, asset D is deemed too distant to be considered, and therefore is not accessed.

[0131] Figure 7 Additional embodiments are illustrated.

[0132] According to one embodiment, the techniques described herein are implemented by one or more general-purpose computing systems that are programmed to perform the techniques according to program instructions in firmware, memory, other storage, or a combination thereof. Special-purpose computing devices, such as desktop computer systems, portable computer systems, handheld devices, network appliances, or any other device that incorporates hard-wired and / or program logic to implement the techniques may be used.

[0133] The operations of the processes described herein may be performed in any appropriate order, unless otherwise noted herein or clearly contradictory to the context. The processes described herein (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as codes (e.g., executable instructions, one or more computer programs, or one or more applications) that are executed together on one or more processors by hardware or a combination thereof. The code may be stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions that may be executed by one or more processors. The computer-readable storage medium may be non-transitory.

[0134] Conjunctions, such as phrases of the form "at least one of A, B, and C" or "at least one of A, B, and C," unless expressly stated otherwise or clearly contradicted by context, should be understood to generally present items, terms, and the like in a context that can be any non-empty subset of the set of A or B or C, or A and B and C. For example, in the illustrative example of a set having three members, the conjunction phrases "at least one of A, B, and C" and "at least one of A, B, and C" refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such conjunction language is generally not intended to imply that certain embodiments require that at least one of A, at least one of B, and at least one of C each be present.

[0135] The use of any and all examples, or exemplary language (e.g., "such as") provided herein, is intended merely to better illuminate embodiments of the invention and does not pose a limitation on the scope of the invention unless otherwise claimed. No language in the specification should be construed as indicating any non-claimed element essential to the practice of the invention.

[0136] In the foregoing description, embodiments of the present invention have been described with reference to numerous specific details, which may vary depending on the embodiment. The specification and drawings should therefore be regarded in an illustrative rather than a restrictive sense. The sole and exclusive indicator of the scope of the invention, and what the applicants intend to be the scope of the invention, is the literal and equivalent range of the claims in the specific form set forth in this application, including any subsequent amendments.

[0137] After reading this disclosure, those skilled in the art may envision further embodiments. In other embodiments, combinations or sub-combinations of the above inventions may be advantageously performed. Example arrangements of components are shown for illustrative purposes, and it should be understood that combinations, additions, rearrangements, etc. are contemplated in alternative embodiments of the present invention. Thus, while the present invention has been described with respect to exemplary embodiments, those skilled in the art will recognize that many modifications are possible.

[0138] For example, the processes described herein may be implemented using hardware components, software components, and / or any combination thereof. Accordingly, the description and drawings should be regarded as illustrative rather than restrictive. However, it will be apparent that various modifications and variations may be made thereto without departing from the broader spirit and scope of the invention as set forth in the claims, and the invention is intended to cover all modifications and equivalents within the scope of the appended claims.

[0139] All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein.< / company> < / tld> < / company>

Claims

1. A method for constructing a graph data structure, the graph data structure representing a prospective asset inventory graph and comprising nodes and edges, wherein each node represents an asset of a network, each edge represents a connection between nodes, and at least some edges have a weight represented in the graph data structure and indicate whether the assets represented in the graph data structure are under common control, the method comprising: storing a representation of at least one seed asset in the graph data structure; extending the expected asset inventory graph to include an additional node based on a common control matching criterion, wherein the common control matching criterion indicates that the additional node is assumed to be under common control of an asset based on the common control matching criterion, wherein the common control matching criterion indicates a match between the additional node and an existing node already present on the expected asset inventory graph; recursively applying the common control matching criteria between the additional node and a third node representing a third asset not already represented on the prospective asset inventory graph; Eliminating branches of the expected asset inventory graph based on identifying edges having weights less than a joint control flag threshold, wherein the joint control flag threshold is a threshold indicating joint control between nodes associated with the edges; and The graph data structure is updated based on the expansion and culling.

2. The method according to claim 1, wherein Extensions include: Reading the expected asset inventory map to identify metadata of known assets, the known assets comprising a set of assets already present on the expected asset inventory map; selecting predetermined metadata about the known asset; Normalizing the predetermined metadata to form normalized metadata; searching an asset database for a match to the normalized metadata; Filter out nodes for assets where the metadata is a product of a registration process rather than a sign of ownership or control; and For each remaining match, the asset is added to the expected asset inventory graph.

3. The method according to claim 1, wherein The weight of the edge of the expected asset inventory graph is the sum of the weights of the individual matches, so that the weight of the edge increases as the number of matches increases.

4. The method according to claim 1, wherein Nodes are assigned confidence levels, where the confidence level of a node corresponds to the likelihood that the asset of the node is a jointly controlled asset.

5. The method according to claim 1, wherein Additional nodes are derived from data structures that indicate asset ownership or asset relationships.

6. The method according to claim 5, wherein: The data structure includes one or more of a Domain Name System (DNS) database, an ASN registry, a historical DNS database, a company ownership document database, a historical lookup database, and / or a WHOIS database, and wherein expanding the graph data structure includes searching the Domain Name System (DNS) database, the historical DNS database, and / or the WHOIS database to identify nodes that meet matching criteria.

7. The method according to claim 5, wherein: The data structure includes one or more of a binary tree, a file shard set, a lookup table, an API and / or an mtbl.

8. The method according to claim 1, wherein Exclusions include obtaining manual input reflecting human input, blacklisting, and / or machine learning output indicating that the object is not a jointly controlled asset.

9. The method according to claim 1, wherein: Recursively applying the common control matching criteria is automatically performed and selecting a subset of available linking methods based on selection depth, confidence, settings, or other criteria, wherein selecting includes including methods and / or excluding methods.

10. The method according to claim 1, wherein Recursively applying the common control matching criteria is performed according to a user defined recursion and selecting a subset of available linking methods based on selection depth, confidence, settings or other criteria, wherein selecting includes including methods and / or excluding methods.

11. The method according to claim 1 , further comprising: formatting the representation of the prospective asset inventory diagram into a display format; as well as The display format is presented on a display.

12. The method according to claim 1, wherein Metadata related to the assets on the expected asset inventory map includes one or more of a host name, a list of vulnerabilities, a list of open ports used, an estimated geographic location of the asset, an operating system used for the asset, a service title for the asset, TLS certificate details for the asset, an IP address, a DNS type indicator, DNS registration data, and / or ASN information for the IP address.

13. The method according to claim 12, wherein: The metadata also includes the content of the technology stack indicating one or more of the language of the web application, the API of the web application, and / or the HTTP link including the social signal.

14. The method according to claim 1, wherein The assets of the network include one or more of domains, internet-connected assets, subdomains, IP addresses, virtual hosts, web servers, name servers, IoT devices, desktop computers, network printers, mail servers, devices connected to the internet or an intranet, content delivery networks, proxies, firewalls, intrusion detection systems, routers and / or switches.

15. The method according to claim 1, wherein The assets of the network include one or more devices capable of receiving network traffic.

16. The method according to claim 1, wherein The first asset is a first domain, the second asset is a second domain, the first asset is on the expected asset inventory map, and the common control matching criteria include one or more of the following: a first test as to whether the second domain shares a common registered email address with the first domain; a second test as to whether the second domain shares a common registered email address domain with the first domain; a third test as to whether the second domain was registered using an email address whose email domain matches the email domain of the first domain; a fourth test as to whether the second domain and the first domain share a common WHOIS field; a fifth test as to whether both the second domain and the first domain include hosted content that refers back to a common host; as well as A sixth test as to whether the second domain and the first domain both use a common authentication authority.

17. The method according to claim 16, wherein Perform at least one test using historical data.

18. The method according to claim 16, wherein At least one domain is a subdomain.

19. The method according to claim 16, wherein Execute at least one test for the state at a specific time.

20. The method according to claim 1, wherein The common control matching criteria includes a test to check whether a site has pages that include links to certain other sites outside of a predetermined set of commonly linked sites.

21. The method according to claim 1, wherein The common control matching criteria include testing for a degree of relatedness between a first site and a second site, and / or common links found in both the first site and the second site.

22. The method of claim 1, further comprising: Filters addresses on a predetermined list of reusable internal addresses.

23. The method of claim 1, further comprising: Filtering is performed on TLDs that are on a predetermined list of commonly reused TLDs, or on a predetermined list of commonly reused TLD portions.

24. The method of claim 1, further comprising: The affinity is adjusted based on whether the affinity is initially determined based on usage of the address space, wherein the resulting addresses are masked by overlapping address ranges that are reusable on multiple unrelated entities.

25. The method according to claim 24, wherein Uses of the address space include internal RFC 1918 addresses, RFC 4193 addresses, RFC 6890 addresses, RFC 3927 addresses, loopback addresses, link-local addresses, broadcast addresses, carrier-grade NAT, unique local addressing, and / or non-routable Internet Protocol addresses.

26. The method of claim 1, further comprising: The affinity is adjusted based on a predetermined list of links between known providers that share an address space.

27. The method of claim 1, further comprising: The relevance is adjusted based on a predetermined list of privacy services whereby unrelated parties using a given privacy service are considered less relevant than if they were related parties.

28. The method according to claim 1, wherein The first asset is a first subdomain, the second asset is a second subdomain, the first asset is on the expected asset inventory map, and the common control matching criteria include one or more of the following: a first test as to whether the second subdomain shares a common IP address with the first subdomain; a second test as to whether the second subdomain shares a common CIDR block with the first subdomain; and A third test is performed as to whether the second subdomain shares a common CIDR signature with the first subdomain.

29. The method of claim 1, further comprising: Reduce the probability of linking caused by sharing IP space of a known provider via privacy services used by multiple unrelated parties.

30. The method according to claim 29, wherein Sharing is the sharing of one or more of an email address, phone number, physical address, whois entry, and / or business address.

31. The method of claim 1 , further comprising: Reduce the probability of links resulting from sharing one or more of the following: domain proxy services, domain privacy services, blank or undefined whois results, and / or placeholder results.

32. The method of claim 1, further comprising: User-defined recursion.

33. The method of claim 1, further comprising: The processing is based on whether two or more IPs both have a PTR record for the hostname of the shared top-level domain.

34. The method of claim 1, further comprising: Reduce the probability of links resulting from top-level domains used on unrelated entities.

35. The method according to claim 34, wherein A valid TLD or TLDs are considered to be associated with a predetermined adjustable relevance weight.

36. The method of claim 1, wherein ASNs designated as reserved or private are considered irrelevant.

37. A method of processing a graph data structure, the graph data structure representing a prospective asset inventory graph and comprising nodes and edges, the nodes each representing an asset of a network, the edges each representing a connection between nodes, and at least some edges having a weight represented in the graph data structure and indicating whether the assets represented in the graph data structure are under common control, the method comprising: obtaining a representation of the graph data structure; obtaining a common control matching criterion, the common control matching criterion indicating common control of an asset between a first node and a second node; Obtaining a set of rules corresponding to possible false positive indications of joint control; as well as When a rule in the rule set indicates that the common control matching criteria indicates a condition that results in a false positive indication, the rule set is applied to the graph data structure to decrease a first weight between the first node and the second node.

38. The method of claim 37, wherein: Applying the rule set to the graph data structure to reduce the first weight includes processing a plurality of rules in the rule set, each rule in the plurality of rules indicating a possible false positive shared control indication.

39. The method of claim 37, further comprising: Manual feedback indicating a false positive is applied to reduce the first weight.

Images