Data processing system and method based on device usage associated with Internet devices

Abstract

The data processing system performs data processing of raw or pre-processed data. The data includes log files, bitstream data, and other network traffic containing cookies or device identifiers. The data processing system associates devices with device activity histories. The data processing system includes a pairing framework for determining device pairs based on device activity histories, a feature vector generation framework for generating feature vectors corresponding to determined candidate device pairs based at least in part on feature values ​​associated with devices of the determined candidate device pairs within the raw or pre-processed data, a scoring engine for determining scores associated with the determined device pairs based on the generated feature vectors, a graph structure including nodes for representing devices of the determined device pairs (wherein edges between pairs of nodes represent the determined device pairs), and a clustering engine for identifying clusters of corresponding n or more nodes within the graph structure representing corresponding groups of devices.

Description

[0001] This application is a divisional application of Chinese application No. 201880020315.9, entitled “Data processing system and method based on device use associated Internet device”, filed on January 23, 2018. Technical Field

[0002] The present disclosure relates to a data processing system for performing data processing for associating Internet devices based on device usage. In addition, the present disclosure relates to a method of using the above data processing system for associating Internet devices based on device usage. In addition, the present disclosure relates to a software product recorded on a machine-readable data storage medium, characterized in that the software product can be executed on computing hardware to perform the above method. Background Art

[0003] Traditionally, the Internet provides a data communications network through which people are able to exchange information using a variety of different types of devices. For example, a given user owns a smart phone, a mobile tablet, a laptop, and a connected television. As users work, socialize, research, and purchase products on multiple different Internet-connected devices, businesses are continuing to shift their focus to more efficiently reaching users through their respective multiple devices. For example, while a given person owns and uses different devices to communicate over the Internet, the relationships between the different devices and the users of the different devices are not obvious to outsiders, such as a business seeking to reach a given person through his multiple devices.

[0004] This above-mentioned relationship may involve a given person using different devices with different device identifiers to communicate over the Internet. For example, a given person communicates anonymously over the Internet without disclosing the given person's personal identifier. The connection of a user device to the Internet is typically temporary and dynamic. Devices are typically connected to the Internet through a connection point associated with an Internet Protocol (IP) address. However, a user device may use different network addresses at different times. During communication over the Internet, user device identification information, such as a device identifier or a user identifier, may be exchanged. However, the identification information used by a given user using one device during Internet communication may be different from the identification information used by the same user during Internet communication using different devices. In addition, the same given device may use different identification information during different Internet communications. Therefore, when communicating with different devices over the Internet, a person may use different IP addresses at different times. For example, a user has multiple different email accounts and may participate in the use of social media under different pseudonyms. Therefore, there is currently no existing readily available reliable deterministic method for identifying users who access the Internet using different devices. Summary of the invention

[0005] Various aspects of the disclosed embodiments attempt to provide an improved system for determining different types of associations between devices based on raw or unprocessed data describing the devices' data network activity, such as Internet browsing activity (e.g., intra-device associations, inter-device associations including ownership, associations in usage shares, household associations, etc.); such an improved system is based on the identity of a given individual being constant across boundaries of platforms, devices, logins, etc., and the identity of a given individual can be inferred based on usage information collected about the given individual.

[0006] According to a first aspect, there is provided a system comprising:

[0007] (i) a computer-readable storage device for storing raw or pre-processed data associating a device identifier with a device activity history;

[0008] (ii) a pairing framework for determining different types of candidate device pairs based at least in part on at least a portion of the device activity history;

[0009] (iii) a feature vector generation framework for generating a multi-feature value feature vector corresponding to the determined candidate device pair based on the one or more types of pairs and based at least in part on a history of device activities associated with the devices of the determined candidate device pair within the raw or pre-processed data,

[0010] Features:

[0011] (iv) the feature vector generation framework is operable to employ at least one rule to generate a feature value for at least one feature represented in the feature vector corresponding to the at least one determined candidate device pair based at least in part on a device activity history associated with a first device identifier of the at least one determined candidate device pair in the raw or preprocessed data and a device activity history associated with a second device identifier of the at least one determined candidate device pair in the raw or preprocessed data;

[0012] (v) a scoring engine operable to determine scores associated with determined candidate device pairs including at least one determined candidate device pair based at least in part on the generated feature vectors associated with the determined candidate device pairs;

[0013] (vi) a computer-readable storage device operable to store a graph structure, wherein nodes within the graph structure represent device identifiers, including a first device identifier and a second device identifier of at least one determined candidate device pair, and wherein edges between pairs of nodes within the graph structure represent determined candidate device pairs; and

[0014] (vii) The clustering engine is operable to identify clusters of respective two or more nodes within the graph structure representing respective groups of devices.

[0015] Advantages provided by aspects of the disclosed embodiments are that the system is able to provide associations that more efficiently identify user devices based on raw or unprocessed data (i.e., avoiding the need to generate any form of index structure, thereby saving computational effort and / or generating association results more quickly).

[0016] It will be appreciated that with respect to “activity history,” for example, each Internet communication from a device may be captured as an event with transmission information, and the sequence of these events may be considered a history of the device’s activity on the Internet.

[0017] It will be appreciated that, for example, with respect to a "framework," there may be different engines for different types of pairs, with at least two such engines being described later in this disclosure (eg, for intra-device pairs and inter-device pairs).

[0018] Optionally, in the system, raw or unprocessed data includes, for example, log files, bitstream data, and other network traffic containing cookies or device identifiers.

[0019] According to a second aspect of the disclosed embodiment, a method for determining association using a system is provided, wherein the method comprises:

[0020] (i) using a computer-readable storage device to store raw or pre-processed data associating a device identifier with a device activity history;

[0021] (ii) using a pairing framework configured to determine candidate device pairs based at least in part on at least a portion of the device activity history;

[0022] (iii) using a feature vector generation framework to generate a multi-eigenvalue feature vector corresponding to the determined candidate device pair based at least in part on an activity history associated with devices of the determined candidate device pair within the raw or pre-processed data, wherein the method further comprises:

[0023] (iv) arranging the feature vector generation framework to employ at least one rule to generate a feature value for at least one feature represented in the feature vector corresponding to the at least one determined candidate device pair based at least in part on a device activity history associated with a first device identifier of the at least one determined candidate device pair in the raw or preprocessed data and a device activity history associated with a second device identifier of the at least one determined candidate device pair in the raw or preprocessed data;

[0024] (v) using a scoring engine to determine scores associated with determined candidate device pairs including at least one determined candidate device pair based at least in part on the generated feature vectors associated with the determined candidate device pairs;

[0025] (vi) using a computer-readable storage device to store a graph structure, wherein nodes within the graph structure represent device identifiers, including a first device identifier and a second device identifier of at least one determined candidate device pair, and wherein edges between pairs of nodes within the graph structure represent determined candidate device pairs; and

[0026] (vii) using a clustering engine to identify clusters of respective two or more nodes within the graph structure representing respective groups of devices.

[0027] According to a third aspect, there is provided a computer program product comprising a non-transitory computer-readable storage medium having computer-readable instructions stored thereon, the computer-readable instructions being executable by a computerized device comprising processing hardware to perform the method of the second aspect.

[0028] It will be appreciated that features of the invention are susceptible to being combined in various combinations without departing from the scope of the invention as defined by the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Embodiments of the present disclosure will now be described, by way of example only, with reference to the following drawings, in which:

[0030] Figure 1A and 1B is a diagrammatic representation of the use of network connection devices at different network addresses associated with different physical embodiments of the present disclosure;

[0031] Figure 2 is a schematic diagram of a data processing system according to an embodiment of the present disclosure, wherein the data processing system includes a digital identity pairing engine;

[0032] Figure 3 It means that according to the embodiment of the present disclosure Figure 2 An illustrative process flow diagram of a configuration of a digital identity pairing engine of a data processing system of;

[0033] Figure 4A The use of the embodiments according to the present disclosure Figure 2 An illustration of a plurality of example device pairs identified by a pairing engine of ;

[0034] Figure 4B is associated with an example feature vector according to an embodiment of the present disclosure Figure 4A An illustration of an example device pair;

[0035] Figure 5 It means that according to the embodiment of the present disclosure Figure 2 An illustrative process flow diagram of a configuration of a feature vector generation engine of a system;

[0036] Fig. 6A It means that according to the embodiment of the present disclosure Figure 5 an illustrative flow chart of a process of applying example details of a first example rule to produce example feature values ​​as a function of feature information associated with a candidate device pair;

[0037] Figure 6B It means that according to the embodiment of the present disclosure Figure 5 an illustrative flow chart of a process of applying example details of a second example rule to produce example feature values ​​as a function of feature information associated with a candidate device pair;

[0038] Figure 7 is a method for providing low-level feature information in an input storage system for generating feature values ​​according to an embodiment of the present disclosure and Figure 2 illustrative diagram of examples of rules within a rule store of a system;

[0039] Fig. 8A It means that according to the embodiment of the present disclosure Figure 2 An illustrative flow chart of the configuration of a scoring engine of a data processing system;

[0040] Figures 8B to 8E It means that according to the embodiment of the present disclosure Figure 2 An illustrative diagram of an alternative configuration of a scoring engine of a data processing system;

[0041] Fig. 9 It means that according to the embodiment of the present disclosure Fig. 8A The scoring engine (optionally also used for Figures 8B to 8E An illustrative flow chart of the details of the training module of the scoring engine of FIG.

[0042] Fig. 10A It means that the unsupervised pre-training module according to the embodiment of the present invention can be operated to train the feature vector based on the sample device. Figure 7 An illustrative diagram of an example of remapping of device pairs to device pair clusters;

[0043] Fig. 10B It is a representation according to an embodiment of the present invention Fig. 9 An illustration of the use of a remapping module of a training module;

[0044] Fig.11 It means that according to the embodiment of the present invention, Figure 2 An illustrative diagram of a data flow process of data transformation occurring within a computer readable storage device within a scoring engine of a data processing system;

[0045] Fig.12 It is a representation according to an embodiment of the present invention Figure 2an illustrative flow chart of a configuration of a clustering engine of a system;

[0046] Fig.13 It means that according to the embodiment of the present invention Fig.12 The clustering engine is used to Figure 4A and 4B An illustration of an example graph generated by a device pair;

[0047] Fig.14 It means that according to the embodiment of the present invention Fig.13 Used in the diagram Fig.12 An illustrative diagram of an example proposed user device clustering set of user device clusters identified by a clustering engine;

[0048] Fig.15 It means that according to the embodiment of the present invention Fig.14 The proposed user equipment clustering uses Fig.12 An illustrative diagram of an example set of final device clusters generated by the clustering engine;

[0049] Fig.16 It means that according to the embodiment of the present invention, Figure 2 an illustrative diagram of a data flow diagram of data transformations occurring during clustering of user devices within a clustering engine of a data processing system;

[0050] Fig.17 It is a device configured to operate in a network environment according to an embodiment of the present disclosure. Figure 2 An illustration of a data processing system.

[0051] In the accompanying drawings, underlined numbers are used to indicate the item in which the underlined number is located or the item to which the underlined number is adjacent. Ununderlined numbers relate to the item identified by the line linking the ununderlined number to the item. When a number is ununderlined and is accompanied by an associated arrow, the ununderlined number is used to identify the general item to which the arrow points. DETAILED DESCRIPTION

[0052] In general, a description of an embodiment of the present disclosure is provided below, which enables any person skilled in the art to create and use a data processing system to associate Internet-connected devices belonging to the same given user based on anonymous Internet user data (i.e., the "activity history" associated with the Internet-connected devices). Various modifications to the embodiments will be apparent to those skilled in the art. The same reference numerals may be used to represent different views of the same item in different figures. The flowcharts in the figures referenced below are used to represent processes. A machine such as a computer system including one or more processors and storage devices is configured to perform these processes. The flowchart represents the configuration of one or more processors and / or storage devices of a computer system configured using computer program code to perform the operations described with reference to the flowchart.

[0053] Embodiments of the present disclosure provide a data processing system that is operable to use raw or pre-processed data to associate devices with device attributes, such as device activity history; optionally, the raw or unprocessed data includes, for example, log files, bitstream data, and other network traffic containing cookies or device identifiers. The data processing system uses a pairing engine in operation to determine a device pair based at least in part on at least a portion of the aforementioned device attributes. In addition, the data processing system uses a feature vector generation engine to generate a feature vector corresponding to the determined device pair based at least in part on feature values ​​associated with the devices of the determined device pair in the raw or pre-processed data. In addition, the data processing system includes a scoring engine for determining a score associated with the determined device pair based at least in part on the generated feature vector associated with the determined device pair. In addition, the data processing system uses a graph structure in operation, the graph structure including nodes representing devices of the determined device pair, and including edges between pairs of nodes indicating the determined device pair. In the data processing system, a clustering engine is also used to identify corresponding node clusters representing corresponding device groups within the graph structure.

[0054] Next, we will refer to Figure 1A Overview of digital identity pairing. In a given example home A, represented by 150, there are living people (i.e., "users") who use multiple mobile Internet-connected digital devices to communicate with the Internet 101 or a similar data communication network. Specifically, notebook computer systems (151 and 161), cellular phones (152 and 162), and tablet computer system 157 are all mobile Internet-connected digital devices, each with a unique device identifier. These mobile devices can and often are taken to other locations that provide wireless (or wired) Internet access with one or more of their users. For example, a given user X of home A can use a notebook computer system 151 and a cellular phone 152, whether in home A 150 or in a workplace W represented by 110.

[0055] When a given user X brings the devices of a notebook computer system 151 and a cell phone 152 to the workplace W 110, those devices are no longer able to use the Wi-Fi router 165 to access the Internet 101. Instead, the given user X connects the notebook computer system 151 to the local area network 129 at the workplace W 110, and configures the cell phone 152 to use the local Wi-Fi network provided by a wireless access point (e.g., of the workplace W 110). With these two Internet connections, the given user X will then be able to use the notebook computer system 151 and the cell phone 152 to access the Internet 101 through the firewall / proxy server 121 at the workplace W 110.

[0056] When a given user X is at home A 150, both the notebook computer system 151 and the cell phone 152 will use a single Internet Protocol (IP) address A 163 on the Wi-Fi router 165. Similarly, when the same given user X is at work W 110, both the notebook computer system 151 and the cell phone 152 will use a single IP address W 123 on the firewall / proxy server 121. Of course, a given user X may selectively send and receive information using one or the other or both of the notebook computer system 151 and the cell phone 152. However, a third party communicating with the given user X via the notebook computer system 151 may not be aware that the given user X may also be reached via the cell phone 152, and a third party communicating with the given user X via the cell phone 152 may not be aware that the given user X may also be reached via the notebook computer system 151.

[0057] However, given the above reference Figure 1A Given the specific Internet usage pattern data described, a keen observer who knows nothing about a given user X and is unaware that the given user X owns both the notebook computer system 151 and the cell phone 152 can reasonably infer that the notebook computer system 151 and the cell phone 152 are very likely used by the same person because the notebook computer system 151 and the cell phone 152 are used together at home, i.e., home A 150, and at work W 110. After making such an inference, the machine can link together the digital identifier associated with the notebook computer system 151 and the digital identifier associated with the cell phone 152. This pairing of different digital identities with a single user is referred to as digital identity pairing.

[0058] Next, we will refer to Figure 2 Provides an overview of the system. Figure 2An illustrative block diagram representing a system 200 according to some embodiments of the present disclosure is provided in ; the system 200 is also referred to as a data processing system. The system 200 includes a computer-readable feature storage device 208 for storing device identifiers and indicia of features associated with the identified devices, a pairing engine 202, a feature vector generation (FVG) engine 212, a scoring engine 204, and a clustering engine 206. The pairing engine 202 identifies a candidate device pair based on low-level preliminary pairing identification features associated with the devices of the candidate device pair indicated in the computer-readable feature storage device 208. It should be understood that the term "candidate" is used herein to indicate that the pairing determination is preliminary and can undergo further evaluation in the scoring engine 204 and the clustering engine 206. The FVG engine 212 generates a feature vector in response to the candidate device pair as a function of additional feature information (i.e., device activity history) associated with each device of the candidate device pair. Specifically, the pairing engine 202 provides the candidate device pair identifier to the FVG engine 212. In response to the received candidate device pair identifier, the FVG engine 212 uses the individual identities of the devices of the candidate device pair to identify additional feature information associated with those individual devices of the candidate device pair in the computer-readable feature storage device 208, namely, the device activity history. The FVG engine 212 generates a feature vector value to be associated with the identified candidate device pair as a function of the additional feature information (i.e., the device activity history) identified as being associated with the individual devices of the identified candidate device pair. The scoring engine 204 determines a score associated with the identified candidate device pair based at least in part on the feature vector generated using the FVG engine 212. The clustering engine 206 generates a graph structure representing at least a portion of the identified candidate device pair and the determined score in the computer-readable storage device. The clustering engine 206 identifies digital identity clusters U.sub.1, U.sub.2, ..., U.sub.N within the graph structure indicating device clusters or device groups associated with the same user.

[0059] According to some embodiments, each individual digital identity cluster stored within a storage device acts as a digital identity group communication structure U.sub.1, U.sub.2, ..., U.sub.N, including a group of devices associated with the same user and that can be used to communicate with the user who owns the grouped devices. Thus, for example, the digital identity group communication structure U.sub.1 includes devices D.sub.A1, D.sub.A2, ..., D.sub.AK. Specifically, the digital identity grouping reveals a plurality of grouped devices that can be used to communicate with the owner of the grouped devices. In contrast, without the digital identity grouping, a third party wishing to communicate with a particular device user cannot easily identify the different devices that communicate with the particular device user. Therefore, the digital identity group communication structure identifies a group of devices associated with the same user in order to expand the available communication pathways with the user, thereby increasing the opportunity to communicate with the user. It should be understood that based on the digital device pairing determined using the pairing engine 202, the scoring using the scoring engine 204 based on the feature vectors generated using the FVG engine 212, and the clustering using the clustering engine 206, a device identity group U.sub.1, U.sub.2, ..., U.sub.N that is previously unknown to certain third parties who wish to communicate with users associated with the group is determined.

[0060] Specifically, in some embodiments of the present disclosure, the computer-readable feature storage device 208 includes one or more storage devices storing the above-mentioned raw or pre-processed data 209 and tag data that associate device identifier information with low-level feature information (i.e., low-level device activity history). The low-level feature information can be stored in the computer-readable feature storage device 208 itself, or can be stored in different storage locations (not shown) indicated in the computer-readable feature storage device 208. For example, the low-level data can be obtained from a server log requested by an Internet-connected device. The system 200 includes a computer-readable rule storage device 213, which stores rules for generating high-level feature values ​​based on the low-level feature information indicated in the computer-readable feature storage device 208. Specifically, the computer-readable rule storage device 213 stores instructions for configuring the FVG engine 212 to identify additional high-level feature information, i.e., high-level device activity history, based on the low-level features associated with each device of the candidate device pair, in response to the pairing engine 202, to associate with the candidate device pair, and use the additional feature information together with the low-level feature information to generate feature values ​​used in the feature vector. The system 200 includes buffer circuits 210, 211, which are configured to receive certain low-level feature information and tag data as input, respectively. In some embodiments, the low-level feature information received in the buffer circuit 210 includes a device ID, a public source / destination identifier (e.g., an IP address), and a timestamp.

[0061] The pairing engine 202 uses preliminary pairing identification feature information, such as device ID, common source / destination identifier (e.g., IP address), and timestamp, to determine preliminary pairings of devices. As used herein, the term "device pair" refers to an information structure stored in a storage device and indicating pairings of different digital identities with a single user. The pairing engine 202 acts as a pre-filter that generates device pairs that represent preliminary "candidate" pairings of devices. The pairing engine 202 also associates a preliminary pair score with each device pair, which provides an indication of the likelihood that the devices in the pair are actually associated with the same user.

[0062] The FVG engine 212 is configured to generate a feature vector to be associated with a candidate device pair using a set of rules from a computer-readable rule storage device 213 as a function of low-level feature information, i.e., low-level device activity history, associated with different individual devices of the candidate device pair within the FVG engine 212. The low-level feature information indicated within the FVG engine 212 includes user profile information associated with the individual devices. The low-level feature information indicated within the computer-readable feature storage device 208 may be collected via the Internet and may include information such as demographic data such as age, gender, or marital status and / or behavioral (British English: "behavior") data such as a user's intent to purchase certain items or personal interests such as likes or dislikes.

[0063] Scoring engine 204 receives as input a candidate device pair identifier (such as a device ID of a device within the pair), an associated preliminary pairing score generated by pairing engine 202, an associated feature vector generated by FVG engine 212, and associated tag information. In some embodiments of the present disclosure, the preliminary pairing score associated with a candidate device pair generated using pairing engine 202 is included as a feature value of a feature vector associated with the candidate device pair. Scoring engine 204 generates a refined pairing score for the received device pair at least in part as a function of its associated feature vector. The refined pairing score generated using scoring engine 204 indicates the likelihood that the device pair is actually associated with the same user. The pairing score generated by scoring engine 204 preferably replaces the preliminary pairing score generated using pairing engine 202.

[0064] In operation, the clustering engine 206 receives as input the device pairs and associated precision pair scores generated using the scoring engine 204. The clustering engine 206 generates a graph structure (not shown) based on the device pairs in a computer-readable storage device. The graph nodes represent devices associated with device identifiers. The edges in the graph represent potential pairings of devices for candidate device pairs. Precision pair scores are associated with the edges of the graph. The clustering engine 206 identifies user device clusters within the graph based on the pair scores associated with the edge connections within the graph. The clustering engine 206 identifies user device groups associated with common users. The clustering engine 206 stores information identifying the respective device groups associated with the respective users in the computer-readable storage device 214.

[0065] Example user device clusters U.sub.1, U.sub.2, ..., U.sub.N, used as digital identity group communication structures for respective owners of devices in the cluster, identified using the clustering engine 206, are shown as being stored in a computer-readable storage device 214 associated with respective user identifier information. Different device identifier groups associated with different respective users. Although the actual identity of the unique user is unknown, each cluster associates a device ID with a unique user. For example, the first user device cluster U.sub.1 indicated by the device ID (D.sub.A1, D.sub.A2, ..., D.sub.AK) is associated with the unique user U.sub.1. Clusters of device IDs may be referred to herein as "device clusters" or "user device clusters".

[0066] In some embodiments of the present disclosure, clustering engine 206 provides feedback F to scoring engine 204. According to some embodiments of the present disclosure, one example feature value considered by scoring engine 204 is an indication of the number of additional devices indicated in the graph as potentially pairing with the device of the candidate device pair. This example indication provides a view of the larger graph that does not necessarily exist alone in the triplet information. This additional potential pairing information is an example of "feedback" F by which clustering engine 206 can influence the precision scores produced using scoring engine 204, which in turn can influence the precision scores associated with the edges of the graph structure.

[0067] In addition, for example, the clustering engine 206 can provide the previous exact pair scores as feedback F to the scoring engine 204 to determine later updated exact pair scores by the scoring engine 204. Using the previous pair scores generated by the scoring engine 204 can allow the scoring engine 204 to determine new updated pair scores to improve on previous pair scores generated when more limited information about the candidate device pairs was available (e.g., when less feature information was available). It should be understood that identifying possible device pairs is an iterative process over time, and as more information becomes available to the system 200, the probability that a prediction of a given pairing is correct is potentially greater.

[0068] Next, the devices and device identifiers employed in the system 200 during operation will be described. As used herein, the terms "device ID" and "device identifier" and "digital identity" refer to, for example, an identifier of a user device, a user device program stored in a computer-readable storage device, a user device information structure (e.g., software cookies) stored in a computer-readable storage device, or a device user identity. For example, as used herein, a "computer-readable" storage device refers to a non-transitory storage device from which stored information can be extracted using a programmable computer or device (e.g., a smart phone). Examples of device IDs include web browser cookies, cell phone device identifiers, MAC addresses, user ids, and other identifiers linked to a specific client device, client program, or device user. For example, as used herein, the terms "device" or "user device" are generally used to refer to digital entities such as notebook computer systems, desktop computer systems, cell phones, tablet computer systems, smart watches, smart devices such as Internet-connected appliances, and web browsers. The teachings of the present disclosure can be used with a variety of different device IDs. In an example of pairing a disclosed digital identity, a notebook computer system, a web browser cookie on a desktop computer, and a device identifier on a cell phone device are used as device IDs. However, the disclosed techniques may be used with any other suitable device that can be used to identify a particular client device, web browser, user, or other digital identity. For example, as used herein, the letter "D" is used to refer to a physical device such as a mobile phone, and the letter "C" is used to refer to a cookie representing a physical device such as a notebook computer. It will be understood from the foregoing that mobile phone and browser cookies encoded in a computer-readable storage device may be referred to herein as "devices."

[0069] Next, features and feature information used in operation in system 200 will be described. As used herein, the term "feature" refers to an attribute associated with a device. As used herein, the term "feature information" refers to information indicating one or more features, such as a device activity history. According to some embodiments of the present disclosure, feature information is collected by observing Internet communications. According to some embodiments, feature information is used to generate feature values ​​that are constituent components of feature vectors.

[0070] For example, low-level feature information includes the following information:

[0071] (i) Source device attributes, e.g., iPad, ID of the installed application (e.g., an application named “Words with Friends” is installed),

[0072] (ii) Device user’s endpoint, device ID, geographic location, date, etc.,

[0073] (iii) target device properties (e.g., installed Mac OS X, three other devices associated, etc.),

[0074] (iv) contextual attributes (e.g., website topic, etc.), and

[0075] (v) IP attributes (e.g., how many unique devices have been observed at this IP address, how many days the IP address has been observed).

[0076] Typically, device-to-device related information comes from several broad categories:

[0077] (a) spatial and temporal observations,

[0078] (b) demographics, and

[0079] (c) Satellite pairs.

[0080] Low-level demographic information includes, for example, the following information: (i) What is the age, gender, home location, and consumer interests of the C and D parties of the pair?

[0081] (ii) Do the descriptions C and D correspond to a unique user?

[0082] Low-level satellite pair information includes, for example, the following information:

[0083] (a) Are there other pairs that are related to either C or D? (Note: It is understood that in graph theory, two edges are said to be “related” if they share a vertex).

[0084] (b) How do these pairs compare to the current pairs?

[0085] Optionally, satellite pair information is injected back into the pairing prediction process from the perspective of the device map (described more fully below).

[0086] The FVG engine 212 generates a feature vector that represents high-level feature information about a candidate device pair as a function of low-level feature information. Specifically, in some embodiments, the FVG engine 212 is configured to use rules to generate feature vectors as a function of device usage information (e.g., device activity history). According to some embodiments of the present disclosure, many high-level features are developed based on spatiotemporal information. In some embodiments, the feature FVG engine 212 uses spatiotemporal heuristics to evaluate low-level features (e.g., collecting all observations of device A on IP address C, how frequently device A appears on IP address C during nighttime or weekends).

[0087] Next, the pairing engine 202 will be described in more detail. Optionally, the pairing engine 202 is implemented according to some embodiments as described in U.S. patent application US 1014 / 0095320 filed on May 10, 2013 and published on April 3, 2014; however, it should be understood that other types of implementations of the pairing engine 202 are also within the scope of the present disclosure, as will be explained in more detail below. Figure 3 In the figure, some embodiments according to the present disclosure are shown. Figure 2 1. An illustrative process flow diagram of an example configuration of a digital identity pairing engine of a system of FIG. Initially, module 510 collects Internet usage data for a number of different digital identities from computer-readable feature storage device 208. In some embodiments of the present disclosure, the Internet usage data collected for analysis includes data triples including a client device ID, a network source / destination address identifier (e.g., an IP address), and a timestamp.

[0088] In embodiments of the present disclosure, a common source / destination identifier is the identity of some source or destination that client devices (as identified by their device IDs) might have in common if the two client devices were related. Figure 1A In the case shown in , an IP address is a common source / destination identifier that can be used to link related client devices. Specifically, the digital identity pairing system can use the fact that laptop computer system 151 (e.g., a given laptop computer system) and cell phone 152 (e.g., a cell phone) share a single IP address W 123 when at work and a single IP address A 163 when at home to infer that laptop computer system 151 and cell phone 152 are related client devices.

[0089] The timestamp in each data triplet can be used to ensure that the data used is relevant. The ownership of a given Internet-connected device may change over time, so that very old Internet usage data, such as device activity history, should optionally not be used. In some embodiments of the present disclosure, IP (Internet Protocol) addresses are used to implement network source / destination addresses. Many IP addresses are "dynamic addresses" that can be used by different entities at different times. Therefore, Internet usage data observations should have a relatively close temporal relationship (when used in embodiments of the present disclosure) in order to provide accurate digital identity pairing results; reference will be made later to Figures 8B to 8E This approach is described in more detail.In addition to ensuring that Internet usage observations are close in time, certain embodiments of the disclosed system 200 use the timestamps in Internet usage data triplets in more sophisticated ways, as will be disclosed later in this disclosure.

[0090] The triplet of Internet usage data described above (device ID, public source / destination identifier, and timestamp) can be collected by an Internet network server that tracks each received Internet server request. Alternatively, for example, in some embodiments of the present disclosure, individual applications (such as games, media aggregators, utilities, etc.) that run on client devices and report usage information to servers on the Internet can also be sources of usage data. For example, it is feasible to use cookie-type software to provide data collected for system 200, that is, to provide device activity history data.

[0091] Reference again Figure 3 After collecting the Internet usage data, according to some embodiments of the present disclosure, the next step of digital identity pairing is to determine a set of candidate digital identity pairs. Typically, in an Internet environment, millions of different digital identities are involved in Internet activities every day. Trying to analyze every possible permutation of digital identities as potential digital identities would be an extremely difficult and potentially futile computational task. Therefore, in order to reduce the size of the digital identity pairing problem, module 520 analyzes the collected Internet usage data to identify a much smaller number of candidate digital identity pairs that have a suitable (i.e., reasonable) probability of being related. That is, the size of the search space used to determine the association between Internet requests and device IDs is limited by applying a pre-filter to the search space; this approach greatly reduces the computational workload in system 200.

[0092] In embodiments of the present disclosure that use IP addresses as public source / destination identifiers, two different techniques are optionally used within system 200 to select potential digital identity pairs for further analysis. The first strategy is to check the number of different digital identities known to use the same IP address. Specifically, if less than a threshold number of digital identities are known to use a particular IP address, all different digital identity logical pairings from that single IP address may be considered potential digital identity pairs. A related reasoning is that if only a small number of different digital identities are associated with a single public IP, then it is likely that some of these different digital identities are associated with the same person, and that person is able to statistically link digital identities belonging to the same person. For example, family members who share a single Internet account may have family members who use more than one digital identity that can be statistically linked in embodiments of the present disclosure.

[0093] In some embodiments of the present disclosure, for example, the threshold is set to 6, so that if six or fewer digital identities are seen at a particular IP address, then various logical combinations of these six or fewer digital identities may be considered potential digital identity pairs. Figure 1A, home A 150 has only six different digital devices (151, 152, 161, 162, 157, and 159) that are coupled to the Internet 101 via a single IP address A 163 on a Wi-Fi router 165, such that the various digital devices in home A 150 can be considered candidate digital identity pairs. In contrast, a very large number of digital devices are coupled to the Internet 101 via a single IP address W 123 at workplace W 110, such that the digital identity pairing system does not immediately consider all combinations of digital devices at workplace W 110 to be potential digital identity pairs. In practice, system 200 identifies household residents (which typically have fewer than six Internet-connected devices) and then attempts to pair digital devices from household residents used by the same user. Thus, this approach employs a simple rule that allows a given client's device to be identified in system 200.

[0094] In another embodiment of the present disclosure, the digital identity pairing system considers a particular IP address source and determines whether the IP address is an address where a paired digital identity may be found (e.g., a family residence as described above). Static IP addresses on the Internet are typically assigned by the Internet Corporation for Assigned Names and Numbers (ICANN). By checking who owns a particular IP address, it can be determined whether it is easy to identify a related digital identity that may be located at that IP address. Thus, for example, an IP address used by an Internet Service Provider (ISP) to provide residential Internet service may be a good IP address to use when identifying potential digital identity pairs. Various other systems for identifying residential home IP addresses may also be used. In addition, in addition to or in lieu of a system for identifying residential homes, other techniques for identifying possible digital identity pairs may also be used in module 520.

[0095] After selecting a set of potential digital identity pairs using module 520, module 540 then processes the collected Internet usage data, i.e., the collected feature information, specifically triple information, to determine preliminary pair scores for candidate digital identity pairings. Since the observed Internet usage data will change over time and certain opportunistic activities may result in the detection of erroneous digital identity associations, the preliminary pair scores generated using pairing engine 202 may be post-processed to remove noise. For example, various techniques may be used to smooth the association scores over time, such as by taking a time moving average. Thus, at stage 560, the association score data generated over time may be post-processed pair scores generated using module 540, such that anomalous data points are largely filtered out. Stage 560, for example implemented as a module, generates a set of high probability digital identity pairings.

[0096] It should be understood that in some embodiments of the present disclosure, the preliminary pairing scores generated using the pairing engine 202 provide preliminary pairing scores that are coarser, i.e., less precise, than the pairing scores generated using the scoring engine 204. In some embodiments of the present disclosure, the scoring engine 204 considers the preliminary pairing scores generated using the pairing engine 202 when determining the precise pairing scores (i.e., “confidence scores”). The pairing engine 202 identifies these candidate device pairs and their associated preliminary pairing scores to the scoring engine 204. In addition, as described more fully below, the device IDs of the identified candidate device pairs are used together with the raw or pre-processed data 209 within the computer-readable feature storage device 208 to retrieve additional feature information, i.e., additional device activity history, for use by the FVG engine 212 to generate feature values ​​corresponding to the candidate device pairs. Reference Figure 1B , showing Figure 1A 150 and a work place W 110 as well as an additional home B 180 and an internet cafe C 190. In home A 150, there are two digital device users: user X and user Y. User X frequently uses a notebook computer system 151 and a cell phone 152, so that the digital identities of the notebook computer system 151 and the cell phone 152 can be identified as paired digital identities. The notebook computer system 151 of user X is digitally identified using a cookie (C) on the web browser, so that it is represented as C.sub.X, and the cell phone b of user X is digitally identified with a device ID (D), so that it is represented as D.sub.X. This digital device reference designation nomenclature will also be used with other laptops and phones. User Y of home A 150 frequently uses a notebook computer system C.sub.Y 161 and a cell phone D.sub.Y 162, so that C.sub.Y 161 and D.sub.Y 162 can also be identified as a related digital identity pair. It will be appreciated that all digital devices (C.sub.X 151, D.sub.X 152, C.sub.Y 161, and D.sub.Y 162) in home A 150 are operable to use the same single IP address A 163 assigned to Wi-Fi router 165 when those digital devices are used at home A 150. Furthermore, it will be appreciated that while this example embodiment is based upon the use of web browser cookies and mobile device identifiers, any other similar identifier that may be associated with a digital device may optionally be used.

[0097] User Z, who lives in home B 180, often uses notebook computer system C.sub.Z 181181 and cellular phone D.sub.Z 182182182. When at home B 180, C.sub.Z 181181 and D.sub.Z 182182182 will use IP address B 183 assigned to Wi-Fi router 185 used at home B 180. User X and user Z work together at workplace W110, so that C.sub.X 151, D.sub.X 152, C.sub.Z 181181, and D.sub.Z 182182182 are often used at workplace W110. When at workplace W 110, those digital devices will all use a single IP address W 123 assigned to firewall / proxy server 121 at workplace W 110. Many other digital devices (111, 112, 113, 114, 115 and 116) will also use a single IP address W 123 at the workplace W 110.

[0098] Finally, reference Figure 1B , shows a diagram of an Internet cafe C 190 that provides free Wi-Fi service to customers of Internet cafe C 190. User Z and User Y frequently visit Internet cafe C 190, so that C.sub.Z 181181, D.sub.Z 182182182, and D.Yub 162 are shown at Internet cafe C 190, where IP address C 193 is used on Wi-Fi router 195. It can be understood that many other visitors (not shown) also frequently visit Internet cafe C 190. However, the various digital devices seen together only at Internet cafe C 190 will not be considered as potential digital identity pairs because too many digital identity pairs are seen together at Internet cafe C 190.

[0099] After collecting the Internet usage data (as explained with reference to module 510), the next step in identifying digital identity pairs is to select a set of potential digital identity pairs, as explained with reference to Figure 3 As explained in 520 of the previous section, various combinations of digital identities associated with IP addresses having six or fewer digital identities may be selected as potential digital identity pairs. Figure 1BIn the example of , various combinations of digital devices at homes A 150 and B 180 are therefore considered candidate digital identity pairs. (Workplace W 110 and Internet cafe C 190 have too many different digital identities associated with them to provide good candidates for digital identity pairs.) For simplicity, only the possible digital identity pairs (C.sub.X, D.sub.X), (C.sub.X, D.sub.Y), (Csub.Y, D.sub.X), and (C.sub.Y, D.sub.Y) from home A 110 will be analyzed in this example.

[0100] In one particular embodiment of the present disclosure, for example, the pairing engine 202 uses a variation of Bayesian probability analysis to calculate a preliminary pair score, also referred to as a "association" score, for each potential cookie and device ID digital identity pair. In some embodiments, a "support" score and a "confidence" score are also determined. The support, confidence, and association scores may be defined as follows:

[0101] Support = P (cookie, device ID)

[0102] Confidence = P(cookie\device ID)

[0103] Association (cookie.fwdarw.device ID) = P (cookie | device ID) / P (cookie)

[0104] These three scores can be used to identify digital identity pairings and assess the confidence of digital identity pairings that have been made. The support score represents the amount of data support for analyzing this particular cookie and device ID pair. The confidence score represents how much confidence there is in the preliminary pair score (also called the association score). The preliminary pair score generated using the pairing engine 202 provides a rating of how closely the cookie and device ID are associated. In some embodiments of the present disclosure, the preliminary pair score generated using the pairing engine 202 is used as a feature in the scoring engine 204.

[0105] In some embodiments of the present disclosure, the above-mentioned Internet usage and triple feature information are optionally used to calculate support scores, confidence scores, and preliminary pair (association) scores. The following relationship describes how to use Internet usage information to calculate support, confidence, and preliminary pair (association) scores according to some embodiments:

[0106] Co-occurrence(cookie, device ID) = the number of times a given cookie and a given device ID are associated with the same location (ie, same source identifier IP address).

[0107] P(cookie, device ID) = co-occurrence (cookie, device ID) / total sample size

[0108] P(cookie\device ID) = co-occurrence(cookie, device ID) / occurrence(device ID)

[0109] P(cookie) = number of occurrences (cookie) / total sample size

[0110] Next reference Figure 4A , showing the use of Figure 2 2 and 3. An illustrative diagram of multiple example candidate device pairs and preliminary pairing scores identified by pairing engine 202 of FIG. 2. For example, the device pair data structure associates preliminary pairing score S3 with a candidate device pair having device identifier D1 and having cookie identifier C2. It should be understood that according to some embodiments of the present disclosure, preliminary pairing score S3 is a value between 0 and 1; however, in some cases, preliminary pairing score S3 may be greater than 1. Candidate device pairs and their corresponding preliminary scores are provided as input to scoring engine 204. In some embodiments of the present disclosure, the preliminary scores are provided as feature values ​​in a feature vector.

[0111] Next, the feature vectors used in the operation of the system 200 will be described in more detail. As used herein, a feature vector refers to a set of multiple feature values ​​associated with a candidate device pair. In addition, a feature vector is an information structure stored in a computer-readable storage. Each feature value represents one or more features associated with a candidate device pair.

[0112] According to some embodiments of the present disclosure, feature vectors are used to evaluate the correlation between identified device pairs during training of the scoring engine 204. The FVG engine 212 generates feature values ​​included in the feature vector as a function of feature information (i.e., activity history), which can be associated with each device of the candidate device pair, or associated with the device pair, and associated with one or more rules. Therefore, in some embodiments of the present disclosure, for example, each feature vector represents N features (e.g., F.sub.1, F.sub.2, F.sub.3, F.sub.4, ..., F.sub.N). In some embodiments of the present disclosure, each candidate device pair is associated with a feature vector (V), wherein each feature has an integer value (F.sub.value1, F.sub.value2, ..., F.sub.valueN), representing the state of the feature as being related to the associated device pair; optionally, each feature can have a score value, such as a probability indication value, and a rule is implemented to take into account such a score value. Thus, for example, the FVG engine 212 uses rules to generate a given feature vector value F.sub.value1 based on feature information (i.e., activity history) associated with the F.sub.1 feature associated with the device pair; then, different rules are used to generate a feature vector value F.sub.value2 based on different feature information associated with the F.sub.2 feature associated with the device pair; and so on.

[0113] Next, the feature vector generation engine will be described in more detail. Figure 5 , an illustrative process flow diagram 300 representing the configuration of the FVG engine 212 according to some embodiments of the present disclosure is shown. The modules of the figure correspond to computer program code (not shown), which is operable to configure a computer system including one or more processors and / or control components to cause the execution of specific operations represented by different modules. In operation, module 302 selects a candidate device pair. In addition, module 303 uses raw or pre-processed data 209 stored in a computer-readable feature storage device 208 to retrieve stored feature information associated with each device ID of the selected pair, i.e., the device activity history. In addition, module 304 selects a rule from the computer-readable rule storage device 213. In addition, module 306 generates feature values ​​for use in the feature vector based on the retrieved feature information (i.e., the activity history) and the currently selected rule.

[0114] Decision module 308 determines whether there are additional rules to be applied to the current feature information (i.e., activity history) associated with the device of the selected candidate device pair. In response to the decision of module 308 to apply the additional rule to the currently selected candidate device pair (based on the application rule), control flows back to module 304 and another rule is selected. For example, which rules are applied to the candidate device pair may depend on which feature information (i.e., activity history) is available for the pair. Conversely, in response to the determination by module 308 not to apply the additional rule to the currently selected candidate device pair, module 310 provides a feature vector including feature values ​​generated based on the individual feature information associated with each device of the currently selected candidate device pair to the scoring engine 204, and decision module 312 determines whether the computer-readable feature storage device 208 includes an additional device pair to be evaluated. In response to the decision module 312 determining that the additional device pair is to be evaluated, control flows to module 302 and another candidate device pair is selected. Conversely, in response to the decision module 312 determining that the additional device pair is not to be evaluated, the process ends and waits for the identification of another candidate device pair.

[0115] It should be understood that according to some embodiments of the present disclosure, the pairing engine 202 provides a preliminary determination of candidate device pairs. The initial candidate pair determination is used as an index for searching the computer-readable feature storage device 208 for separate and possibly different feature information associated with separate and different devices of the currently selected candidate device pair, the activity history. The FVG engine 212 applies one or more rules from the computer-readable rule storage device 213 to the separate and possibly different information identified using the index information provided by the pairing engine to generate feature values ​​for generating a feature vector associated with the selected candidate device pair.

[0116] For example, a candidate device pair may include a laptop and a smart phone. Separate Internet usage information, also referred to herein as low-level feature information, may be collected separately for the laptop and the smart phone. This raw or pre-processed data 209 is then used to identify usage information collected for the laptop and to identify separately collected usage information collected for the smart phone. In module 306, the FVG engine 212 applies one or more rules to generate feature vector values ​​for the device pair based on the separately collected usage information for the two devices in the pair. For example, the usage information optionally includes a device activity history.

[0117] It will also be appreciated that the feature vectors for the many candidate device pairs generated are used to generate exact pair scores for the many corresponding candidate device pairs, as will be explained more fully below. Figure 4B is a representation of the information associated with example feature vectors V1 to V33 according to some embodiments of the present disclosure. Figure 4A In some embodiments of the present disclosure, Figure 4AThe weighted values ​​(i.e., "weights") shown in can be included as a single feature value within the feature vector. Specifically, through supervised and unsupervised training, the scoring engine 204 generates an exact pair score to be associated with the candidate device pairs identified by the pairing engine 202. The exact pair scores generated by the scoring engine 204 are in turn used by the clustering engine 206 to facilitate clustering of user devices.

[0118] Fig. 6A is an illustrative flow chart of a process 650 according to some embodiments of the present disclosure, representing example details of an example first rule applied by module 306 to generate example high-level feature values ​​based on low-level feature information associated with two devices of an example candidate device pair identified using pairing engine 202 in raw or pre-processed data 209. It should be understood that raw or pre-processed data 209 can be diverse in nature and does not resemble any form of index structure in any way; for example, data 209 includes a history of device activity as a time series of Internet events. In addition, it should be understood that module 306 configures FVG engine 212 according to the details of the rule currently selected using module 304; thereby generating feature values ​​based on low-level feature information. Module 306 configures FVG engine 212 differently to apply different rules. Specifically, according to some embodiments, module 306 uses instructions stored in computer-readable rule storage device 213 to configure FVG engine 212 according to the rule selected using module 304 to determine feature values ​​of high-level features based on low-level feature information associated with devices of a candidate device pair selected using module 302 selected using module 303. As used herein, a "high" level feature is a feature whose value depends on the values ​​of one or more other features. The resulting feature value represents an attribute of the relationship between device C and device D.

[0119] Fig. 6A The example first rule process 650 uses multiple low-level feature information indicated in the original or pre-processed data 209 to generate feature values ​​for features that indicate the strength of the relationship between candidate device pairs including device C and device D. As explained herein, the vector includes multiple feature values, each feature value generated using a different rule. The example rules generate one of five possible feature values ​​for the example feature: 0, 1, 2, 3, and 4. The high-level feature values ​​generated using the example rules are included in the feature vector, as more fully described below. For example, the rules are hierarchical. It includes a hierarchy of branching determinations, some of which are predicates of other dependency determinations. Reference Fig. 6A, the first decision module 652 determines whether two devices of the example candidate device pair C and D access information from the same IAB category, such as the Interactive Advertising Bureau (IAB) category. In response to determining that C and D do not access content having the same category, the first decision module 652 generates a feature value of 0 in operation. If C and D do not access content having the same category, the first rule process 650 ends by generating a feature value of 0.

[0120] In response to the first decision module 652 determining in operation that both devices of the example candidate device pair access information from the same category, the second decision module 654 determines whether the category is sports. The second decision module 654 generates a feature value of 1 in response to determining that although C and D both access content having the same category, the estimated geographic locations of devices C and D are more than one hundred miles apart. If C and D access content having the same category, but devices C and D are more than one hundred miles apart, the first rule process 650 ends by generating a feature value of 1.

[0121] In response to the second decision module 654 determining that both devices access the sports category, the third decision module 656 determines whether any of the two devices accesses sports content during weekend time or evening time. In response to determining that neither device C nor device D accesses sports content during weekends or evenings, the third decision module 656 generates a feature value of 2. If both device C and device D access sports content but neither of them accesses sports content during weekends or evenings, the first process 650 ends with generating a feature value of 2.

[0122] In response to the third decision module 656 determining that at least one device accesses the sports category on weekends or evenings, the fourth decision module 658 determines whether only one or both of the two devices access the sports category on weekends or evenings. The fourth decision module 658 generates a feature value of 3 in response to determining that only one or the other of the two devices accesses the sports category on weekends or evenings. In response to determining that both devices access the sports category on weekends or evenings, the fourth decision module 658 generates a feature value of 4.

[0123] Figure 6B is an illustrative flow chart of an example second rule process 660 showing example details of an example second rule applied by module 306 according to some embodiments of the present disclosure. Figure 6B The example second rule process 660 is applied to Fig. 6A, one of six possible feature values ​​is generated for the example feature: 0, 1, 2, 3, 4, and 5. The first decision module 662 determines whether the two devices of the example candidate device pair C and D access information from the same category. In response to determining that C and D do not access content with the same category, the first decision module 662 generates a feature value of 0. If C and D do not access content with the same category, the example second rule process 660 ends by generating a feature value of 0.

[0124] In response to the first decision module 662 determining that both devices in the example candidate device pair access information from the same category, the second decision module 664 determines whether devices C and D are geographically located within one hundred miles of each other. The second decision module 664 generates a feature value of 1 in response to determining that although C and D both access content having the same category, they are not located within one hundred and forty kilometers of each other. If C and D access content having the same category but are not located within one hundred miles of each other, the example second rule process 660 ends by generating a feature value of 1.

[0125] In response to the second decision module 664 determining that the devices are within one hundred and forty kilometers of each other, the third decision module 666 determines whether the category is related to sports. In response to determining that the category is not related to sports, the third decision module 666 generates a feature value of 2. If device C and device D accessing the same category are within one hundred and forty kilometers (100 miles) of each other, but the category is not related to sports, the example second rule process 660 ends by generating a feature value of 2.

[0126] In response to the third decision module 666 determining that the accessed content is related to sports, the fourth decision module 668 determines whether none, only one, or both of the two devices access the football category on weekends or evenings. The fourth decision module 668 generates a feature value of 3 in response to determining that none of the devices access football-related content on weekends or evenings. The fourth decision module 668 generates a feature value of 4 in response to determining that only one or the other of the two devices accesses the sports category on weekends or evenings. The fourth decision module 668 generates a feature value of 5 in response to determining that both devices access the sports category on weekends or evenings.

[0127] It should be understood that Figures 6A to 6B Two example rules applied by the FVG engine 212 are provided to generate example feature values ​​for high-level features generated based on low-level feature information. It will be appreciated that the FVG engine 212 is configured to apply many rules to determine many corresponding feature values ​​to be included in the feature vector based on the observed low-level feature information.

[0128] Thus, the FVG engine 212 generates high-level feature values ​​(or examples, 0, 1, 2, 3, 4, 5) using low-level feature information and a predefined set of rules. In some embodiments, generating high-level feature values ​​involves using heterogeneous low-level features, such as features indicating the following;

[0129] (i) one or more categories (e.g., IAB categories);

[0130] (ii) one or more geographical locations (e.g., within one hundred and forty kilometers (100 miles) of each other); and

[0131] (iii) one or more times (e.g., on weekends or in the evenings).

[0132] Furthermore, in some embodiments of the present disclosure, generating high-level feature values ​​involves applying multiple rules in a sequential order (e.g., the first decision module 662, the second decision module 664, the third decision module 666, and the fourth decision module 668 each implement predefined rules in a predefined order).

[0133] Each rule in the example sequence generates a high-level feature. When the FVG engine 212 applies the example rule sequence, the low-level features are mixed with the higher-level features. Specifically, for example, the first decision module 662 can generate a feature value of 0 based only on one low-level feature (i.e., category). The second decision module 664 can generate a value of 1 to represent a high-level feature representing a combination of a low-level category feature and a geographic proximity feature. The third decision module 666 can generate a feature value of 2 based on a high-level feature representing a combination of a low-level category feature, a geographic proximity feature, and a sports category feature. The fourth decision module 668 can generate feature values ​​3, 4, or 5 based on a combination of a low-level category feature, a geographic proximity feature, a sports category feature, and a time range feature.

[0134] exist Figure 7 In the figure, there is shown a method for generating a characteristic value according to some embodiments of the present disclosure. Figure 2 208 and the rules in the computer-readable rule storage device 213. Many of the example rules in the table are relatively coarse descriptors of the observation history of device C, device D, and the IP address. It should be understood that not all rules are applicable to every potential device pair, and only certain portions of the rules may be applicable to any given device pair.

[0135] Next, the labels used in the operation of the system 200 will be explained in more detail. The label information (i.e., ID pairs) stored in the computer-readable feature storage device 208 is used to train the scoring engine 204 to generate device pair scores as a function of the device pair feature vector. Specifically, a given feature vector includes a label value associated with a corresponding device identifier. The labeled data is used to train the scoring engine 204 to determine the possibility that two devices of the unlabeled device pair are associated with the same given user. The feature vector associated with the labeled device pair is generally referred to as the "ground truth". Specifically, label data is provided to indicate feature vectors that are verifiably known to be associated with the same given user, and label data is also provided, which indicates feature vectors that are verifiably known not to be associated with the same given user. These known device pair relationships are referred to as "labeled" in this article. According to some embodiments of the present disclosure, each labeled device pair is associated with a label -1 or a label +1. All other device pairs, referred to as "unlabeled" in this article, are associated with a label 0. As previously explained with reference to the feature information (i.e., activity history) in the computer-readable feature storage device 208 and the rules in the computer-readable rule storage device 213, the corresponding candidate device pairs are associated with the corresponding feature vectors. Tagged device pairs with label = -1 are associated with feature vectors known to indicate a pair of devices not associated with the same user. Tagged device pairs with label = +1 are associated with feature vectors known to indicate a pair of devices associated with the same user. Tagged device pairs are used in system 200 for learning during the training phase to evaluate whether the feature vectors of corresponding untagged candidate device pairs indicate that the corresponding candidate device pairs are associated with the same user.

[0136] Next, the scoring engine 204 will be described in more detail. Fig. 8A , an illustrative flow chart 700 is shown representing the configuration of the scoring engine 204 according to some embodiments of the present disclosure. Fig. 8A The modules in the middle figure correspond to computer program code (not shown) that configures a computer system including one or more processors and / or control components to cause specific operations represented by different modules to be performed. Module 702 associates candidate device pairs identified using pairing engine 202 with feature vectors generated using feature vector generation (FVG) engine 212. In some embodiments of the present disclosure, the preliminary pair scores generated using pairing engine 202 are merged into the feature vectors used to determine the precise pair scores of the relevant candidate pairs. Module 702 also receives tag information associated with the candidate pairs and feature vectors. Each candidate device pair is also associated with a preliminary pair score. For example, a candidate device pair with device identifier D1 and cookie identifier C2 is associated with feature vector V1 and associated with initial pair score S3.

[0137] Reference again Fig. 8A, module 704 in operation identifies unlabeled and labeled candidate device pairs among the candidate device pairs received from pairing engine 202. Module 704 provides the labeled device pairs to label data selection module 705. Module 704 provides the unlabeled candidate device pairs to train training module 706. Label data selection module 705 specifies a first labeled candidate device pair (referred to as a "training set") to provide to training module 706 for training a scoring model implemented using scoring module 708, and specifies a second labeled candidate device pair (referred to as a "test set") to provide to score testing module 709 for testing the quality of scores generated using the scoring model. In some embodiments of the present disclosure, labeled data selection module 705 specifies sixty to ninety percent, for example, about eighty percent, of the received labeled candidate device pairs for training, and specifies about twenty percent of the received labeled candidate device pairs for testing.

[0138] The training module 706 evaluates the correlation between the feature vector of the unlabeled candidate device pair and the feature vector of the first labeled candidate device pair that has been designated for training, and determines the pair score to be associated with the first candidate device pair based on the correlation evaluation. As described above, the feature vector associated with the unlabeled candidate pair has a label = 0, and the feature vector associated with the labeled candidate device pair has a label = -1 or a label = +1. The training module 706 generates a scoring model used by the scoring module 708. The training module 706 generates a scoring model that associates model parameters with vector features. The training module 706 uses training data to determine model parameters, and the training data includes an array of feature vectors associated with the candidate device pairs and corresponding labels. According to some embodiments of the present disclosure, the determined model parameters indicate the importance of features within the vector for determining whether the devices of the candidate device pair are actually associated with the same user. The goal of training is to generate a training model that minimizes the residual between the model predictions and the observations in the training label data.

[0139] The model parameters used by the scoring model generated using the training module 706 are used to configure the feature-based scoring module 708. The scoring module 708 uses the model parameters in the scoring model, in combination with the feature vectors associated with the candidate device pairs, to generate an exact pair score between 0 and 1 for the unlabeled candidate device pairs. The exact pair score for the candidate device pair represents a more accurate estimate of the likelihood that the devices of the candidate device pair are actually associated with the same user than the corresponding preliminary pair score. According to some embodiments of the present disclosure, the exact pair score is determined as a function of the feature vector associated with each device pair (which may include feature values ​​indicating the preliminary pair score) and the model parameters within the trained scoring model. Therefore, the system 200 optionally employs an iterative process based on the following:

[0140] (i) Propose initial candidate device pairs;

[0141] (ii) determining an association probability based on the device activity history; and

[0142] (iii) selecting based on a probability of a given association for a pair of devices being above one or more threshold criteria, and repeating to (ii).

[0143] The score testing module 709 evaluates the quality of the exact pair scores determined using the scoring model. Specifically, the score testing module 709 determines the exact pair score quality of the candidate device pair as a function of the correlation between the feature vector of the candidate device pair and the second labeled candidate device pair specified for testing by the label data selection module 705. Candidate device pairs whose exact pair scores pass the score quality threshold (i.e., the "one or more threshold criteria" described above) are passed to module 710, which transmits the candidate device pair and its exact pair score to the clustering engine 206. Candidate device pairs whose exact pair scores do not pass the score quality threshold are removed from the feature-based scoring module 708.

[0144] In another exemplary embodiment of the present disclosure, Figures 8B to 8D , the process of determining device pairing is shown in more detail. In operation of system 200, device activity history is advantageously local in time, as old device activity history may be misleading (e.g., a given user sells his / her computer to another person, or buys a new smartphone). Therefore, system 200 employs a logical time T in operation, which has a delta time T+ΔT. At logical time T, the following conditions apply:

[0145] (i) for data set A, identifying unlabeled pairs for logical time T; for example, extracting candidate pairs generated at logical time T that cannot be determined by Pippio data; "Pippio data" refers to analysis data output from a data analysis software product provided by Arbor Technologies Inc. of the United States;

[0146] (ii) for dataset A, identifying labeled pairs for logical time T (i.e., candidate pairs generated at logical time T, which can be determined by Pippio data extracted at logical time T); thereby determining B+ positive pairs in data B and B- negative pairs in data B;

[0147] (iii) the system 200 then generates an ML model at logic time T using B+ positive pairs and B- negative pairs at logic time T and is used to generate a score for the dataset A to obtain an analysis result C;

[0148] (iv) at logical time T + ΔT, for dataset D, identify unlabeled pairs, i.e., candidate pairs that cannot be determined from the Pippio data extracted at logical time T + ΔT;

[0149] (v) at logical time T+ΔT, for dataset E, identify label pairs, i.e., candidate pairs that cannot be determined from the Pippio data extracted at logical time Τ+ΔT, including E+ positive pairs in dataset E and E- negative pairs in dataset E;

[0150] (vi) from (i) to (v) above, for data set F, next calculate the score pairs C at logical time T in the analysis result C, i.e., the score pairs C at logical time T that can be determined by the Pippio data extracted at logical time Τ + ΔT, where there is F+ low confidence score but with positive label, and F- high confidence score but with negative label; and

[0151] (vii) using the identified pairs E+, F+, E-, F- to develop a new ML model at logic time Τ + ΔT, which is then used to generate a score for the dataset D; thus, as described above, a feedback loop is provided by iteratively retraining the ML model using the F+ and F- pairs. Figures 8B to 8E Modules of the processes related to (i) to (vii) shown in .

[0152] refer to Figure 8B , showing an overview of the device connection coverage engine. Server logs, SDK logs, bid streams, etc. are represented by data 2000. Data 2000 is provided to data collector 2010. The processed output data from data collector 2010 is provided to connection generator 2020, which generates connection coverage 2030 in operation; connection coverage 2030 includes intra-device graph 2040, inter-device graph 2050, identity graph 2060 and family graph 2070 in sequence. By analyzing these graphs 2040, 2050, 2060, 2070, it can be understood that device pairings can be identified with a high degree of certainty. The output from connection coverage 2030 is fed back to connection generator 2020 in an iterative manner in operation to provide a pairing result with greater certainty (i.e., representing reliability).

[0153] Next reference Figure 8C , shows an overview of data collection occurring in the data collector 2010 in operation. From a given data source, represented by data source X 2100 for example, activity history data is provided to the ETL module 2110, which provides corresponding processed data to the Botnoise filter 2120, which in operation provides filtered data to the normalization and sampling module 2130. The output from the normalization and sampling module 2130 is then provided to the event store X 2140, the entries of which are used as input data for the rules applied in the system 200 to identify associations between pairs of user devices.

[0154] Next reference Fig.8D , shows a process of data selection, which is performed to generate an event data set for determining a pairing of a user device. Data storage X 2200 provides data to an event access control system 2210, which applies various policy rules, such as policy I 2220, to generate event set I 2230, which is used when determining a pairing indicating an association between devices; "I" is an integer. Policy rules are used to determine which categories of data are used to find associations between devices, for example:

[0155] (i) by owners x, y, z;

[0156] (ii) by country A, and

[0157] (iii) between 2017 / 10 / 20 and 2017 / 11 / 20.

[0158] It should be appreciated that other types of categories may optionally be used instead or in addition to the operation of system 200 when determining pairing of devices.

[0159] Next reference Fig. 8E , showing modules and processes for performing device intra-graph construction. For example, event set I 2300, derived from event set I 2230, provides event data to feature vector generator (FVG) 2310, whose output data enables candidate pair generator 2320 to identify potential pairings of devices. The output pair data from candidate pair generator 2320 is provided to decision module 2330, which is operable to perform deterministic pairing using a subset of feature vectors to provide low confidence (represented by "Z1") and high confidence (represented by "Z2") data results.

[0160] Low confidence results (Z1) are provided in the operation of the ML scoring module 2340 using a subset of the feature vectors. In the event that the ML scoring module 2340 generates low confidence data results, these results are used to indicate possible inter-device candidate pairs 2380. As a function of the results of the ML scoring module 2340 and the high confidence data results (Z2), possible intra-device pairs 2350 are generated and provided to the clustering module 2360 for generating, for example, intra-device graph type 1 (new segments) 2370; however, other types of graphs may be generated thereby, such as Figure 8B shown.

[0161] Fig. 9 It is a method for implementing some embodiments of the present disclosure. Fig. 8A An illustrative flowchart 800 is provided for details of the configuration of the scoring engine 204 of the training module 706. Figures 8B to 8EA substantially similar approach is adopted. The training module 706 implements semi-supervised learning and supervised learning processes. The unsupervised pre-training module 802 performs pre-training based on feature vectors of labeled and unlabeled device pairs; if labels are involved, supervised learning is advantageously adopted, wherein if labeled and unlabeled data are mixed together, semi-supervised learning is adopted.

[0162] According to some embodiments, the unsupervised pre-training module 802 generates a mapping that the remapping module 804 will use to remap features in the feature vectors of the labeled device pairs to the device remapping pair clusters determined by the unsupervised pre-training module 802. The remapping module 804 maps the candidate device pair feature vectors to remap clusters stored in a computer-readable storage device and used to facilitate supervised learning. The remapping module 804 maps the candidate device pair feature vectors to the remapping clusters so that device pairs with greater feature vector similarity determined using the unsupervised pre-training process are clustered together. The supervised training module 808 determines a score associated with the device pair feature vector based on the features of the labeled device pairs. The determined score indicates the likelihood that both devices of the pair are associated with the same user. The device pair remapping clustering performed using the remapping module 804 facilitates supervised learning by improving the labeled feature vectors to target the feature vector remapping clusters to which they have the greatest affinity. Specifically, the unsupervised pre-training module 802 and the remapping module 804 are used to map the feature vectors corresponding to the candidate device pairs to the remapping clusters so as to cluster the candidate device pairs based on their representations in the feature space. As described above, the labeled device pairs have label values ​​of -1 or +1, and the unlabeled device pairs have a label value of 0. It should be understood that according to some embodiments of the present disclosure, the unsupervised pre-training reconstructs the feature vector data with very sparse data to reduce the dimension.

[0163] The supervised training module 808 determines an accurate pair score associated with a candidate device pair feature vector based on the features of the labeled device pair. The determined score indicates the likelihood that the two devices of the candidate device pair are associated with the same user. The device pair remapping clustering performed using the remapping module 804 facilitates supervised learning by improving the labeled feature vectors to target the feature vector remapping clusters for which they have the greatest affinity.

[0164] Fig. 10A It is a representation that the remapping module 804 according to some embodiments of the present disclosure converts the feature vector of the candidate device into Figure 4BIllustration of the remapping of device pairs to an example of a device pair remapping cluster. Each candidate device pair remapping cluster is associated with a set of feature vectors that is different from the set of feature vectors of other device pair remapping clusters determined using an unsupervised training process. This intermediate mapping to the device pair remapping cluster is commonly referred to as "feature space embedding", and the process using unlabeled examples is commonly referred to as the "unsupervised pre-training" step of a semi-supervised learning process.

[0165] In some embodiments of the present disclosure, the unsupervised pre-training module 802 uses a Bayesian network in operation to train the remapping module 804 to produce device pair remapping clusters. In alternative embodiments of the present disclosure, the unsupervised pre-training process may use, for example, PCA, tree embedding, autoencoders, or RBMs. Specifically, according to some embodiments of the present disclosure, the supervised learning process produces a training data set 806 for remapping. In Fig. 10B which, an illustration is shown of the remapping module 804 for remapping an example feature vector Vin from an original feature space spanning {x.sub.l, x.sub.2,..., x.sub.n} to a latent space spanning {l.sub.l, l.sub.2,..., l.sub.k} according to a general non-linear embedding function E, where k is not necessarily > n or < n. The latent space representation E(V) allows for more efficient learning.

[0166] The supervised training module 808 infers a relevant function from the labeled training data, which can be used to estimate the likelihood that an unlabeled device pair belongs to the same user. In some embodiments of the present disclosure, the supervised learning process uses a Bayesian network or a tree ensemble to produce device pair scores and produce a scoring model. In alternative embodiments of the present disclosure, the supervised learning process may use a regression model or a neural network. The goal of this process is to minimize the residual between the model estimate of the possible device pairs and the labeled data. The supervised training module 808 produces a mathematical scoring model, which consists of model parameters in the feature space defined in the remapping module 804. The scoring model is used to configure the above-described scoring module 708 to score all other candidate device pairs.

[0167] Next, the data transformation performed during scoring will be described in more detail. Refer to Fig.11, shows an illustrative diagram of a data flow process 1000 representing data transformations occurring within a computer-readable storage device including within a scoring engine 204 according to some embodiments of the present disclosure. In operation, the scoring engine 204 receives a candidate device pair data structure 1002 as input from the pairing engine 202. It should be understood that during the training phase, the candidate device pairs are used to train the scoring engine 204. The candidate device pair data structure includes a device ID that identifies each device of the device pair and includes a corresponding preliminary pair score for the paired device. In a first data transformation, a feature vector is associated with the candidate device pair, thereby generating a data structure that associates the candidate device pair, its preliminary pair score, and its associated feature vector 1004. In some embodiments of the present disclosure, the preliminary pair score is merged into the feature vector. In a second data transformation, training label data 1005 is associated with the candidate device pair, thereby generating the candidate device pair, its preliminary pair score, its associated feature vector, and its associated training label 1006. It should be understood here that the label value of a labeled feature pair is -1 or +1, and the label value of an unlabeled device pair is 0. In a third data transformation, unsupervised pre-training involving unsupervised pre-training module 802 and remapping 804 uses the device pair feature vectors and their associated training labels to generate affinity scores for candidate device pairs that create clusters (A, B, C, D) 1008, which are then appended to the feature vectors. In a fourth data transformation, supervised training involving supervised training module 808 generates accurate pair scores associated with candidate device pairs 1010 and determines parameters for a scoring model.

[0168] During the fourth data transformation, the score testing module 709 uses the test label data 1012 to perform a precise evaluation of the scores assigned to the device pairs 1008. The test label data 1012 used during the precise evaluation is disjoint from the training label data 1005. That is, the labeled device pairs used as the test label data 1012 are disjoint from the labeled device pairs used for the candidate device pair data structure 1002. The device pairs are placed into different quantized pair score buckets. In the fifth data transformation, the device pair model scores are transformed into precise scores by measuring the precision and recall in each score bucket. The device pairs in the precise pair scores of the buckets below the quality threshold are filtered out. Fig. 8A The final device pairs 1014 with invalid pair scores are removed from the corpus of training results of the feature-based scoring module 708. For example, it is shown that the device pair D.subil-D.sub.j1 is removed by the accurate filter.

[0169] Next, the clustering engine will be described in more detail. Fig.12, an illustrative process flow diagram 1100 representing the configuration of the clustering engine 206 according to some embodiments of the present disclosure is shown. The modules of the figure correspond to computer program code (not shown) that configures a computer system (e.g., a data processing system) including one or more processors and / or control components to cause the execution of specific operations represented by different modules. Module 1102 receives candidate device pairs and associated precise pair scores from the scoring engine 204. It should be understood that some of the first candidate pairs identified using the pairing engine 202 can be filtered out by the scoring engine 204, so that a different set of second candidate device pairs can be received by the clustering engine 206. Module 1104 assembles a graph containing the received device pairs. In some embodiments of the present disclosure, module 1104 configures the computer system to act as a graph generator. Fig.13 It means that according to some embodiments of the present disclosure Fig.12 The clustering engine is used to Figures 4A to 4B 2. An illustrative diagram of an example graph generated for candidate device pairs of FIG. 20. It should be understood that the graph uses the exact pairing scores generated by the scoring module 708 of the clustering engine 206 rather than the preliminary pairing scores generated by the pairing engine 202. The graph nodes correspond to device identifiers. The graph edges connecting the graph nodes indicate device pairs. The exact pairing scores associated with the graph edges indicate the likelihood that the associated devices of the device pair are actually associated with the same user.

[0170] In the process of generating the graph, module 1104 prunes some graph edges from the device graph. Specifically, according to some embodiments, module 1104 implements a local graph sparsification process to clean up the graph before performing graph clustering. The input to the local graph sparsification process includes device pairs (sets of two device IDs) with relevant precision pair scores provided using scoring module 708. The output from the graph sparsification process includes fewer pairs that are wisely selected. A simple way to select pairs to be removed before clustering is to apply a flat cut on the score. For example, all pairs with precision scores <0.1 will be discarded. Although the flat cut method is simple, it is sometimes not optimal because it is often too harsh on the less dense parts of the graph than on the overly dense parts of the graph, destroying our ability to maintain many good clusters. Alternatively, local graph sparsification improves on the flat cut method. A wisely pruned graph can better indicate the relationship between device pairs. Specifically, for example, devices associated with different device pairs that share a common device ID also share common nodes in the graph. In addition, the edge scores represent the relative strengths of different device pairs.

[0171] Module 1106 selects potential user device clusters in the graph to evaluate whether they actually include one or more user device clusters. Fig.14 It is an exemplary embodiment of the user equipment clustering method proposed in some embodiments. Fig.13Schematic diagram of user device clusters U1.sub.A-U9.sub.A identified by the graph of FIG. It should be understood that a number of graph edges have been removed to produce the proposed user device clusters. For example, in Fig.14 Does not exist Fig.13 Graph edge between device D4 and cookie C2 shown in .

[0172] Module 1108 modifies the proposed user device clustering to attempt to identify a valid user device clustering based on one or more cluster fitness requirements. The user device clustering modification may involve adding, deleting, or combining edges within the potential device clustering in an attempt to meet the cluster fitness requirements. Decision module 1110 determines whether the fitness requirements have been met through the modifications of module 1108. In response to the determination of decision module 1110 that the fitness requirements have not been met, decision module 1112 determines whether to continue to modify the potential clustering to meet the fitness requirements. In response to decision module 1112 determining to continue the modification, control flow returns to module 1108. In response to decision module 1112 determining not to continue the modification, module 1114 abandons the proposed clustering.

[0173] In response to the decision module 1110 determining that the cluster fitness requirement has been met, the cluster accuracy filter module 1115 determines which device identifiers of the proposed user device cluster will be associated with the final user device cluster and which devices of the proposed user device cluster are to be removed. Module 1116 outputs the cluster as the final cluster. After module 1114 or after module 1116, depending on the control flow of a given potential user device cluster, control flows to decision module 1118, which determines whether there are more potential device clusters to be evaluated in the graph. In response to determining that there are more potential device clusters to be evaluated in the graph, control flows back to module 1106 and identifies another potential user device cluster. In response to determining that there are no additional device clusters to be evaluated, control flows to module 1120, which causes the decision module to wait for new potential device clusters.

[0174] Fig.15 It means that according to some embodiments, Fig.14 The proposed user equipment clustering uses Fig.12 An illustration of an exemplary final device clustering set U1.sub.B-U9.sub.B generated by the clustering engine of FIG. Fig.14to remove the device indicator determined by the second model training process. Specifically, for example, the device identifier of cookie C13 is present in the proposed user device cluster U5.sub.A, but is missing in the user device cluster U5.sub.B. The final device cluster U1.sub.B-U9.sub.B can act as a digital identity group communication system for communicating with devices that have or are associated with devices identified within the cluster. That is, the cluster acts as a digital identity group communication system for identifying multiple devices that can be used to communicate with related users.

[0175] Next, cluster fitness determination using label propagation will be described in more detail. According to some embodiments of the present disclosure, modules 1108 to 1116 perform user device clustering via label propagation to achieve a final user device clustering including a set of nodes assigned to a unique user identifier. Label propagation according to some embodiments of the present disclosure involves initially assigning a unique label to each vertex in the pairing graph, then propagating the label to other vertices through connected edges, and updating the label associated with each vertex in the graph. Label propagation and label updating are performed iteratively until it no longer updates the label of each node in the graph (if no incoming label has an aggregate score better than the score of the current label, the node will not update its label). Once label propagation stops, any nodes / vertices with the same label will be considered to belong to the same user.

[0176] To propagate labels, according to some embodiments, each node of the graph sends its label to its neighboring (British English: "neighboring") nodes, and at the same time it will also receive labels sent from its neighboring (British English: "neighboring") nodes. In addition to sending and receiving labels, each vertex also updates its label based on its current label and all labels it receives from its neighbors. The entire process is performed in parallel on each individual node. In each node, there are three parameters that control the behavior (British English: "behavior") of sending labels to other nodes. The first is a percentage number that controls how much of the neighboring nodes will receive the label from the current node, the second is an integer that limits the maximum number of nodes that receive the label from the current node, and the third is an integer that controls how far the label moves in the graph.

[0177] Details about the percentage parameter that controls the fraction of neighbor nodes that receive labels are as follows: Each node first computes the maximum of the pair scores associated with all edges connecting the node to its neighbors, and then sends labels only to those neighbors whose scores are above the maximum percentage.

[0178] Regarding the second parameter, even if there are many neighbors eligible to receive the label from the current node, this parameter is used to limit it to the first few nodes. We will sort all eligible edges in descending order and propagate the label only through the first few edges determined by this parameter.

[0179] The third parameter controls how far the label travels throughout the graph. In operation, keep track of how many edges the label has traveled so far, and once the number of edges it has traveled exceeds this integer limit, the label is not allowed to travel further.

[0180] As a given label travels through the graph, it collects pair scores along the path it travels. Each label starts with an initial score of 1. For a label that originates from one node (e.g., node A) to another node (e.g., node B), the score for that label is defined as the score of the label at node A multiplied by the score of the edge connecting nodes A and B, divided by the number of edges the label traveled to reach that node. Each node will aggregate the scores of all the labels it receives, will pick the label with the highest score, and assign that label to itself if the new score is greater than the current label's score. The score of the new label will also be retained for the next iteration. If the label changes, the current node will propagate it to its neighbors so that they can use it in the next iteration.

[0181] It should be understood that the final output of the label propagation module depends on the joint behavior of all the above parameters. Although it is difficult to predict which combination of these parameters will provide the best performance, in practice, user clustering performance indicators (e.g., the accuracy of user clustering) have been defined, a separate set of paired inputs has been assigned as training data, and a grid search has been implemented to find the combination of all parameters that produces the best performance on the output user clustering. The optimal values ​​of these parameters are then applied to future paired inputs and generate the best user clustering.

[0182] Next, cluster fitness determination using simulated annealing will be described in more detail. Alternatively, according to some embodiments, modules 1108 to 1116 perform user device clustering via label simulated annealing. Simulated annealing is an iterative, probabilistic, cluster fitness driven algorithm. Annealing is performed once for each node (device) in the graph because each node acts as a "seed" for the end-user device clustering. The cluster fitness function is maximized when both of the following occur simultaneously:

[0183] (i) the members of the cluster are strongly interconnected, which means that most possible pairs between member nodes are observed to exist and they have very high scores; and

[0184] (ii) The connection between a cluster and devices outside the cluster members is very weak.

[0185] The simulated annealing process proposes to add or remove edges (pairs) between devices randomly, and accepts these proposals with a probability related to the fitness of the resulting cluster. The simulated annealing process iterates and the acceptance probability evolves according to the simulated temperature schedule (starting from a more randomized "high temperature" stage to a more optimally focused "low temperature" stage). The process stops when the temperature is below a certain value and the value of the cluster fitness function has stabilized.

[0186] Simulated annealing is an example of the class of "fitness-driven" clustering methods. "Fitness-driven" methods are one of the three-tuple classifications of clustering algorithms, namely "divisive", "aggregative", or "fitness-driven", although, of course, hybrids can be formed by combining examples of any of these. The process and the final result of annealing require the specification of a clustering fitness function, where, contrary to intuition from popular examples, there is a lot of freedom in customizing the algorithm. As mentioned before, the commonly used clustering fitness function is the product of two factors that are maximized when:

[0187] (i) the devices within the discovered clusters are maximally interconnected; and

[0188] (ii) Devices within a given cluster are maximally isolated from devices outside the given cluster.

[0189] Given this definition of cluster fitness, there are very few free parameters in the annealing algorithm (i.e. what remains is the temperature schedule), so it would appear that there is nothing to tune, and the clustering you get from applying annealing is the "annealed clustering".

[0190] However, in certain problem domains, it is often beneficial to manually add additional terms to the clustering fitness function to penalize or encourage various effects / metrics seen in the extreme clusters. Since these terms are manually added, it is not known which of their various strengths should be compared to each other, or to the original baseline fitness described above, in order to obtain the best clustering. Therefore, a free parameter is usually introduced that needs to be adjusted for each additional term in the fitness function.

[0191] For example, in the example graph described above, there may be several different "types" of candidate device pairs (e.g., desktop cookie to desktop cookie, mobile to mobile, or mobile web cookie to mobile, etc.), each with its own pair scoring model. Performance as a function of score is quite different between pair types, due to many differences in the systematic effects of collecting data about various "devices" (e.g., different activity rates on laptops vs. smartphones). Therefore, it is appropriate to add additional factors to the clustering fitness function to account for these differences (e.g., if your cookie device model is the higher performing of the two, then trust clusters built based on a good mixture of cookie device pairs more than clusters built primarily on cookie-cookie pairs). In this case, the full fitness function can be written as:

[0192]

[0193] Where the sum of “a” is the sum of pairs of type (cookie-device, cookie-cookie, etc.), the various C’s represent the number of inner or outer pairs of type “a”, and λ is a free parameter that we need to tune to find the best performance.

[0194] Tuning these parameters is done as separate steps, namely by optimizing a combination of clustering accuracy (using labeled data) and by using a heuristic metric using a subgraph of the entire device graph. In practice, clustering accuracy is the main factor in choosing the best parameters to adopt, but it is feasible to optionally supplement it with simpler metrics such as average cluster fitness, average number of devices / clusters, etc. Finding the best parameters given a desired quantity to optimize is actually a regular optimization problem. Some of the simplest techniques to solve this problem include grid searches and Monte Carlo methods (where the search region is defined by boundary conditions that limit the size of the search region). For fitness functions with much more than four free parameters, the computation is usually very expensive and it is usually necessary to do something more "intelligent" to find a good optimum (such as Markov chains, Monte Carlo, or their Bayesian variants).

[0195] The partitioning of the data in this optimization phase is similar to that of basic supervised learning: a random subgraph of the entire graph is chosen as a given “test” dataset, and another subgraph is chosen as a given “training set”, whereby there is a metric that is optimized using the clustering in the training set, and this performance is then validated on the test set before proceeding to apply the best clustering to the entire device graph.

[0196] Next, the data transformations employed during device clustering will be described in more detail. Fig.16, an illustration of a data flow process 1500 is provided, which includes data transformations that occur during user device clustering within the clustering engine 206 in some embodiments of the present disclosure. The clustering engine 206 is operable to receive as input candidate device pairs associated with feature vectors and exact pair scores 1502 from the scoring engine 204. In a first data transformation, a graph 1504 is created in which user devices are represented by graph nodes, device pairs are represented by graph edges, and device pair scores are associated with corresponding graph edges. In a second data transformation, graph modifications are performed to produce the proposed user device clusters 1506. It is shown Fig.14 In the third data transformation, the test tag data 1510 is used to determine whether the device pairs within the proposed user device cluster meet the second accuracy threshold accuracy level. The final user device cluster 1512 is generated, wherein if the user device pairs do not meet the second accuracy threshold level, they have been removed. Fig.15 A final device cluster 1512 is created, where, for example, cluster U.sub.5B is removed from the final set of clusters.

[0197] Next, finding a balance between pairing and clustering will be described. The method of pairing candidate devices using pairing engine 202 and in scoring engine 204 is designed to produce a pair score that is as close to "Oracle precision" as possible, meaning that all pairs whose devices are truly owned by the same person have a score = 1 and all pairs that are not have a score = 0. In practice, of course, the scoring model is not perfect, and the vast majority of pairs have scores between these extremes. Some of this imperfection is unmitigated (unpredictable pure random noise, often called "variance"), but much of it is mitigatable (often called "bias").

[0198] There is a lot of other useful information that is not included in the pairing engine. In principle, it is feasible to characterize any of this information and input it directly into the pairing engine to improve the predictions provided by the system 200 in operation. Most of this information is related to information about devices that are not included in the pair (i.e., AB) itself, and the questions are as follows:

[0199] (i) How many of A’s friends are also B’s friends?

[0200] (ii) How many of A’s friends are seen on the same IP as B?

[0201] (iii) How many friends do A and B have respectively?

[0202] (iv) How many of A’s good friends are also B’s friends?

[0203] Some of this information is actually already fed into the pairing engine (hitherto known as "device map feedback to pairing" or "satellite signatures"), etc.

[0204] It will be appreciated that a more efficient way to make sense of this information is obvious: seek to provide the pairing engine with a view of the local properties of the device graph that can be built around the pair. From the perspective of the device graph, all of these questions can be answered at once and with relative ease, whereas the amount of work required to provide a comprehensive encoding of the information in the device graph as a feature of the pairing engine is prohibitive. Thus, as a matter of efficiency and practicality, a line is drawn as to, say, how much information to add to the pairing engine 202 and / or scoring engine 204 to produce a good pairing score and how much information will be considered "just after" the device graph is constructed in the clustering engine 206.

[0205] Adopting this modular structure "pairing engine => clustering engine", rather than trying to construct one large model beyond convenience, actually improves the final result. The fact is that while pairing candidate devices using pairing engine 202 and / or scoring engine 204 have the theoretical ability to completely eliminate the reducible error (bias) in our predictions, the amount of work required to improve the performance of pairing engine 202 and / or scoring engine 204 generally grows exponentially with the desired performance improvement or enhancement. Stopping short and sending the pairing scores of the approximate pairing model to a separate clustering engine 206 allows much less computation to compensate for most of the possible bias reduction.

[0206] Next, the system 200 will be described with respect to its network environment. Fig.17 In the embodiment of the present disclosure, a system configured to operate in a network environment is provided. Figure 2 16. The system 200 in a network environment 1600 includes a first server system 1602 for receiving mobile application usage information and page view information from a user. The first server system 1602 includes a targeting engine 1606 for targeting information delivery to a user across multiple devices and providing user-based frequency capping for information delivered to a user across multiple devices, an insight engine 1608 for providing cross-device attributes across multiple devices of a user and providing offline-online attributes to the user, and an optimization engine 1610 for generating user-based optimizations and platform-based optimizations.

[0207] The system 200 includes a second server system 1604 for generating a user device cluster indicating a device group associated with a user. The second server system 1604 includes a pairing engine 202, a scoring engine 204, a clustering engine 206, an FVG engine 212, raw or pre-processed data 209, and a computer-readable rule storage device 213. According to some embodiments of the present disclosure, the second server is configured to act as a distributed massively parallel processing and distributed storage system according to the Hadoop framework. Therefore, in practice, multiple instances of the pairing engine 202, the scoring engine 204, and the clustering engine 206 engines (as shown) run in parallel.

[0208] The network environment 1600 includes a first interface (indicated within a dashed line) 1612 between a first server system 1602 and a second server system 1604. The first interface 1612 includes a third server 1614 that collects logs containing information related to user interactions with a device, such as mobile application usage and page views, from the first server system 1602. The log optionally includes an ordered sequence of records that are appended only in time order. Typically, a unique sequential log entry number is assigned to each entry. Since the log entries are ordered in time, the log entry number can serve as a "timestamp" for the entry. The third server 1614 is configured to handle high-throughput, low-latency real-time data feeds. According to some embodiments, the third server 1614 is configured to act as a message broker that mediates communications between different applications, such as pushing a bidding model to the first server system 1602 and pushing log files from the optimization engine 1610 to the second server system 1604. According to some embodiments of the present disclosure, the third server 1614 is implemented according to a log collection system document, but other types of system documents may be used instead. The first interface 1612 also includes a log extractor module 1616 within the second server 1604 that extracts logs from the third server 1614 and provides the logs to the pairing engine 202. The log extractor module 1616 acts as a data normalizer that converts unstructured data received in the third server 1614 into structured data such as key-value pairs, for example, suitable for input to the pairing engine 202. It will be appreciated that the information within the logs may be received from different user device platforms that structure the data differently.

[0209] The network environment 1600 includes a second interface (represented within the dashed line) 1618 between the first server system 1602 and the second server system 1604. The second interface 1618 includes a user profile module 1620 within the second server system 1604, which receives and stores user device clustering information generated by the clustering engine 206, which indicates a device group associated with a user. As described above, the clustering engine 206 generates a user device clustering structure that associates a cluster of user device IDs with a unique user. The user profile module 1620 stores information that associates a cluster of user device IDs with a unique user. For example, in some embodiments of the present disclosure, the user profile module 1620 also receives information from the log extraction module 1616, such as user demographic information including one or more of gender, age, income, location, and behavior patterns (e.g., search patterns). The second interface 1618 also includes a fourth server 1622, which obtains the user device clustering structure from the user profile module 1620 and provides it to the first server. The fourth server 1622 is configured to serve many concurrent users. According to some embodiments of the present disclosure, the fourth server 1622 is configured to store, retrieve, and manage document-oriented information, sometimes referred to as semi-structured data. A document-oriented database is a class of NoSQL databases that are designed around the abstract concept of "documents." According to some embodiments, the fourth server 1622 is implemented using a storage data server that can be clustered from a single machine to a very large-scale deployment across many machines, and is configured to provide scalable key-value or document access with low latency and high sustained throughput.

[0210] The network environment 1600 includes a third interface (indicated within the dashed line) 1624 between the first server and the second server. The third interface 1624 includes an activity delivery and execution module 1626 within the second server system 1604, which receives and stores counts of user device clustering structures indicating device groups associated with the user. For example, in some embodiments of the present disclosure, the activity delivery and execution module 1626 also receives information from the log extraction module 1616, such as the number of ad impressions (i.e., "ad impressions") on the user's desktop device, the number of impressions received on the user's mobile device, and the number of impressions received on multiple devices of the user. According to some embodiments of the present disclosure, the activity delivery and execution module is configured based on a Hadoop-based Hive data warehouse infrastructure. For example, the Hive infrastructure is best suited for batch jobs on large data sets (e.g., advertising campaign data). The third interface 1624 also includes a fifth server 1628, which extracts activity execution information from the activity delivery and execution module 1626 and provides it to the first server system 1602. According to some embodiments of the present disclosure, the fifth server 1628 includes a SQL server that provides the second server with access to information related to user-based activities and execution scopes.

[0211] Without departing from the scope of the invention as defined by the appended claims, the embodiments of the invention described above may be modified. Expressions such as "comprises," "comprising," "combined with," "consisting of," "having," and "is" used to describe and claim the present invention are intended to be interpreted in a non-exclusive manner, i.e., to allow for the presence of items, components, or elements that are not explicitly described. References to the singular should also be interpreted as involving the plural. The numbers included in parentheses in the appended claims are intended to aid in understanding the claims and should not be interpreted in any way as limiting the subject matter claimed by these claims.

Claims

1. A data processing system, comprising: Connectivity coverage engine, including: Data collectors are configured to: receiving raw data or pre-processed data from a data source; processing the raw data or pre-processed data received from the data source at an extract transform load (ETL) module; filtering the processed data received from the ETL module at a Botnoise filter; Normalizing and sampling the filtered data received from the Botnoise filter at a normalization and sampling module to generate output data; transferring the output data to a data storage device; A connection generator configured to generate a connection coverage, wherein the connection coverage sequentially includes an intra-device graph and an inter-device graph; The event access control system is configured to: receiving data from the data storage; and generating a set of events, wherein the set of events is based on data received from the data store and at least one first rule; and A feature vector generation framework, executable by one or more processors and configured to generate a multi-feature value feature vector corresponding to a determined candidate device pair based at least in part on a device activity history associated with a device of the determined candidate device pair in the raw or preprocessed data, wherein: the feature vector generation framework is adapted to generate a feature value of at least one feature represented in a feature vector corresponding to the at least one determined candidate device pair based at least in part on (a) a device activity history associated with a first device identifier of at least one determined candidate device pair in the raw or preprocessed data, and (b) a device activity history associated with a second device identifier of the at least one determined candidate device pair in the raw or preprocessed data, and wherein the feature vector corresponding to the at least one determined candidate device pair includes a first confidence result and a second confidence result, the first confidence result being used to generate the intra-device graph, and the second confidence result being used to generate the inter-device graph.

2. A data processing system, comprising: Connectivity coverage engine, including: Data collectors are configured to: receiving raw data or pre-processed data from a data source; processing the raw data or pre-processed data received from the data source at an extract transform load (ETL) module; filtering the processed data received from the ETL module at a Botnoise filter; Normalizing and sampling the filtered data received from the Botnoise filter at a normalization and sampling module to generate output data; transferring the output data to a data storage device; a connection generator configured to generate a connection overlay; The event access control system is configured to: receiving data from the data storage; and generating a set of events, wherein the set of events is based on data received from the data store and at least one first rule; and A feature vector generation framework, capable of being executed by one or more processors and configured to generate a multi-feature value feature vector corresponding to a determined candidate device pair based at least in part on a device activity history associated with a device of the determined candidate device pair in the original or preprocessed data, wherein: the feature vector generation framework is suitable for using at least one second rule to generate a feature value of at least one feature represented in a feature vector corresponding to at least one determined candidate device pair based at least in part on (a) a device activity history associated with a first device identifier of at least one determined candidate device pair in the original or preprocessed data, and (b) a device activity history associated with a second device identifier of the at least one determined candidate device pair in the original or preprocessed data.

3. The system of claim 2, wherein: The feature vector corresponding to the at least one determined candidate device pair includes a first confidence result or a second confidence result, the first confidence result is used to generate the intra-device map, and the second confidence result is used to generate the inter-device map, wherein the first confidence result is associated with the intra-device candidate pair and the second confidence result is associated with the inter-device candidate pair.

4. The system of claim 2, further comprising a data structure for storing information identifying groups of device identifiers associated with respective different users, wherein: The group of device identifiers is used to target information over the Internet to various different users associated with the group of device identifiers.

5. The system of claim 2, wherein: The raw or pre-processed data is used to associate a device identifier with one or more network source / destination address identifiers.

6. The system of claim 5, further comprising a pairing framework, wherein: The pairing framework is adapted to identify a device pair based at least in part on the device identifier and the one or more network source / destination address identifiers.

7. The system of claim 2, wherein: The feature vector generation framework is adapted to determine a feature vector to associate with each determined candidate device pair based at least in part on a device activity history associated with a device identifier of a second device of each determined candidate device pair within the raw or pre-processed data.

8. The system of claim 2, further comprising a scoring engine for determining a score to associate with the determined candidate device pair based at least in part on the generated multi-eigenvalue feature vector associated with the determined candidate device pair.

9. The system of claim 2, further comprising a graph structure storing one or more of an intra-device graph and an inter-device graph, wherein: The nodes within the graph structure represent device identifiers, including a first device identifier and a second device identifier of the at least one determined candidate device pair.

10. The system of claim 9, further comprising a clustering engine adapted to identify clusters of nodes within the graph structure based at least in part on one or more cluster fitness requirements.

11. A method for data processing, comprising: Provides a connection coverage engine, which includes: Use data collectors that can be executed by one or more processes to: receiving raw data or pre-processed data from a data source; processing the raw data or pre-processed data received from the data source at an extract transform load (ETL) module; filtering the processed data received from the ETL module at a Botnoise filter; Normalizing and sampling the filtered data received from the Botnoise filter at a normalization and sampling module to generate output data; transferring the output data to a data storage device; generating a connectivity overlay using a connectivity generator executable by the one or more processors; An event access control system executable by the one or more processors is employed to: receiving data from the data storage; and generating a set of events, wherein the set of events is based on data received from the data store and at least one first rule; and employing a feature vector generation framework executable by the one or more processors for generating a multi-eigenvalue feature vector corresponding to the determined candidate device pair based at least in part on a device activity history associated with devices of the determined candidate device pair within the raw or pre-processed data, In which, the feature vector generation framework is capable of operating to use at least one second rule to generate a feature value of at least one feature represented in a feature vector corresponding to at least one determined candidate device pair based at least in part on (a) a device activity history associated with a first device identifier of at least one determined candidate device pair in the original or preprocessed data, and (b) a device activity history associated with a second device identifier of the at least one determined candidate device pair in the original or preprocessed data.

12. The method of claim 11, wherein: The feature vector corresponding to the at least one determined candidate device pair includes a first confidence result or a second confidence result, the first confidence result is used to generate the intra-device map, and the second confidence result is used to generate the inter-device map, wherein the first confidence result is associated with the intra-device candidate pair and the second confidence result is associated with the inter-device candidate pair.

13. The method of claim 11, further comprising: An information structure is stored that identifies groups of device identifiers associated with various users.

14. The method of claim 13, further comprising: Using the group of device identifiers targets information over the Internet to various different users associated with the group of device identifiers.

15. The method of claim 11, wherein: The raw or pre-processed data is used to associate a device identifier with one or more network source / destination address identifiers.

16. The method of claim 11, wherein: The feature vector generation framework is adapted to determine a feature vector to be associated with each determined device pair based at least in part on a device activity history associated with a device identifier of a second device of each determined device pair within the raw or pre-processed data.

17. The method of claim 11, further comprising: A category of usage data is determined in order to find an association between second devices, and wherein the category of the data includes at least one of an owner of the data, a location of the data, and a time period associated with the data.

18. The method of claim 11, further comprising: A scoring engine is configured to determine a score to associate with the determined candidate device pair based at least in part on the generated multi-eigenvalue feature vector associated with the determined candidate device pair.

19. The method of claim 11, further comprising: A graph structure including one or more of an intra-device graph and an inter-device graph is stored, wherein nodes within the graph structure represent device identifiers, including a first device identifier and a second device identifier of the at least one determined candidate device pair.

20. The method of claim 19, further comprising: A clustering engine is adapted to identify clusters of nodes within the graph structure based at least on one or more cluster fitness requirements.

Images