Machine learning classifier to identify internet service providers through website tracking

Abstract

A method and system for identifying and classifying visitor information tracked on a website to distinguish between Internet Service Providers (ISPs) and non-ISPs. The technology employs machine intelligence to train a classifier on visitor intelligence rich in enterprise statistical structures through website tracking technology. The ISP classifier can differentiate between ISPs and non-ISPs to identify website business attributable to an ISP for a given website.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 838,782, filed April 25, 2019, the entire contents of which are incorporated herein by reference. Background of the publicly disclosed content 1. Technical Field

[0004] This disclosure relates to systems and methods for use in business intelligence, customer relationship management (CRM) systems, marketing automation platforms, and web analytics systems. 2. Background Technology

[0006] Current systems used for online business—such as those used by CRM systems, business intelligence, and web analytics systems—want solutions to differentiate between different types of business on a website (e.g., human business versus non-human business). Summary of the Invention

[0007] The goal is to accurately identify whether the business of a visitor's client computer activity from a website is an Internet Service Provider (ISP) or a non-ISP. Methods and systems that typically rely on simple lists of known ISPs or highly recognizable businesses (e.g., Fortune 500 companies) can be technically flawed in identifying client computer network activity. These methods generate numerous false positives (mislabeled ISPs) and false negatives (mislabeled non-ISPs). They also fail to utilize real, tracked network activity data from the website's Visitor Intelligence (VI). These other methods also cannot identify global businesses using local company names. The implementation described herein uses VI and machine intelligence to determine whether a tracked entity is an ISP within the VI context.

[0008] The following is a brief description of some aspects of the implementation method. Its purpose is merely to present some concepts in a simplified form as an introduction to the more detailed description that will follow.

[0009] The description herein describes implementation methods for computer systems, methods, and computer program products that utilize machine intelligence.

[0010] The arrangement of databases, mappings, and classifiers provides an improved potential positioning platform that includes the following non-limiting advantages compared to current customer relationship management (CRM) systems.

[0011] The following embodiments describe a system and its processes configured to perform analysis for identifying ISP services on a website.

[0012] Therefore, in embodiments, a method and computer system, as well as a computer program product for a method executed by the computer system, are described herein. The computer system includes one or more processors and a computer-readable storage medium encoded with instructions executable by at least one of the processors and operatively coupled to at least one of the processors. The method includes:

[0013] Obtain website data service content, which includes the website's IP address data;

[0014] Map IP address data to business entity identifiers to identify business entities associated with website data business data;

[0015] Obtain attribute data for each business entity associated with a business entity identifier to identify business attributes associated with the business entity mapped to IP address data;

[0016] A training dataset is generated based on the website's data service content and attribute data to train an Internet Service Provider (ISP) classifier to identify ISPs associated with the mapped network service data;

[0017] Applying ISP classifiers to a database of website tracking data; and

[0018] Identify website services that can be attributed to an ISP.

[0019] In an implementation, the method includes:

[0020] The IP address matching acceptance rate is obtained based on website business data mapped to business entity identifiers; and

[0021] A training dataset is generated based on IP addresses from website business data that match business entity identifiers.

[0022] In one implementation, the method includes replacing the IP address of the website business data mapped to the business entity identifier with a previously identified IP address from the website business data previously mapped or matched to other business entities.

[0023] In one implementation, the method includes: generating a model for training an ISP classifier; training the model on a training dataset; determining whether the trained model accurately identifies ISPs; and, if the model is accurate, using the trained model as an ISP classifier.

[0024] In an implementation, the method includes: obtaining website data including website IP address data; placing a tracking code object on the website to be tracked; generating a log file that identifies all IPs tracked by the tracking code and cookies associated with those IPs.

[0025] In this implementation, network service data includes access counts, unique visitor counts, and indexed customer counts for each IP address within a time window. The time window may include weekly, bi-weekly, monthly, bi-monthly, and quarterly intervals. Attribute data for each business entity associated with a business entity identifier may include firmographic data. Firmographic attribute data may include annual sales, industry code, and employee count. Industry codes may include SIC codes, NAICS codes, or both. The feature set of the classifier model may include features from the firmographic data and network service data derived from the attribute data, including: NAICS, SIC, unique visitors, access counts, employee counts, annual sales, and indexed customer counts.

[0026] The classifier model can include models selected from decision trees or gradient boosting classifiers. Training a gradient boosting classifier can involve iterating the gradient boosting classifier approximately 1000 times.

[0027] Therefore, the implementation described herein provides a technical solution that improves conventional CRM systems, business intelligence systems, and web analytics systems that rely on subjective human judgment and less robust rule-based machine recognition.

[0028] Exemplary advantages of the implementation described herein include a framework that runs periodically (e.g., monthly) and reclassifies ISPs using the latest network services. This implementation enhances the credibility of the VI by providing a defensible probability around ISP classification. The system also employs automated methods, applying machine learning by incorporating various target attributes and features to classify large datasets.

[0029] The system described herein represents a significant technological advancement over other systems and methods used for identifying ISPs of online services. Alternative methods typically rely on simple lists of known ISPs or highly recognizable businesses (e.g., Fortune 500 companies). These methods generate many misidentified non-ISPs. They also fail to extrapolate to the long tail of identified businesses. Furthermore, they do not utilize the behavior of genuine online business data tracked on the website. This embodiment tracks the behavior of businesses on the website itself and uses this activity as a signal when determining whether an entity associated with an online business should be considered an ISP within the context of visitor intelligence. Attached Figure Description

[0030] The embodiments will now be further described by way of example only, with reference to the accompanying drawings.

[0031] Figure 1 It is a block diagram of the logical architecture for the implementation method.

[0032] Figure 2 This is a flowchart illustrating the process according to an implementation method.

[0033] Figure 3 This is a flowchart illustrating the process according to an implementation method.

[0034] Figures 4A to 4E An implementation of an environment in which this implementation can be practiced is shown.

[0035] Figure 5 It shows that it can be used in, for example Figure 3 The system shown illustrates an implementation of a network computer.

[0036] Figure 6 It shows that it can be used in, for example Figure 3 The system shown illustrates the implementation of the client computers included in the system.

[0037] Figure 7 An example graphical user interface according to an implementation method is shown. Detailed Implementation

[0038] Various embodiments will now be described more fully below with reference to the accompanying drawings, which form part of the embodiments and illustrate specific embodiments in which the present disclosure can be practiced. However, embodiments may be implemented in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that the disclosure will be thorough and complete and will fully convey the scope of the embodiments to those skilled in the art. Among other things, the various embodiments may be methods, systems, media, or apparatus. Therefore, the following detailed description should not be construed as limiting.

[0039] Throughout the specification and claims, unless the context clearly specifies otherwise, the following terms shall have the meaning explicitly associated with them herein. The term “in this document” refers to the specification, claims, and drawings associated with the present application. The phrases “in one embodiment” or “in an embodiment” as used herein do not necessarily refer to the same embodiment, although they may refer to the same embodiment. As described below, various embodiments of this disclosure can be readily combined without departing from the scope or spirit of this disclosure.

[0040] Furthermore, unless the context clearly specifies otherwise, the term “or” as used herein is inclusive and equivalent to the term “and / or”. Additionally, throughout the specification, the meanings of “a,” “an,” and “the” include plural references. The meaning of “in” includes both “in” and “on”.

[0041] Reference Figure 1 The system is typically indicated by reference numeral 100 in the accompanying drawings, which show a block diagram of the logical architecture and modules for an implementation. Figure 2 This is a flowchart illustrating a process according to an implementation. At block 202, the system is configured to ingest a collection of web data service content 102 from a website. This web data content may be accessed web page content (e.g., product pages, any web page from the site), IP addresses, mobile IDs, and web browser cookies. For example, in an implementation, the system may be configured to include or interface with an identity resolution and / or data loading platform 121. For example, the system may perform identity resolution or interface with platforms 111, 112, 102, 121 (e.g., such as Digital Element). TM LiveRamp TM Neustar TM Acxiom TM The platform (such as IP address resolution platform 111) can be used to access and load network data service content 102, or to perform or obtain identity resolution data for business entities associated with IP addresses. Identity resolution can be performed using IP identity resolution technology platform 111 and tools known in the art, such as matching cookie data with IP addresses, synchronizing cookie pools, etc.

[0042] In this implementation, the tracing object is placed on the website to be traced. For example, a tracing pixel or JavaScript tracing code can be placed on a web server. The following data can be obtained and analyzed using the tracing pixel:

[0043] • The operating system used (provides information about using the mobile device);

[0044] • The type of website or email used, such as on a mobile device or a desktop device;

[0045] • The type of client used, such as a browser or email program;

[0046] • The customer's screen resolution;

[0047] • The time spent reading emails or visiting websites;

[0048] • Activity on the website during the session (using multiple tracking pixels);

[0049] • IP address (provides information about the Internet service provider and location);

[0050] • Identify log files containing all IPs tracked by tracking codes and the cookies associated with those IPs.

[0051] Therefore, in this implementation, the system is configured to acquire Visitor Intelligence (VI) data 105, such as digital element files, web crawler data, or other loading tools. The system is configured to aggregate web traffic data, particularly including access counts for each IP address accessing content on the website, unique visitor and customer index attributes. As described herein, this data can be collected to cover a given time window (e.g., weekly intervals, bi-weekly intervals, monthly intervals, bi-monthly intervals, and / or quarterly intervals). In this implementation, VI 105 web traffic data for multiple websites can be aggregated for aggregate tracking and machine intelligence training.

[0052] At block 203, the system is configured to obtain business entity identifiers from the business entity enterprise statistical structure database 104a. At block 204, the business entity mapping module 106 is configured to map network service data content 102. At block 204, the system is configured to map network data service content 102 to business entity identifiers to identify the business entities accessing each website.

[0053] At block 204, the business entity platform 20 server may include a business entity identity resolution module 103 that matches each network service IP address with a business identifier number, such as a DUNS number (hereinafter referred to as "DUNS"), from the business entity information database 104a. At block 205, the system may be configured to determine the match rate of IP addresses that successfully match a business identifier. The initial mapping in the robust business information database may have a match rate of, for example, between 10% and 20% (e.g., 15%). At block 206, the system may be configured to override the IP address-to-business identifier matching, for example, by replacing or remapping IP addresses from website business data mapped to business entity identifiers with IP addresses from previously mapped or matched website business data of other business entities.

[0054] In implementation, the system can also be configured to identify, for a given business entity identifier, the number of other unique business entity identifiers in the business organization tree for that given business entity identifier. For example, for a given DUNS number, the system can be configured to calculate the number of unique DUNS numbers associated with the appropriate family tree representation associated with that given DUNS number. The family tree representation associated with a given DUNS number may include, for example, ordinary franchisees; DUNS with minority ownership; DUNS all under a headquarters with a high propensity to become members of the following family tree (e.g., DUNS identified by Dun & Bradstreet's decision HQ platform); DUNS with a common headquarters, common domestic ultimate, or common global ultimate based on legal relationships (less than 50% ownership); DUNS with a common headquarters, common domestic ultimate, or common global ultimate based on an analysis of the brands used within a set of DUNS; or DUNS with a common headquarters, common domestic ultimate, or common global ultimate based on a combination of the above. An exemplary system for linking a given business entity identifier to a number of other unique business entity identifiers in a business organization tree for that given business entity identifier is described in U.S. Patent No. 10,152,503, filed October 29, 2015, entitled “Data Communications System and Method that Maximize Efficient Usage of Communications Resources”, the entire contents of which are incorporated herein by reference.

[0055] At block 207, the system may further include a business entity mapping module 106, which is configured to analyze the network business content data of the business entities associated with the business entity identifier and map it to the attribute data of the business associated with the business entity identifier to generate mapping data 107. In an embodiment, the attribute data of each business entity associated with the business entity identifier includes enterprise statistical structure data.

[0056] Non-limiting examples of linking corporate entity data from a business entity information database via a business analytics server, generating a corporate statistical structure database for a company, and performing scoring and data integration are described in U.S. Patent No. 7,822,757, filed February 18, 2003, entitled "System and Method for Providing Enhanced Information," and in U.S. Patent No. 8,346,790, filed September 28, 2010, entitled "Data Integration Method and System." The entire contents of each of these two U.S. patents are incorporated herein by reference. Corporate statistical structure or other attribute data (e.g., company name, address, NAICS, SIC, employee count, annual sales, indexed customer count, and ranking / assessment / risk score) can be associated with an entity that has an IP address.

[0057] At block 208, the system is configured to generate a training dataset 108, consisting of training data 109 and a test dataset 110, based on website data service content and attribute data. This training dataset 108 is used to train an Internet Service Provider (ISP) classifier 101 to identify ISPs associated with the mapped network service data 107. The training data 109 and test data 110 can be stored in the training database 108.

[0058] In this implementation, a randomly selected training set can be obtained from a complete dataset of IP network service data mapped by the Business Entity Mapping Module 106 to business entity identifiers and business entity attribute data. This complete IP dataset is divided into a training set 109 and a test set 110. For example, based on an existing IP dataset 104b containing over 100,000 IP addresses associated with business identifiers, the Business Entity Mapping Module 106 maps 28,000 business entities to IP addresses VI 105 to generate a core / mapping dataset 107. In the mapping dataset 107, approximately 10,000 to 140,000 records overlap with the existing IP address dataset 104b. This overlapping data from the mapping dataset 107 is used as the training set 109, while the remaining non-overlapping data (approximately 12,000 to 18,000 records) of the mapping dataset 107 is used as the test set 110.

[0059] At block 209, the model trainer 120 trains the ISP classifier model using training dataset 108. Supervised classification models, such as gradient boosting classification, can be trained and used to determine whether a business identifier mapped to an IP address is an ISP service or a non-ISP service. In this implementation, the model is a Python-based gradient boosting classifier used to classify each business entity identifier (e.g., DUNS).

[0060] Gradient boosting creates strong learners from a set of weak learners. This classification methodology builds models in stages and analogizes these stages by allowing optimization of arbitrarily differentiable loss functions. The main goal is to "teach" a model {F} that predicts values ​​of the form y = F(x) by minimizing g, the mean squared error. At each stage m (1 <= m <= M) of gradient boosting, it can be assumed that there exists some imperfect model Fm (initially a very weak model that can only predict the mean y in the training set). The gradient boosting algorithm improves Fm by building a new model with an added estimator h to provide a better model: Fm+1(x) = Fm(x) + h(x). To find h, the gradient boosting solution begins with the observation that a perfect h would imply:

[0061] Fm+1(x)=Fm(x)+h(x)=y

[0062] Therefore, gradient boosting fits h to the residual y - Fm(x). As in other boosting variants, each {F(m+1)} attempts to correct the error of its previous Fm. This idea, which extends to loss functions other than squared error and analogies for classification and ranking problems, stems from the observation that the residual yF(x) of a given model is the negative gradient of the squared error loss function 1 / 2(yF(x))^2 (relative to F(x)). Gradient boosting is a gradient descent algorithm, and by analogy, it requires "inserting" different losses and their gradients.

[0063] Advantageously, it was found that at or after 1000 iterations, the gradient boosting model essentially fits the data (almost perfectly) and the residuals decrease to zero. It was also found that gradient boosting fits the data better and produces more accurate ISP labels compared to other classifier models, resulting in a more robust and technically more accurate system.

[0064] At block 210, the system is configured to apply ISP classifier 101 to a database of VIs that includes website tracking data, and to identify which services are ISP services on the website and which are not. At block 210, once the ISP classifier 101 model has been trained and confirmed to be working accurately, the system can then run ISP classifier 101 on the full mapping set of VI network service data to identify ISPs and non-ISPs. At block 211, the data can then be exported to user interface 30, or used for reporting as described below.

[0065] Figure 3This is a flowchart illustrating the process of extracting VI data and generating training dataset 208 (block 208) and training ISP classifier 101 (block 209) according to an embodiment. At block 302, the system ingests network traffic data, extracting log events, such as visitor information log files identifying all IPs tracked by tracking code placed on a website and cookies associated with those IPs. At block 303, the network traffic data is merged with business entity data. For example, as described herein, on a business identity site (e.g., a DUNS site), log events of the network traffic data are added and mapped to a business entity identifier file as described herein. As shown at block 303, the database may include a file containing network visitor information grouped by each visitor's IP address. At block 304, a classifier model is implemented, such as a Python-based gradient boosting classifier as described herein. While decision tree models and logistic regression models can be used, the gradient boosting classifier has been found to be the best classifier during training, producing more accurate results. It has also been determined that the gradient boosting classifier processes results much faster than logistic regression, which requires a longer processing time. Therefore, the gradient boosting classifier improves system latency and accuracy compared to other classifiers.

[0066] In the implementation, the model input comes from a business entity enterprise statistical structure database 104a. At block 305, predictions and training sets for training data are extracted from the merged network service data. For example, for training set 108, the primary key can be Business Identity Site Data (DUNS), and this set can be based on: VI logs, Tri_biz_global_master, Biz_ip, Biz_ip_full, Isp_Core_Set, Isp_Training_Set, Isp_duns_Blacklist, Isp_whitelist_ultduns, and manually tagged records (e.g., up to or more than 3000). As described herein, attributes can be the following features: traffic, visitors, NAICS code (25), SIC code (10), customer ratio, employees, and revenue. These attributes were found to be optimal in training the ISP classifier 101 for ISP identification.

[0067] At block 306, training set 108 is passed to model training step 209 to train the model. At block 307, once the model is trained, the system is configured to evaluate the model at block 312, for example, by checking the recall, accuracy, and error omission rate (FOR) against the model. Exemplary thresholds for accepting the trained model are given below.

[0068] Threshold for percentage rate

[0069] 1. Green – below 7%

[0070] 2. Orange – 7% to 10%

[0071] 3. Red – Above 10%

[0072] Threshold for recall

[0073] 1. Green – Above 85%

[0074] 2. Orange – 70% to 85%

[0075] 3. Red – below 70%

[0076] Threshold for FOR

[0077] 1. Green – FOR below 7%

[0078] 2. Orange – FOR 7% to 10%

[0079] 3. Red – FOR above 10%

[0080] If the threshold is within an appropriate range, then at block 311, the model is pickled and used for model testing of ISP predictions. The pickled model is loaded onto test set 110 and applied to it to obtain the results. The classification results are based on the following probabilities:

[0081] 50% probability: Determined to be non-ISP

[0082] 50% probability: Determine ISP

[0083] If the ISP prediction passes, the system classifier 101 is trained and ready to perform ISP classification on the VI data 107 that is fully mapped at block 210.

[0084] If the ISP training model fails the evaluation at block 307, the system determines whether training set 108 needs more training records. If training set 108 needs more training records, at block 310, additional VI and / or business attribute data are obtained and processed as described herein to further construct the core ISP dataset 107, and training set data is extracted from the core ISP dataset 107 at step 305. However, if training set 108 is sufficient for training, the model is iterated again at block 306 until the training threshold is within an appropriate range.

[0085] Below is an example of the verification results for ISP classification.

[0086] Status: Orange

[0087] Total size of training set: 10649

[0088] Total size of the prediction set: 558284

[0089] The number of ISPs and non-ISPs in the training set

[0090]

[0091] Table 1. Confusion matrix for the test set:

[0092] List value TP_1 532 FP 95 TN_0 1392 FN 111 correct 1924 mistake 206 Percentage example (Error * 100 / Correct) 10％ Recall sensitivity (TP*100 / (TP+FN)) 82％ Error omission rate FOR(FN*100 / (FN+TN)) 7％

[0093] Table 2

[0094] At block 309, the trained model is tested for evaluation.

[0095] In the implementation, the following model confusion matrix precision (TP / (TP+FP)), recall (TP / (TP+FN)), false omission rate (FN / (FN+TN)) and false positive rate (FP / (FP+TN)) can be used to test the model.

[0096] In the example shown in Table 3, the monthly confusion matrix values ​​after manual prediction are as follows:

[0097] List value TP_1 5 FP 12 TN_0 139 FN 1 correct 144 mistake 13 Percentage (Errors * 100 / Correct) 9％ Recall sensitivity (TP*100 / (TP+FN)) 83％ Error omission rate FOR(FN*100 / (FN+TN)) 0％

[0098] Table 3

[0099] As shown in Table 4, the total count of ISPs and non-ISPs among the top 200 records accessed based on ip_duns is as follows:

[0100] Predicted ISPs Forecast of non-ISP 164 36

[0101] Table 4

[0102] Tables 5 and 6 show examples of ISP flag comparisons between ip_de_ranges and ip_de_ranges_future:

[0103]

[0104]

[0105] Table 5

[0106] Classifier Based on rules Accuracy 92％↑ 80％ Recall (TP / TP+FN) 95％↑ 51％ FOR (error omission rate) 4％↓ 25％

[0107] Table 6

[0108] As shown in Table 7, the implementation methods described herein demonstrate a significant improvement in ISP identification compared to rule-based ISP filtering.

[0109] Below are the global match rate and customer match rate based on the ID (customer) provided by the client.

[0110] Status: Orange

[0111] Global match rate: 30.0%

[0112] Customer matching rate

[0113]

[0114]

[0115]

[0116]

[0117] Table 7

[0118] As shown in Tables 8 through 10, exemplary results are obtained by using the matching described herein to identify front-end ISPs and non-ISPs.

[0119] Top 20 ISPs list

[0120]

[0121]

[0122] Table 8

[0123] Top 20 Non-ISP List

[0124]

[0125]

[0126] Table 9

[0127] The top 20 non-ISP responses with null duns ID

[0128]

[0129] Table 10

[0130] Furthermore, the classifier model successfully identified many ISP visitors, which the rule-based method classified as non-ISPs, whereas manual methods failed to identify them. For example, as shown in Tables 11 and 12, the system newly identified US and global ISPs that were previously and conventionally unrecognizable:

[0131] US Business

[0132]

[0133] Table 11

[0134] Non-US business

[0135]

[0136] Table 12

[0137] notify

[0138] Since the system classifier 101 processes real network service data and directly affects the results, the system also employs an automatic notification gate at each stage to maintain classifier optimization and track performance. The system is configured to provide automatic notification when the confusion matrix or output results are inaccurate.

[0139] In the implementation, the system is configured with more than 10 notification gates at each level of model operation, including:

[0140] • A notification is issued regarding the overall record of the training set;

[0141] • Notifications regarding the accuracy of measuring the training set;

[0142] • A notification is issued regarding the overall record for the training set and the test set;

[0143] • Notification regarding the global match rate of VIs after the total ISP is generated;

[0144] • A notification regarding the customer match rate for the VI following the generation of total ISPs (“Reference”). In this implementation, the match rate is a key metric used to measure product performance for the VI;

[0145] • Notifications associated with the front-end ISP and non-ISP, either before or after the model runs.

[0146] Return to Figure 2 At block 210, once the model has been trained and confirmed to operate accurately, the system can then run a classifier on the complete mapping set 107 of the VI network service data to identify ISPs and non-ISPs. In an implementation, rule-based processing can be employed to classify data that cannot be processed by the classifier, such as network tracking data that cannot be mapped to business entity identifiers (e.g., mismatch data for total visitors versus unique visitors). At block 211, the data can then be exported to user interface 30 or used for reporting as described herein.

[0147] In another implementation, the classifier operates on IP addresses that match business identity identifiers but are not mapped to VIs. For example, in one implementation, the system is configured to retrieve a database 104b from a business entity corporate statistics structure database 104a that maps IP addresses to business entity identifiers 102. The system is also configured to map corporate statistics structure attribute data for each business entity associated with a business entity identifier from the corporate statistics structure database 104a as described herein. Attributes may also include, as described herein, features such as: traffic, visitors, NAICS code, SIC code, customer ratio, employees, and revenue. These attributes were found to be optimal for training an ISP classifier 101 for ISP identification. However, it was further found that running the classifier on a mapped database of identified IP addresses rich in corporate statistics structure attributes enabled the classifier to identify ISPs and non-ISPs even without VI data attributes 105. For example, all VI 105 access information is zeroed out (number of visits, number of visitors), but all other attributes are filled in (NAICS code, SIC code, customer ratio, employees, revenue) to run a classifier on a database 104b of over 500,000 mapped, enterprise statistically rich IP addresses that map to business entity identifiers, even without VI 105. Even without VI 105, the classifier can still classify ISPs and non-ISPs based on the remaining attribute information. Therefore, once the model is trained and confirmed to run accurately, in an implementation, the system can be configured to run the classifier on mapped data 107, where network tracking data cannot be mapped to business entity identifiers (e.g., mismatches for total visitors and unique visitors), but IP addresses can still be matched with other attribute data and enriched with other attribute data in the enterprise statistical structure.

[0148] As described above, data can be collected to cover a given time window (e.g., weekly intervals, bi-weekly intervals, monthly intervals, bi-monthly intervals, and / or quarterly intervals). For example, in one implementation, the system can collect data monthly as follows: Figures 2 to 3 The system ingests and maps data and runs classifier 101 to identify ISP and non-ISP network services. It is also configured to continuously maintain the model. For example, in one implementation, for subsequent periodic runs of the classifier on the mapped VI data within a given time period, the system can generate sample records based on model outputs not yet in the training set 108. For example, the classifier runs each month, producing output classification results. For supervised classification, records can be manually reviewed and merged. The modified dataset can then be appended to the next monthly cycle of model runs, updating the model's training.

[0149] Especially through, such as Figure 7The user interface 30 shown is used to provide reports.

[0150] As will be understood, in this implementation, although a database and the data therein are shown in a specific module, the database and the data therein can be shared and accessed between components and modules of the system, and do not need to be located in a specific component for accessing data used for semantic direction value analysis, etc., as described herein. The logical architecture and operational flow disclosed herein are shown to describe implementations in an exemplary manner and are not limited to a specific architecture, as those skilled in the art can modify the architectural design, for example, when implementing the teachings of this disclosure into their own systems.

[0151] Explanatory operating environment

[0152] Figure 4A The diagram illustrates components of an environment 101 in which embodiments of this disclosure can be practiced. Not all components are required to practice the invention, and the arrangement and type of components can be changed without departing from the spirit or scope of this disclosure. As shown, Figure 4 includes a local area network (LAN) / wide area network (WAN) 11, a wireless network 18, client computers 12 to 16, a data analysis server computer 10, a business entity analysis server computer 20, a network service object source 102n, and other data sources 112n.

[0153] The following is combined with Figure 5 At least one embodiment of client computers 12 to 16 is described in more detail. In one embodiment, at least some of client computers 12 to 16 may operate via wired and / or wireless networks, such as networks 11 and / or 18. Generally, client computers 12 to 16 may include virtually any computer capable of communicating over a network to send and receive information, perform various online activities, offline actions, etc. In one embodiment, one or more of client computers 12 to 16 may be configured to operate in a business or other entity to perform various services for that business or other entity. For example, client computers 12 to 16 may be configured to operate as a web server or account server. However, client computers 12 to 16 are not limited to these services and may also be used, for example, as end-user computing nodes in other embodiments. It should be appreciated that more or fewer client computers may be included within a system such as those described herein, and therefore the implementation is not limited to the number or type of client computers employed.

[0154] Computers that can operate as client computers 12 to 16 may include computers typically connected via wired or wireless communication media, such as personal computers, multiprocessor systems, microprocessor-based or programmable electronic devices, network PCs, etc. In some embodiments, client computers 12 to 16 may include virtually any portable personal computer capable of connecting to another computing device and receiving information, such as laptop computers 13, smartphones 12, and tablet computers 15. However, portable computers are not limited to this and may also include other portable devices, such as cellular phones, radio frequency (RF) devices, infrared (IR) devices, personal digital assistants (PDAs), handheld computers, wearable computers, in-vehicle computers, and integrated devices combining one or more of the aforementioned devices. Therefore, client computers 12 to 16 are generally wide-ranging in terms of capabilities and features. Furthermore, client computers 12 to 16 are configured to access various computing applications, including browsers or other web-based applications.

[0155] Network-enabled client computers may include browser applications configured to receive and send web pages, web-based messages, etc. The browser application can be configured to receive and display graphics, text, multimedia, etc., using virtually any web-based language, including Wireless Application Protocol Message (WAP), etc. In one implementation, the browser application can use Handheld Device Markup Language (HDML), Wireless Markup Language (WML), WMLScript, JavaScript, JavaScript Object Notation (JSON), Standard Generalized Markup Language (SGML), Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc., to display and send messages. In one implementation, a user of the client computer can use the browser application to perform various activities over the network (online). However, another application can also be used to perform various online activities.

[0156] Client computers 12 to 16 may also include at least one other client application configured to receive and / or send content with another computer. The client application may include the ability to send and / or receive content, etc. The client application may also provide information identifying itself, including type, capabilities, name, etc. In one embodiment, client computers 12 to 16 may uniquely identify themselves using any of a variety of mechanisms including Internet Protocol (IP) addresses, telephone numbers, mobile identification numbers (MINs), electronic serial numbers (ESNs), or other device identifiers. Such information may be provided in network packets sent between other client computers, data analysis server computer 10, business entity analysis server computer 20, or other computers.

[0157] Client computers 12 to 16 can also be configured to include a client application that enables end users to log in to an end-user account that can be managed by other computers (e.g., data analysis server computer 10, business entity analysis server computer 20, network business object source 102n, and other data sources 112n, etc.). In a non-limiting example, such an end-user account can be configured to enable the end user to manage one or more online activities, which in a non-limiting example include search activities, social networking activities, browsing various websites, communicating with other users, etc. However, participation in such online activities can also be performed without logging into an end-user account.

[0158] Wireless network 18 is configured to couple client computers 14 to 16 and their components to network 11. Wireless network 18 may include any of a variety of wireless subnetworks, which may further cover independent ad-hoc networks, etc., to provide infrastructure-oriented connectivity for client computers 14 to 16. Such subnetworks may include mesh networks, wireless LAN (WLAN) networks, cellular networks, etc. In one embodiment, the system may include more than one wireless network.

[0159] Wireless network 18 may also include autonomous systems such as terminals, gateways, and routers connected via radio links. These connectors can be configured to move freely and randomly and organize themselves arbitrarily, allowing the topology of wireless network 18 to change rapidly.

[0160] Wireless network 18 can also employ various access technologies, including second-generation (2G), third-generation (3G), fourth-generation (4G), and fifth-generation (5G) radio access for cellular systems, WLAN, wireless router (WR) mesh, etc. Access technologies such as 2G, 3G, 4G, 5G, and future access networks can achieve wide-area coverage for mobile devices (e.g., client computers 14 to 16 with varying degrees of mobility). In a non-limiting example, wireless network 18 can achieve radio connectivity via radio network access such as Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Enhanced Data GSM Environment (EDGE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Wideband Code Division Multiple Access (WCDMA), High-Speed ​​Downlink Packet Access (HSDPA), Long Term Evolution (LTE), etc. Essentially, wireless network 18 can include virtually any wireless communication mechanism through which information can be transmitted between client computers 14 to 16 and another computer, network, etc.

[0161] Network 11 is configured to couple network computers to other computers and / or computer devices (including data analysis server computer 10, business entity analysis server computer 20, network business object source 102n and other data sources, client computers 12, 13 and client computers 14 to 16) via wireless network 18. Network 11 can use any form of computer-readable medium for transmitting information from one electronic device to another. Furthermore, network 11 may include the Internet in addition to local area networks (LANs), wide area networks (WANs), direct connections such as via a universal serial bus (USB) port, other forms of computer-readable media, or any combination thereof. On a set of interconnected LANs (including LANs based on different architectures and protocols), routers act as links between LANs, enabling messages to be sent from one to another. Furthermore, communication links within a LAN typically include twisted-pair or coaxial cables, while communication links between networks can utilize analog telephone lines, dedicated digital lines including T1, T2, T3, and T4 and / or other carrier mechanisms (including, for example, E-carriers), Integrated Services Digital Network (ISDN), Digital Subscriber Line (DSL), wireless links including satellite links, or other communication links known to those skilled in the art. Additionally, communication links can employ any of a variety of digital signaling technologies, including but not limited to, DS-0, DS-1, DS-2, DS-3, DS-4, OC-3, OC-12, OC-48, etc. Furthermore, remote computers and other associated electronic devices can be remotely connected to the LAN or WAN via modems and temporary telephone links. In one embodiment, network 11 can be configured to transmit Internet Protocol (IP) information. Essentially, network 11 includes any communication method through which information can be propagated between computing devices.

[0162] Furthermore, communication media typically contain computer-readable instructions, data structures, program modules, or other transmission mechanisms and include any information delivery medium. As examples, communication media include wired and wireless media; wired media include twisted-pair cables, coaxial cables, optical fibers, waveguides, and other wired media; wireless media include acoustic, radio frequency, infrared, and other wireless media.

[0163] An embodiment of a server computer that can be used as a data analysis server computer 10 or a business entity analysis server computer 20 is described in more detail below with reference to Figure 4. In short, a server computer includes virtually any network computer capable of hosting modules as described herein. Computers that can be configured to operate as server computers include a variety of network computers, including but not limited to desktop computers, multiprocessor systems, network PCs, server computers, network devices, etc.

[0164] although Figure 4A Each of the data analytics server computer 10 or the business entity analytics server computer 20 is shown as a single computer, but this disclosure is not limited thereto. For example, one or more functions of the server computer may be distributed across one or more different network computers. Furthermore, the computer server is not limited to a specific configuration. Thus, in one embodiment, the server computer may comprise multiple network computers. In another embodiment, the server computer may comprise multiple network computers operating using a master / slave approach, wherein one of the multiple network computers of the server computer is operable to manage and / or otherwise coordinate the operation of the other network computers. In other embodiments, the server computer may operate as multiple network computers arranged in a cluster architecture, a peer-to-peer architecture, and / or even a cloud architecture. Therefore, this disclosure should not be construed as limited to a single environment, and other configurations and architectures are also contemplated.

[0165] Although shown separately, the data analytics server computer 10 and the business entity analytics server computer 20 can be used as a single network computer or computer platform, a standalone network computer, a cluster of network computers, etc. In some embodiments, either or both of the data analytics server computer 10 and the business entity analytics server computer 20 can be enabled to deliver content, respond to user interactions with content, track user interactions with content, update widgets and widget controllers, etc. Furthermore, while the data analytics server computer 10 and the business entity analytics server computer 20 have been described separately, it will be understood that these servers are hosted on other platforms or can be configured to operate on other platforms.

[0166] As described in this document, the implementation of System 10, its processes, and algorithms can be configured to run on platforms such as Amazon Web Services (AWS). Or Microsoft The cloud computing platform runs on a host machine. The cloud computing architecture is configured for convenient, on-demand network access to a shared pool of configurable computing resources, such as networks, network bandwidth, servers, processing power, storage, applications, virtual machines, and services. The cloud computing platform can be configured to allow the platform provider to unilaterally and automatically provision computing power, such as server time and network storage, as needed, without requiring human interaction with the service provider. Furthermore, cloud computing is available over the network and accessed through standard mechanisms used by heterogeneous thin-client or thick-client platforms, such as mobile phones, laptops, and PDAs. In the cloud computing architecture, the platform's computing resources can be pooled to serve multiple consumers, partners, or other third-party users using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated according to demand. The cloud computing architecture is also configured to enable platform resources to be rapidly and elastically provisioned (in some cases automatically) for rapid outward scaling and rapidly released for rapid inward scaling.

[0167] Cloud computing systems can be configured to automatically control and optimize resource usage by leveraging metering capabilities at some level of abstraction suitable for the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported. As described herein, in one implementation, system 10 is advantageously configured by a platform provider with innovative algorithms and database structures.

[0168] Cloud computing architecture includes multiple services and platform configurations.

[0169] Software as a Service (SaaS) is configured to allow platform providers to use their applications running on cloud infrastructure. These applications can be accessed from various client devices via thin client interfaces such as web browsers (e.g., web-based email). Consumers typically do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application capabilities, with possible exceptions being limited user-specific application configuration settings.

[0170] Platform as a Service (PaaS) is configured to allow platform providers to deploy consumer-created or acquired applications, created using programming languages ​​and tools supported by the provider, to cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but may have control over the deployed applications and potentially over the configuration of the application hosting environment.

[0171] Infrastructure as a Service (IaaS) is configured to allow platform providers to supply processing, storage, networking, and other basic computing resources in which consumers can deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but have control over the operating system, storage, deployed applications, and possibly limited control over the selection of network components (e.g., host firewalls).

[0172] Cloud computing architectures can be offered as private, community, or public cloud computing architectures. A cloud computing architecture can also be configured as a hybrid cloud computing architecture that includes two or more cloud platforms (private, community, or public) – these cloud platforms retain distinct entities but are bound together by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursts for load balancing between clouds).

[0173] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure of a network of interconnected nodes.

[0174] Now refer to Figure 4B This document describes an illustrative cloud computing environment 450. As shown, the cloud computing environment 450 includes one or more cloud computing nodes 455, which are used by cloud consumers to access local computing devices such as, for example, personal digital assistants (PDAs) or cellular phones 13, desktop computers 14, laptop computers 12, sensor data sources 15, network service data sources 102n, and integrated machine data sources 17 and / or other computer system or device data sources 112n. The nodes 455 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described herein. The cloud computing environment 450 is configured to provide infrastructure, platform, and / or software as a service, for which cloud consumers do not need to maintain resources on their local computing devices. It should be understood that... Figure 4B The types of computing devices shown are for illustrative purposes only, and computing node 455 and cloud computing environment 450 can communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0175] Now for reference Figure 4C This demonstrates the 450 (cloud computing environment) Figure 4B It provides a set of functional abstraction layers. Figure 4C The components, layers, and functions shown are illustrative and the embodiments described herein are not limited thereto. As depicted, the following layers and corresponding functions are provided:

[0176] The hardware and software layer 460 may include hardware and software components. Examples of hardware components include, for example: host 461; server 462; server 463; blade server 464; storage device 465; and network and network components 466. In some embodiments, software components include network application server software 467 and database software 468.

[0177] The virtualization layer 470 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 471; virtual storage device 472; virtual network including virtual private network 473; virtual application and operating system 474; and virtual client 475.

[0178] In one example, management layer 480 may provide the following functionalities: Resource Provisioning 481 provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and Pricing 482 provides cost tracking when utilizing resources within the cloud computing environment, as well as billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection for data and other resources. User Portal 483 provides access to the cloud computing environment for consumers and system administrators. Service Level Management 484 provides cloud resource allocation and management to ensure that required service levels are met. Service Level Agreement (SLA) Planning and Fulfillment 485 provides pre-scheduling and procurement of cloud resources based on anticipated future demand for cloud resources according to the SLA.

[0179] The workload layer 490 provides examples of the functionalities that can be leveraged in a cloud computing environment. Examples of workloads and functionalities that can be provided by this layer include mapping 491; input event processing 492; data stream processing 493; identity resolution 494; data analytics processing 495; and data delivery 496.

[0180] Figure 4D The logical architecture for an implementation method is illustrated. The system can be built on an exemplary platform, such as the Amazon Web Services platform, but can also employ other platforms used to support application content delivery, social networks, and network infrastructure. Figure 4DAs shown and described herein, a delivery channel layer 410 can be provided to client computers via a cloud frontend 412. A frontend web server layer 420 can be built on an Elastic Cloud (EC2) architecture 422 and can provide frontend interfaces, such as interfaces built on Angular JS 424 or other JS modules. A backend layer 430 can be operatively connected to the frontend architecture layer 420 via network sockets and can be built on an S3 architecture 432, including data buckets and objects 433 for network-scale data storage and retrieval, and a database layer 440 can, for example, include a database 444 on a relational database structure 442 layer architecture. One or more third-party systems 445 can be integrated or operatively connected to the architecture 450.

[0181] It will be understood by those skilled in the art that the system architecture is a non-limiting example illustrating at least a portion of at least one of various implementations. Therefore, more or fewer components may be employed and / or arranged differently without departing from the scope of the innovations described herein. However, the system at least fully discloses the innovations claimed herein.

[0182] Figure 4E This represents the logical architecture flow 400 of at least one system for visitor intelligent analysis in a cloud computing environment according to various implementations.

[0183] At block 402, the request enters the system via the Elastic Load Balancer (ELB) 421 and is routed to the Visitor Intelligent Cloud Infrastructure (VICI) Autoscaling Cluster Application 434 at block 403.

[0184] At block 404, VICI application 434 polls S3 server 432 for external configuration and pushes the Extract Transform Load (ETL) process and application 434 logs to S3 server 432 at a regular pace.

[0185] At block 405, the VICI application queries several tables in the Dynamo database 444 for IP, DUNS, and cookie data as described in this document.

[0186] At block 406, the ETL process begins with the Java Polling Agent (JPA) 426. JPA 426 polls the S3 server 432 for the ETL logs to be consumed. At block 407, when a new visitor smart file is ready for processing, JPA 426 sends an SNS notification to the Lambda Processing Agent (LPA) 429. LPA 429 runs Java code that transforms the data for loading processing on AWS Lambda.

[0187] At block 408, LPA429 sends the transformed data to a data streaming application configured for data analysis, such as Kinesis Firehose, which streams the data to a cloud data warehouse 440, such as Redshift Database 442, which is configured to support querying petabytes of structured and semi-structured data on the data warehouse 442 and the data lake using standard SQL.

[0188] At block 409, the customer-defined attribute CCA427 interface allows client users to supply first-party data for VICI matching responses. CCA rules are collected from users and filtered through EMR clusters 434 configured to be started and dismantled on demand for ingestion into the Dynamo database 444. At block 410, the seed IPR and DUNS (SID) application 428 seeds new monthly IPs with DUNS and DUNS lookup datasets into the Dynamo database 444 used for VICI. This creates EMR VICI clusters 434 on demand to efficiently seed data.

[0189] At block 411, the Sigmoid Data Extraction (SDE) script 424 extracts hourly VI data to send to a third-party interface for interactive reporting. Specifically, at block 412, the data is encrypted via PGP and pushed to S3 server 432, data bucket 433 for consumption.

[0190] At block 413, the JPA application publishes Simple Queuing Service (SQS) 425 messages to subscribers. These messages contain metadata about the workers performing daily data extraction for client users. At block 414, the workers poll the SQS 425 messages for the work that needs to take action. Then, at block 415, the workers receive instructions from the Redshift database 442 to perform SQL offload for client users who have enabled their daily data export.

[0191] At block 416, the data is then unloaded from database 442 to S3 server 432. At block 417, the SFTP server has an installation on S3 server 432 that includes daily unloaded data for client users. At block 418, client users can connect to the SFTP host using the ELB interface to retrieve their daily files.

[0192] At block 419, the log aggregator agent LAS host has a shell script that runs on a scheduling script (such as Cron) to perform a custom offload from Redshift database 442 for internal teams. At block 420, the offload is then stored in S3 server 432 for consumption.

[0193] Although this disclosure describes implementations on cloud computing platforms, the implementations described herein are not limited to cloud computing environments.

[0194] Explanatory Network Computer

[0195] Figure 5 An embodiment of a network computer 21 according to one embodiment of this disclosure is shown. The network computer 21 may include more or fewer components than those shown. However, the components shown are sufficient to disclose illustrative embodiments for practicing the invention. The network computer 21 may be configured to operate as a server, client, peer, host, or any other computer. The network computer 21 may represent, for example, the data analysis server computer 10 of FIG. 4 and / or the business entity analysis server computer 20, and / or other network computers.

[0196] The network computer 21 includes a processor 22, a processor-readable storage medium 23, a network interface unit 25, an input / output interface 27, a hard disk drive 29, a video display adapter 26, and a memory 24, all of which communicate with each other via a bus 28. In some embodiments, the processor 22 may include one or more central processing units.

[0197] like Figure 5 As shown, the network computer 21 can also communicate with the Internet or other communication networks via the network interface unit 25, which is configured to be used with various communication protocols, including TCP / IP. The network interface unit 25 is sometimes referred to as a transceiver, transceiver device, or network interface card (NIC).

[0198] Network computer 21 also includes a means for connecting to external devices such as a keyboard or Figure 5 The input / output interface 27 communicates with other input or output devices not shown. The input / output interface 27 may utilize one or more communication technologies such as USB, infrared, Bluetooth, etc. TM wait.

[0199] Memory 24 typically includes random access memory (RAM) 54, read-only memory (ROM) 55, and one or more permanent mass storage devices such as hard disk drive 29, magnetic tape drive, optical disk drive, and / or floppy disk drive. Memory 24 stores the operating system 32 used to control the operation of the network computer 21. Any general-purpose operating system can be used. A basic input / output system (BIOS) 42 is also provided for controlling the low-level operation of the network computer 21.

[0200] Although shown separately, memory 24 may include processor-readable storage medium 23. Processor-readable storage medium 23 may be referred to as computer-readable medium, computer-readable storage medium, and / or processor-readable storage device, and / or include computer-readable medium, computer-readable storage medium, and / or processor-readable storage device. Processor-readable storage medium 23 may include volatile, non-volatile, removable, and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Examples of processor-readable storage media include RAM, ROM, EEPROM, flash memory or other storage technologies, CD-ROM, Digital Universal Disc (DVD) or other optical storage devices, magnetic cartridges, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by a computer.

[0201] The memory 24 also includes one or more data storage devices 33, which the network computer 21 can use to store applications 35 and / or other data, etc. For example, the data storage devices 33 can also be used to store information describing various capabilities of the network computer 21. The information can then be provided to another computer based on any of various events, including being sent as part of a header during communication, being sent upon request, etc. The data storage devices 33 can also be used to store messages, web page content, etc. At least a portion of the information can also be stored on another component of the network computer 21, including but not limited to processor-readable storage medium 23, hard disk drive 29, or other computer-readable storage media (not shown) within the network computer 21.

[0202] Data storage device 33 may include databases, text files, spreadsheets, folders, files, etc., and may be configured to maintain and store user account identifiers, user profiles, email addresses, IM addresses and / or other network addresses, etc.

[0203] In at least one of the various implementations, the data storage device 33 may include databases such as training database 108, mapping / core data database 107, business entity database 104a, ISP rules database 118, ISP whitelist 116, ISP blacklist 114, and other databases that may contain information determined based on network analysis and network activity metrics as described herein, such as unique access (date-timestamp, IP address) and unique visitor (different cookie, different IP address).

[0204] The data storage device 33 may also include program code, data, algorithms, etc., for use by a processor, such as processor 22, to execute and perform actions. In one embodiment, at least some of the data storage devices 33 may also be stored on another component of the network computer 21, including but not limited to processor-readable storage medium 23, hard disk drive 29, etc.

[0205] Application 35 may include computer-executable instructions that can be loaded into mass storage 24 and run on operating system 32. Examples of applications may include code converters, schedulers, calendars, database programs, word processing programs, Hypertext Transfer Protocol (HTTP) programs, customizable user interface programs, IPSec applications, encryption programs, security programs, SMS message servers, IM message servers, email servers, account managers, etc. Application 35 may also include a web server 36, an ISP classifier 101, a visitor intelligence or web crawling module 105 for network business data content 102, a business entity identity resolution module 103, a business entity mapping module 106, a model trainer module 120, and a report generator 37.

[0206] Website server 36 can represent any information and service configured to provide content (including messages) to another computer over a network. Therefore, website server 36 may include, for example, a web server, a file transfer protocol (FTP) server, a database server, a content server, etc. Website server 36 can use any format from a variety of formats to provide content, including messages, over the network, including but not limited to WAP, HDML, WML, SGML, HTML, XML, compressed HTML (cHTML), extensible HTML (xHTML), etc.

[0207] Website server 36, ISP classifier 101, visitor intelligence or web crawling module 105 for network business data content 102, business entity identity resolution module 103, business entity mapping module 106, model trainer module 120, and report generator 37 can operate on the data analysis server computer 10 and / or business entity analysis server computer 20 of Figure 4, or can be hosted and operated on the data analysis server computer 10 and / or business entity analysis server computer 20 of Figure 4. Report generator 37 can be combined with... Figures 1 to 2 The description is similar to a process or part of a process to perform at least some of its actions.

[0208] Report generator 37 can be arranged and configured to determine and / or generate reports based on user filters and controls similar to those described above for reference to the user interface 30 controls. Furthermore, report generator 37 can be configured to output customized reports in the form of a publishing software application, where the application prepares and outputs lists in an easily readable format, or in a format suitable for automatic input and processing by other software products, such as plain text for publishing programs (such as LaTeX). In at least one of the various embodiments, report generator 37 can operate on, or be hosted and operated on, the data analysis server computer 10 and / or the business entity analysis server computer 20 of FIG4. Figures 1 to 3 The described content is similar to a process or part of a process, used to perform at least some of its actions. A report generator can be used to target, for example... Figure 7 The interface output report is shown.

[0209] Explanatory client computer

[0210] Reference Figure 6 The client computer 50 may include more or fewer components than those shown in FIG. 4. However, the components shown are sufficient to disclose illustrative embodiments for practicing the invention described herein.

[0211] Client computer 50 may represent at least one of the client computers 12 to 16 of FIG4, for example, in one of the embodiments.

[0212] As shown in the accompanying drawings, the client computer 50 includes a processor 52 that communicates with the mass storage 24 via a bus 51. In some embodiments, the processor 52 includes one or more central processing units (CPUs). The client computer 50 also includes a power supply 65, one or more network interfaces 68, an audio interface 69, a display 70, a keypad 71, a lighting device 72, a video interface 73, an input / output interface 74, a haptic interface 75, and a global positioning system (GPS) receiver 67.

[0213] Power supply 65 provides power to client computer 50. Rechargeable or non-rechargeable batteries can be used to provide power. Power can also be provided by an external power source such as an AC adapter or a power docking station for replenishing and / or recharging the battery.

[0214] Client computer 50 may optionally communicate with a base station (not shown) or directly with another computer. Network interface 68 includes circuitry for coupling client computer 50 to one or more networks and is configured for use with one or more communication protocols and technologies, including but not limited to GSM, CDMA, TDMA, GPRS, EDGE, WCDMA, HSDPA, LTE, User Datagram Protocol (UDP), Transmission Control Protocol / Internet Protocol (TCP / IP), Short Message Service (SMS), WAP, Ultra Wideband (UWB), IEEE 802.16 Global Microwave Access Interoperability (WiMax), Session Initiation Protocol / Real-Time Transport Protocol (SIP / RTP), or any of various other wireless communication protocols. Network interface 68 is sometimes referred to as a transceiver, transceiver device, or network interface card (NIC).

[0215] Audio interface 69 is configured to generate and receive audio signals, such as human voice. For example, audio interface 69 may be coupled to a speaker and microphone (not shown) to enable other telecommunications communication and / or generate audio confirmation of some action.

[0216] The display 70 may be a liquid crystal display (LCD), a gas plasma display, a light-emitting diode (LED), an organic LED, or any other type of display used with a computer. The display 70 may also include a touch-sensitive screen arranged to receive input from an object, such as a stylus or the fingers of a human hand.

[0217] Keypad 71 may include any input device arranged to receive input from the user. For example, keypad 71 may include a push-button numeric dial or a keyboard. Keypad 71 may also include command buttons associated with selecting and sending images. Illuminator 72 may provide status indication and / or provide light. Illuminator 72 may remain active for a specific period of time or in response to an event. For example, when illuminator 72 is active, it may illuminate the buttons on keypad 71 from behind and remain lit when the client computer is powered on. Furthermore, when a specific action (such as dialing another client computer) is performed, illuminator 72 may illuminate these buttons from behind in various modes. Illuminator 72 may also cause a light source located within the transparent or translucent casing of the client computer to illuminate in response to an action.

[0218] Video interface 73 is configured to capture video images such as still photographs, video clips, infrared video, etc. For example, video interface 73 can be coupled to a digital camcorder, webcam, etc. Video interface 73 may include a lens, image sensor, and other electronic components. The image sensor may include a complementary metal-oxide-semiconductor (CMOS) integrated circuit, a charge-coupled device (CCD), or any other integrated circuit used for sensing light.

[0219] The client computer 50 also includes an input / output interface 74 for communicating with external devices such as headphones or other input or output devices not shown in Figure 4. The input / output interface 74 may utilize one or more communication technologies such as USB, infrared, Bluetooth, etc. TM wait.

[0220] The haptic interface 75 is arranged to provide haptic feedback to the user of the client computer 50. For example, the haptic interface 75 can be used to vibrate the client computer 50 in a specific manner when another user of the computer is making a call. In some embodiments, the haptic interface 75 is optional.

[0221] The client computer 50 may also include a GPS transceiver 67 to determine the physical coordinates of the client computer 50 on the Earth's surface. In some embodiments, the GPS transceiver 67 is optional. The GPS transceiver 67 typically outputs the location as latitude and longitude values. However, the GPS transceiver 67 may also employ other geolocation mechanisms, including but not limited to triangulation, assisted GPS (AGPS), enhanced time difference of observation (E-OTD), cell identifier (CI), service area identifier (SAI), enhanced timing advance (ETA), base station subsystem (BSS), etc., to further determine the physical location of the client computer 50 on the Earth's surface. It should be understood that, under different conditions, the GPS transceiver 67 can determine the physical location of the client computer 50 within millimeters. In other cases, the determined physical location may be less precise, for example, within meters or significantly greater distances. However, in one embodiment, the client computer 50 may provide additional information that can be used to determine the physical location of the computer through other components, including, for example, a media access control (MAC) address, an IP address, etc.

[0222] Mass storage 24 includes random access memory (RAM) 54, read-only memory (ROM) 55, and other storage devices. Mass storage 24 illustrates an example of a computer-readable storage medium (device) for storing information such as computer-readable instructions, data structures, program modules, or other data. Mass storage 24 stores a basic input / output system (BIOS) 57 for controlling the low-level operations of client computer 50. Mass storage 24 also stores an operating system 56 for controlling the operation of client computer 50. It will be understood that this component may include UNIX or LINUX. TM Versions of general-purpose operating systems or such as Microsoft Windows TM OS, Apple's iOS TM Google's Android TM or A dedicated client communication operating system. The operating system may include or interface with a Java Virtual Machine module, which can control hardware components and / or the operation of the operating system via Java applications.

[0223] The mass storage device 24 also includes one or more data storage devices 58, which the client computer 50 can utilize to store application 60 and / or other data, etc. For example, the data storage device 58 can also be used to store information describing various capabilities of the client computer 50. This information can then be provided to another computer based on any of various events, including being sent as part of a header during communication, being sent upon request, etc. The data storage device 58 can also be used to store social network information, including address books, friend lists, aliases, user profile information, etc. Furthermore, the data storage device 58 can also store any content from messages, web page content, or various user-generated content. At least a portion of the information can also be stored on another component of the client computer 50, including but not limited to processor-readable storage medium 66, disk drives, or other computer-readable storage devices (not shown) within the client computer 50.

[0224] Processor-readable storage medium 66 may include volatile, non-volatile, removable, and non-removable media implemented in any method or technology for storing information such as computer-readable instructions or processor-readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media include RAM, ROM, electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, optical disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage devices, magnetic cartridges, magnetic tape, disk storage devices or other magnetic storage devices, or any other physical medium that can be used to store desired information and is accessible by a computer. Processor-readable storage medium 66 is also referred to herein as a computer-readable storage medium and / or a computer-readable storage device.

[0225] Application 60 may include computer-executable instructions that, when executed by client computer 50, send, receive, and / or otherwise process network data. Network data includes, but is not limited to, messages (e.g., SMS, Multimedia Messaging Service (MMS), Instant Messaging (IM), email, and / or other messages), audio, video, and enables telecommunications communication with another user on another client computer 50.

[0226] Application 60 may include, for example, a browser 61 and other applications 62. Other applications 62 include, but are not limited to, calendars, search programs, email clients, IM applications, SMS applications, Voice over Internet Protocol (VoIP) applications, contact managers, task managers, code converters, database programs, word processors, security applications, spreadsheet programs, games, search programs, etc.

[0227] Browser 61 can include virtually any application configured to receive and display graphics, text, multimedia, messages, etc., using virtually any web-based language. In one embodiment, the browser application uses HDML, WML, WMLScript, JavaScript, JSON, SGML, HTML, XML, etc., to display and send messages. However, any of a variety of other web-based programming languages ​​can be used. In one embodiment, browser 61 enables the user of client computer 50 to communicate with and interface with another network computer, such as data analysis server computer 10, business entity analysis server computer 20, web business object source 102n, and other data sources 112n, as shown in Figure 4, allowing the user to operate the user interface 30 as described herein.

[0228] Application 60 may also include a widget controller 63 and one or more widgets 64. Widgets 64 may be a collection of content provided to client computers by the data analytics server computer 10, business entity analysis server computer 20, network business object source 102n, and other data sources 112n. (Figure 4 shows the data analytics server computer 10, business entity analysis server computer 20, network business object source 102n, and other data sources 112n.) Widget controller 63 and widgets 64 may run as native client computer applications, or they may run as web browser-based applications in browser 61. Furthermore, widget controller 63 and widgets 64 may be configured to run as native applications, web browser applications, or a combination thereof. In one embodiment, browser 61 employs widget controller 63 and widgets 64 to enable the user of client computer 50 to communicate and interface with other network computers such as the data analytics server computer 10, business entity analysis server computer 20, network business object source 102n, and other data sources 112n, allowing the user to operate the user interface 30 as described herein.

[0229] Explanatory graphical user interface

[0230] Reference Figure 7In at least one of the various embodiments, a user interface other than the user interface 30 described below may be used without departing from the spirit and / or scope of this disclosure. Such a user interface may have more or fewer user interface elements arranged in various ways. In some embodiments, the user interface may be generated using web pages, mobile applications, emails, PDF documents, text messages, etc.

[0231] The user interface unit 30 will now be described in more detail. For example... Figure 7 As shown, interface 30 can be configured specifically for VI reporting for ISP identification.

[0232] In one implementation, the system can be configured to provide daily log-based reports showing audiences based on Business Identity Specifiers (DUNS)—indicating non-human business activity and potentially useful for further email targeting, CRM, and retargeting. In another implementation, the system can be configured to provide intelligence to understand the performance and ROI of existing campaigns initiated by clients under their activity by identifying non-human audiences. Furthermore, this information can be used to effectively understand which products or services individuals are interested in on the website, and these insights can be used to determine the content of retargeting advertisements. Additionally, the system can be configured to identify suspicious non-human activity on the website that may indicate malicious or unwanted online business activity.

[0233] User interface 30 has been described using an example of a dashboard suitable for personal computers, as this is a preferred form for illustrative purposes. Similar graphical user interfaces with a dashboard format can also be provided as mobile apps, such as mobile apps for Android or iPhone operating systems, where the term "mobile app" primarily refers to a module of application software capable of running on a smartphone or tablet device or other client computer. Other types of user interfaces can also be provided. An alternative type of user interface is an Application Programming Interface (API), which is a type of user interface suitable for developers who wish to integrate the system described herein with third-party software applications, for example, by merging outputs in a flexible manner suitable for the integrated third-party application software. Another type of user interface is a report writing software application that outputs customized reports based on user filters and controls similar to those described above with reference to the dashboard.

[0234] The flowchart illustrations have already described certain aspects of the operation of this disclosure. In at least one of the various embodiments, in conjunction with Figures 1 to 7The described processes can be implemented and / or executed on a single network computer. In other embodiments, these processes or portions of these processes can be implemented and / or executed on multiple network computers. Similarly, in at least one of the various embodiments, a process or portion of a process can operate on one or more client computers, such as client computer 50. However, the embodiments are not limited thereto, and various combinations of network computers, client computers, virtual machines, etc., can be used. Furthermore, in at least one of the various embodiments, the processes described in conjunction with the flowchart illustrations can operate in a system having, for example, the logical architecture described herein.

[0235] It will be understood that each block of the flowchart illustrations described herein, and combinations of blocks in the flowchart illustrations, can be implemented by computer program instructions. These program instructions can be provided to a processor to generate a machine, such that the instructions, which execute on the processor, create means for implementing the actions specified in one or more flowchart blocks. The computer program instructions can be executed by the processor to cause the processor to perform a series of operational steps to produce a computer-implemented process, such that the instructions, when executed on the processor, provide steps for implementing the actions specified in one or more flowchart blocks. The computer program instructions can also cause at least some of the operational steps shown in the flowchart blocks to be executed in parallel. Furthermore, some steps can also be executed on multiple processors, for example, in a multiprocessor computer system or even a group of multiple computer systems. Moreover, without departing from the scope or spirit of this disclosure, one or more blocks or combinations of blocks in the flowchart illustrations can also be executed simultaneously with other blocks or combinations of blocks, or even in an order different from that shown.

[0236] Therefore, the blocks illustrated in the flowchart support combinations of actions to perform specified actions, combinations of steps to perform specified actions, and program instruction means to perform specified actions. It will also be understood that each block of the flowchart and combinations of blocks in the flowchart can be implemented by a system based on dedicated hardware, or a combination of dedicated hardware and computer instructions, to perform specified actions or steps. The foregoing examples should not be construed as limiting and / or exhaustive, but rather as illustrative use cases illustrating implementations of at least one of the various embodiments of this disclosure.

Claims

1. A method for identifying an Internet Service Provider (ISP) executed by a computer system, the computer system comprising one or more processors and a computer-readable storage medium encoded with instructions executable by at least one of the processors and operatively coupled to at least one of the processors, the method comprising: Obtain website data service content, which includes multiple IP addresses of the website; The multiple IP addresses are mapped to multiple business entity identifiers to identify multiple business entities associated with the website's data service content; Obtain the attribute data of each business entity associated with each business entity identifier to identify the business attributes associated with each business entity mapped to each IP address; A training dataset is generated based on the website data service content and the attribute data to train an ISP classifier to identify ISPs associated with the mapped website service data. The training dataset is generated based on the website data service content having an IP address that successfully matches the business entity identifier. The attribute data includes enterprise statistical structure data. The feature set for training the model of the ISP classifier includes features from the enterprise statistical structure data and the website service data, including: industry code, unique visitors, access count, employee count, revenue, and customer data. The website business data includes access counts for each IP address within a time window, unique visitor counts, and indexed customer counts. The time window includes the following intervals: ingesting the website business data, mapping the website business data, running the ISP classifier to identify ISP and non-ISP website businesses, and configuring the continuous maintenance of the model through aggregated tracking and machine intelligence. Generate the model, The model includes supervised classification and is a gradient boosting classifier. The gradient boosting classifier training includes iterating the gradient boosting classifier approximately 1000 times. Train the model on the training dataset; Determine whether the trained model accurately identifies ISPs; and If the trained model is accurate, the ISP classifier is trained using the trained model; and the trained ISP classifier is applied to a database of website tracking data; and Identify website services that can be attributed to an ISP.

2. The method according to claim 1, further comprising: The matching acceptance rate of IP addresses is obtained based on website business data mapped to the business entity identifier; as well as The training dataset is generated based on the IP addresses of website business data that match the business entity identifier.

3. The method according to claim 2, further comprising: Replace the IP address of the website business data mapped to the business entity identifier with the previously identified IP address from the website business data that was previously mapped or matched to other business entities.

4. The method according to claim 1, further comprising: The trained ISP classifier is applied to a database of IP addresses of business entity identifiers mapped from a database of business entity enterprise statistical structures, which does not include visitor smart VI access information.

5. The method according to claim 1, wherein, Website data services that obtain website IP address data also include: Place the tracing code object on the website you want to track; and A log file is generated that identifies all IPs tracked by the tracing code and the cookies associated with those IPs.

6. The method according to claim 1, wherein, The time window includes: Select from the following time windows: weekly interval, bi-weekly interval, monthly interval, bi-monthly interval, and quarterly interval.

7. The method according to claim 6, wherein, The time window includes the monthly interval.

8. The method according to claim 1, wherein, The revenue mentioned includes annual sales.

9. The method according to claim 1, wherein, The industry codes include SIC codes, NAICS codes, or both.

10. The method according to claim 1, wherein, The model {F} predicts the value of the form y = F(x) by minimizing the mean square error of g.

11. The method according to claim 1, wherein, The gradient boosting classifier includes: At each stage m (1 <= m <= M) of gradient boosting for a model Fm that predicts the mean y in the training dataset, the gradient boosting algorithm adds an estimator h[Fm+1(x) = Fm(x) + h(x)], where Fm+1(x) = Fm(x) + h(x) = y, and the gradient boosting makes h fit the residual y - Fm(x).

12. The method according to claim 1, wherein, The model achieves data fitting.

13. The method according to claim 12, wherein, Compared to other classifier models, this model produces accurate ISP identifiers.

14. The method according to claim 1, wherein, The gradient boosting classifier includes processing real website business data and directly affecting the results, while employing automatic notification gates at each stage.

15. The method according to claim 14, wherein, The gradient boosting classifier is configured to provide automatic notifications when the confusion matrix or output results are inaccurate.

16. The method according to claim 1, wherein, The gradient boosting classifier includes a classifier that runs at monthly intervals to generate classification outputs.

17. The method according to claim 16, wherein, The gradient boosting classifier also includes manual review and merging of records.

18. The method according to claim 17, wherein, The gradient boosting classifier also includes a modified dataset that is appended to the next monthly interval of model operation and the training of the model is updated using machine intelligence.

19. A computer program product stored on a computer-readable medium and loadable into the internal memory of a digital computer, the computer program product comprising a software code portion that, when the computer program product is run on a computer, performs the method according to any one of claims 1 to 18.

20. A computer system for identifying Internet Service Providers (ISPs), comprising: A network computer, the network computer comprising: A transceiver for communicating over a network; A memory, the memory being used to store at least an instruction and a word database; and A processor device, operable to execute instructions to perform the following actions, including: Obtain website data service content, which includes multiple IP address data of the website; Map the multiple IP address data of the website to at least one business entity identifier to identify at least one business entity associated with the data business content of the website; Obtain attribute data for each business entity associated with the at least one business entity identifier to identify business attributes associated with the at least one business entity mapped to the plurality of IP address data; A training dataset is generated based on the website data service content and the attribute data to train an ISP classifier to identify ISPs associated with the mapped website service data. The training dataset is generated based on the website data service content having an IP address that successfully matches the business entity identifier. The attribute data includes enterprise statistical structure data. The feature set for training the model of the ISP classifier includes features from the enterprise statistical structure data and the website service data, including: industry code, unique visitors, access count, employee count, revenue, and customer data. The website business data includes access counts for each IP address within a time window, unique visitor counts, and indexed customer counts. The time window includes the following intervals: ingesting website business data, mapping the website business data, running the ISP classifier to identify ISP and non-ISP website businesses, and configuring the continuous maintenance of the model through aggregated tracking and machine intelligence; generating the model, The model includes supervised classification and is a gradient boosting classifier. The gradient boosting classifier training includes iterating the gradient boosting classifier approximately 1000 times; and training the model on the training dataset. Determine whether the trained model accurately identifies ISPs; and If the trained model is accurate, the ISP classifier is trained using the trained model; and the trained ISP classifier is applied to a database of website tracking data; and Identify website services that can be attributed to an ISP.

Images