Visitor Forecast

Abstract

To provide systems and methods for visit prediction using machine learning (ML) attribution techniques.SOLUTION: Data relating to users and their venue visits is collected and merged with data relating to various directed information impressions. Features of the merged data are identified for one or more time intervals and assigned values and / or labels. The identified features and corresponding values / labels may be used to train a ML model to provide a visit probability for each user represented in the merged data. On the basis of the visit probabilities provided by the ML model, the percentage increase (or "lift") in venue visit rates attributable to the directed information impressions can be accurately estimated.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a PCT international patent application filed on May 7, 2020, and claims priority to U.S. patent application Ser. No. 16 / 405,481, entitled "VISITPREDICTION," filed on May 7, 2019, which is incorporated herein by reference in its entirety. [Background technology]

[0002] Generally, marketing attribution refers to identifying the set of actions or events that contribute to the effectiveness of directed information and assigning a value to each action or event. Often, the set of actions or events is based on a staggering number of variables (demographics, location, date, length of exposure, exposure medium, etc.) for the various users associated with the directed information. Accurately quantifying the respective influences of the various variables on user behavior is a complex and often unachievable task.

[0003] It is with respect to these and other general considerations that the aspects disclosed herein have been made, and while relatively particular problems may be discussed, it should be understood that the examples should not be limited to solving the particular problems identified in the background or elsewhere in this disclosure. Summary of the Invention [Means for solving the problem]

[0004] Examples of the present disclosure describe systems and methods for visitation prediction using machine learning (ML) attribution techniques. In one aspect, data related to users and their venue visits is collected and merged with data related to various directed content impressions. Features of the merged data are identified for one or more time intervals and assigned values ​​and / or labels. The identified features and corresponding values / labels can be used to train an ML model to provide visit probabilities for each user represented in the merged data. Based on the visit probabilities provided by the ML model, the increase (or "lift") in venue visit rates attributed to directed content impressions can be accurately estimated.

[0005] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Additional aspects, features, and / or advantages of examples will be set forth in part in the description that follows, and in part will be obvious from the description, or may be learned by practice of the disclosure.

[0006] Non-limiting and non-exhaustive examples are described with reference to the following drawings: [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 illustrates an overview of an exemplary system for visitation prediction using the ML techniques described herein. [Figure 2] FIG. 1 illustrates an exemplary input processing unit for visitation prediction using the ML techniques described herein. [Figure 3] FIG. 1 illustrates an exemplary method for training a visit prediction model described herein. [Figure 4] FIG. 1 illustrates an exemplary method for determining user visit lift described herein. [Figure 5] FIG. 1 illustrates an example of a suitable operating environment in which one or more of the present embodiments may be implemented. DETAILED DESCRIPTION OF THE INVENTION

[0008] Various aspects of the present disclosure are described more fully below with reference to the accompanying drawings, which form a part of this specification and which show specific exemplary aspects. However, different aspects of the present disclosure may be implemented in many different forms and should not be construed as limited to the aspects set forth herein. Rather, these aspects are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the aspects to those skilled in the art. The aspects may be embodied as methods, systems, or devices. Thus, the aspects may take the form of hardware implementations, entirely software implementations, or implementations combining software and hardware aspects. Therefore, the following detailed description should not be construed in a limiting sense.

[0009] Visit probability (e.g., the probability that a person will visit or has visited a place or venue) is often influenced by several demographic and psychographic factors. One potentially important factor may be a person's exposure to directed information related to a particular place or venue. The importance (or validity) of such directed information is based on several variables. Accurately attributing the individual causal influence of these variables on visit decisions is often difficult, if not impossible. However, attributing the individual causal influence of variables is essential for determining an exposed person's expected visit rate (e.g., what would an exposed person's visit behavior have been if they had not been exposed to the directed information). Therefore, without an accurate expected visit rate, the actual importance / validity of the viewed directed information generally cannot be accurately quantified.

[0010] To address such issues, this disclosure describes systems and methods for determining user visitation lift using machine learning (ML) attribution techniques. Visitation lift, as used herein, may refer to an increase in location visitation rate attributing to one or more events or actions. As a particular example, visitation lift may refer to a percentage increase in venue visitation rate attributing to directed information. The directed information may be content (e.g., text, audio, and / or video content), metadata, instructions to perform an action, haptic feedback, or any other form of information that can be transmitted and / or displayed by a device. In an aspect, user identification data and / or user visitation data for one or more locations may be collected. The collected data may be labeled and / or unlabeled. In examples, the user identification data and / or user visitation data may be associated with the directed information. Information regarding impressions of the directed information may also be collected. In examples, the impression information may comprise, among other things, a directed information identifier and an indication of the number of times the directed information (or a medium comprising the directed information) was fetched and / or loaded. The identification data and / or user visit data and impression information may be merged into one or more datasets. An unlimited number of features of the merged data may then be identified. Examples of features include, but are not limited to, user age, user gender, user language, household income, user or mobile device location, number of children in the household, date, day of the week, recency of the previous visit, distance to the venue or visit location, application generating the visit data, device features generating the visit data, directed information identifier, directed information exposure date / time, etc. In examples, allowing an unlimited number of features to be used in visit prediction analysis allows the resulting ML model (described below) to be easily and dynamically modified as additional features are added to the analysis. Furthermore, allowing an unlimited number of features to be used may enable more detailed and accurate attribution analysis.

[0011] In one aspect, the identified features of the merged data may be organized into groups corresponding to individual users and / or individual days. Using one or more featurization techniques, a feature value may be calculated and / or assigned for each feature in each group. The feature value may be a numerical representation of the feature, a value paired with the feature in the merged data, an indication of one or more condition states of the feature, an indication of how well the feature predicts visits, etc. Alternatively or additionally, each group may be assigned a value for each feature corresponding to that group. In examples, the featurization technique may include the use of ML processing, normalization operations, binning operations, and / or vectorization operations. In some aspects, each group may be assigned a visit indication value. The visit indication value may indicate whether the user visited a location or venue. The visit indication value may also indicate whether the user was exposed to directional information and / or whether the visit occurred within a statistically relevant time period of exposure.

[0012] In one aspect, a first set of data comprising identified features, feature values, and / or visit indication values ​​may be provided to a model to train the model to determine whether or not, or the probability that, a user visited a location / venue on a particular date. As used herein, a model may refer to a predictive or statistical model that may be used to determine a probability distribution over one or more character sequences, classes, objects, outcome sets, or events, and / or predict a response value from one or more predictors. The model may be based on or incorporate one or more rule sets, machine learning, neural networks, etc. In some examples, the model may be trained primarily (or exclusively) using data from unexposed users (e.g., users not exposed to directed information). In other examples, the model may be trained primarily (or exclusively) using data from exposed users (e.g., users exposed to directed information). In yet other examples, the model may be trained using data from both exposed and unexposed users. In such examples, the trained model may be configured to accurately estimate / measure typical or expected visiting behavior of users not exposed to directed information.

[0013] In one embodiment, after the model is trained, users exposed to the above-mentioned directional information are identified. User identification data, user visit data, and impression information of the exposed users are collected. The collected data is merged as described above and provided to the trained model. Based on the collected data, a time period for analyzing the merged data can be identified. The analysis time period can correspond to the eligibility date of the users identified in the merged data. The eligibility date, as used herein, can refer to the date on which the effect of the directional information is calculated. In an example, the eligibility date can be determined using the date on which the user was exposed to the directional information (e.g., the exposure date of the directional information) and the period after the exposure date of the directional information. Collectively, the eligibility dates can define an attribution window. The attribution window, as used herein, can refer to the period including the exposure date of the directional information and the period after the exposure date of the directional information. As a specific example, a five-day attribution window can include the exposure date of the directional information and the four days immediately following the exposure date of the directional information.

[0014] In one aspect, for each identified eligible date, the model may calculate and / or output a result set comprising visit decisions and / or visit probabilities for each exposed user. The users' visit decisions / probabilities may be summed to calculate a value indicative of the users' expected total visit rate to the location or venue. In at least one example, the expected total visit rate is based on the assumption that the exposed users were not exposed to the directed information. That is, the expected total visit rate represents a best estimate of the number of visits that would have occurred in the absence of exposure to the directed information.

[0015] In one aspect, the total number of actual visits (e.g., actual total visit rate) occurred by users exposed on eligible dates may be identified. Identifying actual visits may include querying one or more local and / or remote data sources. As a particular example, a visit detection and / or outage detection system may be queried for actual visit data corresponding to a user or set of users for one or more dates. The actual total visit rate may then be evaluated against the expected total visit rate to calculate a percentage increase in visit rate (e.g., visit lift) that attributes directional information associated with the set of collected data. In some aspects, the visit lift may be presented on a user interface, transmitted to one or more devices, or a report or notification may be generated.

[0016] Thus, the present disclosure provides multiple technical benefits, including, but not limited to, quantifying total incremental lift in visit rates attributing to one or more actions or events, creating feature sets from visit and directed impression data, quantifying the importance of various individual variables that influence visit decisions, generating / training visit prediction models with an unlimited number of control variables, using ML techniques to calculate expected visit rates, and leveraging existing visit data and stop detection data, among other examples.

[0017] FIG. 1 illustrates an overview of an exemplary system for visitation prediction using the ML techniques described herein. The presented exemplary system 100 is a combination of interdependent components that interact to form an integrated whole for a venue detection system. The components of the system may be hardware components or software implemented on and / or executed by the system's hardware components. In an example, system 100 may include either hardware components (e.g., used to execute / run an operating system (OS)) and software components (e.g., applications, application programming interfaces (APIs), modules, virtual machines, runtime libraries, etc.) that execute on the hardware. In one example, exemplary system 100 may provide an environment in which the software components execute, adhere to set constraints for operation, and utilize the resources or facilities of system 100, and the components may be software (e.g., applications, programs, modules, etc.) that execute on one or more processing devices. For example, software (e.g., applications, operating procedures, modules, etc.) may execute on processing devices such as computers, mobile devices (e.g., smartphones / phones, tablets, laptops, personal digital assistants (PDAs), etc.), and / or any other electronic devices. For an example of an operating environment for a processing device, see the exemplary operating environment shown in Figure 5. In other examples, components of the systems disclosed herein may be distributed across multiple devices. For example, input may be entered on a client device, and information may be processed or accessed from other devices in the network, such as one or more server devices.

[0018] As an example, system 100 includes computing device 102, distributed network 104, visitation prediction system 106, and storage 108. Those skilled in the art will understand that the scale of a system such as system 100 may vary and may include more or fewer components than those depicted in Figure 1. In some examples, interfaces between components of system 100 may occur remotely, for example, where components of system 100 may be distributed across one or more devices in a distributed network.

[0019] The computing device 102 may be configured to receive and / or access information from or related to one or more users. The information may include, for example, user and / or device identification data (e.g., username / identifier, device name, etc.), demographic data (e.g., age, gender, income, etc.), user visit data (e.g., venue name, geographic location coordinates, Wi-Fi information, length of stop / visit, date / time of visit, etc.), directed information data (e.g., directed information identifier, directed information impression date, number of exposures, etc.), user feedback signals (e.g., active / passive venue check-in data, purchase or shopping events, etc.). Examples of computing device 102 may include client devices (e.g., laptops or PCs, mobile devices, wearable devices, etc.), server devices, web-based appliances, etc.

[0020] In one aspect, at least a portion of the data may be associated with directional information for one or more venues or locations. As a particular example, one or more sensors of computing device 102 may be operable to collect Wi-Fi information, accelerometer data, and check-in data when a user visits a venue that the user was previously exposed to directional information for that venue. The information (or a representation thereof) may be stored locally on computing device 102 or remotely in a remote data store, such as storage 108. In some aspects, computing device 102 may transmit at least a portion of the data to a system, such as visitation prediction system 106, over network 104.

[0021] The visit prediction system 106 may be configured to process and / or characterize the information. In one aspect, the visit prediction system 106 may access the information received / accessed by the computing device 102. Upon accessing the information, the visit prediction system 106 may process (or cause the information to be processed) the information to identify one or more features. The features may be divided into groups representing different users and / or different dates / time periods. For example, a set of features may be created for each user and for each date identified in the information. For each set of features, a corresponding set of feature values ​​may be calculated or identified and assigned to the set of features using one or more characterization techniques. Alternatively, each group may be assigned a value for each feature in that group's feature set. The visit prediction system 106 may further assign a visit indication value to one or more of the groups. The visit indication value may indicate whether a user visited a location or venue on a particular day. In at least one aspect, the visit prediction system 106 may also assign (or otherwise associate) an exposure indication value to one or more groups, indicating whether a user was exposed to directional information within a statistically relevant time period. For example, the exposure display value may classify a user as not exposed, exposed and eligible for the visit analysis (e.g., the user was exposed within the relevant time period of the visit analysis), or exposed and ineligible for the visit analysis (e.g., the user was exposed, but the exposure was not within the relevant time period of the visit analysis).

[0022] The visit prediction system 106 may be further configured to train and / or maintain one or more predictive models. In one aspect, the visit prediction system 106 may have access to one or more predictive models / algorithms or a model generation component for generating one or more predictive models. As a particular example, the visit prediction system 106 may comprise an ML model using one or more k-nearest neighbor, gradient boosted tree, or logistic regression algorithms. Upon identifying / generating a relevant predictive model, the visit prediction system 106 may use the identified features, feature values, visit indication values, and / or exposure indication values ​​to train the predictive model to determine whether or the probability that a user visited a location / venue on a particular day. After the predictive model is trained, the visit prediction system 106 may provide additional information to the trained model from one or more data sources. In one example, the data sources may include the computing device 102, other client devices associated with the user of the computing device 102, client devices of other users, one or more cloud-based services / applications, local and / or remote storage locations (such as the storage 108), etc. In at least one embodiment, the additional information may include data for users exposed to the directional information. As part of the analysis / processing performed on the additional information by the predictive model, one or more attribution windows and / or eligibility dates for the directional information associated with the additional information may be identified. For each identified eligibility date, the predictive model may calculate and / or output a visit decision and / or visit probability for each exposed user. The user visit decisions / probabilities may be summed to calculate the user's expected total visit rate for a location or venue.

[0023] The visit prediction system 106 may be further configured to calculate a visit rate lift for the directed information. In one aspect, the visit prediction system 106 may access data indicating the total number of actual visits (e.g., actual visit rate) occurred by users exposed on the eligible dates identified by the predictive model. The visit prediction system 106 may store the actual total visit rate data locally and may query one or more external data sources or services to access the actual total visit rate data. After accessing the actual total visit rate data, the predictive model or another component of (or accessible to) the visit prediction system 106 may evaluate the actual total visit rate data against a previously calculated expected total visit rate. As a result of the evaluation, a visit rate lift (e.g., a percentage increase in visit rate attributing to the directed information associated with the data collected by the computing device 102) may be calculated. In some aspects, after the visit rate lift is calculated, the visit prediction system 106 may cause one or more actions to be performed. As an example, the visit prediction system 106 may generate a report measuring the effectiveness of the directed information in directing consumers to a physical location. The report may also comprise data relating to causal influences attributing to individual characteristics / factors of various users or user groups.

[0024] 2 shows an overview of an example input processing system 200 for visit prediction using the ML techniques described herein. The visit prediction technology implemented by input processing system 200 may comprise the visit detection techniques and data described in the system of FIG. 1. In some examples, one or more components (or functionality thereof) of input processing system 200 may be distributed across multiple devices. In other examples, a single device (including at least a processor and / or memory) may comprise the components of input processing system 200.

[0025] With reference to FIG. 2 , the input processing system 200 may comprise a data collection engine 202, a processing engine 204, a predictive model 206, and a data store 208. The data collection engine 202 may be configured to collect or receive information related to the directional information. In one aspect, the data collection engine 202 may collect or receive visit information from one or more data sources or computing devices, such as the computing device 102. The visit information may include, for example, user and / or device identification data, user demographic data, user visit and / or stop data, user behavior data, etc. The data collection engine 202 may further collect or receive impression information related to the directional information. The impression information may include, for example, directional information identification data, exposure data, etc. The data collection engine 202 may store the collected data in one or more storage locations and / or make the collected data accessible to one or more applications, services, or components accessible to the input processing system 200. In at least one example, the collected data may be accessed via an interface (not shown) provided by or accessible to the input processing system 200. The interface may allow the collected data to be navigated and / or manipulated by a user. For example, the interface may allow the collected data to be labeled, annotated, and / or classified.

[0026] The processing engine 204 may be configured to process the collected data. In one aspect, the processing engine 204 may access data collected by the data collection engine 202. The processing engine 204 may perform one or more operations on the collected data to process and / or format the collected data. For example, processing the collected data may include a merging operation. The merging operation may merge visit information and impression information according to a user identification and / or date. For example, a user's venue visit data may be matched with the user's directional information exposure using a combination of a user identifier and a date. Processing the collected data may additionally or alternatively include a characterization operation. The characterization operation may identify various features of the collected data. The identified features may be grouped according to one or more criteria, such as a user identifier and / or a date. The value of each feature within a group may be determined using one or more ML techniques. Furthermore, a visit indicator value may be assigned to one or more of the groups. The visit indicator value may indicate whether a user visited a location or venue on a particular day. For example, a group may be assigned a "1" if the user visited the location on a particular day and a "0" if the user did not visit the location on a particular day. In at least one aspect, the characterization operation may further include assigning an exposure indication value to one or more of the groups. The exposure indication value may indicate whether the user was exposed to the directed information within a statistically relevant time period of the visit analysis. For example, a group may be assigned or otherwise associated with a "U" to indicate that the user was not exposed to the directed information, an "EE" to indicate that the user was exposed to the directed information within a statistically relevant time period of the visit analysis, or an "EI" to indicate that the user was exposed to the directed information outside of a statistically relevant time period of the visit analysis.

[0027] The predictive model 206 may be configured to output a visitation prediction value. In one aspect, the processing engine 204 may provide the processed data to the predictive model 206. The predictive model 206 may implement one or more ML algorithms, such as a k-nearest neighbor algorithm, a gradient boosted tree algorithm, or a logistic regression algorithm. The processed data may be used to train the predictive model 206 to determine the probability that a particular user (indicated in the processed data) visited a location / venue on a particular day. For example, based on the processed data provided to the predictive model 206, the predictive model 206 may determine an attribution window within which the processed data should be analyzed. In such an example, the processed data may comprise information for users primarily (or exclusively) exposed to the above-mentioned directional information. The attribution window may define a period during which the impact of exposure to the directional information is statistically relevant to visitation decisions. For each day identified in the attribution window, the predictive model 206 may calculate the probability that a user identified in the processed data visited the target venue or location on that day. The visit probabilities per user and per day may be summed to calculate a value representing a user's expected total visit rate for a location or venue. In one aspect, the predictive model 206 may access actual visit data indicating the total number of actual visits (e.g., actual total visit rate) generated by users exposed to the directed information during the attribution window. The actual visit data may be accessed locally at a data source, such as the data store 208, or may be accessed remotely by querying one or more external data sources or services. After accessing the actual total visit rate data, the predictive model 206 may evaluate the actual total visit rate data against the expected total visit rate to calculate the directed information's visit rate lift. In some aspects, after calculating the directed information's visit rate lift, the predictive model 206 may cause one or more actions to be performed. For example, the predictive model 206 may provide report generation instructions to a reporting component of the input processing system 200.

[0028] Having described various systems that may be employed by aspects disclosed herein, the present disclosure next describes one or more methods that may be performed by various aspects of the present disclosure. In one aspect, methods 300 and 400 may be performed by a visitation prediction system, such as system 100 of FIG. 1 or system 200 of FIG. 2. However, methods 300 and 400 are not limited to such examples. In other aspects, methods 300 and 400 may be performed for an application or service for performing visitation prediction. In at least one aspect, methods 300 and 400 may be performed (e.g., computer-implemented operations) by one or more components of a distributed network, such as a web service / distributed network service (e.g., a cloud service).

[0029] 3 illustrates an example method 300 for training a visitation prediction model described herein. The example method 300 begins at operation 302, where information related to directional information is received. In one aspect, a data collection component, such as the data collection engine 202, may receive visit information from one or more computing devices, such as the computing device 102. The visit information may include, for example, user and / or device identification data, user demographic data, user visit and / or stop data, date / time data, user behavior data, etc. In an example, the time period represented by the visit information may correspond to at least a portion of the directional information. The data collection component may also receive impression information of the directional information from one or more data sources. The impression information may include, for example, directional information identification data, directional information exposure date / time, user and / or device identification data, etc.

[0030] At operation 304, the received information may be merged. In one aspect, a data processing component such as processing engine 204 may merge the visit information and the impression information into a single data set. Merging the information may include matching data in the visit information to data in the impression information using one or more pattern matching techniques, such as regular expressions, fuzzy logic, or the like. For example, a visit information data object and an impression information data object may both comprise a user identifier "X." Normal expression utilities may be used to identify commonalities in both data objects (i.e., the user identifier "X"). Based on the identified commonalities, the two data objects may be merged into a new third data object comprising at least a portion of the information from each of the two data objects.

[0031] In operation 306, features of the merged information may be grouped. In one aspect, a data processing component, such as processing engine 204, may identify various features of the merged information. The identified features may be organized into groups corresponding to individual users and / or individual days. For example, features of the merged information corresponding to user identifier “X” and day “1” may be organized into a first group, and features of the merged information corresponding to user identifier “X” and day “2” may be organized into a second group. In some aspects, group names may be assigned to the groups. The group names may be based on the information used to organize the groups. As an example, for a group comprising information for user identifier “X” and day “1,” the group name “X:1” may be automatically generated and assigned by the data processing component. Alternatively, the group name may be assigned randomly and not immediately (or at all) indicate the information comprised in the group. In at least one aspect, the group names may be manually assigned and / or modified using an interface accessible to the data processing component.

[0032] At operation 308, one or more feature values ​​may be assigned. In one aspect, feature values ​​may be calculated and / or identified for the features in each group using one or more characterization techniques. For example, feature-value pairings in the merged information and information data objects may be identified and evaluated. The evaluation may include identifying and / or extracting values ​​for one or more features, normalizing the values, and assigning the normalized values ​​to each feature. As another example, a value representing the causal influence of an impression feature on a user's visiting behavior may be calculated. For example, the merged data (or groups therein) may comprise gender, age, and income features. Based on one or more attribution models / algorithms, it may be determined that gender attributes a 70% influence to visiting behavior, age attributes a 25% influence to visiting behavior, and income attributes a 70% influence to visiting behavior. As a result, the feature value for gender may be set to 0.70, the feature value for age may be set to 0.25, and the feature value for income may be set to 0.05. Alternatively, each feature value may be weighted according to the influence of the corresponding feature or the tendencies of one or more users. For example, features may be categorized into ranges having specific values. As a specific example, the age range 18-30 may be categorized as a first bucket having a value of 3, the age range 31-45 may be categorized as a second bucket having a value of 2, and the age range 46-60 may be categorized as a third bucket having a value of 1. The bucket values ​​(e.g., 3, 2, 1) may represent the estimated influence of each age range on visiting behavior. A weight may be applied to each bucket value to reflect the combination of the feature's influence value and the associated feature range. Thus, if age attributes a 25% influence on visiting behavior, the age bucket values ​​for buckets 1, 2, and 3 may be calculated to have a total influence of 0.75, 0.50, and 0.25, respectively.

[0033] In operation 310, one or more group values ​​may be assigned. In one aspect, the data processing component may assign each group a visitation indicator value indicating whether the user visited a location or venue on a particular day. For example, a group designated by "X:1" (corresponding to user identifier "X" and day "1") may be assigned a "1" if the user visited the location on a particular day and a "0" if the user did not visit the location on a particular day. As a result, the group designation may be modified accordingly, e.g., to "X:1:1" or "X:1:0." In some aspects, the data processing component may assign each group an exposure indicator value indicating whether the user was exposed to directed information within a statistically relevant time period of the visitation prediction analysis. For example, each group may be assigned (or otherwise associated) with "U" to indicate that the user was not exposed to directed information, "EE" to indicate that the user was exposed to directed information within a statistically relevant time period of the visitation analysis, or "EI" to indicate that the user was exposed to directed information outside of a statistically relevant time period of the visitation analysis. In such an example, the statistically relevant time period may be predefined as a specific number of days from (or including) the date of user exposure to the directional information. In some aspects, the impact of days within the statistically relevant time period on relevance decreases increasingly as the days are further from the date of exposure. For example, the statistically relevant time period for directional information may be defined as four days (e.g., the date of exposure and the three days thereafter). The relevance of the exposed directional information may be determined to have decreased by 25% each day since the date of exposure. As a result, a multiplier of 1.0 may be applied on the date of exposure, a multiplier of 0.75 may be applied one day after the date of exposure, a multiplier of 0.50 may be applied two days after the date of exposure, and a multiplier of 0.25 may be applied three days after the date of exposure. In at least one aspect, the relevance multiplier may be applied to feature values ​​and / or group values.

[0034] At operation 312, a model may be trained using the merged data. In one aspect, a predictive model such as predictive model 206 may be identified or generated. Alternatively, multiple predictive models may be identified or generated. For example, a first predictive model may be trained primarily (or exclusively) using information from exposed users, and a second predictive model may be trained primarily (or exclusively) using information from unexposed users. The predictive model may be a binary bias-corrected logistic regression model trained using the merged data and / or group data (e.g., grouped features and values, group values ​​and / or names, etc.) to determine whether or the probability that one or more users identified in the merged data visited a location / venue on a particular day. In an example, using bias-corrected logistic regression techniques allows the model to account for unfair sampling bias in the data used to train the model. That is, a significant number of rare positive outcome examples (e.g., venue / location visits) may be included in the training dataset, while ensuring that the model's analysis is based on the actual base rates of positive and negative visit outcomes. In certain embodiments, the particular bias-corrected logistic regression technique employed may be described by introducing the notation s0 to represent the sampling rate applied to negative training instances (non-visits) and s1 to represent the sampling rate for positive training instances (visits). In such embodiments, the practical goal is to adjust s0 low (e.g., less than 0.01) while making s1 very large (often exactly equal to 1, meaning no downsampling) to preserve discriminatory information from rare visit data. This may ensure that downsampling of negative training data is controlled to maintain an overall training data size set that meets size constraints related to computer memory limitations, processing time, or other operational constraints applied to model fitting.

[0035] In one aspect, the predictive model may be subject to a specific confidence interval. For example, because the lift calculation may not consider statistical significance, a probability distribution over all possible lift values ​​may be generated. The probability distribution may incorporate a priori knowledge of the lift distribution. In some aspects, a statistical model or algorithm, such as a Markov Chain Monte Carlo (MCMC) algorithm, may be used to sample data from the probability distribution. As used herein, MCMC may refer to a random walk-based algorithm that moves through data points in a manner that depends on the probability distribution. Using the sampled data, various values ​​(e.g., mean, median, percentile, standard deviation, variance, etc.) may be calculated as an expression of the distribution order statistics of lift. For example, the median of the probability distribution may be identified, and a confidence interval bounded by, for example, the 5th and 95th percentiles may be established.

[0036] FIG. 4 illustrates an example method 400 for determining user visitation lift as described herein. The example method 400 begins at operation 402, where information for users exposed to the directional information is identified. In one aspect, a data collection component, such as the data collection engine 202, may receive visitation information for one or more users exposed to the directional information (e.g., exposed users). In some aspects, the visitation information may further include information for one or more users not exposed to the directional information (e.g., unexposed users). The visitation information may be received from one or more computing devices, such as the computing device 102, or one or more data sources, such as the data store 208. In at least one particular example, the visitation information may be collected from a context awareness engine that records users' visitation patterns to venues and locations. The visitation information may include, for example, user and / or device identification data, user demographic data, user visit and / or stop data, date / time data, user behavior data, etc. In one aspect, the data collection component may also receive impression information associated with the user from one or more data sources. Impression information may include, for example, directed information identification data, directed information exposure date / time, user and / or device identification data, and the like.

[0037] In some aspects, the received visit information and / or impression information may correspond to a set of users having particular characteristics or attributes. The characteristics of the set of users may be the same as (or substantially similar to) the characteristics of the set of training data used to train the predictive model described in method 300 of FIG. 3. For example, the predictive model may be trained using five characteristics of users in the set of training data (e.g., age, gender, metropolitan area, recency of visit, and language). As a result, for each user in the training data, one or more users having characteristics matching (or similar to) the user in the training data may be identified, and visit information for the identified set of users may be received / collected. In at least one aspect, the received visit information and / or impression information may be merged. Merging the information may comprise identifying various characteristics of the information and grouping the information into one or more groups. As described in method 300 of FIG. 3, merging the information may also include generating values ​​for the characteristics and / or groups.

[0038] At operation 404, an attribution window may be identified. In one aspect, an attribution window for the directed information exposed to the exposed user may be identified. The attribution window may comprise exposure data for the directed information and the number of days from the exposure date. In one example, the attribution window may be pre-selected by a user associated with the administration or management of the directed information. In another example, the attribution window may be pre-defined by a data collection component or a component of a visitation prediction system. In yet another example, the attribution window may be dynamically determined based on received visitation information and / or impression information. For example, one or more ML techniques may be used to define a time period during which the impact of the directed information remains statistically relevant after a user is exposed to the directed information. The ML techniques may assign a value to each day in the attribution window to represent a decrease in the impact of the directed information on relevance for additional days after the exposure date of the directed information.

[0039] In operation 406, the received information may be provided as input to a predictive model. In one aspect, the received visit information, impression information, and / or corresponding feature and group data may be provided as input to a predictive model, such as predictive model 206. The predictive model may be, for example, a binary logistic regression model trained to determine whether or the probability that a user identified in the received information visited a location / venue on a particular day. For example, the information input to the predictive model may be organized into groups corresponding to users and / or dates. The feature data for each group may be provided to the predictive model. As a result, the predictive model may output a probability that a particular user visited a target venue or location on a particular day. In one aspect, the probabilities output by the predictive model may be summed to calculate a value indicative of an expected total visit rate for the location or venue. The expected total visit rate may be based on the assumption that the users represented in the information input to the predictive model had not been exposed to the directional information.

[0040] At operation 408, an actual visit rate for the location or venue may be determined. In one aspect, a total number of actual visits occurred by users during the attribution window may be identified. In one example, the total number of actual visits may correspond to the number of users exposed to the directed information, the number of users not exposed to the directed information, or some combination thereof. Identifying the total number of actual visits may comprise querying one or more services and / or remote data sources. Alternatively, identifying the total number of actual visits may comprise receiving input manually entered by a user using an interface.

[0041] At operation 410, a visit rate lift may be calculated. In one aspect, to calculate the visit rate lift of the directed information (e.g., the percentage increase in visit rate attributing the directed information), the total number of actual visits (e.g., the actual total visit rate) may be evaluated against the expected total visit rate. In one particular example, the visit rate lift may be calculated using the following formula:

[0042]

number

[0043] In the above equation, d is a single eligible day (representing both the user and the date before which the user was most recently exposed to the directed information), D is the set of all eligible D-days in the analysis, visited?(d) is whether the user encoded in d visited the target chain on that day, probVisited?(d) is the probability that an unexposed user will visit on date d, and visits actual is the total number of visits actually made on the eligibility date, and visits estimated is the estimated total number of visits made by undisclosed users on the eligibility dates.

[0044] At optional operation 412, one or more actions may be performed in response to the calculation of the visit lift rate. In one aspect, one or more actions or events may be performed in response to the calculation of the visit lift rate. The actions / events may include generating a report, providing information to a predictive model, comparing the results of two or more predictive models, calculating one or more confidence intervals for the calculated visit lift rate, and adjusting the statistical significance of various features and / or feature values. As one particular example, a report measuring the effectiveness of the directed information may be generated and displayed to one or more users. The report may include the various features analyzed, the estimated causal impact of the features on visit behavior, and / or the attribution window for which the visit prediction analysis was performed.

[0045] FIG. 5 illustrates an exemplary suitable operating environment for the venue detection system described in FIG. 1. In its most basic configuration, the operating environment 500 typically includes at least one processing unit 502 and memory 504. Depending on the exact configuration and type of computing device, the memory 504 (which stores instructions for executing embodiments of the visitation prediction disclosed herein) may be volatile (e.g., RAM), non-volatile (e.g., ROM, flash memory, etc.), or some combination of the two. This most basic configuration is illustrated in FIG. 5 by dashed line 506. Additionally, the environment 500 may also include storage devices (removable 508 and / or non-removable 510), including, but not limited to, magnetic or optical disks or tape. Similarly, the environment 500 may also have input devices 514, such as a keyboard, mouse, pen, voice input, etc., and / or output devices 516, such as a display, speakers, printer, etc. Additionally included in the environment may be one or more communication connections 512, such as a LAN, WAN, point-to-point, etc. In embodiments, the connection may be operable to facilitate point-to-point communication, connection-oriented communication, connectionless communication, and the like.

[0046] The operating environment 500 typically includes at least some form of computer-readable media. Computer-readable media may be any available media that can be accessed by the processing unit 502 or other devices that comprise the operating environment. By way of example, and not limitation, computer-readable media may comprise computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Computer storage media includes RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVDs) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store the desired information. Computer storage media does not include communication media.

[0047] Communication media embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, microwave, and other wireless media. Combinations of any of the above should also be included within the scope of computer-readable media.

[0048] Operating environment 500 may be a single computer operating in a networked environment using logical connections to one or more remote computers. The remote computers may be personal computers, servers, routers, network PCs, peer devices, or other common network nodes, and typically include many or all of the elements listed above, as well as other elements not specifically mentioned. The logical connections may include any method supported by available communications media. Such networked environments are commonplace in offices, enterprise-wide computer networks, intranets, and the Internet.

[0049] The embodiments described herein may be employed using software, hardware, or a combination of software and hardware to implement and perform the systems and methods disclosed herein. While specific devices have been cited throughout this disclosure as performing particular functions, those skilled in the art will understand that these devices are provided for illustrative purposes and that other devices may be employed to perform the functions disclosed herein without departing from the scope of the present disclosure.

[0050] This disclosure describes several embodiments of the technology with reference to the accompanying drawings, which illustrate only some of the possible embodiments. However, other aspects may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of possible embodiments to those skilled in the art.

[0051] Although specific embodiments are described herein, the scope of the present technology is not limited to those specific embodiments. Those skilled in the art will recognize other embodiments or improvements that fall within the scope and spirit of the present technology. Therefore, specific structures, acts, or mediums are disclosed only as exemplary embodiments. The scope of the present technology is defined by the following claims and equivalents therein. [Explanation of symbols]

[0052] 100 systems 102 Computing Devices 104 Decentralized Network 106 Visitor Prediction System 108 Storage 200 Input Processing System 202 Data Collection Engine 204 Processing Engine 206 Predictive Model 208 Datastore 300 ways 400 ways 500 Operating environment 502 Processing Unit 504 memory 506 dashed line 508 Removable Storage Devices 510 Non-removable storage device 512 communication connections 514 input devices 516 output devices

Claims

1. one or more processors; a memory coupled to at least one of the one or more processors; wherein the memory, when executed by the at least one processor, receiving visitation information associated with one or more users exposed to the directed information; receiving impression information related to the directional information, the impression information being associated with at least a portion of the one or more users; identifying an attribution window associated with the directional information; providing the visit information and the impression information within the attribution window to a machine learning model to calculate an expected visit rate for the one or more users; determining an actual visit rate for said one or more users; evaluating the actual visit rate against the expected visit rate to calculate a visit lift rate; A system comprising computer-executable instructions for performing a method comprising:

2. The system of claim 1 , wherein the visit information is collected from a context awareness engine that records a user's visitation patterns to locations.

3. The system of claim 2 , wherein the attribution window defines a date of exposure to the directional information and a number of days since the date of exposure.

4. The system of claim 2 , wherein the machine learning model is a binary logistic regression model.

5. The system of claim 1 , wherein the expected visitation rate represents a probability that the one or more users visited one or more locations on one or more days.

6. The system of claim 1 , wherein the actual visit rate represents the number of visits actually generated by a user during the attribution window.

7. The system of claim 1 , wherein the visit lift rate represents an increase in visit rate attributing the directed information.

8. The system of claim 1 , wherein calculating the visit lift rate comprises dividing the actual visit rate by the expected visit rate.

9. The method comprises:

10. The system of claim 1, further comprising: performing one or more actions in response to calculating the visit lift rate, the one or more actions including automatically generating a report.

10. A computer-implemented method comprising: receiving visit information associated with one or more users, wherein the one or more users are exposed to directed information; receiving impression information related to the directional information, the impression information being associated with at least a portion of the one or more users; identifying an attribution window associated with the directional information; providing the visit information and the impression information within the attribution window to a machine learning model to calculate an expected visit rate for the one or more users; determining an actual visit rate of a user during the attribution window; calculating a visit lift rate using the expected visit rate and the actual visit rate, the visit lift rate representing an increase in visit rate attributable to exposure to the directed information; A method comprising:

Images