Visit prediction

Abstract

To provide systems and methods for visit prediction using machine learning (ML) attribution techniques.SOLUTION: Data relating to users and their venue visits is collected and merged with data relating to various directed information impressions. Features of the merged data are identified for one or more time intervals and assigned values and / or labels. The identified features and corresponding values / labels may be used to train a ML model to provide a visit probability for each user represented in the merged data. On the basis of the visit probabilities provided by the ML model, the percentage increase (or "lift") in venue visit rates attributable to the directed information impressions can be accurately estimated.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application was filed as a PCT international patent application on May 7, 2020, and claims priority to U.S. patent application Ser. No. 16 / 405,481, entitled "VISITPREDICTION," filed May 7, 2019, which is incorporated by reference in its entirety herein. [Background technology]

[0002] Generally, marketing attribution refers to the identification of a set of actions or events that contribute to the effectiveness of a directed piece of information, and the assignment of a value to each action or event. Often, the set of actions or events is based on a staggering amount of variables (demographics, location, date, length of exposure, exposure medium, etc.) of the various users associated with the directed piece of information. Accurately quantifying the respective influences of the various variables on user behavior is a complex and often unachievable task.

[0003] It is with respect to these and other general considerations that the aspects disclosed herein have been made, and while relatively particular problems may be discussed, it should be understood that the examples should not be limited to solving the particular problems identified in the background or elsewhere in this disclosure. Summary of the Invention [Means for solving the problem]

[0004] Examples of the present disclosure describe systems and methods for visitation prediction using machine learning (ML) attribution techniques. In one aspect, data related to users and their venue visits are collected and merged with data related to various directed content impressions. Features of the merged data are identified for one or more time intervals and assigned values ​​and / or labels. The identified features and corresponding values / labels can be used to train an ML model to provide visit probabilities for each user represented in the merged data. Based on the visit probabilities provided by the ML model, the increase (or "lift") in venue visit rates attributing to directed content impressions can be accurately estimated.

[0005] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Additional aspects, features, and / or advantages of the examples will be set forth in part in the description that follows, and in part will be obvious from the description, or may be learned by practice of the disclosure.

[0006] Non-limiting and non-exhaustive examples are described with reference to the following drawings: [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 illustrates an overview of an exemplary system for visitation prediction using ML techniques described herein. [Diagram 2] FIG. 2 illustrates an example input processing unit for visitation prediction using the ML techniques described herein. [Diagram 3] FIG. 1 illustrates an example method for training a visit prediction model described herein. [Figure 4] FIG. 2 illustrates an example method for determining user visitation lift described herein. [Diagram 5] FIG. 1 illustrates an example of a suitable operating environment in which one or more of the present embodiments may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Various aspects of the present disclosure are described more fully below with reference to the accompanying drawings, which form a part of this specification and show certain exemplary aspects. However, different aspects of the present disclosure can be implemented in many different forms and should not be construed as being limited to the aspects set forth herein. Rather, these aspects are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the aspects to those skilled in the art. The aspects may be implemented as a method, a system, or a device. Thus, the aspects may take the form of a hardware implementation, a fully software implementation, or an implementation combining software and hardware aspects. Thus, the following detailed description should not be construed in a limiting sense.

[0009] Visitation probability (e.g., the probability that a person will visit or has visited a place or venue) is often influenced by several demographic and psychographic factors. One potentially important factor may be a person's exposure to directed information related to a particular place or venue. The importance (or validity) of such directed information is based on several variables. It is often difficult, if not impossible, to accurately attribute the individual causal influence of these variables on the visit decision. However, attributing the individual causal influence of the variables is essential to determine an exposed person's expected visitation rate (e.g., what would an exposed person's visiting behavior have been if that person had not been exposed to the directed information). Thus, without an accurate expected visitation rate, the actual importance / validity of the viewed directed information generally cannot be accurately quantified.

[0010] To address such issues, the present disclosure describes systems and methods for determining user visitation lift using machine learning (ML) attribution techniques. Visitation lift, as used herein, may refer to an increase in a location visitation rate attributing to one or more events or actions. As a particular example, visitation lift may refer to a percentage increase in a venue visitation rate attributing to directional information. The directional information may be content (e.g., text, audio, and / or video content, etc.), metadata, instructions to perform an action, haptic feedback, or any other form of information that can be transmitted and / or displayed by a device. In an aspect, user identification data and / or user visitation data for one or more locations may be collected. The collected data may be labeled and / or unlabeled. In an example, the user identification data and / or user visitation data may be associated with the directional information. Information regarding impressions of the directional information may also be collected. In an example, the impression information may comprise, among other things, a directional information identifier and an indication of the number of times the directional information (or a medium comprising the directional information) was fetched and / or loaded. The identification data and / or user visit data and impression information may be merged into one or more data sets. An unconstrained number of features of the merged data may then be identified. Examples of features include, but are not limited to, the user's age, the user's gender, the user's language, household income, the user or mobile device's location, the number of children in the household, the date, the day of the week, the recency of the last visit, the distance to the venue or visit location, the application generating the visit data, the features of the device generating the visit data, the directed information identifier, the directed information exposure date / time, etc. In an example, allowing an unconstrained number of features to be used in the visit prediction analysis allows the resulting ML model (described below) to be easily and dynamically modified as additional features are added to the analysis. Additionally, allowing an unconstrained number of features to be used may enable more detailed and accurate attribution analysis.

[0011] In one aspect, the identified features of the merged data may be organized into groups corresponding to individual users and / or individual days. Using one or more characterization techniques, a feature value may be calculated and / or assigned for each feature in each group. The feature value may be a numerical representation of the feature, a value paired with the feature in the merged data, an indication of one or more condition states of the feature, an indication of how well the feature can predict a visit, etc. Alternatively or additionally, each group may be assigned a value for each feature corresponding to that group. In examples, the characterization technique may include the use of ML processing, normalization operations, binning operations, and / or vectorization operations. In some aspects, each group may be assigned a visit display value. The visit display value may indicate whether the user visited a location or venue. The visit display value may also indicate whether the user was exposed to directional information and / or whether the visit occurred within a statistically relevant time period of exposure.

[0012] In one aspect, a first set of data comprising the identified features, feature values, and / or visit indication values ​​may be provided to a model to train the model to determine whether or not, or the probability that, a user visited a place / venue on a particular date. A model as used herein may refer to a predictive or statistical model that may be used to determine a probability distribution over one or more character sequences, classes, objects, outcome sets, or events, and / or to predict a response value from one or more predictors. The model may be based on or incorporate one or more rule sets, machine learning, neural networks, and the like. In some examples, the model may be trained using data of primarily (or exclusively) unexposed users (e.g., users not exposed to directed information). In other examples, the model may be trained using data of primarily (or exclusively) exposed users (e.g., users exposed to directed information). In still other examples, the model may be trained using data of both exposed and unexposed users. In such examples, the trained model may be configured to accurately estimate / measure typical or expected visiting behavior of users not exposed to directed information.

[0013] In one aspect, after the model is trained, users exposed to the above-mentioned directional information are identified. User identification data, user visit data, and impression information of the exposed users are collected. The collected data is merged as described above and provided to the trained model. Based on the collected data, a time period during which the merged data is analyzed can be identified. The analysis time period can correspond to the eligibility date of the user identified in the merged data. The eligibility date as used herein may refer to the date on which the effect of the directional information is calculated. In an example, the eligibility date can be determined using the date on which the user was exposed to the directional information (e.g., the exposure date of the directional information) and the period after the exposure date of the directional information. Collectively, the eligibility date can define an attribution window. The attribution window as used herein may refer to the period including the exposure date of the directional information and the period after the exposure date of the directional information. As a specific example, a 5-day attribution window can include the exposure date of the directional information and the 4 days immediately following the exposure date of the directional information.

[0014] In one aspect, for each identified eligible date, the model may calculate and / or output a result set comprising visit decisions and / or visit probabilities for each exposed user. The visit decisions / probabilities for the users may be summed to calculate a value indicative of the users' expected total visit rate to the location or venue. In at least one example, the expected total visit rate is based on the assumption that the exposed users were not exposed to the directional information. That is, the expected total visit rate represents a best estimate of the number of visits that would have occurred in the absence of exposure to the directional information.

[0015] In one aspect, a total number of actual visits (e.g., actual total visit rate) that occurred by a user exposed on a qualifying date may be identified. Identifying the actual visits may include querying one or more local and / or remote data sources. As a particular example, a visit detection and / or outage detection system may be queried for actual visit data corresponding to a user or set of users on one or more dates. The actual total visit rate may then be evaluated against the expected total visit rate to calculate a percentage increase in visit rate (e.g., visit lift) that attributes to the directional information associated with the set of collected data. In some aspects, the visit lift may be presented on a user interface, transmitted to one or more devices, or a report or notification may be generated.

[0016] Thus, the present disclosure provides multiple technical benefits including, but not limited to, quantifying total incremental lift in visitation rates attributing to one or more actions or events, creating feature sets from visit and directed impression data, quantifying the importance of various individual variables that influence visit decisions, generating / training visitation prediction models with an unlimited number of control variables, using ML techniques to calculate predicted visitation rates, and leveraging existing visitation and stop detection data, among other examples.

[0017] FIG. 1 shows an overview of an exemplary system for visitation prediction using ML techniques described herein. The presented exemplary system 100 is a combination of interdependent components that interact to form an integrated whole for a venue detection system. The components of the system may be hardware components or software implemented on and / or executed by the hardware components of the system. In an example, the system 100 may include either hardware components (e.g., used to execute / run an operating system (OS)) and software components (e.g., applications, application programming interfaces (APIs), modules, virtual machines, runtime libraries, etc.) that execute on the hardware. In an example, the exemplary system 100 may provide an environment in which the software components execute, follow constraints set for operation, and utilize resources or facilities of the system 100, and the components may be software (e.g., applications, programs, modules, etc.) that execute on one or more processing devices. For example, the software (e.g., applications, operating procedures, modules, etc.) may execute on processing devices such as computers, mobile devices (e.g., smartphones / phones, tablets, laptops, personal digital assistants (PDAs), etc.) and / or any other electronic devices. For an example of an operating environment for a processing device, see the exemplary operating environment illustrated in Figure 5. In other examples, components of the systems disclosed herein may be distributed across multiple devices. For example, input may be entered on a client device and information may be processed or accessed from other devices in a network, such as one or more server devices.

[0018] As an example, system 100 includes a computing device 102, a distributed network 104, a visitation prediction system 106, and storage 108. Those skilled in the art will appreciate that a system such as system 100 may vary in size and may include more or fewer components than those depicted in Figure 1. In some examples, interfaces between components of system 100 may occur remotely, for example, where components of system 100 may be distributed across one or more devices of a distributed network.

[0019] The computing device 102 may be configured to receive and / or access information from or related to one or more users. The information may include, for example, user and / or device identification data (e.g., username / identifier, device name, etc.), demographic data (e.g., age, gender, income, etc.), user visit data (e.g., venue name, geographic location coordinates, Wi-Fi information, length of stop / visit, date / time of visit, etc.), directed information data (e.g., directed information identifier, date of directed information impression, number of exposures, etc.), user feedback signals (e.g., active / passive venue check-in data, purchase or shopping events, etc.). Examples of computing devices 102 may include client devices (e.g., laptops or PCs, mobile devices, wearable devices, etc.), server devices, web-based appliances, etc.

[0020] In an aspect, at least a portion of the data may be associated with directional information for one or more venues or locations. As a particular example, one or more sensors of the computing device 102 may be operable to collect Wi-Fi information, accelerometer data, and check-in data when the user visits a venue that the user was previously exposed to directional information for that venue. The information (or a representation thereof) may be stored locally on the computing device 102 or remotely in a remote data store, such as the storage 108. In some aspects, the computing device 102 may transmit at least a portion of the data to a system, such as the visit prediction system 106, over the network 104.

[0021] The visit prediction system 106 may be configured to process and / or characterize the information. In one aspect, the visit prediction system 106 may access the information received / accessed by the computing device 102. Upon accessing the information, the visit prediction system 106 may process (or cause the information to be processed) the information to identify one or more features. The features may be divided into groups representing different users and / or different dates / time periods. For example, a set of features may be created for each user and for each date identified in the information. For each set of features, a corresponding set of feature values ​​may be calculated or identified using one or more characterization techniques and assigned to the set of features. Alternatively, each group may be assigned a value for each feature in the group's feature set. The visit prediction system 106 may further assign a visit indication value to one or more of the groups. The visit indication value may indicate whether a user visited a place or venue on a particular date. In at least one aspect, the visit prediction system 106 may also assign (or otherwise associate) an exposure indication value to one or more groups that indicates whether a user was exposed to directional information within a statistically relevant time period. For example, the exposure indication value may classify a user as not exposed, exposed and eligible for the visit analysis (e.g., the user was exposed within the relevant time period of the visit analysis), or exposed and ineligible for the visit analysis (e.g., the user was exposed, but the exposure was not within the relevant time period of the visit analysis).

[0022] The visit prediction system 106 may further be configured to train and / or maintain one or more predictive models. In one aspect, the visit prediction system 106 may access one or more predictive models / algorithms or a model generation component for generating one or more predictive models. As a particular example, the visit prediction system 106 may comprise an ML model using one or more k-nearest neighbors, gradient boosted trees, or logistic regression algorithms. Upon identifying / generating a relevant predictive model, the visit prediction system 106 may use the identified features, feature values, visit display values, and / or exposure display values ​​to train the predictive model to determine whether or the probability that a user visited a location / venue on a particular day. After the predictive model is trained, the visit prediction system 106 may provide additional information to the trained model from one or more data sources. In one example, the data sources may include the computing device 102, other client devices associated with the user of the computing device 102, client devices of other users, one or more cloud-based services / applications, local and / or remote storage locations (such as the storage 108), and the like. In at least one embodiment, the additional information may include data for users exposed to the directional information above. As part of the analysis / processing performed on the additional information by the predictive model, one or more attribution windows and / or eligibility dates for the directional information associated with the additional information may be identified. For each identified eligibility date, the predictive model may calculate and / or output a visit decision and / or visit probability for each exposed user. The visit decisions / probabilities of the users may be summed to calculate the users' expected total visit rate to a location or venue.

[0023] The visit prediction system 106 may be further configured to calculate a visit rate lift of the directed information. In one aspect, the visit prediction system 106 may access data indicating the total number of actual visits (e.g., actual visit rate) that occurred by users exposed to the eligible date identified by the predictive model. The visit prediction system 106 may store the actual total visit rate data locally and may query one or more external data sources or services to access the actual total visit rate data. After accessing the actual total visit rate data, the predictive model or another component of (or having access to) the visit prediction system 106 may evaluate the actual total visit rate data against the previously calculated expected total visit rate. As a result of the evaluation, a visit rate lift (e.g., a percentage increase in visit rate attributing to the directed information associated with the data collected by the computing device 102) may be calculated. In some aspects, after the visit rate lift is calculated, the visit prediction system 106 may cause one or more actions to be performed. As an example, the visit prediction system 106 may generate a report that measures the effectiveness of the directed information in directing consumers to a physical location. The reports may also comprise data relating to causal influences attributable to individual characteristics / factors of various users or user groups.

[0024] 2 shows an overview of an example input processing system 200 for visit prediction using the ML techniques described herein. The visit prediction techniques implemented by the input processing system 200 may comprise the visit detection techniques and data described in the system of FIG. 1. In some examples, one or more components (or functions thereof) of the input processing system 200 may be distributed across multiple devices. In other examples, a single device (including at least a processor and / or memory) may comprise the components of the input processing system 200.

[0025] With reference to FIG. 2, the input processing system 200 may comprise a data collection engine 202, a processing engine 204, a predictive model 206, and a data store 208. The data collection engine 202 may be configured to collect or receive information related to the directional information. In an aspect, the data collection engine 202 may collect or receive visit information from one or more data sources or computing devices, such as the computing device 102. The visit information may include, for example, user and / or device identification data, user demographic data, user visit and / or stop data, user behavior data, and the like. The data collection engine 202 may further collect or receive impression information related to the directional information. The impression information may include, for example, directional information identification data, exposure data, and the like. The data collection engine 202 may store the collected data in one or more storage locations and / or make the collected data accessible to one or more applications, services, or components accessible to the input processing system 200. In at least one example, the collected data may be accessed via an interface (not shown) provided by or accessible to the input processing system 200. The interface may allow the collected data to be navigated and / or manipulated by a user. For example, the interface may allow the collected data to be labeled, annotated, and / or classified.

[0026] The processing engine 204 may be configured to process the collected data. In an aspect, the processing engine 204 may access data collected by the data collection engine 202. The processing engine 204 may perform one or more operations on the collected data to process and / or format the collected data. For example, processing the collected data may include a merging operation. The merging operation may merge visit information and impression information according to a user identification and / or date. For example, a user's venue visit data may be matched with a user's exposure of directional information using a combination of a user identifier and a date. Processing the collected data may additionally or alternatively include a characterization operation. The characterization operation may identify various features of the collected data. The identified features may be grouped according to one or more criteria, such as a user identifier and / or a date. The value of each of the features in the group may be determined using one or more ML techniques. Additionally, a visit display value may be assigned to one or more of the groups. The visit display value may indicate whether a user visited a location or venue on a particular day. For example, a group is assigned a "1" if the user visited the location on a particular day and a "0" if the user did not visit the location on a particular day. In at least one aspect, the characterization operation may further include assigning an exposure indication value to one or more of the groups. The exposure indication value may indicate whether the user was exposed to the directional information within a statistically relevant time period of the visit analysis. For example, a group may be assigned or otherwise associated with a "U" to indicate that the user was not exposed to the directional information, an "EE" to indicate that the user was exposed to the directional information within a statistically relevant time period of the visit analysis, or an "EI" to indicate that the user was exposed to the directional information outside of the statistically relevant time period of the visit analysis.

[0027] The predictive model 206 may be configured to output a visitation prediction value. In an aspect, the processing engine 204 may provide the processed data to the predictive model 206. The predictive model 206 may implement one or more ML algorithms, such as a k-nearest neighbor algorithm, a gradient boosted tree algorithm, or a logistic regression algorithm. The processed data may be used to train the predictive model 206 to determine the probability that a particular user (indicated in the processed data) visited a place / venue on a particular day. For example, based on the processed data provided to the predictive model 206, the predictive model 206 may determine an attribution window in which the processed data should be analyzed. In such an example, the processed data may comprise information for users who are primarily (or exclusively) exposed to the above-mentioned directional information. The attribution window may define a period during which the impact of exposure of the directional information is statistically relevant to the visitation decision. For each day identified in the attribution window, the predictive model 206 may calculate the probability that a user identified in the processed data visited the target venue or place on that day. The visit probabilities per user and per day may be summed to calculate a value representing the expected total visit rate of the user to the location or venue. In one aspect, the predictive model 206 may access actual visit data indicating the total number of actual visits (e.g., actual total visit rate) that occurred by users exposed to the directed information during the attribution window. The actual visit data may be accessed locally at a data source such as the data store 208 or may be accessed remotely by querying one or more external data sources or services. After accessing the actual total visit rate data, the predictive model 206 may evaluate the actual total visit rate data against the expected total visit rate to calculate the directed information's visit rate lift. In some aspects, after calculating the directed information's visit rate lift, the predictive model 206 may cause one or more actions to be performed. For example, the predictive model 206 may provide report generation instructions to a reporting component of the input processing system 200.

[0028] Having described various systems that may be employed by the aspects disclosed herein, the present disclosure will now describe one or more methods that may be performed by various aspects of the present disclosure. In one aspect, the methods 300 and 400 may be performed by a visitation prediction system, such as the system 100 of FIG. 1 or the system 200 of FIG. 2. However, the methods 300 and 400 are not limited to such examples. In other aspects, the methods 300 and 400 may be performed for an application or service for performing visitation prediction. In at least one aspect, the methods 300 and 400 may be performed (e.g., computer-implemented operations) by one or more components of a distributed network, such as a web service / distributed network service (e.g., cloud service).

[0029] FIG. 3 illustrates an example method 300 for training a visitation prediction model described herein. The example method 300 begins at operation 302, where information related to directional information is received. In one aspect, a data collection component, such as the data collection engine 202, may receive visit information from one or more computing devices, such as the computing device 102. The visit information may include, for example, user and / or device identification data, user demographic data, user visit and / or stop data, date / time data, user behavior data, etc. In an example, a time period represented by the visit information may correspond to at least a portion of the directional information. The data collection component may also receive impression information of the directional information from one or more data sources. The impression information may include, for example, directional information identification data, directional information exposure date / time, user and / or device identification data, etc.

[0030] In operation 304, the received information may be merged. In one aspect, a data processing component such as the processing engine 204 may merge the visit information and the impression information into a single data set. Merging the information may include matching data in the visit information to data in the impression information using one or more pattern matching techniques such as regular expressions, fuzzy logic, and the like. For example, the visit information data object and the impression information data object may both comprise a user identifier "X". Regular expression utilities may be used to identify commonalities in both data objects (i.e., the user identifier "X"). Based on the identified commonalities, the two data objects may be merged into a new third data object comprising at least a portion of the information from each of the two data objects.

[0031] In operation 306, features of the merged information may be grouped. In one aspect, a data processing component, such as the processing engine 204, may identify various features of the merged information. The identified features may be organized into groups corresponding to individual users and / or individual days. For example, features of the merged information corresponding to user identifier "X" and day "1" may be organized into a first group, and features of the merged information corresponding to user identifier "X" and day "2" may be organized into a second group. In some aspects, a group name may be assigned to the group. The group name may be based on the information used to organize the group. As an example, for a group comprising information of user identifier "X" and day "1", the group name "X:1" may be automatically generated and assigned by the data processing component. Alternatively, the group name may be assigned randomly and not immediately (or at all) indicative of the information comprised in the group. In at least one aspect, the group name may be manually assigned and / or modified using an interface accessible to the data processing component.

[0032] In operation 308, one or more feature values ​​may be assigned. In one aspect, feature values ​​may be calculated and / or identified for features in each group using one or more characterization techniques. For example, feature-value pairings in the merged information and information data objects may be identified and evaluated. The evaluation may include identifying and / or extracting values ​​for one or more features, normalizing the values, and assigning the normalized values ​​to the respective features. As another example, values ​​representing the causal influence of the impression features on the user's visiting behavior may be calculated. For example, the merged data (or groups therein) may comprise gender, age, and income features. Based on one or more attribution models / algorithms, it may be determined that gender attributes 70% influence to visiting behavior, age attributes 25% influence to visiting behavior, and income attributes 70% influence to visiting behavior. As a result, the feature value of gender may be set to 0.70, the feature value of age may be set to 0.25, and the feature value of income may be set to 0.05. Alternatively, each feature value may be weighted according to the influence of the corresponding feature or tendencies of one or more users. For example, features may be categorized into ranges having specific values. As a specific example, the age range 18-30 may be categorized as a first bucket having a value of 3, the age range 31-45 may be categorized as a second bucket having a value of 2, and the age range 46-60 may be categorized as a third bucket having a value of 1. The bucket values ​​(e.g., 3, 2, 1) may represent the estimated influence of each age range on the visiting behavior. A weight may be applied to the value of each bucket to reflect the combination of the feature's influence value and the associated feature range. Thus, if age attributes a 25% influence on visiting behavior, the age bucket values ​​for buckets 1, 2, and 3 may be calculated to have a total influence of 0.75, 0.50, and 0.25, respectively.

[0033] In operation 310, one or more group values ​​may be assigned. In one aspect, the data processing component may assign each group a visit indicator value indicating whether the user visited a location or venue on a particular day. For example, a group designated with "X:1" (corresponding to user identifier "X" and day "1") may be assigned a "1" if the user visited the location on a particular day and a "0" if the user did not visit the location on a particular day. As a result, the group designation may be modified accordingly, for example, to "X:1:1" or "X:1:0". In some aspects, the data processing component may assign each group an exposure indicator value indicating whether the user was exposed to the directional information within the statistically relevant time period of the visit prediction analysis. For example, each group may be assigned (or otherwise associated) with a "U" to indicate that the user was not exposed to the directional information, an "EE" to indicate that the user was exposed to the directional information within the statistically relevant time period of the visit analysis, or an "EI" to indicate that the user was exposed to the directional information outside the statistically relevant time period of the visit analysis. In such an example, the statistically relevant time period may be predefined as a certain number of days from (or including) the date of exposure of the user to the directional information. In some aspects, the impact of days within the statistically relevant time period on the relevance decreases more and more as the days are further from the date of exposure. For example, the statistically relevant time period of the directional information may be defined as 4 days (e.g., the date of exposure and 3 days thereafter). It may be determined that the relevance of the exposed directional information has decreased by 25% every day since the date of exposure. As a result, a multiplier of 1.0 may be applied to the date of exposure, a multiplier of 0.75 may be applied 1 day after the date of exposure, a multiplier of 0.50 may be applied 2 days after the date of exposure, and a multiplier of 0.25 may be applied 3 days after the date of exposure. In at least one aspect, the relevance multiplier may be applied to the feature value and / or the group value.

[0034] In operation 312, a model may be trained using the merged data. In one aspect, a predictive model such as predictive model 206 may be identified or generated. Alternatively, multiple predictive models may be identified or generated. For example, a first predictive model may be trained primarily (or exclusively) using information of exposed users, and a second predictive model may be trained primarily (or exclusively) using information of unexposed users. The predictive model may be a binary bias-corrected logistic regression model trained using the merged data and / or group data (e.g., grouped features and values, group values ​​and / or names, etc.) to determine whether or the probability that one or more users identified in the merged data visited a place / venue on a particular day. In an example, the use of bias-corrected logistic regression techniques allows the model to account for unfair sampling bias in the data used to train the model. That is, a significant number of rare positive outcome examples (e.g., venue / location visits) may be included in the training dataset while ensuring that the model's analysis is based on the actual base rate of positive and negative visit outcomes. In certain embodiments, the particular bias-corrected logistic regression technique employed may be described by introducing the notation s0 to represent the sampling rate applied to negative training instances (non-visits) and s1 to represent the sampling rate of positive training instances (visits). In such embodiments, the actual goal is to make s1 very large (often exactly equal to 1, meaning no downsampling) to preserve discriminatory information from rare visit data, while adjusting s0 low (e.g., less than 0.01). This ensures that the downsampling of negative training data is controlled to maintain an overall training data size set that meets size constraints related to computer memory limitations, processing time, or other operational constraints applied to model fitting.

[0035] In one aspect, the predictive model may be subject to a particular confidence interval. For example, a probability distribution over all possible lift values ​​may be generated, since lift calculation may not take statistical significance into account. The probability distribution may incorporate a priori knowledge of the lift distribution. In some aspects, a statistical model or algorithm, such as a Markov Chain Monte Carlo (MCMC) algorithm, may be used to sample data from the probability distribution. MCMC as used herein may refer to a random walk-based algorithm that moves through data points in a manner that depends on the probability distribution. Using the sampled data, various values ​​(e.g., mean, median, percentile, standard deviation, variance, etc.) may be calculated as an expression of the distribution order statistics of lift. For example, the median of the probability distribution may be identified, and a confidence interval bounded by, for example, the 5th and 95th percentiles may be established.

[0036] FIG. 4 illustrates an example method 400 for determining user visitation lift as described herein. The example method 400 begins at operation 402, where information for users exposed to the directional information is identified. In one aspect, a data collection component, such as the data collection engine 202, may receive visitation information for one or more users exposed to the directional information (e.g., exposed users). In some aspects, the visitation information may further include information for one or more users not exposed to the directional information (e.g., unexposed users). The visitation information may be received from one or more computing devices, such as the computing device 102, or one or more data sources, such as the data store 208. In at least one particular example, the visitation information may be collected from a context awareness engine that records users' visitation patterns to venues and locations. The visitation information may include, for example, user and / or device identification data, user demographic data, user visit and / or stop data, date / time data, user behavior data, etc. In one aspect, the data collection component may also receive impression information associated with the user from one or more data sources. The impression information may include, for example, directional information identification data, directional information exposure date / time, user and / or device identification data, and the like.

[0037] In some aspects, the received visit information and / or impression information may correspond to a set of users having certain characteristics or attributes. The characteristics of the set of users may be the same (or substantially similar) to the characteristics of the set of training data used to train the predictive model described in the method 300 of FIG. 3. For example, the predictive model may be trained using five characteristics (e.g., age, gender, metropolitan area, recency of visit, and language) of the users in the set of training data. As a result, for each user in the training data, one or more users having characteristics matching (or similar) to the user in the training data may be identified, and visit information for the set of identified users may be received / collected. In at least one aspect, the received visit information and / or impression information may be merged. Merging the information may comprise identifying various characteristics of the information and grouping the information into one or more groups. As described in the method 300 of FIG. 3, merging the information may also include generating values ​​for the characteristics and / or groups.

[0038] In operation 404, an attribution window may be identified. In one aspect, an attribution window of the directed information exposed to the exposed user may be identified. The attribution window may comprise the exposure data of the directed information and the number of days from the exposure date. In one example, the attribution window may be preselected by a user associated with the administration or management of the directed information. In another example, the attribution window may be predefined by a data collection component or a component of a visit prediction system. In yet another example, the attribution window may be dynamically determined based on the received visit information and / or impression information. For example, one or more ML techniques may be used to define a period of time during which the impact of the directed information remains statistically relevant after the user is exposed to the directed information. The ML techniques may assign a value to each day of the attribution window to represent a decrease in the impact of the directed information on the relevance for additional days from the exposure date of the directed information.

[0039] In operation 406, the received information may be provided as input to a predictive model. In one aspect, the received visit information, impression information, and / or corresponding feature and group data may be provided as input to a predictive model, such as predictive model 206. The predictive model may be, for example, a binary logistic regression model trained to determine whether or not, or the probability that, a user identified in the received information visited a place / venue on a particular date. For example, the information input to the predictive model may be organized into groups corresponding to users and / or dates. The feature data of each group may be provided to the predictive model. As a result, the predictive model may output a probability that a particular user visited a target venue or location on a particular date. In one aspect, the probabilities output by the predictive model may be summed to calculate a value indicative of an expected total visit rate for the location or venue. The expected total visit rate may be based on the assumption that the users represented in the information input to the predictive model had not been exposed to the directional information.

[0040] At operation 408, an actual visit rate for the location or venue may be determined. In an aspect, a total number of actual visits occurring by users during the attribution window may be identified. In an example, the total number of actual visits may correspond to a number of users exposed to the directed information, a number of users not exposed to the directed information, or some combination thereof. Identifying the total number of actual visits may comprise querying one or more services and / or remote data sources. Alternatively, identifying the total number of actual visits may comprise receiving input manually entered by a user using an interface.

[0041] In operation 410, a visit rate lift may be calculated. In one aspect, to calculate the visit rate lift of a directed information (e.g., the percentage increase in visit rate attributing the directed information), the total number of actual visits (e.g., the actual total visit rate) may be evaluated against the expected total visit rate. In one particular example, the visit rate lift may be calculated using the following formula:

[0042]

number

[0043] In the above formula, d is a single eligible day (representing both the user and the date before which the user was recently exposed to the directed information), D is the set of all D days that are eligible in the analysis, visited?(d) is whether the user encoded in d visited the target chain on that day, probVisited?(d) is the probability that an unexposed user will visit on date d, and visits actual is the total number of visits actually made on the eligibility date, and visits estimated is the estimated total number of visits made by non-public users on the eligibility dates.

[0044] In an optional operation 412, one or more actions may be performed in response to the calculation of the visit lift rate. In one aspect, one or more actions or events may be performed in response to the calculation of the visit lift rate. The actions / events may include generating a report, providing information to a predictive model, comparing the results of two or more predictive models, calculating one or more confidence intervals of the calculated visit lift rate, and adjusting the statistical significance of various features and / or feature values. As one particular example, a report measuring the effectiveness of the directed information may be generated and displayed to one or more users. The report may include the various features analyzed, the estimated causal impact of the features on the visit behavior, and / or the attribution window over which the visit prediction analysis was performed.

[0045] FIG. 5 illustrates an exemplary suitable operating environment for the venue detection system described in FIG. 1. In its most basic configuration, the operating environment 500 typically includes at least one processing unit 502 and memory 504. Depending on the exact configuration and type of computing device, the memory 504 (which stores instructions for carrying out embodiments of the visit prediction disclosed herein) may be volatile (such as RAM), non-volatile (such as ROM, flash memory, etc.), or some combination of the two. This most basic configuration is illustrated in FIG. 5 by dashed line 506. Additionally, the environment 500 may also include storage devices (removable 508 and / or non-removable 510), including, but not limited to, magnetic or optical disks or tapes. Similarly, the environment 500 may also have input devices 514, such as a keyboard, mouse, pen, voice input, etc., and / or output devices 516, such as a display, speakers, printer, etc. Included in the environment may also be one or more communication connections 512, such as a LAN, WAN, point-to-point, etc. In embodiments, a connection may be operable to facilitate point-to-point communication, connection-oriented communication, connectionless communication, and the like.

[0046] The operating environment 500 typically includes at least some form of computer-readable media. Computer-readable media may be any available media that can be accessed by the processing unit 502 or other devices that comprise the operating environment. By way of example, and not limitation, computer-readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVDs) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store the desired information. Computer storage media does not include communication media.

[0047] Communication media embodied computer readable instructions, data structures, program modules or other data in a modulated data signal, such as a carrier wave or other transport mechanism, includes any information delivery media. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, RF, infrared, microwave and other wireless media. Combinations of any of the above should also be included within the scope of computer readable media.

[0048] The operating environment 500 may be a single computer operating in a networked environment using logical connections to one or more remote computers. The remote computers may be personal computers, servers, routers, network PCs, peer devices, or other common network nodes, and typically include many or all of the above elements, as well as other elements not specifically mentioned. The logical connections may include any manner supported by available communications media. Such networked environments are commonplace in offices, enterprise-wide computer networks, intranets, and the Internet.

[0049] The embodiments described herein may be employed using software, hardware, or a combination of software and hardware to implement and execute the systems and methods disclosed herein. Although specific devices have been cited throughout this disclosure as performing specific functions, those skilled in the art will understand that these devices are provided for illustrative purposes and that other devices may be employed to perform the functions disclosed herein without departing from the scope of the present disclosure.

[0050] This disclosure describes several embodiments of the technology with reference to the accompanying drawings, which illustrate only some of the possible embodiments. However, other aspects may be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of possible embodiments to those skilled in the art.

[0051] Although specific embodiments are described herein, the scope of the technology is not limited to those specific embodiments. Those skilled in the art will recognize other embodiments or improvements that are within the scope and spirit of the technology. Thus, specific structures, acts, or media are disclosed only as exemplary embodiments. The scope of the technology is defined by the following claims and the equivalents therein. [Explanation of symbols]

[0052] 100 Systems 102 Computing Devices 104 Distributed Network 106 Visitor Prediction System 108 Storage 200 Input Processing System 202 Data Collection Engine 204 Processing Engine 206 Predictive Model 208 Datastore 300 ways 400 ways 500 Operating environment 502 Processing Unit 504 Memory 506 Dashed Line 508 Removable Storage Devices 510 Non-removable storage device 512 Communication Connection 514 Input Devices 516 Output Devices

Claims

1. one or more processors; a memory coupled to at least one of the one or more processors; wherein the memory, when executed by the at least one processor, receiving visit information associated with one or more users; receiving impression information related to directional information, the impression information being associated with at least a portion of the one or more users; merging the visit information and the impression information to create merged data, the merged data comprising a set of features; grouping said set of features into a set of groups; assigning one or more feature values ​​to said set of features; assigning one or more group values ​​to said set of groups; training a machine learning model using the merged data; A system comprising computer-executable instructions for performing a method comprising:

2. The system of claim 1 , wherein the visit information comprises at least two of user identification data, demographic data, or user visit behavior data.

3. The system of claim 1 , wherein the impression information comprises at least two of directional information identification data, directional information exposure data, or user identification data.

4. The system of claim 1 , wherein creating the merged data comprises matching a portion of the visit information to a portion of the impression information.

5. The system of claim 1 , wherein grouping the set of features comprises organizing the set of groups according to at least one of a user or a day.

6. The system of claim 5 , wherein the set of groups is assigned a name according to at least one of the user or the day.

7. The system of claim 1 , wherein assigning one or more feature values ​​comprises calculating a value representing a causal influence of the impression information feature on a user's visiting behavior.

8. The system of claim 1 , wherein assigning one or more group values ​​comprises determining a visit indication value indicating whether a user visited the location on a specified date.

9. The system of claim 1 , wherein assigning one or more group values ​​comprises determining an exposure indication value indicating whether a user has been exposed to the directional information.

10. 2. The system of claim 1, wherein the machine learning model is a binary logistic regression model used to determine a probability that the one or more users identified in the merged data visited a location on a specified date.

11. one or more processors; a memory coupled to at least one of the one or more processors; wherein the memory, when executed by the at least one processor, receiving visit information associated with one or more users, the one or more users being exposed to directional information; receiving impression information related to the directional information, the impression information being associated with at least a portion of the one or more users; identifying an attribution window associated with the directional information; providing the visit information and the impression information within the attribution window to a machine learning model to calculate an expected visit rate for the one or more users; determining an actual visit rate of the one or more users; evaluating the actual visit rate against the expected visit rate to calculate a visit lift rate; A system comprising computer-executable instructions for performing a method comprising:

12. The system of claim 11 , wherein the visit information is collected from a context awareness engine that records a user's visiting patterns to locations.

13. 12. The system of claim 11, wherein the attribution window defines a date of exposure to the oriented information and a number of days from the date of exposure.

14. The system of claim 11 , wherein the machine learning model is a binary logistic regression model.

15. The system of claim 11 , wherein the expected visitation rate represents a probability that the one or more users visited one or more locations on one or more days.

16. The system of claim 11 , wherein the actual visit rate represents a number of visits actually generated by a user during the attribution window.

17. The system of claim 11 , wherein the visit lift rate represents an increase in visit rate attributing the directed information.

18. The system of claim 11 , wherein calculating the visit lift rate comprises dividing the actual visit rate by the expected visit rate.

19. The method further comprising:

12. The system of claim 11, further comprising: performing one or more actions in response to calculating the visit lift rate, the one or more actions including automatically generating a report.

20. receiving visit information associated with one or more users, the one or more users being exposed to directional information; receiving impression information related to the directional information, the impression information being associated with at least a portion of the one or more users; identifying an attribution window associated with the directional information; providing the visit information and the impression information within the attribution window to a machine learning model to calculate an expected visit rate for the one or more users; determining an actual visit rate of a user during the attribution window; calculating a visit lift rate using the predicted visit rate and the actual visit rate, the visit lift rate representing an increase in visit rate attributable to exposure to the directed information; A method comprising:

Images