System and method for determining a customer lifetime value of a user of a client application

Abstract

A method, system, and apparatus including a computer program encoded on a computer storage medium for determining a customer lifetime value of a user of a client application. The method can include receiving first user data characterizing an interaction between a first user and a client application executing on a client device of the first user, first time interval data characterizing a first time interval associated with the first user, and second time interval data characterizing a second time interval associated with the first user, wherein the first time interval characterizes a length of time shorter than a length of time characterized by the second time interval.

Description

Background Art

[0001] (Cross - reference to related applications) This application claims the benefit of U.S. Provisional Application No. 63 / 347,329, filed May 31, 2022, the entire contents of which are incorporated herein by reference.

[0002] Customer Lifetime Value (LTV) is a prediction of the revenue resulting from an ongoing relationship between a user and a product such as a mobile application. By providing a continuous estimate of the amount a particular user is likely to spend on that product, LTV can help ensure that a company pursues its most effective users. If a company can accurately predict a user's LTV, LTV can provide the company with an excellent basis for making decisions, which can help the company maximize the effectiveness of its efforts directed at that user.

Summary of the Invention

Means for Solving the Problems

[0003] The present invention is directed to a system and method for determining the customer lifetime value (LTV) of users of a client application. According to the present invention, short-term data can be used to predict the long-term LTV of users of a client application. In some implementations of the present invention, the prediction can be performed in a transformed space. The resulting prediction can be inverse-transformed from the transformed space to generate a long-term LTV prediction for the user. In one embodiment, based on the predicted long-term LTV, such short-term data can be used to make an estimate of the amount or potential revenue that can be generated by the user for the client application over their lifetime. Thus, an enhanced engagement campaign targeted at the user may be justified if the estimated potential revenue generated by the user is above a certain threshold or within a certain threshold, and additional engagement campaigns can be presented to the user in or associated with the client application.

[0004] A system and method for determining the customer lifetime value of users of a client application are provided. Related apparatus, techniques, and articles are also described.

[0005] In one aspect, first user data characterizing an interaction between a first user and a client application running on the first user's client device, first time interval data characterizing a first time interval associated with the first user, and second time interval data characterizing a second time interval associated with the first user can be received by at least one data processor. The first time interval can characterize a length of time shorter than a length of time characterized by the second time interval. The first time interval customer lifetime value (LTV) of the first user can be determined by at least one data processor based on the received first user data and the received first time interval data. Historical long-term LTV data for a plurality of users of the client application can be received by at least one data processor, and each of the plurality of users can have a respective first time interval LTV. The received historical long-term LTV data can be transformed by at least one data processor into a transformation space. A second time interval LTV for each of the plurality of users can be determined by at least one data processor using the transformed historical long-term LTV data in the transformation space and the received second time interval data, and a distribution of the second time interval LTVs can be generated. A prediction of the second time interval LTV for the first user can be generated by at least one data processor in the transformation space by matching the first time interval LTV of the first user with each respective first time interval LTV of one of the plurality of users, based on the distribution. The prediction of the second time interval LTV for the first user can be inverse-transformed by at least one data processor from the transformation space. The display of information to the first user within the client application can be modified by at least one data processor when the prediction of the second time interval LTV for the first user meets a pre-determined LTV threshold.

[0006] One or more of the following features can be included within any executable combination. For example, the prediction of the second time interval LTV can be generated using a random forest model. For example, the random forest model can be trained using the received historical long-term LTV data. For example, the received historical long-term LTV data can be transformed into a transformed space using the Lambert W function. For example, multiple predictions of the second time interval LTV can be determined for a first user by at least one data processor. For example, graph data characterizing multiple predictions of the second time interval LTV for a first user can be generated by at least one data processor. For example, a random forest regression model can be generated by at least one data processor in a transformed space based on the transformed historical long-term LTV data. For example, the distribution of the predictions of the second time interval LTV for a first user can be generated using the random forest regression model generated in the transformed space by at least one data processor. For example, the generated distribution of the predictions of the second time interval LTV for a first user can be sampled by at least one data processor, and a second distribution of the predictions of the second time interval LTV for a first user can be generated by at least one data processor. For example, a confidence interval characterizing the generated distribution of the predictions of the second time interval LTV for a first user can be determined by at least one data processor based on the second distribution of the predictions of the second time interval LTV for a first user.

[0007] In another aspect, a system is provided, the system can include at least one data processor and a memory storing instructions, and when the instructions are executed by the at least one data processor, cause the at least one data processor to perform the operations described herein.The operation is characterized by receiving, by at least one data processor, first user data characterizing an interaction between a first user and a client application running on the client device of the first user, first time interval data characterizing a first time interval associated with the first user, and second time interval data characterizing a second time interval associated with the first user, wherein the first time interval characterizes a length of time shorter than the length of time characterized by the second time interval, and determining, by at least one data processor, a first time interval customer lifetime value (LTV) of the first user based on the received first user data and the received first time interval data, receiving, by at least one data processor, historical long-term LTV data for a plurality of users of the client application, wherein each of the plurality of users has a respective first time interval LTV, transforming, by at least one data processor, the received historical long-term LTV data into a transformed space, using, by at least one data processor, the transformed historical long-term LTV data in the transformed space and the received second time interval data to determine a second time interval LTV for each of the plurality of users and generate a distribution of the second time interval LTVs, generating, by at least one data processor, a prediction of the second time interval LTV for the first user based on the distribution by matching the first time interval LTV of the first user with the respective first time interval LTVs of one of the plurality of users in the transformed space, transforming back, by at least one data processor, the prediction of the second time interval LTV for the first user from the transformed space, and modifying, by at least one data processor, the display of information to the first user within the client application when the prediction of the second time interval LTV for the first user meets a pre-determined LTV threshold.

[0008] One or more of the following features can be included in any executable combination. For example, the prediction of the second time interval LTV can be generated using a random forest model. For example, the operation can further include training a random forest model using the received historical long-term LTV data. For example, the received historical long-term LTV data can be transformed into a transformation space using Lambert's W function. For example, the operation can further include determining, by at least one data processor, a plurality of predictions of the second time interval LTV for a first user. For example, the operation can further include generating, by at least one data processor, graph data characterizing the plurality of predictions of the second time interval LTV for a first user. For example, the operation can further include generating, by at least one data processor, a random forest regression model based on the transformed historical long-term LTV data in the transformation space. For example, the operation can further include generating, by at least one data processor, a distribution of predictions of the second time interval LTV for a first user using the random forest regression model generated in the transformation space. For example, the operation can further include sampling, by at least one data processor, the generated distribution of predictions of the second time interval LTV for a first user and generating, by at least one data processor, a second distribution of predictions of the second time interval LTV for a first user.

[0009] Non-transitory computer program products (i.e., physically embodied computer program products) are also described, and a non-transitory computer program product stores instructions that, when executed by one or more data processors of one or more computing systems, cause at least one data processor to perform the operations herein. Similarly, computer systems are also described, and a computer system can include one or more data processors and a memory coupled to the one or more data processors. The memory can store instructions temporarily or permanently, and the instructions cause at least one processor to perform one or more of the operations described herein. Additionally, a method can be implemented by one or more data processors, and the one or more data processors can be within a single computing system or distributed among two or more computing systems. Such computing systems can be connected and can exchange data and / or commands or other instructions, etc., via one or more connections including connections via a network (e.g., the Internet, a wireless wide area network, a local area network, a wide area network, a wired network, etc.), connections via a direct connection between one or more of the plurality of computing systems, etc.

Brief Description of the Drawings

[0010] The embodiments described above will be more fully understood from the following detailed description considered in conjunction with the accompanying drawings. The drawings are not intended to be drawn to exact scale. For purposes of clarity, not all components may be labeled in all of the drawings.

[0011]

Figure 1

[0012]

Figure 2

[0013]

Figure 3

[0014]

Figure 4

[0015]

Figure 5

[0016]

Figure 6

[0017]

Figure 7

[0018] Certain exemplary embodiments will now be described in order to provide a thorough understanding of the structure, function, manufacture, and use principles of the devices and methods disclosed herein. One or more examples of these embodiments are illustrated in the accompanying drawings. Those skilled in the art will specifically understand that the devices and methods described herein and illustrated in the accompanying drawings are non-limiting exemplary embodiments, and that the scope of the present invention is defined only by the claims. Features illustrated or described in connection with one exemplary embodiment may be combined with the features of other embodiments. Such modifications and variations are intended to be included within the scope of the present invention. Further, in the present disclosure, components having the same name in the embodiments generally have similar features, and thus, within a particular embodiment, each feature of each component having the same name is not necessarily fully detailed therein.

[0019] The present invention is directed to a system and method for determining the customer lifetime value (LTV) of users of a client application. According to the present invention, short-term data can be used to predict the long-term LTV of users of a client application. In some implementations of the present invention, the prediction can be performed in a transformed space. The resulting prediction can be inverse-transformed from the transformed space to generate a long-term LTV prediction for the user. In one embodiment, based on the predicted long-term LTV, the present invention can use such short-term data to estimate the amount or potential revenue that can be generated by the user for the client application over their lifetime. Thus, if the estimated potential revenue generated by the user is above a certain threshold, or within a certain threshold, an enhanced engagement campaign targeted at the user would be justified, and additional engagement campaigns can be presented to the user in or associated with the client application. Embodiments of the present invention can support predictions regardless of the definition of LTV selected, can predict at the user level, can vary the prediction period, and can be updated periodically, such as on a weekly basis or other appropriate period.

[0020] Merely for purposes of discussion, and not by way of limitation, the present disclosure may refer to a digital game as an exemplary client application for illustrating various aspects of the present invention. However, the present invention can be used in and with any suitable type of client application (e.g., a mobile application, a desktop application, or any other suitable type of client application) in which a user may expend currency (e.g., physical, electronic, virtual, digital, etc.) while engaging with and interacting with the client application. For example, the LTV of a player in a digital game can be defined as the estimated revenue that the player will generate in the digital game over their lifetime. However, the present invention can be used in and with any suitable type of client application in which the calculation of the long-term LTV of a user is desired. Merely for purposes of discussion, and not by way of limitation, the present disclosure will refer to "revenue" as a basis for LTV calculations for purposes of illustrating various aspects of the present invention, but the revenue can be replaced with any suitable quantity. For example, merely for purposes of discussion, the LTV can be the sum of the discounted daily revenues, using the first day of installation of the client application, or the day of the first deposit in the client application, as day 1 (D1).

[0021] FIG. 1 is a block diagram illustrating an exemplary system 100 for predicting the long - term LTV for users of a client application by using short - term LTV. The server system 114 can provide functionality for receiving and collecting data (e.g., revenue data, etc.) associated with the characteristics of users in a client application such as players in a digital game. The server system 114 can include, for example, software components and databases that can be deployed in one or more data centers 112 in one or more geographical locations. The software components of the server system 112 can include a customer lifetime value analysis module 116 and a client device information display module 118. The software components can include sub - components that can be executed on the same or different individual data processing devices. The databases of the server system 114 can include, for example, a user data database 120 and a client application data database 122, although other databases are also possible. The databases can reside within one or more physical storage systems or can be cloud - based. The software components and databases will be further described below.

[0022] As shown in FIG. 1, the Customer Lifetime Value Analysis Module 116 and the Client Device Information Display Module 118 can communicate with each other and with the User Data Database 120 and the Client Application Data Database 122. The User Data Database 120 can include, for example, one or more users of the client application and any appropriate information related to the interactions between those users and the client application: for example, user LTV data (predicted, actual, short-term, long-term, etc.), user characteristics, user interaction history (e.g., in the context of a digital game, the digital games played, the number of games won in each digital game, the number of games lost in each digital game, the number of games played for each digital game, the score in each digital game, the time played for each digital game, etc.), user identification information (e.g., user name), the history of user connections to the system 100, user purchases, user achievements, user tasks, user interactions with other users (e.g., chat), user deposits / withdrawals, user virtual item acquisitions or usage amounts, other conditions in the client application, etc. The Client Application Data Database 122 can include, for example, information related to the client applications implemented using the system 100. The Client Application Data Database 122 can include information related to each client application, such as the virtual environment, images, videos, text, and / or audio data for each client application, event data corresponding to past, current, or future events, client application state data defining the current state of each client application, etc.

[0023] A software application (e.g., a digital game or other web-based or suitable client application, etc.) can be provided as an end-user client application to enable a user to interact with the server system 114. The software application can be related to and / or provide a wide variety of functions and information including, for example, entertainment (e.g., games, music, videos, etc.), business (e.g., word processing, accounting, spreadsheets, etc.), news, weather, finance, sports, etc. In some implementations of the present invention, the software application can provide a digital game. The digital game can be, for example, a sports game, an adventure game, a card game on virtual play, a virtual board game, a puzzle game, a racing game, or any other suitable type of digital game, or can include them. In one embodiment, the digital game can be an asynchronous competitive skill-based game, in which players can compete with each other in the digital game, but do not need to play the digital game simultaneously. In an alternative embodiment, the digital game can be a synchronous competitive skill-based game, in which players can play the digital game simultaneously and can compete with each other in real time in the digital game. Other suitable software applications are also possible.

[0024] A software application or its components can be accessed by users of client devices such as client device A102, client device B104, client device C106, ···, client device N108, etc. through network 110 (e.g., the Internet), where N can be any appropriate natural number. Each of the client devices can be any appropriate type of electronic device that can execute the software application and communicate with server system 114 through network 110, such as a smartphone, tablet computer, laptop computer, desktop or personal computer, etc. Other client devices are also possible (e.g., portable or desktop game consoles, smart TVs, smartwatches, and other similar computing devices). In an alternative embodiment, user data database 120, client application data database 122, or any portion thereof can be stored on one or more client devices. Additionally, or alternatively, software components for system 100 (e.g., customer lifetime value analysis module 116 and / or client device information display module 118) or any portion thereof can reside on one or more client devices or be used to perform operations thereon.

[0025] In some implementations of the present invention, the LTV of a user can represent the amount of currency (e.g., physical, electronic, virtual, digital, etc.) that the user has spent in the client application. For example, in the context of a digital game, the LTV of a user can represent the amount obtained by subtracting any prizes or awards that the user may have received from the money the user has spent in the digital game. In certain embodiments, each user can have their own LTV, and the value of the user's LTV can change with each transaction the user makes in the client application. In certain embodiments, a user who has not made a transaction, or otherwise deposited currency into the client application, will not have an LTV. In some implementations of the present invention, the calculation of the user's LTV can start after the user makes their first deposit into the client application, from the first payment transaction in the client application (e.g., in the context of a digital game, the first payment for participation in a competition). Such an event can be referred to as a "post-first-deposit transaction". Alternatively, the calculation of the user's LTV can start from the first installation of the client application. In certain embodiments, the length of the LTV to be calculated can be defined with reference to an appropriate period such as several hours, days, weeks, months, etc. Merely for illustrative purposes, and not by way of limitation, the period can be several days, and the number of days can be counted starting from the user's post-first-deposit transaction or first installation. For example, D1 LTV can refer to the sum of all relevant transactions within 1 day (24 hours) of the user's post-first-deposit transaction or first installation, D2 LTV can refer to the sum of all relevant transactions within the first 2 days (48 hours) of the user's post-first-deposit transaction or first installation, D7 LTV can refer to the sum of all relevant transactions within the first 7 days of the user's post-first-deposit transaction or first installation, and so on.Transaction components that can be used to calculate a user's LTV (i.e., transactions that can be considered when calculating a user's LTV) can depend, for example, on the client application and the types of transactions supported by the client application. For example, in the context of a digital game on a competitive skill-based digital game platform, transactions such as entry fees paid from matches and tournaments can increase a user's LTV, while transactions such as cash prizes, refunds, physical prizes (e.g., from rewards or gift stores) awarded or exchanged for winning a match or tournament can decrease a user's LTV. Other transaction components of a user's LTV are also possible.

[0026] In some implementations of the present invention, the Customer Lifetime Value Analysis Module 116 can use an XGBoost model or the like to predict the long-term LTV of a user based on the short-term LTV (e.g., starting from the first transaction after the user's first deposit or the first installation), but other suitable machine learning / artificial intelligence models can also be used. FIG. 2 is a flowchart illustrating an exemplary method 200 for predicting the long-term LTV for users of a client application by using the short-term LTV according to an embodiment of the present disclosure. In some implementations of the present invention, method 200 can be implemented by, for example, the Customer Lifetime Value Analysis Module 116 using an XGBoost model or the like. In one embodiment, the Customer Lifetime Value Analysis Module 116 can monitor a plurality of users of a client application that runs on each client device of the user. At block 205, the Customer Lifetime Value Analysis Module 116 can select (or receive the selection of) a short-term date (e.g., D7, D30, or any other suitable time interval) that should be a prediction criterion for the first user, and a long-term date (e.g., D90, D180, or any other suitable time interval) that should be the prediction target. Merely for illustrative purposes, and not by way of limitation, the Customer Lifetime Value Analysis Module 116 can select D180 for the long-term date and D7 for the short-term date for the first user. Thus, in this illustration, the Customer Lifetime Value Analysis Module 116 can predict the LTV on the 180th day after the first installation of the client application by the first user or the first deposit in the application (including the 7th day) based on the data up to the 7th day after the first installation of the client application by the first user or the first deposit in the application, but any other suitable date or period or interval can also be selected. At block 210, the Customer Lifetime Value Analysis Module 116 can determine the cohort or group that the first user belongs to (or is in), and the short-term LTV of the first user, such as D7 LTV in this illustration.For example, the customer lifetime value analysis model 116 can use data collected or received for a user (e.g., deposit and / or expenditure data read from the user data database 120) to determine the user's short-term LTV (e.g., in this illustration, the D7 LTV). For illustrative purposes, but not limited to, in the context of digital games, users can be grouped into cohorts by: digital game, platform, network, country, advertising network, time to first deposit, paid withdrawals and maintenance fees, paid participation fees, first digital game played, number of digital games played, and / or any combination thereof. Other cohorts or groups are also possible and will depend, for example, on the type of client application, the type of user of the client application, the way the user interacts with or engages with the client application, etc. Each cohort or group can consist of any suitable number of users. In certain embodiments, the cohorts can be organized or optimized to reduce cohort-level errors in the predicted long-term LTV. In block 215, the customer lifetime value analysis module 116 can determine the percentile of the short-term LTV (e.g., D7 LTV) of a first user and identify other users within the cohort having the same short-term LTV (e.g., D7 LTV) percentile as the first user, and they can be all users with the same percentile or a suitable-sized portion of such users within the cohort. In an alternative embodiment, instead of using an exact percentile, the customer lifetime value analysis module 116 can use, for example, the decile within which the user falls. In certain embodiments, the width of the selected percentile range can be a hyperparameter and can be selected, for example, by appropriate cross-validation (CV) / tuning. In a further alternative embodiment, the customer lifetime value analysis module 116 can directly predict the long-term LTV as a continuous value.

[0027] In block 220, the customer lifetime value analysis module 116 can determine the long-term LTV (e.g., D180 LTV) for a set of users having the same or similar short-term LTV (e.g., D7 LTV) percentiles (calculated in block 215) as the first user, and generate a distribution of possible long-term values for the first user. In some embodiments, the customer lifetime value analysis module 116 can collect, read, or receive historical long-term LTV data regarding a set of users for which long-term LTV data is available. Data from the set of users can be used for the purpose of training a machine learning model (e.g., XGBoost, etc.). In some implementations of the present invention, historical long-term LTV data for additional or alternative users other than those within a defined cohort or group of users can be used to train the machine learning model. In some embodiments, the customer lifetime value analysis module 116 can read long-term LTV data from the user data database 120 regarding the set of users. The customer lifetime value analysis module 116 can delete any users from the set of users that should not be considered (e.g., the user data is fraudulent, damaged, invalid, or unusable) from the collected, read, or received long-term LTV data. In some implementations of the present invention, the customer lifetime value analysis module 116 can perform hyperparameter optimization on the long-term LTV data for the set of users. In hyperparameter optimization, multiple models can be generated, and each of the multiple models involves random values regarding the number of trees, tree depth, regularization strength, etc. Hyperparameters that result in the highest validation score as judged by a primary metric (e.g., total error within each cohort) can be used to retrain against the complete dataset (e.g., training set + validation set). In some implementations of the present invention, the primary metric can be calculated post hoc rather than being directly optimized by the model. The model can be trained using, for example, mean squared error, etc.

[0028] In one embodiment, the determination made by the customer lifetime value analysis module 116 can generate a distribution (e.g., a histogram) of the possible long-term LTV (e.g., D180 LTV) for a first user for which a prediction is to be made. Thus, the customer lifetime value analysis module 116 can use the collected, retrieved, or received historical long-term LTV data regarding a group of users having the same or similar short-term LTV as the first user to generate a distribution of the possible long-term LTV. In other words, the short-term LTV of the first user can be used to match the first user to a distribution of long-term LTVs having the same or similar short-term LTV as the first user. According to an alternative embodiment, instead of using exact percentiles, the customer lifetime value analysis module 116 can use, for example, the decile to which the first user belongs. The width of the selected percentile range can be a hyperparameter and can be selected, for example, by appropriate CV / tuning. In an alternative embodiment, the customer lifetime value analysis module 116 can directly predict the long-term LTV from the short-term LTV using, for example, an appropriate regression model or the like. In block 225, the customer lifetime value analysis module 116 can determine the most appropriate percentile to use for the prediction for the first user by using the long-term LTV (e.g., D180 LTV) distribution (e.g., a histogram) determined by the customer lifetime value analysis module 116 in block 220. For illustrative purposes and without limitation, the customer lifetime value analysis module 116 can select the 50th percentile and use the median (or other appropriate statistical calculation) of this distribution to make the prediction. Alternatively, for example, the customer lifetime value analysis module 116 can instead select the 70th percentile and consistently predict higher values. Other percentiles are also possible. According to an embodiment, the percentile to be predicted can also be considered a hyperparameter and can be investigated using appropriate CV / tuning.In block 230, the customer lifetime value analysis module 116 can determine the value of the determined percentile at a long-term date (e.g., D180) (e.g., dollar amount or other monetary value), and can use that value as the predicted long-term LTV (e.g., D180 LTV) of the first user. For example, if the value is in dollars (or other appropriate currency) amount, the predicted long-term LTV of the first user can be the estimated revenue that the first user can generate in the client application over their lifetime. The customer lifetime value analysis module 116 can store the results in the user data database 120 for later retrieval and use.

[0029] In some implementations of the present invention, the Customer Lifetime Value Analysis Module 116 can perform alternative steps to determine a prediction using a group-level historical LTV distribution. The Customer Lifetime Value Analysis Module 116 can select a short-term date (e.g., D7, D30, or any other appropriate time interval) that should be a prediction criterion for a first user, and a long-term date (e.g., D90, D180, or any other appropriate time interval) that should be the subject of the prediction. Merely for illustrative purposes, without limitation, the Customer Lifetime Value Analysis Module 116 can select D180 as the long-term date and D7 as the short-term date for the first user. Thus, in this illustration, the Customer Lifetime Value Analysis Module 116 can predict the LTV at the 180th day after the first installation of the client application by the first user or the first payment in the application (including the 7th day) based on the data up to the 7th day after the first installation of the client application by the first user or the first payment in the application, although any other appropriate date can also be selected. The Customer Lifetime Value Analysis Module 116 can determine the group (e.g., grouping users into cohorts by mobile application, platform, network, etc. as discussed above) that the first user belongs to and the short-term LTV (e.g., D7 LTV) of the first user in the manner discussed above. The Customer Lifetime Value Analysis Module 116 can determine the percentile of the short-term LTV (e.g., D7 LTV) of the first user. In some implementations of the present invention, the LTV of the first user can be the same percentile at the long-term date (e.g., D180). The Customer Lifetime Value Analysis Module 116 can determine the value of the determined percentile (e.g., dollar amount or other monetary value) at the long-term date (e.g., D180) and use that value as the predicted long-term LTV (e.g., D180 LTV) of the first user. For example, if the value is in dollar (or other appropriate currency) amount, the predicted long-term LTV of the first user can be the estimated revenue that the user can generate in the client application over their lifetime.In an alternative embodiment, the customer lifetime value analysis module 116 can directly predict the long-term LTV, for example, using a regression model or the like, based on or using the features collected for the first user on a short-term day.

[0030] In some implementations of the present invention, the historical LTV can be segmented by relevant groups. For illustrative purposes, and not by way of limitation, users of a digital game company can be segmented, for example, by site, network, platform, and the group of games installed first, although any other suitable groups and groupings of users can also be considered. After all users are grouped, then for each group, the LTV curve for each user within that group can be calculated (e.g., by the customer lifetime value analysis module 116). As a result, for each day, each user can have their own LTV on that day, so there can be a distribution of LTV on that day. In one embodiment, the LTV can potentially be non-monotonic. For example, when the information is graphed for a particular group, "day" can be on the x-axis and the LTV up to that day can be on the y-axis. After the LTV distribution for each day is calculated for each group, then training of an appropriate machine learning model can be performed. According to the present invention, two types of data can be used within such a model: time-independent (all features fall into this category) (e.g., platform, advertising network, etc.); and time-dependent (e.g., target variables such as revenue).

[0031] LTV prediction can encounter extreme distributions of LTV data, which can exhibit thick or heavy tails and / or skewness. The embodiments illustrated and discussed with respect to FIG. 2 can use such data to successfully reduce cohort-level errors. However, in some implementations of the present invention, the customer lifetime value analysis module 116 is implemented using a random forest model or the like and can predict a user's long-term LTV based on short-term LTV (e.g., starting from a user's first deposit transaction or first installation), although other suitable machine learning / artificial intelligence models can also be used (e.g., neural networks, etc.). Such embodiments can be used to more easily perform LTV prediction in the presence of wide or extreme distributions of LTV data. Models trained using data exhibiting such extreme distributions can be biased by outliers and can compromise overall prediction accuracy. In some implementations of the present invention, a technique for "Gaussianizing" heavy-tailed distributions using the Lambert W function (also called the omega function or product logarithm) can be used for LTV data with wide or extreme distributions. The Lambert W function can provide an inverse transformation, which can be estimated by the maximum likelihood method. Such an inverse transformation can remove heavy tails from the data and can also provide analytical expressions for the cumulative distribution function and probability density function. Thus, the methodology according to some implementations of the present invention is performed in the transformed space and can more easily handle LTV data with wide or extreme distributions. In one embodiment, the Lambert transformation can be adapted to the training data (e.g., LTV data with wide or extreme distributions), and a suitable model can be trained in the transformed space where the values "behave better", and then the transformation can be reversed to return to the normal space (e.g., the "cash" or "currency" space, i.e., the "heavy-tailed world") for use in predicting the user's long-term LTV.Such techniques can enable embodiments of the present invention to make accurate predictions regarding a wide variety of users within a data set (e.g., from low spenders to high spenders who can result in extreme distributions of LTV data) without truncating values or biasing the model towards one group or the other. The Lambert W function can be used for thick or heavy tailed data, but the choice of transformation will depend on the type of data being analyzed. For example, a log transformation can be used for data without negative values, while a power transformation can be used for skewed data. Other transformations are possible.

[0032] Figure 3 is a flowchart illustrating an exemplary method 300 for predicting the long - term LTV for users of a client application by using short - term LTV according to an embodiment of the present disclosure. In some implementations of the present invention, method 300 can be implemented by the customer lifetime value analysis module 116, for example, using a random forest model or the like. Compared with the XGBoost model, the random forest model can enable the use of an increased number of trees without overfitting. The random forest regression model can average the predictions from each tree and generate a final output. In an embodiment, the customer lifetime value analysis module 116 can treat each tree as an independent estimator of the average LTV and use the independent estimators as an approximate long - term LTV distribution for each user. In some implementations of the present invention, it should be noted that the median of the approximate distribution can be a more appropriate estimate of the user's LTV than the mean value due to, for example, skewed and heavy - tailed data within the data, and the quantiles of the distribution can provide a higher confidence interval for LTV prediction. However, although the median can provide a more appropriate estimate for each individual user, the median may not provide a more appropriate value for calculating the total LTV of the user cohort. In situations where the individual LTV distributions can have a strong right - hand skew, the median may tend to underestimate the mean value. However, in some implementations of the present invention where the cohort LTV should take such skewness into account, even when the median may function more appropriately for an individual, the mean value is used, but it may function more appropriately than the median for the user cohort.

[0033] In one embodiment, the Customer Lifetime Value Analysis Module 116 can monitor multiple users of a client application that runs on each user's client device. At block 305, the Customer Lifetime Value Analysis Module 116 can select (or receive the selection of) a short-term date that should be a prediction criterion for a first user and a long-term date that should be the subject of prediction. Merely for illustrative purposes and without limitation, the Customer Lifetime Value Analysis Module 116 can select D180 as the long-term date and D30 as the short-term date for the first user. Thus, in this illustration, based on data up to and including the 30th day after the first installation of the client application by the first user or the first payment in the application, the Customer Lifetime Value Analysis Module 116 can predict the LTV on the 180th day after the first installation of the client application or the first payment in the application. However, any other appropriate date or period or interval can be selected for the long-term and short-term dates. At block 310, the Customer Lifetime Value Analysis Module 116 can use the selected short-term date and the received or collected data for the user (e.g., payment and / or expenditure data read from the User Data Database 120) to determine the short-term LTV (e.g., D30 LTV) of the first user. At block 315, the Customer Lifetime Value Analysis Module 116 can receive, collect, or read historical long-term LTV data for a set of other users who have the same or similar short-term LTV as the first user and for whom long-term LTV data is available using the selected long-term date. The set of other users can be used for the purpose of training a machine learning model (e.g., a random forest model, etc.), and the set of other users can consist of users other than one user. In one embodiment, the Customer Lifetime Value Analysis Module 116 can receive, collect, or read long-term LTV data from the User Data Database 120 for a set of other users.In one embodiment, the customer lifetime value analysis module 116 can remove any user among other sets of users that should not be considered (e.g., the user data is fraudulent, corrupted, invalid, or unusable) from the collected or retrieved long-term LTV data.

[0034] In block 320, the customer lifetime value analysis module 116 can transform or convert the received historical long-term LTV data for other sets of users into an appropriate transformation space. In one embodiment, the Lambert W function can be used as the transformation so that long-term LTV data (e.g., LTV data with a wide or extreme distribution) can be more easily processed, but other appropriate transformations can be used. In such an embodiment, the root mean square error of the model can be used within the transformed space as this can represent the extent to which the model functions properly across the entire dataset. In some implementations of the present invention, the customer lifetime value analysis module 116 can perform hyperparameter optimization on the long-term LTV data for the sets of users. In hyperparameter optimization, multiple models can be generated, and each of the multiple models has random values with respect to the number of trees, tree depth, regularization strength, etc. The hyperparameters that result in the highest validation score as determined by a primary metric can be used to retrain on the complete dataset (e.g., training set + validation set). However, hyperparameter tuning is not necessarily required in the method 300 illustrated and discussed with respect to FIG. 3 as the random forest model, for example, has far fewer parameters than an XGBoost model and is much less sensitive to tuning, but hyperparameter tuning can be used.

[0035] In block 325, the Customer Lifetime Value Analysis Module 116 can determine the long-term LTV (e.g., D180 LTV) for a set of other users having the same or similar short-term LTV (e.g., D30 LTV) as the first user (calculated for the first user in block 310), and generate a distribution of possible long-term values for the first user. In some embodiments, the determination made by the Customer Lifetime Value Analysis Module 116 is made within a transformed space, and more easily, can generate a distribution (e.g., a histogram) of possible long-term LTV (e.g., D180 LTV) for the first user for which a prediction is to be made. Thus, the Customer Lifetime Value Analysis Module 116 can use the received, collected, or retrieved historical long-term LTV data for a set of other users having the same or similar short-term LTV as the first user to generate a distribution of possible long-term LTV for the first user. In other words, the short-term LTV of the first user can be used to match the first user to the distribution of long-term LTVs of other users having the same or similar short-term LTV as the first user. According to an alternative embodiment, the Customer Lifetime Value Analysis Module 116 can predict the long-term LTV, for example, using an appropriate regression model or the like, based on or using features received or collected from the short-term days for the first user. In block 330, the Customer Lifetime Value Analysis Module 116 can use the long-term LTV distribution to determine the value of the predicted long-term LTV (e.g., D180 LTV) of the first user at a long-term day. For example, the Customer Lifetime Value Analysis Module 116 can employ the mean, median, percentile, or other appropriate measure of the long-term LTV distribution to determine the value of the first user's long-term LTV. In block 335, the Customer Lifetime Value Analysis Module 116 can transform or convert the predicted long-term LTV of the first user so as to return from the transformed space. For example, the Customer Lifetime Value Analysis Module 116 can use the inverse of the transformation used in step 325 (e.g., an inverse function such as the Lambert W function) to transform or convert the prediction back to the non-transformed (i.e., normal) space.For illustrative purposes, but not limited to, with respect to predictions based on amounts, the customer lifetime value analysis module 116 can reverse the transformation and return the prediction to the "cash" space. For example, after the inverse transformation, if the predicted value is in dollars (or other appropriate currency) amount, the predicted long-term LTV of the first user can be the estimated revenue that the first user can generate in the client application over their lifetime. The customer lifetime value analysis module 116 can store the results in the user data database 120 for later retrieval and use.

[0036] Figure 4 is a flowchart illustrating an exemplary method 400 for predicting the long-term LTV for users of a client application by using the short-term LTV according to an embodiment of the present disclosure. In some implementations of the present invention, method 400 can be implemented by the customer lifetime value analysis module 116 using, for example, a random forest model or the like. In an embodiment, the customer lifetime value analysis module 116 can monitor a first user among a plurality of users of a client application running on a client device of the first user. At block 405, the customer lifetime value analysis module 116 can select a short-term date that should be a prediction criterion for the first user and a long-term date that should be the prediction target. Merely for illustrative purposes, but not limited to, the customer lifetime value analysis module 116 can select D180 as the long-term date and D30 as the short-term date for the first user. Thus, in this illustration, based on the data up to and including the 30th day after the first installation of the client application by the first user or the first deposit in the application, the customer lifetime value analysis module 116 can predict the LTV on the 180th day after the first installation of the client application or the first deposit in the application, but any other suitable date or period or interval can also be selected. At block 410, the customer lifetime value analysis module 116 can receive, collect, or read appropriate input features for the first user based on the selected short-term date. For example, the customer lifetime value analysis module 116 can collect or generate a short-term LTV (e.g., D30 LTV) for the first user, the number of interactions with the client application (e.g., in the context of a digital game, the number of games played, etc.), chat messages, or any other appropriate input features depending on the type of the client application, the model being developed and used, and the like.In block 415, the customer lifetime value analysis module 116 can receive, collect, or read historical long-term LTV data for other groups of users for which long-term LTV data is available. The other groups of users can be used for the purpose of training a machine learning model (e.g., a random forest regression model, etc.) and can consist of users other than the first user for which long-term LTV predictions are being made. In certain embodiments, the customer lifetime value analysis module 116 can read long-term LTV data from the user data database 120 for other groups of users. In certain embodiments, the customer lifetime value analysis module 116 can delete any users among other groups of users that should not be considered (e.g., the user data is fraudulent, damaged, invalid, or unusable) from the collected or read long-term LTV data.

[0037] In block 420, the customer lifetime value analysis module 116 can transform or convert the collected historical long-term LTV data into an appropriate transformation space. In one embodiment, the Lambert W function can be used as the transformation so that long-term LTV data (e.g., LTV data with a wide or extreme distribution) can be processed more easily, although other appropriate transformations can be used. In such an embodiment, the root mean square error of the model can be used within the transformed space as this can represent the extent to which the model functions properly across the entire dataset. In block 425, the customer lifetime value analysis module 116 can develop or create or construct a model (e.g., a random forest regression model, etc.) based on the transformed historical long-term LTV data. In block 430, the customer lifetime value analysis module 116 can provide the received or collected input features of the first user to the model and generate the distribution of the predicted long-term LTV for the first user at a selected long-term date (e.g., D180 LTV). In one embodiment, the model can be developed or constructed when the LTV data exhibits a wide or extreme distribution, and the distribution can be generated more easily by the customer lifetime value analysis module 116 in the transformation space. In block 435, the customer lifetime value analysis module 116 can transform or convert the distribution back from the transformation space. For example, the customer lifetime value analysis module 116 can use the inverse of the transformation used in step 420 (e.g., the inverse function of the Lambert W function, etc.) to transform the distribution back to the non-transformed (i.e., normal) space. In block 440, the customer lifetime value analysis module 116 can measure the distribution (within the non-transformed space) and generate the value of the predicted long-term LTV for the first user. For example, the customer lifetime value analysis module 116 can adopt the mean value, median, percentile, or other appropriate measurement of the long-term LTV distribution to determine the value of the long-term LTV for the first user. The customer lifetime value analysis module 116 can store the results in the user database 120 for later retrieval and use.

[0038] Figure 5 is a flowchart illustrating an exemplary method 500 for predicting the long-term LTV for users of a client application by using the short-term LTV according to an embodiment of the present disclosure. In some implementations of the present invention, method 500 can be implemented by the customer lifetime value analysis module 116, for example, using a random forest model or the like. In an embodiment, the customer lifetime value analysis module 116 can monitor a first user among a plurality of users of a client application running on the client device of the first user. At block 505, the customer lifetime value analysis module 116 can receive first user data characterizing the interaction between the first user and the client application running on the client device of the first user, first time interval data characterizing a first time interval associated with the first user, and second time interval data characterizing a second time interval associated with the first user. The first time interval can characterize a length of time shorter than the length of time characterized by the second time interval. At block 510, the customer lifetime value analysis module 116 can determine the first time interval LTV of the first user based on the received first user data and the received first time interval data. At block 515, the customer lifetime value analysis module 116 can receive historical long-term LTV data for a plurality of users of the client application. Each of the plurality of users can have a respective first time interval LTV. At block 520, the customer lifetime value analysis module 116 can transform or convert the received historical long-term LTV data into a transformed space. At block 525, the customer lifetime value analysis module 116 can use the transformed historical long-term LTV data in the transformed space and the received second time interval data to determine the second time interval LTV for each of the plurality of users and generate a distribution of the second time interval LTV.In block 530, the customer lifetime value analysis module 116 can generate a prediction of the second time interval LTV for the first user based on the distribution by matching the first time interval LTV of the first user with the respective first time interval LTVs of one of the plurality of users in the transformation space. In block 535, the customer lifetime value analysis module 116 can transform or transform back the prediction of the second time interval LTV for the first user from the transformation space. In block 540, the customer lifetime value analysis module 116 can modify the display of information to the first user within the client application when the prediction of the second time interval LTV for the first user meets (e.g., is greater than) a pre-determined LTV threshold.

[0039] In all embodiments illustrated and discussed with respect to FIGS. 3, 4, and 5, the customer lifetime value analysis module 116 can predict directly with respect to the long-term LTV (e.g., D180 LTV). In an alternative embodiment, the customer lifetime value analysis module 116 can predict the difference between the long-term LTV (e.g., D180 LTV) and the short-term LTV (e.g., D30 LTV) and then plug back in the short-term LTV (e.g., D30 LTV) in the conclusion to generate the long-term LTV (e.g., D180 LTV). Such an alternative embodiment can be used when many of the users have permanently canceled in the client application by the short-term date (e.g., D30), whereby the difference for such users is zero. As a result, for the random forest model, it may be easier to separate the canceling users and assign them a zero value than to predict that their long-term LTV is the same as their short-term LTV.

[0040] According to embodiments of the present invention, the predicted long-term LTV of a user can be used to determine information that can be displayed within or associated with a client application executing on the user's client device, if applicable. In certain embodiments, when the prediction of the user's long-term LTV is above a pre-determined threshold, or within a certain pre-determined threshold (e.g., a pre-determined dollar amount or estimated revenue amount), the client device information display module 118 illustrated in FIG. 1 can be used to modify the display of information to the user within or associated with the client application. Alternatively, when the prediction of the user's long-term LTV is below a pre-determined threshold, or outside of a certain pre-determined threshold (e.g., a pre-determined dollar amount or estimated revenue amount), the client device information display module 118 illustrated in FIG. 1 can be used to modify the display of information to the user within or associated with the client application. The pre-determined threshold can be any suitable amount or quantity. For example, if the predicted long-term LTV of a user is the estimated revenue that the user can generate in the client application over their lifetime, the pre-determined threshold can be a suitable dollar or revenue amount (e.g., $10, $50, $100, or a similar dollar amount).

[0041] In some implementations of the present invention, the client device information display module 118 illustrated in FIG. 1 can update, customize, modify, or personalize the graphical user interface, features, and / or functionality of a client application in any suitable manner. In certain embodiments, the display of such personalized information can be used to increase engagement and participation in the client application for users with a predicted long-term LTV that meets or exceeds one or more pre-determined thresholds. In alternative embodiments, the display of such information can be used to increase engagement and participation in the client application for users with a predicted long-term LTV that is below one or more pre-determined thresholds (e.g., in an attempt to increase their predicted long-term LTV). For example, the client device information display module 118 can personalize any or all aspects of the graphical display of the client application (e.g., any aspect such as one or more graphical elements of the client application, the "look and feel" of the graphical interface presented by the client application, the information displayed within the client application, the features and / or functionality of the client application, etc.). Merely for illustrative purposes, and not by way of limitation, in the context of a digital game, the client device information display module 118 can personalize the graphical display of the digital game and, for example, display player incentives, special offers (e.g., limited-time offers or LTOs), advertisements, etc. to the player within or in the digital game. In this illustration, different player incentives, special offers, advertisements, etc. can be displayed to the player in the digital game based on the player's predicted long-term LTV.Additionally, or alternatively, the client device information display module 118 can display or present additional and / or alternative prizes, rewards, and / or gifts to the player in an associated gift store for the client application. For example, a player with a predicted long-term LTV above a pre-determined threshold or within a certain pre-determined threshold can be presented with prizes, rewards, gifts, etc. different from those that can be presented to another player in a digital game with a predicted long-term LTV below the pre-determined threshold or outside a pre-determined threshold (or vice versa). Thus, the menu or list of prizes, rewards, and / or gifts in the associated gift store presented to the player can be tailored to the player with a predicted long-term LTV relative to one or more pre-determined thresholds. Additionally, or alternatively, the client device information display module 118 can also personalize, for example, graphical information displayed to the user outside the client application (such as advertisements or offers presented to the user on their client device outside the client application). However, in some embodiments, if the prediction of the user's long-term LTV is below a pre-determined threshold, the information may not be displayed (or hardly displayed) to the user by the client device information display module 118.

[0042] As already discussed, appropriate machine learning / artificial intelligence techniques can be used to dynamically predict a user's long-term LTV from their short-term LTV. For example, one or more machine learning models can be trained based on historical revenue data from all users of a client application (such as that read from user data database 120). One or more machine learning models can then be used to dynamically predict a particular user's long-term LTV using revenue data from the user population, according to the methods described herein. One or more machine learning models can be updated or adapted as the user's revenue data changes or evolves over time. In alternative embodiments, any of the pre-determined values discussed above can be dynamically selected. According to an alternative embodiment, the customer lifetime value analysis module 116 can use appropriate machine learning / artificial intelligence techniques to dynamically select or choose appropriate values for any of the pre-determined values and / or parameters discussed above. For example, one or more machine learning models can be trained based on data from one or both of user data database 120 and client application data database 122. One or more machine learning models can then be used to dynamically select appropriate values for each or any of the aforementioned variables, based on, for example, user characteristics (such as user revenue), client application characteristics (such as the type of client application), and other similar characteristics or data. One or more machine learning models can be updated or adapted as the characteristics, results, and other similar data associated with the user and client application change and evolve over time.

[0043] Accordingly, the embodiments illustrated and discussed above can use short-term data (e.g., D7, D30, or any other appropriate time interval) to predict the long-term LTV for a user (e.g., D180 or any other appropriate time interval). In some implementations of the present invention, based on the predicted long-term LTV, the present invention can use such short-term data to estimate the dollar amount or potential revenue that can be generated by a user in a client application throughout their lifetime. Thus, if the estimated potential revenue generated by a user is above a certain threshold or within a certain threshold, additional marketing campaigns and offers can be presented to the user within or associated with the client application, as the increased marketing expenditure targeting the user would be justified. Note that some implementations of the present invention do not have model parameters for estimation, and in some implementations, up to three or more hyperparameters can be used for alternative ways of making predictions. Accordingly, in some implementations of the present invention, the predictions can be made using the methods discussed and described in this disclosure, using the appropriate CV / adjustments discussed and described in this disclosure. Further, note that the present invention can make predictions in a prediction period that is of a similar length to the training data.

[0044] Since the long-term LTV is predicted at the user level, in some implementations of the present invention, individual users can be grouped or organized into cohorts in any suitable manner, such as with respect to a client application or across different (but potentially related) client applications. Such grouping can support variations in experiments and tests that cannot be performed, for example, when the long-term LTV is predicted at the cohort level. For example, individual users can be grouped according to any suitable characteristics associated with the user, the client application, or the client device on which the client application runs: user short-term LTV, user long-term LTV, user location, user age, user gender, type of client device (manufacturer, model, operating system, etc.), genre of the client application, length of time the user has interacted with or engaged in the client application, progress made by the user in the client application, etc. When appropriately grouped, suitable experiments and tests can be performed on the cohorts (e.g., testing new features, new offers, etc.). The users can then be regrouped into different cohorts to support variations in experiments and tests.

[0045] In some implementations of the present invention, bootstrapping can be performed on a user. Bootstrapping is a statistical procedure that resamples a single dataset to create many simulated samples. Such a process can enable the calculation of standard errors, the construction of confidence intervals, and the performance of hypothesis tests for many types of sample statistics. In the bootstrapping approach, a sample of size n is drawn from the population. The sample can be called S. Then, rather than using theory to determine every possible estimate, the sampling distribution is created by resampling observations with m replacements from S, and each resampled set has n observations. As a result, if properly sampled, S should represent the population. Thus, by resampling S m times with replacement, it appears as if m samples have been drawn from the original population, and the derived estimates will represent the theoretical distribution under the conventional approach. Increasing the number of resamples m will not increase the amount of information in the data. That is, for example, resampling the original set 100,000 times is not more useful than resampling the original set 1,000 times. The amount of information within the set depends on the sample size n, and it will remain constant throughout each resample. The benefit of more resamples is thus to derive a more appropriate estimate of the sampling distribution. In some implementations of the present invention, the customer lifetime value analysis module 116 can sample a pre-determined number of times (e.g., 1,000 times, etc.) using replacement from a dataset for long-term LTV prediction (e.g., as stored in and read from the user data database 120) and create sets with different long-term LTV predictions for the same user with an equal or similar number of times (e.g., 1,000 times, etc.). The customer lifetime value analysis module 116 can then construct or configure a confidence interval (e.g., a 95% confidence interval from the 2.5th percentile to the 97.5th percentile, although other confidence intervals are also possible) and provide more and additional insights about the user's long-term LTV to stakeholders.

[0046] Figure 6 is a graph illustrating an exemplary extrapolation graph for predicting the long - term LTV for users of a client application by using the short - term LTV according to an embodiment of the present disclosure. In some implementations of the present invention, the graph can be generated and used by the customer lifetime value analysis module 116. For example, "day" is represented on the x - axis, and the predicted long - term LTV (e.g., in dollars or other currency units) is represented on the y - axis. For illustrative purposes, and not by way of limitation, graph 600 can illustrate the long - term LTV value predictions determined for users of a client application according to the techniques and methodologies discussed above. For example, a user may have a first predicted long - term LTV 602 of $10 on the 30th day. A user may have a second predicted long - term LTV 604 of $30 on the 60th day. A user may have a third predicted long - term LTV 606 of $40 on the 90th day. A user may have a fourth predicted long - term LTV 608 of $45 on the 180th day. Additional and / or alternative long - term LTV data points and values within graph 600 are also possible, and graph 600 can include any suitable number of long - term LTV data points. In some implementations of the present invention, one or more of the first predicted long - term LTV 602, the second predicted long - term LTV 604, the third predicted long - term LTV 606, and the fourth predicted long - term LTV 608 can be determined based on the machine - learning models and / or transformation techniques discussed above using the short - term LTV to predict the long - term LTV. In an alternative embodiment, one or more of the first predicted long - term LTV 602, the second predicted long - term LTV 604, the third predicted long - term LTV 606, and the fourth predicted long - term LTV 608 can be the actual long - term LTV (not a prediction) at each data point. In a further alternative embodiment, the first predicted long - term LTV 602, the second predicted long - term LTV 604, the third predicted long - term LTV 606, and the fourth predicted long - term LTV 608 can be any suitable mixture or combination of predicted long - term LTV data points and actual long - term LTV data points.

[0047] In some implementations of the present invention, the shape or slope 614 of the curve within graph 600 in the most recent prediction (in this illustration, the fourth predicted long-term LTV 608) can then be used to extrapolate the long-term LTV prediction for the user for any suitable period in the future. For illustrative purposes, but not by way of limitation, based on the slope 614 of the curve of graph 600 at the fourth predicted long-term LTV 608, the first extrapolated long-term LTV 610 at the 540th day could be $50, while the second extrapolated long-term LTV 612 at the 1080th day could be $55. Additional or alternative future extrapolations regarding the predicted long-term LTV are also possible, based on the slope 614 of the curve within graph 600. Additionally, or alternatively, the extrapolated predictions of long-term LTV can be used for new users of the client application, in accordance with the techniques and methodologies discussed above (e.g., new users with short-term and / or long-term LTV profiles similar or identical to those of existing users of graph 600, or based on other suitable similar characteristics between new and existing users).

[0048] FIG. 7 is a block diagram of an exemplary computing device 700 that can perform one or more of the operations described herein according to this embodiment. The computing device 700 can be connected to other computing devices within a LAN, intranet, extranet, and / or the Internet. The computing device 700 can operate as a server machine within a client-server network environment or as a client within a peer-to-peer network environment. The computing device 700 can be provided by a personal computer (PC), a set-top box (STB), a server, a network router, a switch or bridge, or any machine capable of executing (sequentially or) a set of instructions that define the actions to be taken by that machine. Further, although only a single computing device 700 is illustrated, the term "computing device" shall be construed to include any set of computing devices that individually or together execute a set (or multiple sets) of instructions for implementing the methods discussed herein.

[0049] The exemplary computing device 700 may include a computer processing device 702 (e.g., a general-purpose processor, ASIC, etc.), a main memory 704, a static memory 706 (e.g., flash memory, etc.), and a data storage device 708, which may communicate with each other via a bus 730. The computer processing device 702 may be provided by one or more general-purpose processing devices such as a microprocessor, a central processing unit, etc. In an illustrative example, the computer processing device 702 may comprise a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor implementing another instruction set or a combination of instruction sets. The computer processing device 702 may also comprise one or more special-purpose processing devices such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), a network processor, etc. The computer processing device 702 may be configured to execute the operations described herein according to one or more aspects of the present disclosure to perform the operations and steps discussed herein.

[0050] The computing device 700 may further include a network interface device 712, which may communicate with a network 714. The data storage device 708 may include a machine-readable storage medium 728, in which one or more instruction sets, e.g., instructions for performing the operations described herein according to one or more aspects of the present disclosure, may be stored. The instructions 718 implementing the core logic instructions 726 may be fully or at least partially resident in the main memory 704 and also in the computer processing device 702 during their execution by the computing device 700. The main memory 704 and / or the computer processing device 702 also constitute a computer-readable medium. The instructions may be further transmitted or received via the network 714 through the network interface device 712.

[0051] The machine-readable storage medium 728 is illustrated in the illustrative example as a single medium, but the term "computer-readable storage medium" should be interpreted to include a single medium or multiple media (e.g., a centralized or distributed database and / or associated cache and server) that store one or more instruction sets. The term "computer-readable storage medium" should be interpreted to include any medium that can store, encode, or carry an instruction set for machine execution and cause a machine to perform the methods described herein. The term "computer-readable storage medium" should, therefore, be interpreted to include, without limitation, solid state memory, optical media, magnetic media, etc.

[0052] The subject matter described herein provides many technical advantages. For example, the server system 114 is scaled to support simultaneous long-term LTV determinations for a large number of individual users, such as hundreds of thousands, millions, tens of millions, or more users, thereby substantially improving computer resource allocation and processing efficiency. Accordingly, some implementations of the present invention can support and provide a more substantially faster long-term LTV determination time, particularly with respect to client applications with a large number of users, improving the efficiency and processing power of computer hardware resources (e.g., computer processing and memory) and enabling the long-term LTV of individual users to be determined. For example, some implementations of the present invention can more efficiently handle the determination of long-term LTV simultaneously for a large number of users. By improving the long-term LTV determination speed and efficiency for client applications with a large number of users, computer hardware resources can be more quickly freed up and used for other tasks and processes, resulting in a significant improvement in computer resource utilization.

[0053] In addition, some implementations of the present invention can be used to more easily perform LTV predictions in the presence of wide or extreme distributions of LTV data. Models trained on data exhibiting such extreme distributions can be biased by outliers that impair overall prediction accuracy. Embodiments of the present invention make correct predictions regarding a wide variety of individual users within a data set (e.g., from low spenders to high spenders who can result in extreme distributions of LTV data) without truncating values or biasing the model towards one group or the other, thereby improving the processing power and efficiency of the server system 114. Embodiments of the present invention also provide granularity for long-term LTV determinations that cannot be accomplished when long-term LTV predictions are performed at the cohort level by enabling individual users to be grouped in any manner, thereby further improving the functionality and processing of the server system 114. Additionally, some implementations of the present invention can enable testing of new features in a client application and user responses to those new features. For example, when a new feature is introduced into a client application, if the corresponding predicted long-term LTV for a user demonstrates an upward trend or other increase in long-term LTV, such a determination can indicate a positive user response to the new feature. Conversely, if the corresponding predicted long-term LTV for a user demonstrates a downward trend or other decrease in long-term LTV, such a determination can indicate a negative user response to the new feature. Such feature testing can support improved functionality, processing, and efficiency in the server system 114, the client device executing the client application, and the client application itself by reducing or eliminating unnecessary, undesirable, unused, and / or unneeded features and / or functionality from the client application.

[0054] The subject matter and embodiments of the operations described in this disclosure can be implemented in digital electronic circuitry, or in computer software, firmware, hardware, or combinations of one or more of them that include the structures disclosed in this disclosure and their structural equivalents. Embodiments of the subject matter described in this disclosure can be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a computer storage medium for execution by, or to control the operation of, a data processing apparatus. Alternatively, or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to a suitable receiver apparatus for execution by a data processing apparatus. A computer storage medium can be, or can be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Further, a computer storage medium is not a propagated signal, but a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. A computer storage medium can also be, or can be included in, one or more separate physical components or media (e.g., a plurality of CDs, disks, or other storage devices).

[0055] The operations described in this disclosure can be implemented as operations performed by a data processing apparatus based on data stored on one or more computer-readable storage devices or received from other sources.

[0056] The term "data processing apparatus" includes, by way of example, any kind of apparatus, device, and machine for processing data, including programmable processors, computer processing devices, computers, systems on a chip, or a plurality or combination of the foregoing. A computer processing device can include one or more processors, including special-purpose logic circuitry such as an FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit), a central processing unit (CPU), a multi-core processor, and the like. The apparatus can also include, in addition to the hardware, code for generating an execution environment for the computer program, such as processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or code constituting one or more combinations thereof. The apparatus and the execution environment can implement various different computing model infrastructures such as web services, distributed computing, and grid computing infrastructures.

[0057] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiler-type or interpreter-type languages, declarative, procedural, or functional languages. It can be deployed in any form, either as a stand-alone program or as appropriate modules, components, subroutines, objects, or other units for use within a computing environment. A computer program may or may not correspond to a file in a file system. The program can be stored as part of a file that holds other programs or data (e.g., one or more scripts stored within a markup language resource), in a single file dedicated to the program, or in multiple cooperating files (e.g., files that store one or more modules, subprograms, or portions of code). A computer program can be deployed to be executed on one computer, located at one site, or distributed across multiple computers located at multiple sites and interconnected by a communication network.

[0058] The processes and logical flows described in this disclosure can be implemented by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logical flows can also be implemented by special-purpose logic circuitry, such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit), and the apparatus can also be implemented as such.

[0059] Suitable processors for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. Essential elements of a computer are a processor for performing actions in accordance with instructions, and one or more memory devices for storing the instructions and data. Generally, a computer also includes, or is operatively coupled to receive data from, or transfer data to, one or more mass storage devices for storing data, such as, magnetic disks, magneto-optical disks, optical disks, solid state devices, etc. However, a computer need not have such devices. Further, a computer can be embedded in another device, such as, by way of a few examples, a smartphone, a mobile audio or media player, a gaming console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive). Devices suitable for storing computer program instructions and data include, by way of example, semiconductor memory devices, such as, EPROM, EEPROM, and flash memory devices, magnetic disks, such as, internal hard disks or removable disks, magneto-optical disks, and all forms of non-volatile memory, media, and memory devices including CD-ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0060] To provide interaction with a user, embodiments of the subject matter described herein can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, light emitting diode (LED) monitor, etc., and a keyboard and a pointing device by which the user can provide input to the computer, such as a mouse, trackball, touchpad, stylus, etc. Other types of devices can likewise be used to provide interaction with the user. For example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and the input received from the user can be in any form including acoustic, speech, or tactile input. Other possible input devices include touch screens or other touch sensor-based devices such as single or multi-point resistive or capacitive trackpads, speech recognition hardware and software, optical scanners, optical pointers, digital image capture devices and associated interpretation software, etc. In addition, the computer can interact with the user by sending resources to and receiving resources from devices used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0061] Embodiments of the subject matter described in this disclosure can be implemented in a computing system that includes, for example, a data server, including backend components, or including middleware components, such as an application server, or a frontend component, such as a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this disclosure, or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication, such as by a communication network. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), the Internet (e.g., the Internet), peer-to-peer networks (e.g., ad hoc peer-to-peer networks), and the like.

[0062] A computing system can include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by computer programs that run on respective computers and have a client-server relationship to each other. In some embodiments, the server transmits data (e.g., an HTML page) to the client device (e.g., for the purpose of displaying data to a user interacting with the client device and receiving user input from the user). Data generated at the client device (e.g., as a result of user interaction) can be received at the server from the client device.

[0063] One or more computer systems can be configured to perform certain operations or actions by having software, firmware, hardware, or combinations thereof installed on the system that cause the system to perform the actions during operation. One or more computer programs can be configured to perform certain operations or actions by including instructions that, when executed by a data processing apparatus, cause the apparatus to perform the actions.

[0064] Throughout this disclosure, references to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic is described in connection with an embodiment that includes at least one such embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this disclosure are not necessarily all referring to the same embodiment. Additionally, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or".

[0065] This disclosure includes many specific implementation details, but these are to be construed as descriptions of features specific to particular embodiments of a particular invention rather than as limitations on the scope of any invention or of what may be claimed. Features described in the context of separate embodiments in this disclosure can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented separately in multiple embodiments or in any suitable sub-combination. Further, although a feature may be described above as acting in a certain combination and may even be initially claimed as such, one or more features from the claimed combination can in some cases be deleted from the combination, and the claimed combination can be directed to a sub-combination or a variation of a sub-combination.

[0066] Similarly, the operations and / or logical flows are depicted in the drawings and / or described in the specification in a particular order, but this should not be construed as requiring that such operations and / or logical flows be performed in the particular order shown, or in sequential order, or that all of the illustrated operations be performed. In some situations, multitasking and parallel processing may be advantageous. Further, the separation of various system components in the embodiments described above should not be construed as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated together in a single software product or packaged into multiple software products.

[0067] Accordingly, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve desirable results. Additionally, the processes depicted in the accompanying figures do not necessarily require the particular order or sequential order shown to achieve desirable results. In some implementations, multitasking and parallel processing may be advantageous.

[0068] As used herein, the words "example" or "exemplary" are used to mean serving as an example, instance, or illustration. Any aspect or design described herein as "example" or "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects or designs. Rather, the use of the words "example" or "exemplary" is intended to present concepts in a concrete manner. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X includes A or B" is intended to mean any of the natural inclusive permutations. That is, "X includes A or B" is satisfied under any of the foregoing instances where X includes A, X includes B, or X includes both A and B. Additionally, as used in this application and the appended claims, the articles "a" and "an" generally should be construed to mean "one or more" unless otherwise specified or clear from the context when referring to the singular form. Further, the use throughout of the terms "an embodiment" or "one embodiment" or "an implementation" or "one implementation" is not intended to mean the same embodiment or implementation unless so described. Further, as used herein, terms such as "first", "second", "third", "fourth", etc. are meant to distinguish between different elements as labels and do not necessarily have an ordinal meaning in accordance with their numerical designation.

[0069] In the foregoing description and claims, phrases such as "at least one of" or "one or more of" may occur followed by a sequential list of elements or features. The term "and / or" may also occur within a list of two or more elements or features. Absent an express or implied contrary indication to the contrary by the context in which it is used, such phrases are intended to mean any of the recited elements or features individually or in combination with any of the other recited elements or features. For example, each of the phrases "at least one of A and B," "one or more of A and B," and "A and / or B" is intended to mean "only A, only B, or A and B together." Similar interpretations are also intended with respect to lists containing three or more items. For example, each of the phrases "at least one of A, B, and C," "one or more of A, B, and C," and "A, B, and / or C" is intended to mean "only A, only B, only C, A and B together, A and C together, B and C together, or A and B and C together." Additionally, the use of the term "based on" in the foregoing description and claims is intended to mean "at least in part based on" such that features or elements not recited are also permissible.

[0070] The above description of the illustrated implementations of the invention is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Specific implementations of the invention and examples thereof are described herein for illustrative purposes, but various equivalent modifications are possible within the scope of the invention, as will be recognized by those of ordinary skill in the art. The subject matter described herein can be embodied in a system, apparatus, method, and / or article, as appropriate for the desired configuration. The implementations described in the foregoing description do not represent all implementations consistent with the subject matter described herein. Instead, they are merely some examples that are consistent with aspects related to the subject matter described. Some variations have been described in detail above, but other modifications or additions are possible. In particular, further features and / or variations can be provided in addition to those described herein. For example, the implementations described above can be directed to various combinations and sub-combinations of the disclosed features and / or combinations and sub-combinations of some of the additional features disclosed above. Other implementations can also be within the scope of the following claims.

Claims

1. A method, the method comprising: receiving, by at least one data processor, first user data characterizing an interaction between a first user and a client application to be executed on the client device of the first user, first time interval data characterizing a first time interval associated with the first user, and second time interval data characterizing a second time interval associated with the first user, wherein the first time interval characterizes a length of time shorter than a length of time characterized by the second time interval; determining, by the at least one data processor, a first time interval customer lifetime value (LTV) of the first user based on the received first user data and the received first time interval data; receiving, by the at least one data processor, historical long-term LTV data for a plurality of users of the client application, each of the plurality of users having a respective first time interval LTV; transforming, by the at least one data processor, the received historical long-term LTV data into a transformed space; using, by the at least one data processor, the transformed historical long-term LTV data in the transformed space and the received second time interval data to determine a second time interval LTV for each of the plurality of users and generate a distribution of the second time interval LTVs; generating, by the at least one data processor, a prediction of a second time interval LTV for the first user based on the distribution by matching the first time interval LTV of the first user with each respective first time interval LTV of one of the plurality of users in the transformed space; transforming back, by the at least one data processor, the prediction of the second time interval LTV for the first user from the transformed space; When the prediction of the second time interval LTV for the first user meets a pre-determined LTV threshold by the at least one data processor, modifying the display of information to the first user within the client application A method comprising. **Claim 2** The method according to claim 1, wherein the prediction of the second time interval LTV is generated using a random forest model. **Claim 3** The method according to claim 2, further comprising training the random forest model using the received historical long-term LTV data. **Claim 4** The method according to claim 1, wherein the received historical long-term LTV data is transformed into the transformation space using the Lambert W function. **Claim 5** The method according to claim 1, further comprising determining a plurality of predictions of the second time interval LTV for the first user by the at least one data processor. **Claim 6** The method according to claim 5, further comprising generating graph data characterizing the predictions of a plurality of second time interval LTVs for the first user by the at least one data processor. **Claim 7** The method according to claim 1, further comprising generating a random forest regression model based on the transformed historical long-term LTV data in the transformation space by the at least one data processor. **Claim 8** The method according to claim 7, further comprising generating a distribution of predictions of the second time interval LTV for the first user using the random forest regression model generated in the transformation space by the at least one data processor. **Claim 9** Sampling the generated distribution of the prediction of the second time interval LTV for the first user by the at least one data processor, Generating, by the at least one data processor, a second distribution of the prediction of the second time interval LTV for the first user The method according to claim 1, further comprising: **Claim 10** The method according to claim 9, further comprising determining, by the at least one data processor, a confidence interval characterizing the generated distribution of the prediction of the second time interval LTV for the first user based on the second distribution of the prediction of the second time interval LTV for the first user. **Claim 11** A system, the system comprising: At least one data processor; A memory storing instructions And comprising: When the instructions are executed by the at least one data processor: Receiving, by the at least one data processor, first user data characterizing an interaction between a first user and a client application running on the client device of the first user, first time interval data characterizing a first time interval associated with the first user, and second time interval data characterizing a second time interval associated with the first user, wherein the first time interval characterizes a length of time shorter than a length of time characterized by the second time interval; Determining, by the at least one data processor, a first time interval customer lifetime value (LTV) of the first user based on the received first user data and the received first time interval data; Receiving, by the at least one data processor, historical long-term LTV data for a plurality of users of the client application, wherein each of the plurality of users has a respective first time interval LTV, Converting, by the at least one data processor, the received historical long-term LTV data into a transformed space, Using, by the at least one data processor, the transformed historical long-term LTV data in the transformed space and the received second time interval data to determine a second time interval LTV for each of the plurality of users and generate a distribution of the second time interval LTV, Generating, by the at least one data processor, a prediction of a second time interval LTV for the first user based on the distribution by matching the first time interval LTV of the first user with respective first time interval LTVs of one of the plurality of users in the transformed space, Converting back, by the at least one data processor, the prediction of the second time interval LTV for the first user from the transformed space, Modifying, by the at least one data processor, the display of information to the first user within the client application when the prediction of the second time interval LTV for the first user meets a pre-determined LTV threshold value A system that causes the at least one data processor to perform operations including the above.

12. The system according to claim 11, wherein the prediction of the second time interval LTV is generated using a random forest model.

13. The system according to claim 12, wherein the operations further include training the random forest model using the received historical long-term LTV data.

14. The system according to claim 11, wherein the received historical long-term LTV data is converted into the conversion space using the Lambert W function. **Claim 15** The system according to claim 11, wherein the operation further comprises determining, by the at least one data processor, a plurality of predictions of the second time interval LTV for the first user. **Claim 16** The system according to claim 15, wherein the operation further comprises generating, by the at least one data processor, graph data characterizing the plurality of predictions of the second time interval LTV for the first user. **Claim 17** The system according to claim 11, wherein the operation further comprises generating, by the at least one data processor, a random forest regression model in the conversion space based on the converted historical long-term LTV data. **Claim 18** The system according to claim 17, wherein the operation further comprises generating, by the at least one data processor, a distribution of predictions of the second time interval LTV for the first user using the random forest regression model generated in the conversion space. **Claim 19** The operation is sampling, by the at least one data processor, the generated distribution of predictions of the second time interval LTV for the first user, and generating, by the at least one data processor, a second distribution of predictions of the second time interval LTV for the first user and further comprises the system according to claim 11. **Claim 20** A non-transitory computer program product storing executable instructions, wherein the executable instructions, when executed by at least one data processor forming part of at least one computing system, receive, by at least one data processor, first user data characterizing an interaction between a first user and a client application executing on the client device of the first user, first time interval data characterizing a first time interval associated with the first user, and second time interval data characterizing a second time interval associated with the first user, wherein the first time interval characterizes a length of time shorter than a length of time characterized by the second time interval, determine, by the at least one data processor, a first time interval customer lifetime value (LTV) of the first user based on the received first user data and the received first time interval data, receive, by the at least one data processor, historical long-term LTV data for a plurality of users of the client application, wherein each of the plurality of users has a respective first time interval LTV, transform, by the at least one data processor, the received historical long-term LTV data into a transformed space, use, by the at least one data processor, the transformed historical long-term LTV data in the transformed space and the received second time interval data to determine a second time interval LTV for each of the plurality of users and generate a distribution of the second time interval LTVs, generate, by the at least one data processor, a prediction of a second time interval LTV for the first user based on the distribution by matching the first time interval LTV of the first user with a respective first time interval LTV of one of the plurality of users in the transformed space, Converting back, by the at least one data processor, the prediction of the second time interval LTV for the first user from the transformed space; Modifying, by the at least one data processor, the display of information to the first user within the client application when the prediction of the second time interval LTV for the first user meets a pre-determined LTV threshold; A non-transitory computer program product that performs operations including the above.

Images