System and method for determining lifecycle value of user of client application

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining a lifecycle value of a user of a client application. The method may include receiving first user data characterizing an interaction of a first user with a client application executing on a client device of the first user, first time interval data characterizing a first time interval associated with the first user, and second time interval data representing a second time interval associated with the first user, the time length represented by the first time interval being shorter than the time length represented by the second time interval; determining a first time interval lifecycle value (LTV) of the first user based on the received first user data and the received first time interval data; receiving historical long-term LTV data for a plurality of users of the client application, wherein each of the plurality of users has a respective first time interval LTV; converting the received historical long-term LTV data into a conversion space; determining a second time interval, LTV, for each of the plurality of users using the transformed historical long-term, LTV data in the transformation space and the received second time interval data to generate a distribution of the second time interval, LTV; generating a prediction of a second time interval LTV of the first user based on the distribution in the transformation space by matching the first time interval LTV of the first user with a corresponding first time interval LTV of one of the plurality of users; transforming the prediction of the second time interval LTV of the first user back from the transformation space; and modifying the display of information to the first user within the client application when the prediction of the second time interval, LTV, of the first user satisfies a predetermined LTV threshold.

Description

[0001] Cross - reference to related applications

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 347,329, filed on May 31, 2022, the entire content of which is incorporated herein by reference. Background Art

[0003] Lifetime value (LTV) is a prediction of the revenue generated by the ongoing relationship between a user and a product (e.g., a mobile application). By providing a real - time estimate of how much a particular user might spend on a product, LTV can help ensure that a company pursues the most effective users. If a company can successfully predict a user's LTV, it can provide a better basis for the company's decision - making, which can help the company maximize the efficiency of its user acquisition efforts. Summary of the Invention

[0004] The present invention relates to a system and method for determining the lifetime value (LTV) of a user of a client application. According to the present invention, short - term data can be used to predict the long - term LTV of a user of a client application. In some embodiments of the present invention, the prediction can be performed in a transformed space. The resulting prediction can be transformed back from the transformed space to generate a long - term LTV prediction for the user. In an embodiment, based on the predicted long - term LTV, such short - term data can be used to estimate the amount or potential revenue that a user can generate over the lifetime of the user with respect to the client application. Accordingly, if the estimated potential revenue generated by the user is greater than or equal to a certain threshold or within a certain threshold, additional promotional activities can be presented to the user within the client application or associated with the client application, because enhanced promotional activities for the user are necessary.

[0005] Systems and methods for determining the lifetime value of a user of a client application are provided. Related devices, techniques, and articles of manufacture are also described.

[0006] In one aspect, a first user data representing an interaction between a first user and a client application executed on a client device of the first user, a first time interval data representing a first time interval associated with the first user, and a second time interval data representing a second time interval associated with the first user can be received by at least one data processor. The time length represented by the first time interval can be shorter than the time length represented by the second time interval. A first time interval lifetime value (LTV) of the first user can be determined by at least one data processor based on the received first user data and the received first time interval data. Historical long-term LTV data of a plurality of users of the client application can be received by at least one data processor, and each of the plurality of users can have a corresponding first time interval LTV. The received historical long-term LTV data can be transformed by at least one data processor into a transformed space. A second time interval LTV of each of the plurality of users can be determined by at least one data processor using the transformed historical long-term LTV data in the transformed space and the received second time interval data to generate a distribution of the second time interval LTV. A prediction of the second time interval LTV of the first user can be generated by at least one data processor in the transformed space based on the distribution by matching the first time interval LTV of the first user with a corresponding first time interval LTV of one of the plurality of users. The prediction of the second time interval LTV of the first user can be transformed back by at least one data processor from the transformed space. When the prediction of the second time interval LTV of the first user meets a predetermined LTV threshold, the information display for the first user within the client application can be modified by at least one data processor.

[0007] One or more of the following features may be included in any feasible combination. For example, a random forest model may be utilized to generate a prediction of the second time interval LTV. For example, the received historical long-term LTV data may be utilized to train the random forest model. For example, the Lambert W function may be utilized to transform the received historical long-term LTV data into a transformed space. For example, multiple predictions of the second time interval LTV of the first user may be determined by at least one data processor. For example, graphical data representing the multiple predictions of the second time interval LTV of the first user may be generated by at least one data processor. For example, a random forest regression model may be generated by at least one data processor in the transformed space and based on the transformed historical long-term LTV data. For example, a distribution of the predictions of the second time interval LTV of the first user may be generated by at least one data processor and by utilizing the random forest regression model generated in the transformed space. For example, the generated distribution of the predictions of the second time interval LTV of the first user may be sampled by at least one data processor, and a second distribution of the predictions of the second time interval LTV of the first user may be generated by at least one data processor. For example, a confidence interval representing the generated distribution of the predictions of the second time interval LTV of the first user may be determined by at least one data processor and based on the second distribution of the predictions of the second time interval LTV of the first user.

[0008] In another aspect, a system is provided and the system can include at least one data processor and a memory that stores instructions which, when executed by the at least one data processor, cause the at least one data processor to perform the operations described herein. The operations can include: receiving, by the at least one data processor, first user data representing an interaction of a first user with a client application executing on a client device of the first user, first time interval data representing a first time interval associated with the first user, and second time interval data representing a second time interval associated with the first user, the first time interval representing a shorter length of time than the second time interval; determining, by the at least one data processor, a first time interval lifetime value (LTV) of the first user based on the received first user data and the received first time interval data; receiving, by the at least one data processor, historical long-term LTV data of a plurality of users of the client application, wherein each of the plurality of users has a corresponding first time interval LTV; transforming, by the at least one data processor, the received historical long-term LTV data into a transformed space; determining, by the at least one data processor, a second time interval LTV of each of the plurality of users using the transformed historical long-term LTV data in the transformed space and the received second time interval data to generate a distribution of the second time interval LTV; generating, by the at least one data processor, a prediction of the second time interval LTV of the first user in the transformed space by matching the first time interval LTV of the first user with a corresponding first time interval LTV of one of the plurality of users, based on the distribution; transforming, by the at least one data processor, the prediction of the second time interval LTV of the first user back from the transformed space; and modifying, by the at least one data processor, the display of information about the first user within the client application when the prediction of the second time interval LTV of the first user meets a predetermined LTV threshold.

[0009] One or more of the following features may be included in any feasible combination. For example, a random forest model can be utilized to generate a prediction of the second time interval LTV. For example, the operation may further include: training a random forest model using the received historical long-term LTV data. For example, the Lambert W function can be utilized to transform the received historical long-term LTV data into a transformed space. For example, the operation may further include determining, by at least one data processor, multiple predictions of the second time interval LTV of a first user. For example, the operation may further include generating, by at least one data processor, graphical data representing the multiple predictions of the second time interval LTV of a first user. For example, the operation may further include: generating, by at least one data processor in the transformed space and based on the transformed historical long-term LTV data, a random forest regression model. For example, the operation may further include: generating, by at least one data processor and using the random forest regression model generated in the transformed space, a distribution of the predictions of the second time interval LTV of a first user. For example, the operation may further include: sampling, by at least one data processor, the generated distribution of the predictions of the second time interval LTV of a first user, and generating, by at least one data processor, a second distribution of the predictions of the second time interval LTV of a first user.

[0010] A non-volatile computer program product (i.e., a physically embodied computer program product) storing instructions is also described, the instructions, when executed by one or more data processors of one or more computing systems, cause at least one data processor to perform the operations herein. Similarly, a computer system that may include one or more data processors and a memory connected to the one or more data processors is also described. The memory may store the instructions temporarily or permanently, the instructions causing at least one processor to perform one or more of the operations described herein. Additionally, the method may be implemented by one or more data processors within a single computing system or distributed across two or more computing systems. Such computing systems may be connected and may exchange data and / or commands or other instructions, etc., via one or more connections (including connections via a network (e.g., the Internet, wireless wide area network, local area network, wide area network, wired network, etc.)), via a direct connection between one or more of the multiple computing systems, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above-described embodiments will be more fully understood from the following detailed description in conjunction with the accompanying drawings. The drawings are not intended to be drawn to scale. For clarity, not every component will be labeled in each drawing. In the drawings:

[0012] Figure 1is a block diagram showing an example system for predicting the long - term lifecycle value of users of a client application by leveraging short - term lifecycle value;

[0013] Figure 2 is a flowchart showing an example method for predicting the long - term lifecycle value of users of a client application by leveraging short - term lifecycle value;

[0014] Figure 3 is a flowchart showing an example method for predicting the long - term lifecycle value of users of a client application by leveraging short - term lifecycle value;

[0015] Figure 4 is a flowchart showing an example method for predicting the long - term lifecycle value of users of a client application by leveraging short - term lifecycle value;

[0016] Figure 5 is a flowchart showing an example method for predicting the long - term lifecycle value of users of a client application by leveraging short - term lifecycle value;

[0017] Figure 6 is a graph showing an example extrapolation graph for predicting the long - term lifecycle value of users of a client application by leveraging short - term lifecycle value; and

[0018] Figure 7 is a block diagram of an example computing device that can perform one or more of the operations described herein according to the present implementation. Detailed Description

[0019] Certain exemplary embodiments will now be described to provide a general understanding of the structures, functions, manufacturing, and principles of use of the devices and methods disclosed herein. One or more examples of these embodiments are shown in the accompanying drawings. Those skilled in the art will understand that the devices and methods specifically described and shown in the drawings are non - limiting exemplary embodiments, and the scope of the present invention is defined only by the claims. Features shown or described in connection with one exemplary embodiment can be combined with the features of other embodiments. Such modifications and variations are intended to be included within the scope of the present invention. Additionally, in the present disclosure, components with similar names in the embodiments generally have similar features, and thus, in a particular embodiment, each feature of each similarly named component is not necessarily fully elaborated.

[0020] The present invention relates to a system and method for determining the lifetime value (LTV) of a user of a client application. According to the present invention, short-term data can be used to predict the long-term LTV of a user of a client application. In some embodiments of the present invention, the prediction can be performed in a transformed space. The resulting prediction can be transformed back from the transformed space to generate a long-term LTV prediction of the user. In an embodiment, based on the predicted long-term LTV, the present invention can utilize such short-term data to estimate the amount or potential revenue that a user can generate during the lifetime with respect to the client application. Accordingly, if the estimated potential revenue generated by the user is greater than or equal to a certain threshold or within a certain threshold, additional promotional activities can be presented to the user in or associated with the client application, because enhanced promotional activities for the user are necessary. Embodiments of the present invention can support prediction for any definition of the selected LTV, can be predicted at the user level, are capable of changing the prediction scope, and are capable of being updated regularly, such as weekly or other suitable time periods.

[0021] For purposes of discussion and not limitation, the present invention can illustrate various aspects of the present invention by taking a digital game as an exemplary client application. However, the present invention can be used for any suitable type of client application (e.g., mobile application, desktop application, or any other suitable type of client application) where a user can consume currency (e.g., physical, electronic, virtual, digital, etc.) when participating in and interacting with the client application. For example, the LTV of a player in a digital game can be defined as the expected revenue that the player will generate during the lifetime in the digital game. However, the present invention can be used for any suitable type of client application that requires calculating the long-term LTV of a user. For purposes of discussion and not limitation, the present invention takes "revenue" as the basis for LTV calculation to illustrate various aspects of the present invention, but any suitable parameter can also be used instead of revenue. For example, for purposes of discussion only, using the initial installation date of the client application or the date of the first deposit in the client application as the first day (D1), the LTV can be the sum of the discounted daily revenues.

[0022] Figure 1FIG. 0 is a block diagram illustrating an example system 100 for predicting the long-term LTV of users of a client application by leveraging short-term LTV. The server system 114 may provide functionality for receiving and collecting data associated with characteristics of users (such as players in a digital game) in the client application (e.g., revenue data, etc.). The server system 114 may include software components and databases that may be deployed at one or more data centers 112 in, for example, one or more geographical locations. The software components of the server system 112 may include a lifetime value analysis module 116 and a client device information display module 118. The software components may include sub-components that may be executed on the same or different individual data processing devices. The databases of the server system 114 may include, for example, a user data database 120 and a client application data database 122, although other databases are possible. The databases may exist in one or more physical storage systems or may be cloud-based. The software components and databases will be described further below.

[0023] As Figure 1 shown, the lifetime value analysis module 116 and the client device information display module 118 may communicate with each other and with the user data database 120 and the client application data database 122. The user data database 120 may include, for example, any suitable information related to one or more users of the client application and the interactions between these users and the client application, such as user LTV data (predicted, actual, short-term, long-term, etc.), user characteristics, user interaction history (e.g., in the case of a digital game, which digital games have been played, the number of game sessions won in each digital game, the number of game sessions lost in each digital game, the number of game sessions played in each digital game, the score in each digital game, the time played in each digital game, etc.), user identification information (e.g., username), the history of the user connecting to the system 100, user purchases, user achievements, user tasks, interactions between the user and other users (e.g., chat), user deposits / withdrawals, user virtual item acquisitions or uses, other conditions in the client application, etc. The client application data database 122 may include, for example, information related to the client applications implemented using the system 100. The client application data database 122 may include information related to each client application, such as the virtual environment of each client application, the image, video, text, and / or audio data of each client application, event data corresponding to previous, current, or future events, client application status data defining the current state of each client application, etc.

[0024] A software application (e.g., a digital game or other web-based or suitable client application) can be provided as an end-user client application to allow a user to interact with the server system 114. The software application can involve and / or provide a variety of functions and information, including, for example, entertainment (e.g., games, music, video, etc.), business (e.g., word processing, accounting, spreadsheets, etc.), news, weather, finance, sports, etc. In some embodiments of the present invention, the software application can provide a digital game. The digital game can be or include, for example, a sports game, an adventure game, a virtual card game, a virtual board game, a puzzle game, a racing game, or any other suitable type of digital game. In an embodiment, the digital game can be an asynchronous skill-based game, where players can compete with each other in the digital game but do not have to play the digital game at the same time. In an alternative embodiment, the digital game can be a synchronous skill-based game, where players can play the digital game at the same time and can compete with each other in the digital game in real time. Other suitable software applications are also possible.

[0025] The software application or its components can be accessed by a user of the client device via a network 110 (e.g., the Internet), the client devices being such as client device A 102, client device B 104, client device C 106, ……, client device N 108, where N can be any suitable natural number. Each of the client devices can be any suitable type of electronic device capable of executing the software application and communicating with the server system 114 via the network 110, such as a smart phone, a tablet computer, a laptop computer, a desktop or personal computer, etc. Other client devices are also possible (e.g., portable or desktop game consoles, smart TVs, smart watches, and other similar computing devices). In an alternative embodiment, the user data database 120, the client application data database 122, or any part thereof can be stored on one or more client devices. Additionally or alternatively, the software components of the system 100 (e.g., the lifetime value analysis module 116 and / or the client device information display module 118) or any part thereof can reside on one or more client devices or can be used to perform operations on one or more client devices.

[0026] In some embodiments of the present invention, the LTV of a user may represent how much currency (e.g., physical, electronic, virtual, digital, etc.) the user has consumed in the client application. For example, in the case of a digital game, the LTV of a user may represent how much money the user has spent in the digital game, minus any bonuses or rewards that the user may have been granted. In an embodiment, each user may have their own LTV, and the value of their LTV may change with each transaction the user makes in the client application. In an embodiment, a user who has not made a transaction or otherwise deposited currency into the client application will not have an LTV. In some embodiments of the present invention, the calculation of the LTV of a user may start from the first paid transaction (e.g., in the case of a digital game, the first paid entry fee for a competition) in the client application after the user makes their first deposit. Such an event may be referred to as a "transaction after first deposit". Alternatively, the calculation of the LTV of a user may start from the initial installation of the client application. In an embodiment, the length of the calculated LTV may be specified with reference to a suitable time period (such as hours, days, weeks, months, etc.). For illustrative purposes only and not by way of limitation, the time period may be a day, and the number of days may be counted from the user's transaction after first deposit or initial installation. For example, D1 LTV may refer to the sum of all relevant transactions within one day (24 hours) of the user's transaction after first deposit or initial installation, D2 LTV may refer to the sum of all relevant transactions within the first two days (48 hours) of the user's transaction after first deposit or initial installation, D7 LTV may refer to the sum of all relevant transactions within the first seven days of the user's transaction after first deposit or initial installation, and so on. The transaction components that can be used to calculate the LTV of a user (i.e., the transactions that can be considered when calculating the LTV of a user) may depend on, for example, the client application and the types of transactions supported by the client application. For example, in the case of a digital game on a digital game platform based on competitive skills, transactions such as paid entry fees for competitions and tournaments may increase the LTV of the user, while transactions such as cash rewards granted for winning a competition or tournament, refunds, physical rewards granted or redeemed (e.g., from a prize or gift store), etc. may decrease the LTV of the user. Other transaction components of the LTV of a user are also possible.

[0027] In some embodiments of the present invention, the Lifetime Value Analysis Module 116 may utilize an XGBoost model or the like to predict the long-term LTV of a user based on the short-term LTV (e.g., starting from the user's transaction after first deposit or initial installation), but other suitable machine learning / artificial intelligence models may also be utilized. Figure 2It is a flowchart showing an example method 200 for predicting the long-term LTV of users of a client application by leveraging short-term LTV according to an embodiment of the present invention. In some embodiments of the present invention, method 200 may be performed, for example, by the Lifecycle Value Analysis Module 116 using an XGBoost model or the like. In an embodiment, the Lifecycle Value Analysis Module 116 may monitor multiple users of the client application executed on the respective client devices of the users. At block 205, the Lifecycle Value Analysis Module 116 may select (or receive a selection of) a short-term day (e.g., D7, D30, or any other suitable time interval) from which to make a prediction and a long-term day (e.g., D90, D180, or any other suitable time interval) to which to make a prediction for the first user. For purposes of illustration and not limitation, the Lifecycle Value Analysis Module 116 may select D180 as the long-term day for the first user and D7 as the short-term day for the first user. Thus, in this example, the Lifecycle Value Analysis Module 116 may predict the LTV on the 180th day after the initial installation of the client application or the first deposit in the client application based on data up to and including the 7th day after the initial installation of the client application or the first deposit in the client application, but any other suitable number of days or time periods or intervals may also be selected. At block 210, the Lifecycle Value Analysis Module 116 may determine which cohort or group the first user belongs to or otherwise belongs to and the short-term LTV of the first user (e.g., D7 LTV in this example). For example, the Lifecycle Value Analysis Module 116 may utilize user data collected or otherwise received (e.g., deposit and / or consumption data retrieved from the User Data Database 120) to determine the short-term LTV of the user (e.g., D7 LTV in this example). For purposes of illustration and not limitation, in the case of a digital game, users may be divided into cohorts according to digital game, platform, network, country, advertising network, time of first deposit, withdrawal and maintenance fees paid, entry fees paid, first digital game played, number of digital games played, etc. or any combination thereof. Other cohorts or groups are possible and will depend, for example, on the type of client application, the type of users of the client application, how the users interact with or otherwise participate in the client application, etc. Each cohort or group may consist of any suitable number of users. In an embodiment, the cohorts may be organized or otherwise optimized to reduce cohort-level errors in the predicted long-term LTV. At block 215, the Lifecycle Value Analysis Module 116 may determine the percentile of the short-term LTV of the first user (e.g., D7 LTV) and identify other users in the cohort having the same percentile of short-term LTV (e.g., D7 LTV) as the first user, which may be all users in the cohort having the same percentile or a suitably sized subset of such users.In an alternative embodiment, the lifetime value analysis module 116 may utilize, for example, the decile in which the user is located instead of the exact percentile. In an embodiment, the width of the selected percentile range may be a hyperparameter and may be selected, for example, by appropriate cross-validation (CV) / tuning. In another alternative embodiment, the lifetime value analysis module 116 may directly predict the long-term LTV as a continuous value.

[0028] At block 220, the lifetime value analysis module 116 may determine the long-term LTV (e.g., D180 LTV) of a set of users having the same or similar short-term LTV (e.g., D7 LTV) percentile (calculated at block 215) as the first user to generate a distribution of possible long-term values for the first user. In an embodiment, the lifetime value analysis module 116 may collect, retrieve, or otherwise receive historical long-term LTV data for the set of users for which the long-term LTV data is available. Data from the set of users may be used for the purpose of training a machine learning model (e.g., XGBoost, etc.). In some embodiments of the present invention, in addition to or instead of the users in the specified user cohort or group, the historical long-term LTV data of other users may be used to train the machine learning model. In an embodiment, the lifetime value analysis module 116 may retrieve the long-term LTV data of the set of users from the user data database 120. The lifetime value analysis module 116 may remove any users from the collected, retrieved, or received long-term LTV data that should not be considered in the set of users (e.g., the user data is fraudulent, corrupted, disabled, or otherwise unavailable). In some embodiments of the present invention, the lifetime value analysis module 116 may perform hyperparameter optimization on the long-term LTV data of the set of users. In hyperparameter optimization, multiple models may be generated, each model having random values for the number of trees, depth of the trees, regularization strength, etc. The hyperparameters that result in the highest validation score as judged by the primary metric (e.g., total error in each cohort) may be used to retrain the complete dataset (e.g., training set plus validation set). In some embodiments of the present invention, the primary metric may be calculated post hoc rather than directly optimized by the model. For example, the mean squared error, etc. may be used to train the model.

[0029] In an embodiment, the determination made by the Lifetime Value Analysis Module 116 can generate a distribution (e.g., a histogram) of possible long-term LTVs (e.g., D180 LTV) for a first user for which a prediction is to be made. Thus, the Lifetime Value Analysis Module 116 can utilize historical long-term LTV data of a set of users collected, retrieved, or received, where the set of users has a short-term LTV that is the same as or similar to that of the first user. In other words, the short-term LTV of the first user can be used to match the first user to a distribution of long-term LTVs that have the same or similar short-term LTV as the first user. According to an alternative embodiment, the Lifetime Value Analysis Module 116 can utilize, for example, the decile in which the first user falls, rather than an exact percentile. The width of the selected percentile range can be a hyperparameter and can be selected, for example, through appropriate CV / tuning. In an alternative embodiment, the Lifetime Value Analysis Module 116 can directly predict long-term LTV from short-term LTV using a suitable regression model or the like. At block 225, the Lifetime Value Analysis Module 116 can determine which percentile is most suitable for use in the prediction for the first user by utilizing the distribution (e.g., histogram) of long-term LTVs (e.g., D180 LTV) determined by the Lifetime Value Analysis Module 116 at block 220. For illustrative purposes and not limitation, the Lifetime Value Analysis Module 116 can select the 50th percentile and use the median (or other suitable statistical calculation) of the distribution for the prediction. Alternatively, for example, the Lifetime Value Analysis Module 116 can select the 70th percentile to consistently predict higher values. Other percentiles are possible. According to an embodiment, the percentile to be predicted can also be considered a hyperparameter and studied with appropriate CV / tuning. At block 230, the Lifetime Value Analysis Module 116 can determine the value (e.g., dollar amount or other currency value) of the determined percentile at the long-term day (e.g., D180), and can use that value as the predicted long-term LTV (e.g., D180 LTV) for the first user. For example, if the value is a dollar (or other suitable currency) amount, the predicted long-term LTV for the first user can be the expected revenue that the first user can generate over the course of its lifetime in the client application. The Lifetime Value Analysis Module 116 can store the result in the User Data Database 120 for later retrieval and use.

[0030] In some embodiments of the present invention, the Lifetime Value Analysis Module 116 may perform alternative steps for determining a prediction of the historical LTV distribution at the cohort level. The Lifetime Value Analysis Module 116 may select, for a first user, a short-term day from which to make a prediction (e.g., D7, D30, or any other suitable time interval) and a long-term day to which to make a prediction (e.g., D90, D180, or any other suitable time interval). For purposes of illustration only and not limitation, the Lifetime Value Analysis Module 116 may select D180 as the long-term day for the first user and D7 as the short-term day for the first user. Thus, in this example, the Lifetime Value Analysis Module 116 may predict the LTV at the 180th day after the initial installation of the client application or the first deposit in the client application, based on data up to and including the 7th day after the initial installation of the client application or the first deposit in the client application, but any other suitable number of days may also be selected. The Lifetime Value Analysis Module 116 may determine which cohort the first user is in (e.g., dividing users into queues according to mobile applications, platforms, networks, etc., as described above) and determine what the short-term LTV of the first user is (e.g., D7 LTV). The Lifetime Value Analysis Module 116 may determine the percentile of the short-term LTV of the first user (e.g., D7 LTV). In some embodiments of the present invention, the LTV of the first user may be at the same percentile at the long-term day (e.g., D180). The Lifetime Value Analysis Module 116 may determine the value (e.g., a dollar amount or other currency amount) of the determined percentile at the long-term day (e.g., D180) and use that value as the predicted long-term LTV of the first user (e.g., D180 LTV). For example, if the value is a dollar (or other suitable currency) amount, the predicted long-term LTV of the first user may be the expected revenue that the user can generate during the lifetime of the client application. In an alternative embodiment, the Lifetime Value Analysis Module 116 may directly predict the long-term LTV using, for example, a regression model, based on or otherwise leveraging the characteristics collected on the short-term day of the first user.

[0031] In some embodiments of the present invention, historical LTV can be segmented according to relevant groups. For purposes of illustration and not limitation, users of a digital game company can be considered, for example, as a site group, a network group, a platform group, and a first-install game group, but any other suitable groups and user groupings are also possible. After all users are grouped, the LTV curve of each user within each group can be calculated (e.g., by the lifetime value analysis module 116). Thus, for each day, there can be a distribution of LTV on that day because each user can have its own LTV on that day. In an embodiment, the LTV can be non-monotonic. For example, if information is plotted for a particular group, "day" can be on the x-axis and the LTV up to that day can be on the y-axis. After calculating the daily LTV distribution for each group, then training of an appropriate machine learning model can be performed. According to the present invention, two types of data can be utilized in such a model: time-independent data, where all features belong to this category (e.g., platform, advertising network, etc.); and time-dependent data (e.g., target variables such as revenue).

[0032] LTV prediction can encounter extreme distributions of LTV data, which may exhibit fat or heavy tails and / or skewness. Regarding Figure 2The embodiments shown and discussed can advantageously utilize this data to reduce queue level errors. However, in some embodiments of the present invention, a random forest model or the like can be utilized to implement the lifetime value analysis module 116 to predict a user's long-term LTV based on short-term LTV (e.g., starting from transactions after the user's first deposit or initial installation), but other suitable machine learning / artificial intelligence models (e.g., neural networks, etc.) can also be utilized. Such embodiments can be used to more easily perform LTV predictions in cases where there is a large distribution or other extreme distribution in the LTV data. Models trained using data exhibiting such extreme distributions may be biased by outliers, thus compromising overall prediction accuracy. In some embodiments of the present invention, a technique of "Gaussianizing" a heavy-tailed distribution using the Lambert W function (also known as the omega function or product logarithm) can be used for LTV data with a large distribution or extreme distribution. The Lambert W function can provide an explicit inverse transformation, which can be estimated by maximum likelihood. This inverse transformation can remove the heavy tails in the data and can also provide analytical expressions for the cumulative distribution function and the probability density function. Therefore, the methods according to some embodiments of the present invention can be performed in the transformed space to more easily handle LTV data with a large distribution or extreme distribution. In an embodiment, the Lambert transformation can be fitted to the training data (e.g., LTV data with a large distribution or extreme distribution), an appropriate model can be trained in the transformed space where the values are more "well-behaved", and then the transformation can be inverted to return the predictions to the normal space (e.g., the "cash" or "currency" space - the "heavy-tailed world") for predicting the user's long-term LTV. This technique can allow embodiments of the present invention to make correct predictions for a wide variety of users in the dataset (e.g., from smaller consumers who may cause extreme distributions in the LTV data to larger consumers) without cutting off values or biasing the model towards one group or another. Although the Lambert W function can be used for fat-tailed or heavy-tailed data, the choice of transformation will depend on the type of data being analyzed. For example, a logarithmic transformation can be used for data without negative values, and a power transformation can be used for skewed data. Other transformations are also possible.

[0033] Figure 3is a flowchart showing an example method 300 for predicting the long-term LTV of users of a client application by leveraging short-term LTV. In some embodiments of the present invention, method 300 may be performed, for example, by the Lifecycle Value Analysis Module 116 using a random forest model or the like. Compared with the XGBoost model, the random forest model may allow for the use of a larger number of trees without overfitting. The random forest regression model may average the predictions of each tree to produce a final output. In an embodiment, the Lifecycle Value Analysis Module 116 may treat each tree as an independent estimator of the average LTV and use the independent estimators as an approximate long-term LTV distribution for each user. It should be noted that in some embodiments of the present invention, due to, for example, skewness and fat tails in the data, the median of the approximate distribution may better estimate the LTV of a user than the mean, and the quantiles of the distribution may provide a higher confidence interval for LTV prediction. However, although the median may provide a better estimate for each individual user, the median may not provide a better value for calculating the total LTV of a user cohort. In cases where the individual LTV distributions may have a strong right skew, the median may underestimate the mean. However, in some embodiments of the present invention, although the median is more effective for individuals, the cohort LTV should take this skew into account, and using the mean may be more effective than using the median for a user cohort.

[0034] In an embodiment, the lifetime value analysis module 116 may monitor multiple users of a client application executing on a respective client device of a user. At block 305, the lifetime value analysis module 116 may select (or receive a selection of) a short-term day (from which to make a prediction) and a long-term day (to which to make a prediction) for a first user. For illustrative purposes only and not by way of limitation, the lifetime value analysis module 116 may select D180 as the long-term day for the first user and D30 as the short-term day for the first user. Thus, in this example, the lifetime value analysis module 116 may predict the LTV at the 180th day after the initial installation of the client application or the first deposit in the client application, based on data up to the 30th day (including the 30th day) after the initial installation of the client application or the first deposit in the client application by the first user, but any other suitable number of days or time periods or intervals may also be selected for the long-term day and the short-term day. At block 310, the lifetime value analysis module 116 may utilize the selected short-term day and user data received or otherwise collected (e.g., deposit and / or consumption data retrieved from the user data database 120) to determine the short-term LTV (e.g., D30LTV) of the first user. At block 315, the lifetime value analysis module 116 may utilize the selected long-term day to receive, collect, or otherwise retrieve historical long-term LTV data for a set of other users that have the same or similar short-term LTV as the first user and for which long-term LTV data is available. The set of other users may be used to train a machine learning model (e.g., a random forest model, etc.) and may include users other than the first user for which the long-term LTV prediction is being made. In an embodiment, the lifetime value analysis module 116 may receive, collect, or otherwise retrieve the long-term LTV data for the set of other users from the user data database 120. In an embodiment, the lifetime value analysis module 116 may remove any users from the long-term LTV data collected or retrieved that should not be considered in the set of other users (e.g., the user data is fraudulent, corrupted, disabled, or otherwise unavailable).

[0035] At block 320, the Lifetime Value Analysis module 116 can transform or otherwise convert the received historical long-term LTV data of other user sets into a suitable transformed space. In an embodiment, the Lambert W function can be utilized as the transformation, such that it becomes easier to process the long-term LTV data (e.g., LTV data with large or extreme distributions), but other suitable transformations can also be utilized. In such an embodiment, the root mean square error of the model can be used in the transformed space, as this can indicate how well the model performs across the entire data set. In some embodiments of the present invention, the Lifetime Value Analysis module 116 can perform hyperparameter optimization on the long-term LTV data of the user sets. In hyperparameter optimization, multiple models can be generated, each model having random values for the number of trees, the depth of the trees, the regularization strength, etc. The hyperparameters that result in the highest validation score based on the primary metric can be used to retrain the complete data set (e.g., the training set plus the validation set). However, since the random forest model has significantly fewer parameters than, for example, the XGBoost model and is less sensitive to tuning, hyperparameter tuning is not necessary in the method 300 Figure 3 shown and discussed, but hyperparameter tuning can also be utilized.

[0036] At block 325, the lifetime value analysis module 116 can determine the long-term LTV (e.g., D180 LTV) of other user sets having the same or similar short-term LTV (e.g., D30 LTV) as the first user (calculated for the first user at block 310) to generate a distribution of possible long-term values for the first user. In an implementation, the determination made by the lifetime value analysis module 116 can be made in a transformed space to more easily generate a distribution (e.g., histogram) of possible long-term LTVs (e.g., D180 LTV) for the first user for which a prediction is to be made. Thus, the lifetime value analysis module 116 can utilize historical long-term LTV data of other user sets (which have the same or similar short-term LTV as the first user) received, collected, or otherwise retrieved to generate a distribution of possible long-term LTVs for the first user. In other words, the short-term LTV of the first user can be used to match the first user to the distribution of long-term LTVs of other users having the same or similar short-term LTV as the first user. According to an alternative implementation, the lifetime value analysis module 116 can predict the long-term LTV based on or otherwise utilize features received or collected from the short-term days of the first user, using, for example, a suitable regression model or the like. At block 330, the lifetime value analysis module 116 can utilize the long-term LTV distribution to determine the value of the predicted long-term LTV (e.g., D180 LTV) of the first user on a long-term day. For example, the lifetime value analysis module 116 can employ the mean, median, percentile, or other suitable measure of the long-term LTV distribution to determine the value of the long-term LTV of the first user. At block 335, the lifetime value analysis module 116 can transform or otherwise convert the predicted long-term LTV of the first user back from the transformed space. For example, the lifetime value analysis module 116 can utilize the inverse transform of the transform utilized in step 325 (e.g., the inverse transform of the Lambert W function or the like) to transform or otherwise convert the prediction back to the non-transformed (i.e., normal) space. For purposes of illustration and not limitation, for a prediction based on a monetary amount, the lifetime value analysis module 116 can reverse the transform to return the prediction to the "cash" space. For example, after the inverse transform, if the predicted value is in dollars (or other suitable currency) amount, the predicted long-term LTV of the first user can be the expected revenue that the first user can generate over the course of its lifetime in the client application. The lifetime value analysis module 116 can store the result in the user data database 120 for later retrieval and use.

[0037] Figure 4is a flowchart showing an example method 400 for predicting the long - term LTV of users of a client application by leveraging short - term LTV according to an embodiment of the present invention. In some embodiments of the present invention, method 400 may be performed, for example, by the lifetime value analysis module 116 using a random forest model or the like. In an embodiment, the lifetime value analysis module 116 may monitor a first user among a plurality of users of a client application executing on a client device of the first user. At block 405, the lifetime value analysis module 116 may select a short - term day from which to make a prediction and a long - term day to which to make a prediction for the first user. For illustrative purposes only and not by way of limitation, the lifetime value analysis module 116 may select D180 as the long - term day for the first user and D30 as the short - term day for the first user. Thus, in this example, the lifetime value analysis module 116 may predict the LTV on the 180th day after the initial installation of the client application or the first deposit in the client application based on data up to the 30th day (including the 30th day) after the first user initially installed the client application or made the first deposit in the client application, but any other suitable number of days or time periods or intervals may also be selected. At block 410, the lifetime value analysis module 116 may receive, collect, or otherwise retrieve suitable input features for the first user based on the selected short - term day. For example, the lifetime value analysis module 116 may collect or otherwise generate the short - term LTV of the first user (e.g., D30 LTV), the number of interactions with the client application (e.g., in the case of a digital game, the number of game sessions played, etc.), chat messages, or any other appropriate input features, depending on the type of the client application, the model being developed and used, etc. At block 415, the lifetime value analysis module 116 may receive, collect, or otherwise retrieve historical long - term LTV data for a set of other users for which the long - term LTV data is available. The set of other users may be used to train a machine - learning model (e.g., a random forest regression model or the like) and may include users other than the first user for whom the long - term LTV prediction is being made. In an embodiment, the lifetime value analysis module 116 may retrieve the long - term LTV data for the set of other users from the user data database 120. In an embodiment, the lifetime value analysis module 116 may remove any users from the long - term LTV data collected or retrieved that should not be considered in the set of other users (e.g., the user data is fraudulent, corrupted, disabled, or otherwise unavailable).

[0038] At block 420, the Lifetime Value Analysis Module 116 can transform or otherwise convert the collected historical long-term LTV data into a suitable transformed space. In an implementation, the Lambert W function can be utilized as the transformation so that it can be easier to process the long-term LTV data (e.g., LTV data with large or extreme distributions), but other suitable transformations can also be used. In such an implementation, the root mean square error of the model can be used in the transformed space as this can indicate how well the model performs across the entire dataset. At block 425, the Lifetime Value Analysis Module 116 can develop or otherwise create or construct a model (e.g., a random forest regression model, etc.) based on the transformed historical long-term LTV data. At block 430, the Lifetime Value Analysis Module 116 can provide the input features of the first user received or collected to the model to generate a distribution of the predicted long-term LTV (e.g., D180 LTV) of the first user at the selected long-term day. In an implementation, when the LTV data exhibits a large or extreme distribution, the model can be developed or otherwise constructed, and the Lifetime Value Analysis Module 116 can more easily generate the distribution in the transformed space. At block 435, the Lifetime Value Analysis Module 116 can transform or otherwise convert the distribution back from the transformed space. For example, the Lifetime Value Analysis Module 116 can utilize the inverse transformation of the transformation utilized in step 420 (e.g., the inverse of the Lambert W function, etc.) to convert the distribution back to the non-transformed (i.e., normal) space. At block 440, the Lifetime Value Analysis Module 116 can measure the distribution (in the non-transformed space) to generate a value of the predicted long-term LTV of the first user. For example, the Lifetime Value Analysis Module 116 can employ the mean, median, percentile, or other suitable measure of the long-term LTV distribution to determine the value of the long-term LTV of the first user. The Lifetime Value Analysis Module 116 can store the result in the User Data Database 120 for later retrieval and use.

[0039] Figure 5is a flowchart showing an example method 500 for predicting the long-term LTV of users of a client application by leveraging short-term LTV. In some embodiments of the present invention, method 500 may be performed, for example, by the lifecycle value analysis module 116 using a random forest model or the like. In an embodiment, the lifecycle value analysis module 116 may monitor a first user among a plurality of users of a client application executed on a client device of the first user. At block 505, the lifecycle value analysis module 116 may receive first user data characterizing the interaction of the first user with the client application executed on the client device of the first user, first time interval data characterizing a first time interval associated with the first user, and second time interval data characterizing a second time interval associated with the first user. The time length characterized by the first time interval may be shorter than the time length characterized by the second time interval. At block 510, the lifecycle value analysis module 116 may determine the first time interval LTV of the first user based on the received first user data and the received first time interval data. At block 515, the lifecycle value analysis module 116 may receive historical long-term LTV data of a plurality of users of the client application. Each of the plurality of users may have a corresponding first time interval LTV. At block 520, the lifecycle value analysis module 116 may transform or otherwise transform the received historical long-term LTV data into a transformed space. At block 525, the lifecycle value analysis module 116 may use the transformed historical long-term LTV data in the transformed space and the received second time interval data to determine the second time interval LTV of each of the plurality of users, to generate a distribution of the second time interval LTV. At block 530, the lifecycle value analysis module 116 may generate a prediction of the second time interval LTV of the first user in the transformed space by matching the first time interval LTV of the first user with the corresponding first time interval LTV of one of the plurality of users, based on the distribution. At block 535, the lifecycle value analysis module 116 may transform or otherwise transform back the prediction of the second time interval LTV of the first user from the transformed space. At block 540, when the prediction of the second time interval LTV of the first user meets (e.g., is greater than or equal to) a predetermined LTV threshold, the lifecycle value analysis module 116 may modify the information display of the first user within the client application.

[0040] In connection with Figure 3 、 Figure 4 and Figure 5In any or all of the embodiments shown and discussed, the Lifetime Value Analysis Module 116 can directly predict the long-term LTV (e.g., D180 LTV). In an alternative embodiment, the Lifetime Value Analysis Module 116 can predict the difference between the long-term LTV (e.g., D180 LTV) and the short-term LTV (e.g., D30 LTV), and then add back the short-term LTV (e.g., D30 LTV) at the end to generate the long-term LTV (e.g., D180 LTV). This alternative embodiment can be utilized when a large proportion of users have permanently churned in the client application before the short-term day (e.g., D30), such that the difference for such users is zero. Thus, for a random forest model, it may be easier to separate the churned users and assign them a zero value than to predict that their long-term LTV is the same as their short-term LTV.

[0041] According to an embodiment of the present invention, the predicted long-term LTV of a user can be used to determine what information (if any) can be displayed to the user within or associated with a client application that can be executed on the user's client device. In an embodiment, when the prediction of the user's long-term LTV is greater than or equal to a predetermined threshold or within certain predetermined thresholds (e.g., a predetermined dollar amount or an expected revenue amount), Figure 1 the Client Device Information Display Module 118 shown can be used to modify the information display to the user within or associated with the client application. Alternatively, when the prediction of the user's long-term LTV is less than the predetermined threshold or outside of certain predetermined thresholds (e.g., a predetermined dollar amount or an expected revenue amount), Figure 1 the Client Device Information Display Module 118 shown can be used to modify the information display to the user within or associated with the client application. The predetermined threshold can be any suitable amount or quantity. For example, if the predicted long-term LTV of a user is the expected revenue that the user can generate during the lifetime of the client application, the predetermined threshold can be an appropriate dollar amount or revenue amount (e.g., $10, $50, $100, or a similar dollar amount).

[0042] In some embodiments of the present invention, Figure 1The client device information display module 118 shown can update, customize, modify, or otherwise personalize the graphical user interface, features, and / or functionality of the client application in any suitable manner. In an embodiment, the display of such personalized information can be used to increase the engagement and involvement of users having a predicted long-term LTV that meets or exceeds one or more predetermined thresholds within the client application. In an alternative embodiment, the display of such information can be used to increase the engagement and involvement of users having a predicted long-term LTV that is below one or more predetermined thresholds (e.g., attempting to increase their predicted long-term LTV) within the client application. For example, the client device information display module 118 can personalize any or all aspects of the graphical display of the client application (e.g., one or more graphical elements of the client application, such as any aspect of the "look and feel" of the graphical interface displayed by the client application), the information displayed within the client application, the features and / or functionality of the client application, etc. For purposes of illustration only and not limitation, in the case of a digital game, the client device information display module 118 can personalize the graphical display of the digital game to display, for example, player incentives, special offers (e.g., limited-time offers or LTOs), advertisements, etc. to the player within or in the digital game. In this example, different player incentives, special offers, advertisements, etc. can be displayed to players in the digital game based on the player's predicted long-term LTV. Additionally or alternatively, the client device information display module 118 can display or otherwise present additional and / or alternative rewards, prizes, and / or gifts to the player in the associated gift store of the client application. For example, rewards, prizes, gifts, etc. can be presented to players in the digital game or otherwise in the gift store associated with the digital game having a predicted long-term LTV greater than a predetermined threshold or within certain predetermined thresholds that are different from the rewards, prizes, gifts, etc. that can be presented to another player in the digital game having a predicted long-term LTV less than a predetermined threshold or outside certain predetermined thresholds, and vice versa. In this way, the menu or list of rewards, prizes, and / or gifts displayed to the player in the associated gift store can be customized for players having a predicted long-term LTV relative to one or more predetermined thresholds. Additionally or alternatively, the client device information display module 118 can personalize graphical information displayed to the user outside of the client application, such as advertisements or offers displayed to the user on their client device outside of the client application. However, in an embodiment, if the prediction of the user's long-term LTV is less than a predetermined threshold, the client device information display module 118 may not display any information (or display less information) to the user.

[0043] As described above, suitable machine learning / artificial intelligence techniques can be used to dynamically predict a user's long-term LTV from short-term LTV. For example, one or more machine learning models can be trained based on historical revenue data for all users from the client application (e.g., as retrieved from the user data database 120). Then, one or more machine learning models can be used to dynamically predict the long-term LTV of a particular user using revenue data from the user population according to the methods described herein. The one or more machine learning models can be updated or otherwise adapted as the user's revenue data changes or otherwise evolves over time. In an alternative embodiment, any or all of the predetermined values discussed above can be dynamically selected. According to an alternative embodiment, the lifetime value analysis module 116 can use suitable machine learning / artificial intelligence techniques to dynamically select or otherwise choose appropriate values for any or all of the predetermined values and / or parameters discussed above. For example, one or more machine learning models can be trained based on data from either or both of the user data database 120 and the client application data database 122. Then, the one or more machine learning models can be used to dynamically select appropriate values for each or any of the above variables based on, for example, characteristics of the user (e.g., user revenue), characteristics of the client application (e.g., type of client application), and other similar characteristics or data. The one or more machine learning models can be updated or otherwise adapted as characteristics, results, and other similar data associated with the user and the client application change and evolve over time.

[0044] Accordingly, the embodiments shown and discussed above can use short-term data (e.g., D7, D30, or any other suitable time interval) to predict a user's long-term LTV (e.g., D180 or any other suitable time interval). In some embodiments of the present invention, based on the predicted long-term LTV, the present invention can use such short-term data to estimate the dollar amount or potential revenue that a user can generate over the lifetime of the client application. Accordingly, if the estimated potential revenue generated by the user is greater than or equal to a certain threshold or within a certain threshold, additional marketing campaigns and offers can be presented to the user within or associated with the client application, as enhanced marketing spending on the user is warranted. Note that some embodiments of the present invention do not have model parameters to be estimated, and in some embodiments, up to three or more hyperparameters can be used as an alternative method to make predictions. Accordingly, in some embodiments of the present invention, predictions can be made using appropriate CV / adjustments using the methods discussed and described in the present invention. Further note that the present invention can make predictions within a prediction range of a similar length to the training data.

[0045] Since the long-term LTV is predicted at the user level, in some embodiments of the present invention, individual users can be grouped or otherwise organized into cohorts in any suitable manner, such as relative to a client application or on different (but potentially related) client applications. Such grouping can support, for example, variations in experiments and tests that would not be possible if the long-term LTV were predicted at the cohort level. For example, individual users can be grouped based on any appropriate characteristics related to the user, the client application, or the client device on which the client application is executed, such as the user's short-term LTV, the user's long-term LTV, the user's location, the user's age, the user's gender, the type of client device (brand, model, operating system, etc.), the type of client application, the length of time the user has interacted with or otherwise participated in the client application, the progress the user has made in the client application, etc. Once the appropriate grouping has been done, suitable experiments and tests (e.g., testing new features, new offers, etc.) can be performed on the cohorts. Then, the users can be regrouped into different cohorts to support variations in the experiments and tests.

[0046] In some embodiments of the present invention, bootstrapping can be performed on a user. Bootstrapping is a statistical process that resamples a single data set to create many simulated samples. This process can allow for the calculation of standard errors, the construction of confidence intervals, and hypothesis testing on various types of sample statistics. In the bootstrapping method, a sample of size n is drawn from a population. The sample can be referred to as S. Then, instead of using theory to determine all possible estimates, a sampling distribution is created by resampling the observations m times with replacement, where each resampled set has n observations. Thus, if the sampling is done properly, S should represent the population. Therefore, by performing m resamplings with replacement on S, it is equivalent to drawing m samples from the original population, and the resulting estimates will represent the theoretical distribution under traditional methods. Increasing the number of resamplings m does not increase the amount of information in the data. That is, resampling the original set 100,000 times, for example, is no more useful than resampling it 1,000 times, for example. The amount of information in the data set depends on the sample size n, which will remain constant in each resampling. Then, the benefit of more resamplings is to obtain a better estimate of the sampling distribution. In some embodiments of the present invention, the lifetime value analysis module 116 can perform replacement sampling a predetermined number of times (e.g., 1,000, etc.) from a data set of long-term LTV predictions (e.g., stored in and retrieved from the user data database 120) for the same user to create a set of different long-term LTV predictions with the same or similar number (e.g., 1,000, etc.). Then, the lifetime value analysis module 116 can establish or otherwise construct a confidence interval (e.g., for a 95% confidence interval, from the 2.5th percentile to the 97.5th percentile, but other confidence intervals are possible) to provide stakeholders with more and additional insights into the user's long-term LTV.

[0047] Figure 6A graph showing an example extrapolation graph for predicting the long-term LTV of users of a client application by leveraging short-term LTV, in accordance with the present embodiment. In some embodiments of the present invention, the lifetime value analysis module 116 may generate and utilize a graph where, for example, "days" are represented on the x-axis and the predicted long-term LTV (e.g., in dollars or other currency) is represented on the y-axis. For illustrative purposes and not limitation, graph 600 may illustrate the long-term LTV value predictions determined for users of a client application in accordance with the techniques and methods discussed above. For example, a user may obtain a first predicted long-term LTV 602 of $10 on day 30. The user may obtain a second predicted long-term LTV 604 of $30 on day 60. The user may obtain a third predicted long-term LTV 606 of $40 on day 90. The user may obtain a fourth predicted long-term LTV 608 of $45 on day 180. Since graph 600 may include any suitable number of long-term LTV data points, additional and / or alternative long-term LTV data points and values in graph 600 are possible. In some embodiments of the present invention, one or more of the first predicted long-term LTV 602, second predicted long-term LTV 604, third predicted long-term LTV 606, and fourth predicted long-term LTV 608 may be determined based on the machine learning models and / or transformation techniques for predicting long-term LTV by leveraging short-term LTV discussed above. In an alternative embodiment, one or more of the first predicted long-term LTV 602, second predicted long-term LTV 604, third predicted long-term LTV 606, and fourth predicted long-term LTV 608 may be the actual (rather than predicted) long-term LTV at each data point. In another alternative embodiment, the first predicted long-term LTV 602, second predicted long-term LTV 604, third predicted long-term LTV 606, and fourth predicted long-term LTV 608 may be any suitable mixture or combination of predicted and actual long-term LTV data points.

[0048] In some embodiments of the present invention, the shape or slope 614 of the curve in graph 600 at the most recent prediction (in this example, the fourth predicted long-term LTV 608) can be used to extrapolate the long-term LTV prediction of a user for any suitable future time period. For purposes of illustration and not limitation, based on the slope 614 of the curve of graph 600 at the fourth predicted long-term LTV 608, the first extrapolated long-term LTV 610 at day 540 can be $50, and the second extrapolated long-term LTV 612 at day 1080 can be $55. Additional or alternative future extrapolations of the predicted long-term LTV are possible based on the slope 614 of the curve in graph 600. Additionally or alternatively, according to the techniques and methods discussed above, the extrapolated prediction of the long-term LTV can be used for new users of the client application (e.g., new users having short-term and / or long-term LTV profiles similar to or the same as those of the existing users of graph 600, or new users based on other suitable similarity characteristics between the new users and the existing users).

[0049] Figure 7 is a block diagram of an example computing device 700 that can perform one or more of the operations described herein. The computing device 700 can be connected to other computing devices in a LAN, intranet, extranet, and / or the Internet. The computing device 700 can operate as a server machine in a client-server network environment or as a client in a peer-to-peer network environment. The computing device 700 can be provided by a personal computer (PC), set-top box (STB), server, network router, switch or bridge, or any machine capable of executing a set of instructions (sequentially or otherwise) that specify actions to be taken by that machine. Moreover, although only a single computing device 700 is shown, the term "computing device" should also be regarded as including any collection of computing devices that individually or jointly execute one (or more) sets of instructions to perform the methods discussed herein.

[0050] Example computing device 700 may include: a computer processing device 702 (e.g., a general-purpose processor, ASIC, etc.), a main memory 704, a static memory 706 (e.g., flash memory, etc.), and a data storage device 708, which may communicate with each other via a bus 730. The computer processing device 702 may be provided by one or more general-purpose processing devices such as a microprocessor, a central processing unit, etc. In an illustrative example, the computer processing device 702 may include: a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets or a combination of instruction sets. The computer processing device 702 may also include one or more dedicated processing devices, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, etc. According to one or more aspects of the present invention, the computer processing device 702 may be configured to perform the operations described herein for performing the operations and steps discussed herein.

[0051] The computing device 700 may further include a network interface device 712, which may communicate with a network 714. According to one or more aspects of the present invention, the data storage device 708 may include a machine-readable storage medium 728, on which one or more instruction sets may be stored, e.g., instructions for performing the operations described herein. During execution by the computing device 700, the main memory 704, and the computer processing device 702, which also constitutes a computer-readable medium, the instructions 718 implementing the core logic instructions 726 may also be present in whole or at least in part in the main memory 704 and / or the computer processing device 702. The instructions may further be sent or received over the network 714 via the network interface device 712.

[0052] Although the machine-readable storage medium 728 is shown as a single medium in the illustrative example, the term "computer-readable storage medium" should be considered to include a single medium or multiple media (e.g., a centralized or distributed database and / or associated cache and server) storing one or more instruction sets. The term "computer-readable storage medium" should also be considered to include any medium capable of storing, encoding, or carrying an instruction set executable by a machine and causing the machine to perform the methods described herein. Accordingly, the term "computer-readable storage medium" should be considered to include, but not be limited to, solid-state memory, optical media, magnetic media, etc.

[0053] The subject matter described herein provides many technical advantages. For example, the server system 114 can be scaled to support concurrent long-term LTV determination for a larger number of individual users (e.g., hundreds of thousands, millions, tens of millions, or more users), thereby significantly improving computer resource allocation and processing efficiency. Thus, some embodiments of the present invention can improve the efficiency and processing performance of computer hardware resources (e.g., computer processing and memory) for determining the long-term LTV of individual users, and in particular provide significantly faster long-term LTV determination times for client applications with a larger number of users. For example, some embodiments of the present invention can more efficiently handle the determination of the long-term LTV of a larger number of users simultaneously. By increasing the speed and efficiency of long-term LTV determination for client applications with a larger number of users, computer hardware resources can be released more quickly and used for other tasks and processes, thereby significantly increasing computer resource utilization.

[0054] Additionally, some embodiments of the present invention can be used to more easily perform LTV prediction in cases where there is a large distribution or other extreme distribution of LTV data. Models trained using data that exhibits such extreme distributions may be biased by outliers, thereby compromising overall prediction accuracy. Embodiments of the present invention can make accurate predictions for a wide variety of individual users in a dataset (e.g., from smaller consumers who may contribute to extreme distributions of LTV data to larger consumers) without truncating values or biasing the model towards one group or another, thereby improving the processing performance and efficiency of the server system 114. Embodiments of the present invention can also allow individual users to be grouped in any way to provide granularity with respect to long-term LTV determination, which cannot be achieved when performing long-term LTV prediction at the group level, thereby further enhancing the functionality and processing of the server system 114. Furthermore, some embodiments of the present invention can allow testing of new features in client applications and how users respond to those new features. For example, when a new feature is introduced into a client application, if the corresponding predicted long-term LTV of a user exhibits an upward trend or other increase in long-term LTV, such determination can indicate a positive user response to the new feature. Conversely, if the corresponding predicted long-term LTV of a user exhibits a downward trend or other decrease in long-term LTV, such determination can indicate a negative user response to the new feature. By reducing or eliminating unwanted, undesired, unused, and / or unnecessary features and / or functions from the client application, such feature testing can support improved functionality, processing, and efficiency of the server system 114, the client device executing the client application, and the client application itself.

[0055] Implementations of the subject matter and the operations described in this disclosure can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this disclosure and their structural equivalents thereof, or in a combination of one or more of them. Implementations of the subject matter described in this disclosure can be implemented as one or more computer programs encoded on a computer storage medium for execution by, or to control the operation of, a data processing apparatus, i.e., one or more modules of computer program instructions. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a suitable receiver apparatus for execution by the data processing apparatus. A computer storage medium can be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them, or be included in them. Further, although a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. A computer storage medium can also be one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices), or be included in them.

[0056] The operations described in this disclosure can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0057] The term “data processing apparatus” encompasses all types of devices, apparatuses, and machines for processing data, e.g., including programmable processors, computer processing devices, computers, systems on a chip, or combinations of the foregoing. A computer processing device can include one or more processors, which can include dedicated logic circuitry, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC), a central processing unit (CPU), a multi-core processor, etc. In addition to hardware, the device can also include code that creates an execution environment for the computer programs being discussed, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The device and the execution environment can implement various different computing model infrastructures, such as network services, distributed computing, and grid computing infrastructures.

[0058] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative languages, procedural languages, or functional languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program can, but need not, correspond to a file in a file system. The program can be stored in a part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language resource), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, subroutines, or portions of code). A computer program can be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0059] The processes and logical flows described in this disclosure can be performed by one or more programmable processors that execute one or more computer programs to perform actions by operating on input data and generating output. The processes and logical flows can also be performed by, and the apparatus can also be implemented as, special purpose logic circuitry, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC).

[0060] For example, processors suitable for executing a computer program include general and special purpose microprocessors, and any one or more processors of any type of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for performing actions in accordance with the instructions and one or more memory devices for storing the instructions and data. Generally, a computer will also include one or more mass storage devices for storing data (e.g., magnetic disks, magneto-optical disks, optical disks, solid state drives, etc.), or operatively connected to receive data from or transfer data to, or both receive and transfer data from, the one or more mass storage devices. However, a computer need not have such devices. In addition, a computer can be embedded in another device, such as a smartphone, a mobile audio or media player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including semiconductor storage devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0061] To provide for interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a cathode ray tube (CRT), a liquid crystal display (LCD) monitor, a light emitting diode (LED) monitor, etc.) for displaying information to the user, and a keyboard and a pointing device (e.g., a mouse, a trackball, a touchpad, a stylus, etc.) by which the user may provide input to the computer. Other types of devices may also be used to provide for interaction with the user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic input, voice input, or tactile input. Other possible input devices include a touch screen or other touch-sensitive device, such as a single-point or multi-point resistive or capacitive touchpad, speech recognition hardware and software, an optical scanner, an optical pointer, a digital image capture device, and associated interpretation software, etc. Additionally, the computer may interact with the user by sending resources to and receiving resources from the device used by the user; for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0062] Embodiments of the subject matter described in this disclosure may be implemented in a computing system that includes backend components, such as a data server; or includes middleware components, such as an application server; or includes frontend components, such as a client computer having a graphical user interface or a web browser through which a user may interact with embodiments of the subject matter described in this disclosure; or any combination of one or more such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), the Internet (e.g., the Internet), a peer-to-peer network (e.g., an ad hoc peer-to-peer network), etc.

[0063] The computing system may include a client and a server. The client and the server are typically remote from each other and typically interact through a communication network. The relationship of the client and the server is generated by computer programs running on the respective computers and having a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to the client device (e.g., to display data to a user interacting with the client device and to receive user input from the user). Data generated at the client device (e.g., the result of a user interaction) may be received at the server from the client device.

[0064] A system of one or more computers can be configured to perform particular operations or actions by installing software, firmware, hardware, or a combination thereof on the system, which, when operating, cause or enable the system to perform the actions. One or more computer programs can be configured to perform particular operations or actions by including instructions that, when executed by a data processing apparatus, cause the apparatus to perform the actions.

[0065] References throughout this disclosure to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" throughout this disclosure are not necessarily all referring to the same embodiment. Additionally, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or".

[0066] Although this disclosure contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features that are described in the context of separate embodiments in this disclosure may also be implemented in combination within a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments. Additionally, although features may be described above as acting in certain combinations and even initially claimed as such, in some cases, one or more features from a claimed combination may be excluded from the combination, and the claimed combination may cover a sub-combination or a variation of a sub-combination.

[0067] Similarly, although operations and / or logical flows are depicted in the figures in a particular order and / or described herein in a particular order, this should not be understood as requiring that such operations and / or logical flows be performed in the particular order shown or in sequential order, or that all of the illustrated operations be performed, to achieve a desired result. In some cases, multitasking and parallel processing may be advantageous. Additionally, the separation of various system components in the embodiments described above should not be understood as required in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0068] Accordingly, particular implementations of the subject matter have been described. Other implementations are within the scope of the appended claims. In some cases, the acts recited in the claims can be performed in a different order and still achieve the desired results. Additionally, the processes depicted in the figures need not be in the particular order or sequential order shown to achieve the desired results. In certain implementations, multitasking and parallel processing may be advantageous.

[0069] The word "example" or "exemplary" is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as "example" or "exemplary" is not necessarily to be construed as superior or preferable to other aspects or designs. Rather, the use of the word "example" or "exemplary" is intended to present concepts in a concrete fashion. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X includes A or B" is intended to mean any of the natural inclusive permutations. That is, if X includes A; X includes B; or X includes both A and B, then "X includes A or B" is satisfied in any of the foregoing instances. Additionally, unless otherwise specified or clearly indicated by the context to the contrary, the articles "a" and "an" as used in this application and the appended claims generally should be construed to mean "one or more". Further, unless so described, the terms "implementation" or "an implementation" or "realization" or "a realization" used throughout are not intended to refer to the same implementation or realization. Moreover, as used herein, the terms "first", "second", "third", "fourth", etc. are labels used to distinguish between different elements and may not necessarily have an ordinal meaning in accordance with their numerical designation.

[0070] In the foregoing description and claims, phrases such as "at least one of" or "one or more of" may appear after a list of elements or features. The term "and / or" may also appear in a list of two or more elements or features. Unless implicitly or explicitly contradicted by the context in which the phrase is used otherwise, such phrases are intended to mean any of the individually listed elements or features, or any combination of any of the listed elements or features with any other of the listed elements or features. For example, the phrases "at least one of A and B"; "one or more of A and B"; and "A and / or B" each are intended to mean "A alone, B alone, or A and B together". Similar interpretations are intended for lists that include three or more items. For example, the phrases "at least one of A, B, and C"; "one or more of A, B, and C"; and "A, B, and / or C" each are intended to mean "A alone, B alone, C alone, A and B together, A and C together, B and C together, or A and B and C together". Additionally, the term "based on" as used above and in the claims is intended to mean "at least in part based on", such that unrecited features or elements are also permitted.

[0071] The foregoing description of the illustrated embodiments of the invention is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Although specific embodiments and examples of the invention are described herein for illustrative purposes, various equivalent modifications will be recognized by those skilled in the relevant art to be within the scope of the invention. The subject matter described herein can be embodied in a system, apparatus, method, and / or article, depending on the desired configuration. The embodiments set forth in the foregoing description do not represent all embodiments consistent with the subject matter described herein. Instead, they are merely some examples consistent with aspects of the subject matter described. Although some variations have been described in detail above, other modifications or additions are possible. In particular, further features and / or variations can be provided in addition to those features and / or variations set forth herein. For example, the embodiments described above can relate to various combinations and subcombinations of the disclosed features and / or combinations and subcombinations of multiple further features disclosed above. Other embodiments are within the scope of the appended claims.

Claims

1. A method, comprising: Receiving, by at least one data processor, first user data representing an interaction of a first user with a client application executing on a client device of the first user, first time interval data representing a first time interval associated with the first user, and second time interval data representing a second time interval associated with the first user, wherein the time length represented by the first time interval is shorter than the time length represented by the second time interval; Determining, by at least one data processor, a first time interval lifetime value (LTV) of the first user based on the received first user data and the received first time interval data; Receiving, by at least one data processor, historical long-term LTV data of a plurality of users of the client application, wherein each of the plurality of users has a corresponding first time interval LTV; Converting, by at least one data processor, the received historical long-term LTV data into a transformed space; Determining, by at least one data processor, a second time interval LTV of each of the plurality of users using the transformed historical long-term LTV data in the transformed space and the received second time interval data to generate a distribution of the second time interval LTV; Generating, by at least one data processor, a prediction of the second time interval LTV of the first user in the transformed space based on the distribution by matching the first time interval LTV of the first user with a corresponding first time interval LTV of one of the plurality of users; Converting, by at least one data processor, the prediction of the second time interval LTV of the first user back from the transformed space; and Modifying, by at least one data processor, the information display for the first user within the client application when the prediction of the second time interval LTV of the first user meets a predetermined LTV threshold.

2. The method according to claim 1, wherein, Generating a prediction of the second time interval LTV using a random forest model.

3. The method according to claim 2, further comprising: Training the random forest model using the received historical long-term LTV data.

4. The method according to claim 1, wherein, Converting the received historical long-term LTV data into the transformed space using the Lambert W function.

5. The method according to claim 1, further comprising: Determining, by at least one data processor, multiple predictions of the second time interval LTV of the first user.

6. The method according to claim 5, further comprising: Generating, by at least one data processor, graphical data representing multiple predictions of the second time interval LTV of the first user.

7. The method according to claim 1, further comprising: Generating, by at least one data processor, a random forest regression model in the transformed space and based on the transformed historical long-term LTV data.

8. The method according to claim 7, further comprising: Generating, by at least one data processor and using the random forest regression model generated in the transformed space, a distribution of the prediction of the second time interval LTV of the first user.

9. The method according to claim 1, further comprising: Sampling, by at least one data processor, the generated distribution of the prediction of the second time interval LTV of the first user; And Generating, by at least one data processor, a second distribution of the prediction of the second time interval LTV of the first user.

10. The method according to claim 9, further comprising: Determining, by at least one data processor and based on the second distribution of the prediction of the second time interval LTV of the first user, a confidence interval representing the generated distribution of the prediction of the second time interval LTV of the first user.

11. A system, comprising: At least one data processor; and a memory storing instructions which, when executed by the at least one data processor, cause the at least one data processor to perform operations including the following: Receiving, by the at least one data processor, first user data characterizing an interaction of a first user with a client application executing on a client device of the first user, first time interval data characterizing a first time interval associated with the first user, and second time interval data characterizing a second time interval associated with the first user, the first time interval having a shorter time length than the second time interval; Determining, by the at least one data processor, a first time interval lifetime value (LTV) of the first user based on the received first user data and the received first time interval data; Receiving, by the at least one data processor, historical long-term LTV data of a plurality of users of the client application, wherein each of the plurality of users has a corresponding first time interval LTV; Converting, by the at least one data processor, the received historical long-term LTV data into a transformed space; Determining, by the at least one data processor, a second time interval LTV of each of the plurality of users using the transformed historical long-term LTV data in the transformed space and the received second time interval data to generate a distribution of the second time interval LTV; Generating, by the at least one data processor, a prediction of the second time interval LTV of the first user in the transformed space based on the distribution by matching the first time interval LTV of the first user with a corresponding first time interval LTV of one of the plurality of users; Converting back, by the at least one data processor, the prediction of the second time interval LTV of the first user from the transformed space; and Modifying, by the at least one data processor, the display of information about the first user within the client application when the prediction of the second time interval LTV of the first user meets a predetermined LTV threshold.

12. The system according to claim 11, wherein, Using a random forest model to generate the prediction of the second time interval LTV.

13. The system according to claim 12, wherein, The operations further include: training the random forest model using the received historical long-term LTV data.

14. The system according to claim 11, wherein Converting the received historical long-term LTV data into the transformed space using the Lambert W function.

15. The system according to claim 11, wherein, The operations further include: determining, by the at least one data processor, multiple predictions of the second time interval LTV of the first user.

16. The system according to claim 15, wherein, The operations further include: generating, by the at least one data processor, graphical data characterizing the multiple predictions of the second time interval LTV of the first user.

17. The system according to claim 11, wherein, The operations further include: generating, by the at least one data processor and in the transformed space and based on the transformed historical long-term LTV data, a random forest regression model.

18. The system according to claim 17, wherein, The operations further include: generating, by the at least one data processor and using the random forest regression model generated in the transformed space, a distribution of the prediction of the second time interval LTV of the first user.

19. The system according to claim 11, wherein, The operations further include: Sampling, by the at least one data processor, the generated distribution of the prediction of the second time interval LTV of the first user; and A second distribution of a prediction of a second time interval LTV of a first user is generated by at least one data processor.

20. A non - volatile computer program product storing executable instructions which, when executed by at least one data processor forming part of at least one computing system, implement operations comprising: Receiving, by at least one data processor, first user data characterizing an interaction of a first user with a client application executing on a client device of the first user, first time interval data characterizing a first time interval associated with the first user, and second time interval data characterizing a second time interval associated with the first user, wherein the time length characterized by the first time interval is shorter than the time length characterized by the second time interval; Determining, by at least one data processor, a first time interval lifetime value (LTV) of the first user based on the received first user data and the received first time interval data; Receiving, by at least one data processor, historical long - term LTV data of a plurality of users of the client application, wherein each of the plurality of users has a corresponding first time interval LTV; Transforming, by at least one data processor, the received historical long - term LTV data into a transformed space; Determining, by at least one data processor, a second time interval LTV of each of the plurality of users using the transformed historical long - term LTV data in the transformed space and the received second time interval data to generate a distribution of the second time interval LTV; Generating, by at least one data processor, a prediction of a second time interval LTV of the first user in the transformed space by matching the first time interval LTV of the first user with a corresponding first time interval LTV of one of the plurality of users, based on the distribution; Transforming back, by at least one data processor, the prediction of the second time interval LTV of the first user from the transformed space; and Modifying, by at least one data processor, the display of information about the first user within the client application when the prediction of the second time interval LTV of the first user meets a predetermined LTV threshold.

Images