User behavior prediction modeled via generation of event sequences

Abstract

Systems and methods are provided for generating a customizable model for predicting user behavior. The method comprises: obtaining first structured event data representing a first event performed by one or more users; training a generative language model using the first structured event data; obtaining second structured event data representing one or more second events performed by the user; and predicting third structured event data representing a behavior associated with the user at least in part by applying the second structured event data as input to the trained generative language model according to the one or more customizable analysis outputs.

Description

Technical Field

[0001] The present disclosure relates to generative models, and more particularly, to generating language models that utilize non-human event language to predict user outcomes associated with events. Background Art

[0002] The background description provided herein is for the purpose of generally presenting the context of the present disclosure. The work of the presently named inventors (to the extent that it is described in this background section) and aspects of the specification that might not have been considered prior art at the time of filing are neither explicitly nor implicitly admitted to be prior art with respect to the present disclosure.

[0003] In various use cases, it is desirable to generate models that can predict user outcomes, including behaviors, actions, consequences, and / or other user-focused determinations (e.g., for digital advertising, content design, layout optimization, etc.). While some models have such predictive capabilities, they are generally not flexible because the models need to be retrained for specific desired outputs and / or time horizons.

[0004] For example, in digital advertising, advertisers may want to determine whether a user is likely to perform a specific action and / or trigger a specific event. In traditional systems, models can be trained to perform this analysis. However, these techniques require training a unique model for each timeframe of interest (e.g., a 7-day model, a 28-day model, etc.), each specific output of interest (e.g., a specific predicted user behavior), and so on. Summary of the Invention

[0005] In one example implementation, a method for generating a customizable model for predicting user behavior includes: (i) obtaining, by a computing system, first structured event data representing a first event performed by one or more users; (ii) training, by the computing system, a generative language model using the first structured event data; (iii) obtaining, by the computing system, second structured event data representing one or more second events performed by the users; and (iv) predicting, by the computing system, third structured event data representing behavior associated with the users based on one or more customizable analysis outputs and at least in part by applying the second structured event data as input to the trained generative language model.

[0006] In another example implementation, a computing system includes one or more processors and one or more non-transitory tangible memories storing instructions that, when executed by the one or more processors, cause the computing system to: (i) obtain first structured event data representing a first event performed by one or more users; (ii) train a generative language model using the first structured event data; (iii) obtain second structured event data representing one or more second events performed by the users; and (iv) predict third structured event data representing a behavior associated with the users based on one or more customizable analysis outputs at least in part by applying the second structured event data as input to the trained generative language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a block diagram of an example system in which techniques for efficiently generating and training models using an event language may be implemented.

[0008] Figure 2 A block diagram depicting an example of using a generative language model to train and output predictions.

[0009] Figure 3A Depicts an exemplary process for embedding input features into vectors in a dimensional space for use in comparing and determining relationships between events.

[0010] Figure 3B Describes an exemplary process for operating and / or training a sequence text generation model, where an output prediction is iteratively used as the next input value.

[0011] Figure 4 Describe how to Figure 1 An exemplary large language model implemented in the generation device.

[0012] Figure 5 is a flowchart of an example method for using an event language to train a model and generate predictions about user behavior. DETAILED DESCRIPTION

[0013] In the disclosed invention, a generative language model (e.g., a large language model (LLM)) is trained on event data tokens rather than human language. The event data is structured according to the grammatical rules of the non-human language, allowing the generative language model to interpret and predict the event data using techniques that are somewhat similar to how existing generative language models interpret and predict human language text. For example, the generative language model can treat different user experience flows in a manner similar to a collection of documents, treat a user profile consisting of multiple conversations in a manner similar to a single document, treat conversations in a manner similar to sentences or paragraphs (e.g., an ordered grouping of closely related events), and treat events in a manner similar to words or tokens. In some implementations, the disclosed invention uses long-range dependency training techniques to train the generative language model.

[0014] The disclosed invention can use data indicating specific events and possibly associated metadata (e.g., data indicating the relative timing of the events) to train a generative language model to generally understand the event data and predict further / subsequent event data in a customizable manner. For example, a generative language model can receive event data associated with a user (e.g., via Google Analytics) to generate a prediction about whether the user in question will exhibit a specific behavior (e.g., perform a specific action, such as interacting with a website or advertisement in a specific way) within a specific time window (e.g., the next hour, next day, next week, etc.), where both the behavior and the time window are customizable without having to retrain the model. Thus, the disclosed invention provides a flexible means that can be modified relatively quickly and easily (e.g., by advertisers or by analytics teams, etc.) without the delays and other costs associated with training new models or retraining existing models.

[0015] In particular, the disclosed invention avoids the need to retrain existing models for specific expected outputs or train completely new models for specific expected outputs by utilizing generative language models rather than conventional models to process structured event data. As mentioned herein, "structured event data" can refer to event data that is structured according to one or more grammatical rules of a non-human language. Because generative language models are typically used to process human languages, it is unintuitive to use them to process structured event data. However, advantageously, generative language models can be trained to understand the sequence of input events and predict / output a sequence of any future events with any length (i.e., the number of predicted events) without the need to conduct new training or retraining for the specific output. For example, although conventional models may need to be specially trained to predict whether event A will occur after a past event sequence (e.g., the probability of occurrence), generative language models can be trained more generally and can predict, for example, whether event A will occur, whether event B will occur, whether event C then event D will occur, etc. Accordingly, the predicted future events or event sequences provide a rich source of information that can be processed in any desired manner (e.g., using various known analytical techniques) to predict or calculate any specific outcome or metric. For example, if a generative language model predicts that a user who performed events A and B will next perform events X, Y, and Z, then post-processing can be specifically applied to determine that event XZ (i.e., the combination of X and Z) will likely occur, or to determine that event C will likely not occur, etc.

[0016] Event data can be structured and generated to include additional information, such as event times or relative timing between events, beyond the information included in human language that is conventionally used to train standard generative language models. By training a generative language model on the additional information included in the structure of the event data, the model can take advantage of additional dimensions of information that are not possible in conventional use of language models (i.e., when processing human language).

[0017] However, the differences between the structure of the event language used for event data and the conventional language used to train standard language models present unique challenges. In particular, (i) the length of the event sequence of event data may be different from the length of the training data for conventional language models, and (ii) the event structure and vocabulary cardinality of event data are different from the event structure and vocabulary cardinality of the training data for conventional language models. Training data for conventional language models may include documents of relevant human text, which may be or include hundreds or thousands of relevant word tokens. However, event data may be only a few event tokens long. For example, a user may load a news site, read an article, subscribe to a newsletter, and never return to the site. Therefore, compared to conventional systems, there may be less event data to train a language model with. Similarly, event data includes rich metadata, such as page URLs, parameters (e.g., purchase amounts), time elapsed since the previous event, type of computing device, etc. Training data for conventional language models does not include this rich metadata, and therefore training the model should take this into account.

[0018] As described above, these differences can present challenges. For example, the shortened sequence length reduces the amount of available context that the model can use in training, which is problematic because contextual token disambiguation is important for training many types of models (e.g., such as generative language models). In addition, deep event metadata structures can lead to errors in the vocabulary space. For example, a conventional English LLM may contain 30,000 unique subword tokens. For event data, multiplying the cardinality of each metadata dimension (e.g., URL, time gap, floating point parameter value, etc.) can result in a nearly infinite number of unique tokens, each of which may appear too rarely in the corpus of historical event data to be effectively learned. In addition, event metadata is not a context that can be discarded after analysis and / or training. Instead, the model can generate predictions for the event language output that include new metadata. This hierarchical generation is not part of the conventional model architecture.

[0019] The instant techniques described herein overcome such challenges by remedying and exploiting these differences in structure. For example, to address vocabulary cardinality, a computing system can generate embeddings of event metadata components (e.g., as described herein). The encoded feature vectors become part of language model training and prediction. Thus, by using the structure of event data as described herein to train a language model on the data, the dimensionality of each metadata component becomes more manageable for such a vocabulary.

[0020] As a further example, to address challenges related to short sequence lengths and to enable the language model to produce complete events with all appropriate metadata, the computing system can further exploit the structure of the metadata. Because the model has access to all contextual metadata tokens within a single event, encoding and representing each component separately allows for reasonable predictive power even for short sequences.

[0021] When generating output, some categories of metadata are easier to predict than others due to dimensionality differences. In particular, metadata has an implicit hierarchy. The computing system can split the generation of the output event into generating each segment of metadata in sequence, where later components leverage already predicted segments. For the event currently being generated, the output segments that are more difficult to predict (e.g., URL embeds or event names) can leverage previously made predictions for components that are easier to generate and have lower cardinality. In other words, the language model can avoid generating events and associated metadata at the same time. In some such implementations, the language model instead takes as input a previous, fully complete event token and a partially complete output token, with a subset of the metadata filled in. The computing system then iterates through all remaining metadata categories, adding a new prediction for one element of the metadata during each iteration.

[0022] In some implementations, the disclosed invention can additionally simulate theoretical scenarios to generate predicted scenario outcomes. For example, a computing system can present a trained generative language model with a scenario in which a user returns to a site 24 hours after the user's last known / actual visit to the site, and then use the generative language model to predict the likelihood that the user will purchase something within the next three days if the return visit actually occurs. Depending on the implementation, the simulated / theoretical scenario can be in the future (e.g., a future event that the user might perform) or a branching past path (e.g., a hypothetical change to past data).

[0023] Figure 1 An example system 100 is shown in which one or more techniques for generating customizable models for predicting user behavior can be implemented. Example system 100 includes a client device 102, a computing system 104, a publisher 106, an analysis server 108, and a network 110. In some implementations, computing system 104 is remote from client device 102 and / or analysis server 108 and is communicatively coupled to client device 102 and / or analysis server 108 via network 110. Figure 1The communication / network connections shown for publisher 106 in FIG. 1 represent communication / network connections with computing devices or systems associated with publisher 106. It should be understood that example system 100 is exemplary and that other systems may include additional, fewer, or alternative components. Similarly, the arrangement of the components of system 100 may be modified. For example, some elements of system 100 may be combined, split, swapped, etc.

[0024] The network 110 may be a single communication network (e.g., the Internet), and in some implementations may also include one or more additional networks. As just one specific example, the network 110 may include a cellular network, the Internet, and a server-side local area network (LAN). Figure 1 Only a single client device 102, publisher 106, and analytics server 108 are shown, but it should be understood that system 100 may include any suitable number of similar client devices, publishers, and / or content sponsors operating according to the principles disclosed herein.

[0025] Generally, client device 102 may access one or more information resources supplied or published by publisher 106, and computing system 104 generates and trains a machine learning model to receive event data about a user at client device 102, and subsequently generate predictions about the user's future behavior based on the data provided by client device 102.

[0026] In some implementations, the information resource may be a web page of a website hosted by publisher 106, and analytics server 108 may analyze and / or store event data associated with user interactions with the information resource (e.g., from client device 102). Alternatively, computing system 104 may include analytics server 108 and / or store historical data associated with users in addition to or in lieu of analytics server 108.

[0027] In some implementations and / or scenarios, the computing system 104 (or Figure 1 Another computing system (not shown in the figure) uses the techniques described herein to train a machine learning model. For example, the machine learning model can be a large language model (LLM) configured to receive event data in a non-human language for events. In some implementations, the event data can consist of one or more tokens indicating a specific interaction (e.g., an event) between a user and an information resource provided by the publisher 106.

[0028] As described herein, a non-human language can refer to a language that humans do not use for communication or that humans cannot naturally read. For example, a non-human language can describe events and the relative timing of events, and a non-human language can use events as individual tokens to represent interactions. Furthermore, events can be structured according to one or more grammatical rules (e.g., similar to grammatical rules used for human languages). For example, various events can be arranged in sequences or conversations, indicating events that occur within a single predetermined time period (e.g., within a single visit to a website, within a day, within an hour, etc.). Similarly, multiple conversations can be arranged within a user profile corresponding to a user over a period of time, which in turn can be one of a series of user profiles in an event data stream associated with a particular grouping that includes the user (e.g., based on location, device, age, etc.). Furthermore, a non-human language can include metadata for events (e.g., indicating event time signatures, relative times between events, device IDs associated with events, user IDs associated with events, locations associated with events, past user history associated with the user who caused the event, etc.). Similarly, a non-human language can include an indication of the event type as part of a grammatical rule.

[0029] In some implementations, the information resource with which the user interacts can be a screen / user interface / page of an application (e.g., a mobile app) provided by publisher 106 or another entity to client device 102 for installation, where the screen / user interface / page includes content slots to be populated (e.g., by computing system 104) with content (e.g., image content, video content, text content, etc.). As another example, the information resource can be a video played by a video player of client device 102, and the content slots can be distributed temporally throughout the video.

[0030] The client device 102 may be or include any fixed, mobile, or portable computing device with wired and / or wireless communication capabilities (e.g., a smartphone, tablet computer, laptop computer, desktop computer, smart wearable device (such as smart glasses or smart watch), vehicle head unit computer, etc.). Figure 1 In an example implementation of , the client device 102 includes a network interface 120, a processor 122, a memory 124, and a display 126. The processor 122 may be a single processor (e.g., a central processing unit (CPU)) or may include a set of processors (e.g., multiple CPUs, or one or more CPUs and one or more graphics processing units (GPUs)).

[0031] Memory 124 includes one or more computer-readable, non-transitory storage units or devices, which may include persistent memory components (e.g., a hard disk) and / or non-persistent memory components. Memory 124 stores instructions that can be executed by processing unit 122 to perform various operations, including instructions for various software applications and data generated and / or used by such applications. Figure 1 In an example implementation of , the memory 124 stores at least an application 130 , which may be, for example, a web browser application, a mobile application downloaded from an application store, or a video player application.

[0032] Generally, the application 130 is executed by the processor 122 to display the video via the display 126 (and possibly one or more speakers of the client device 102). Figure 1 106 ). In one embodiment, the information resource is presented to a user of the client device 102 by means of a web browser (not shown). In a further implementation, at least one of the information resources includes one or more spatial and / or temporal content slots for dynamically presenting video content, image content, text content, and / or any other such information resource. In an implementation in which the application 130 is a web browser application, for example, the information resource may be a web page hosted by the publisher 106, where the web browser causes the client device 102 to download the web page's hypertext markup language (HTML), scripts, and / or other code for presentation to the user via the display 126.

[0033] The display 126 includes hardware, firmware, and / or software configured to enable a user to view the visual output of the client device 102 and may use any suitable display technology (e.g., LED, OLED, LCD, etc.). In some implementations, the display 126 is incorporated into a touch screen having both display and manual input capabilities. Furthermore, in some implementations where the client device 102 is a wearable device, the display 126 is a transparent viewing component with integrated electronic components (e.g., the lens of smart glasses). For example, the display 126 may include micro-LED or OLED electronics embedded in the lens of smart glasses.

[0034] The network interface 120 includes hardware, firmware, and / or software configured to enable the client device 102 to exchange electronic data with the computing system 104 via the network 110. For example, the network interface 120 may include a cellular communication transceiver, a Wi-Fi transceiver, and / or a transceiver for one or more other wired and / or wireless communication technologies.

[0035] Although Figure 1 The client device 102 is shown as a single component that communicates directly with the computing system 104 (ie, via the network 110), but in some implementations, Figure 1The subcomponents of the client device 102 shown in FIG are alternatively divided between two or more user-side devices. As just one example, a pair of smart glasses may include a processor 122, a memory 124, and a display 126, while a smartphone may include another processing unit, another memory, another display, and a network interface 120. The smart glasses (or smart helmet, etc.) may then communicate with the smartphone (e.g., via Bluetooth) as needed to implement the operations described herein.

[0036] Computing system 104 includes a network interface 140, a processor 142, and a memory 144. Network interface 140 includes hardware, firmware, and / or software configured to enable computing system 104 to exchange electronic data with client device 102 and other similar client devices via network 110. For example, network interface 140 may include a wired or wireless router and modem. Processor 142 may be a single processor, may include two or more processors, etc. Computing system 104 may include, for example, one or more servers that may reside in a single location or in multiple locations.

[0037] Memory 144 is a computer-readable, non-transitory storage unit or device or collection of units / devices, which may include a persistent memory component and / or a non-persistent memory component. Memory 144 stores instructions for prediction generator 150, scenario generator 152, and training module 154, each of which is executable by processor 142. Prediction generator 150 includes prompt generator 160 and generates artificial intelligence (AI) and / or machine learning (ML) models 162. In some implementations, the instructions are omitted. Figure 1 Some of the software modules / units shown in . For example, the prediction generator may omit the hint generator 160, or the scenario generator 152 may be omitted entirely.

[0038] The prediction generator 150, scenario generator 152, and training module 154 are software modules that include instructions that are executed by the processor 142 to generate, train, configure, and / or otherwise facilitate the generation of an AI and / or ML model 162 (e.g., an LLM). The AI / ML model 162 is trained so that the AI / ML model 162 can read a customizable amount of data associated with a user and generate predictions for similarly customizable future time periods. For example, the computing system 104 can generate and / or train the AI / ML model 162 to read data associated with a user from the previous 3 days, the previous 7 days, the previous 14 days, the previous 28 days, the previous month, and the like. As another example, the computing system 104 can generate and / or train the AI / ML model 162 to predict user behavior within the next 3 days, the next 7 days, the next 14 days, the next 28 days, the next month, and the like.

[0039] In general, prediction generator 150 generates predictions about user behavior based on past behavior and using AI / ML models 162, as described herein. Furthermore, scenario generator 152 uses scenario data 164 (e.g., received from publisher 106, analytics server 108, client device 102, and / or another computing device (not shown)) to implement additional scenarios when generating possible predictions. For example, scenario generator 152 may receive scenario data 164 indicating a scenario in which a user returns to view an information resource 24 hours from the current time. Prediction generator 150 then adjusts the prediction accordingly.

[0040] In addition, the training module 154 uses training data 168 as described herein to train the AI ​​and / or ML model 162 (e.g., including the LLM). In some implementations, the training data is or includes data associated with the user (e.g., historical data in the historical data database 174). In further implementations, the training data is or includes data associated with other users (e.g., historical data in the historical data database 174). In still further implementations, the training data is or includes data provided by the publisher 106 (e.g., manually generated historical data).

[0041] The operation of prediction generator 150, scenario generator 152, and training module 154, and their components, are discussed in further detail below in conjunction with various example implementations.

[0042] exist Figure 1 In an example implementation of , the analysis server 108 is communicatively coupled to a historical data database 174. In some implementations, the computing system 104 includes and / or is communicatively coupled to a database (not shown) for storing the scenario data 164, the training data 168, and / or other related forms of data. Depending on the implementation, each of the databases (e.g., the historical data database 174 and / or the databases for the scenario data 164, the training data 168, etc.) can be stored in local memory (e.g., the memory 144) or can be stored in memory remote from the coupled device / system.

[0043] Historical data database 174 includes event data generated, collected, created, and / or otherwise stored, for example, by analysis server 108. In some implementations, analysis server 108 interfaces with publishers 106, computing systems 104, and / or client devices 102 to retrieve and / or generate event data. In other implementations, analysis server 108 provides remote access to computing system 104, and computing system 104 generates historical data and / or modifies the data before sending it to analysis server 108 for storage in historical data database 174. In still other implementations, historical data database 174 is stored / maintained by computing system 104 rather than analysis server 108. Thus, in some such implementations, computing system 104 may perform the functionality of analysis server 108.

[0044] In some implementations, publishers (including publisher 106) maintain accounts associated with services provided by computing system 104. For example, publishers may create such accounts in order to monetize information resources that they publish or otherwise make available (e.g., by selling advertisements in content slots on web pages hosted by the publisher). In these implementations, information associated with publisher accounts may be stored in an account database ( Figure 1 The account database may be stored in memory 144 or in one or more memories remote from computing system 104, for example. The account information may include information such as entity name, subscription level, entity preferences (e.g., brand control preferences), etc. In some implementations, the account information includes selection parameters (e.g., bid amounts or maximum bid amounts) associated with different content sponsors for use by computing system 104 or a different computing system in selecting content to include in a content slot of a publisher's information resource.

[0045] Figure 2 Describes a method for performing a task by training and / or using a machine learning model (e.g., Figure 1 Example process 200 for generating an output (such as an output prediction of user behavior) using an AI / ML model 162. For example, process 200 may be performed by Figure 1 144 when executing instructions stored in memory 144).

[0046] Model 210 (e.g., AI / ML model 162) (e.g., from Figure 1The historical data database 174 of the computing system 104 receives the historical event data 202. In some implementations, the model 210 receives the historical event data 202 via an application programming interface (API) for the model 210 (e.g., running on the computing system 104, the client device 102, the analysis server 108, etc.).

[0047] In some implementations, historical event data 202 is used to train model 210 (e.g., by computing system 104). In particular, historical event data 202 is structured to follow the grammatical rules of a non-human language (such as the event language described herein). In other words, computing system 104 is able to use historical event data 202 without converting the data into a human-readable language (e.g., English, French, Japanese, etc.), but rather analyzes the event data in the non-human language (e.g., in native form). Depending on the implementation, historical event data 202 can include labeled data (e.g., data with a label indicating the next event in a sequence, the overall conversation / sequence context, etc.), unlabeled data, and / or any other data representing past events. Thus, computing system 104 can train model 210 according to supervised techniques (e.g., using labeled data), unsupervised techniques (e.g., using unlabeled data), semi-supervised techniques, etc. The following will discuss the historical event data 202 with respect to FIG. 3 to FIG. Figure 4 B describes training the model 210 in more detail.

[0048] Although historical event data 202 is not converted into human-readable language, computing system 104 can use historical event data 202 to train model 210 in a manner similar to training a conventional LLM. For example, model 210 can treat different user experience flows in a manner similar to a collection of documents, treat a user profile including multiple conversations in a manner similar to a single document, treat conversations in a manner similar to sentences or paragraphs (e.g., an ordered grouping of closely related events), and treat events in a manner similar to words or tokens.

[0049] In addition, historical event data 202 includes metadata associated with the events. Metadata includes information that does not exist in human language, such as the relative timing of events (e.g., how long between events occurred), user identity (e.g., user account ID, user IP address (or an anonymous identifier serving as a substitute), user customer number, etc.), user location, an indication of the user's event history (e.g., past events associated with the user), one or more device characteristics of the user's device (e.g., client device 102), and / or other such metadata. Therefore, conventional models trained on human language data rarely or cannot utilize such information.

[0050] Depending on the implementation, model 210 may be or include a deep neural network, and more specifically, may be or include a large language model (LLM). The LLM may have a transformer model architecture with an encoder and a decoder, and may tokenize the input / text, as described below with respect to Figure 4 A and Figure 4 B. The transformer model can incorporate a self-attention mechanism to facilitate faster learning / training and / or more accurate output. In some implementations, the LLM includes multiple layers of a neural network, potentially including multiple embedding layers, multiple feedforward layers, and multiple recurrent layers. In some implementations, model 210 includes a generative model other than the LLM. For example, model 210 can instead include a less complex neural network.

[0051] Model 210 may have been trained by computing system 104 or another computing system using supervised or semi-supervised learning and utilizing training data of an appropriate modality (e.g., historical event data 202) (i.e., event data). Model 210 may be a general model (e.g., trained on a large dataset of publicly available event data) or may be a domain-specific model (e.g., trained on a custom and / or proprietary dataset of event data).

[0052] The difference between the format of the event language of historical event data 202 and the techniques used to train conventional models creates unique challenges for training model 210 on historical event data 202. Specifically, (i) the length of the event sequences of historical event data 202 may differ from the length of the training data for conventional models, and (ii) the event structure and vocabulary cardinality of historical event data 202 differ from the event structure and vocabulary cardinality of the training data for conventional models. Training data for conventional language models may include documents of relevant human text, which may be or include hundreds or thousands of relevant word tokens. However, historical event data 202 may be only a few event tokens long. For example, a user may load a news site, read an article, subscribe to a newsletter, and never return to the site. Therefore, compared to conventional systems, computing system 104 may have less event data with which to train model 210. Similarly, event data includes rich metadata such as page URLs, parameters (e.g., purchase amount), time elapsed since the previous event, type of computing device, etc. Training data for conventional systems does not include such rich metadata, and therefore training the model 210 should take this into account.

[0053] As described above, these differences can present challenges. For example, the shortened sequence length reduces the amount of available context that the model 210 can use in training, which is problematic because contextual token disambiguation is important for training many types of models (e.g., such as LLMs). In addition, the deep event metadata structure can lead to errors in the vocabulary space. For example, a conventional English LLM may contain 30,000 unique subword tokens. For event data, multiplying the cardinality of each metadata dimension (e.g., URL, time interval, floating point parameter value, etc.) can result in a nearly infinite number of unique tokens, each of which may appear too rarely in the corpus of historical event data to be effectively learned. In addition, event metadata is not a context that can be discarded after analysis and / or training. Instead, the model 210 can generate predictions for the event language output that include new metadata. This hierarchical generation is not part of the conventional language model architecture.

[0054] The instant techniques described herein overcome such challenges by remedying and exploiting these differences. For example, to address vocabulary cardinality, the computing system 104 may generate embeddings for event metadata components (e.g., as described below with respect to Figure 3A and Figure 3B The encoded feature vectors become part of the model training and prediction. Therefore, the dimensionality of each metadata component can become more manageable for such a vocabulary.

[0055] As a further example, to address challenges with short sequence lengths and to enable the model 210 to generate complete events with all appropriate metadata, the computing system 104 can exploit the structure of the metadata. Since the model has access to all context-related metadata tokens within a single event, encoding and representing each component separately allows for reasonable predictive power even for short sequences.

[0056] When generating output, some categories of metadata are easier to predict than others due to dimensionality differences. In particular, metadata has an implicit hierarchy. The computing system 104 can split the generation of output events into each segment of the sequentially generated data, where later components leverage already predicted segments. For the event currently being generated, the output segments that are more difficult to predict (e.g., URL embeds or event names) can leverage previously made predictions for components that are easier to generate and have lower cardinality. In other words, the model 210 can avoid generating events and associated metadata at the same time. In some such implementations, the model 210 instead takes as input a previous, fully complete event token and a partially complete output token, with a subset of the metadata filled in. The computing system 104 then iterates through all remaining metadata categories, adding a new prediction for one element of the metadata during each iteration.

[0057] The computing system 104 provides model-based generation of event sequences by using embeddings to adjust the vocabulary space and iteratively generate constituent segments of event metadata.

[0058] In some implementations, the model 210 is an AI / ML model 162 stored at the computing system 104 (e.g., Figure 1 In further implementations, model 210 is another model stored on a remote server and / or other component of system 100 (not shown) but accessed by computing system 104. For example, model 210 may be an LLM that a remote server makes available to computing systems (including computing system 104) via an application programming interface (API).

[0059] The model 210 additionally receives user data 204 and / or user metadata 206. Depending on the implementation, user data 204 is data associated with a particular user (e.g., using the client device 102). User metadata 206 may include metadata as described above with respect to the metadata for the historical event data 202. User data 204 and user metadata 206 are similarly in the same non-human language and follow the same grammatical structure as the historical event data 202.

[0060] The model 210 then uses the user data 204 and the user metadata 206 to perform various tasks regarding the user, such as generating output predictions 220 (e.g., including output scenarios). For example, the model 210 may utilize the user data 204 to determine output predictions 220 regarding the user's behavior in the window over a specific period of time. The output of the model 210 may be customized based on customization parameters 208 received at the model 210. Depending on the implementation, the customization parameters 208 may indicate the type of prediction or other operation (e.g., predicting the likelihood of a user performing a specified event or action, predicting the likelihood of a user triggering an event, determining the number of expected users within a time period, determining a specific revenue forecast associated with a user, etc.), the window in which the user is predicted to perform an action (e.g., 3 days, 7 days, 28 days, a month, etc.), and / or any other such parameters.

[0061] In traditional systems, a computing system can generate a model to perform this analysis. However, by generating models according to conventional techniques, a unique model must typically be generated and trained for each timeframe (e.g., 7-day model, 28-day model, etc.), metric (e.g., predicted user behavior, branching scenarios in user history, etc.), and / or other desired such results. By training model 210 on event data as described herein, computing system 104 can generate, train, and otherwise provide a single model with customizable capabilities.

[0062] In some implementations, computing system 104 additionally simulates theoretical scenarios to generate output scenarios as part of output 220. In particular, model 210 uses scenario data 215 to modify, enhance, replace, and / or otherwise adjust user data 204 used for model 210 to generate output scenarios. For example, computing system 104 may provide scenario data 215 representing a scenario in which a user returns to a site 24 hours after the user's last known or actual visit to the site, and computing system 104 (e.g., via model 210) may then predict an output scenario representing the likelihood that the user will purchase something within the next three days if the return visit actually occurs. Depending on the implementation, the output scenario may be based on scenario data 215 representing the future (e.g., future events that the user may perform) or a branched past path (e.g., a hypothetical change to past data).

[0063] Figure 3A Depicts a process 300A for embedding input data when training a machine learning model, according to one implementation. Depending on the implementation, for example, the process 300A may be performed by Figure 1 144 when executing instructions stored in memory 144).

[0064] To generate an embedding for input data, computing system 104 receives one or more input features 335 (e.g., historical data and / or user data as described herein). Computing system 104 then generates one or more vector encodings 330 representing the input features. Computing system 104 generates a D-dimensional (e.g., 2D, 3D, 4D, etc.) space indicating similarities between input features 335 and vector encodings 330. Computing system 104 plots vector encodings 330 in the D-dimensional space to generate embedding 320. In some implementations, computing system 104 also generates encodings for additional features 340 of event data. For example, additional features 340 may include metadata associated with one or more input features 335 (e.g., location of a device associated with the user, time of event, account ID associated with the user, time of day, time of week, observable device information (e.g., mobile or desktop, browser, specific device type, device language, etc.), source of the user (search engine, direct link, ad interaction, etc.), demographic information of the user, rank of the user (e.g., in a loyalty rewards program), membership start date of the user, lifetime value of the user, past behavior of the user, device type of device, screen size of the device, browser type of the device, location of the device (e.g., country or region), URL of the webpage, page title or screen name of the webpage, content on the webpage (e.g., products shown), known metadata about the product (e.g., price or category), type of business associated with the publisher, name or type of event, time of the event (e.g., day, week, month, year), parameters associated with the event, and / or any other such features). In other implementations, the computing system avoids generating encodings for additional features 340. Depending on the implementation, embedding 320 represents one or more long-range dependencies of tokens in the event language. The computing system 104 can utilize the embeddings 320 (and, by extension, the long-range dependencies) to train the model, as described herein with respect to Figure 4 A and Figure 4 As described in B.

[0065] In some implementations, the computing system 104 further generates a log probability layer 310 that represents a probability distribution for the likelihood of various next outcomes (e.g., the next event determined in the sequence). The computing system 104 determines a probability output 305 based on the embedding layer. In further implementations, the computing system 104 determines the probability output 305 using one or more labels 325 associated with the input features to teach the model to embed the input data.

[0066] Figure 3BDepicts an exemplary process 300B for operating and / or training a sequence text generation model, wherein an output prediction is iteratively used as the next input value. In particular, model 350 receives an initial input / output value (e.g., I / O 360) and performs a prediction 352 to generate I / O 362. I / O 362 is fed into model 350, and another prediction 354 is performed to generate I / O 364. The process is iteratively repeated, wherein each I / O is fed into model 350 to perform a prediction (e.g., prediction 352, 354, 356, 358, etc.) and the output is then used as an I / O value (e.g., I / O 362, 364, 366, 368, etc.) as input to model 350. In this way, the model is able to iteratively predict larger strings of events, as described herein. Depending on the implementation, the prediction can be a multinomial distribution process and / or another classification process, as described below with respect to Figure 4 As stated.

[0067] Depending on the implementation, model 350 can be or include a neural network, such as an artificial neural network (ANN), a convolutional neural network (CNN), or a recurrent neural network (RNN). In further implementations, model 350 can incorporate, include, be, and / or otherwise use techniques in a manner reminiscent of language model techniques (e.g., large language models (LLMs), bag-of-words models, etc.). For example, model 350 can utilize events such that model 350 treats different user experience flows in a manner similar to a collection of documents, treats a user profile comprising multiple conversations in a manner similar to a single document, treats conversations in a manner similar to sentences or paragraphs (e.g., an ordered grouping of closely related events), and treats events in a manner similar to words or tokens. Similarly, model 350 can incorporate, include, be, and / or otherwise use a transformer architecture to utilize appropriate language model techniques, as described below with respect to Figure 4 As stated.

[0068] Figure 4 An exemplary large language model 400 trained as a generative model as described herein is shown. In particular, a generator model 410 receives an input vector 405 and outputs an output 460. In some implementations, the input vector 405 can be a random vector of fixed length. In further implementations, the input vector 405 can be randomly drawn from a Gaussian distribution so that a point in the vector space corresponding to the input vector 405 can correspond to a point in the problem domain representing the data distribution (e.g., as described with respect to Figure 3A and Figure 3B Depending on the implementation, the vector space corresponding to the input vector 405 may include one or more hidden variables (e.g., variables that cannot be directly observed). In some implementations, the input vector 405 can be used to provide a seed for the generation process.

[0069] In some implementations, the generated AI and / or ML model can be based on an LLM that is trained to predict events in a sequence of events. For example, the LLM can be trained to predict the next event following a given sequence of events. For example, the generated AI and / or ML model can be given the following sequence: "(i) visit webpage; (ii) wait 3 days; (iii) visit webpage; and (iv) purchase an item from the webpage," and the generated AI and / or ML model can predict the next event, such as "return to webpage within 3 days," "purchase another item," "do not return to webpage," etc.

[0070] Advantageously, some implementations use transformers to train the model (e.g., by using a generative pre-trained transformer (GPT) model). More specifically, some implementations use a GPT model that includes (i) an encoder that processes an input sequence and (ii) a decoder that generates an output sequence. Both the encoder and the decoder can include a multi-head self-attention mechanism that allows the GPT model to weight parts of the input sequence differently to infer meaning and context (e.g., using metadata in history and / or training data).

[0071] In particular, the input vector 405 can be a vector representing the relationship between events, sequences, etc. in the input. The large language training module 450 may include a self-attention block 452 component (e.g., referred to as a multi-self-attention block, a multi-head attention block, a multi-head self-attention block, a masked multi-self-attention block, a masked multi-head self-attention block, etc.) for simultaneously or nearly simultaneously focusing on different parts of the input to capture the relationship and / or dependency between different parts of the input. In particular, the self-attention block 452 correlates different positions of the sequence to calculate a representation of the sequence. Therefore, the self-attention block 452 can weigh the influence of different events in the sequence when sorting. Therefore, the model 400 learns to emphasize different parts of the input vector 405. In some implementations, the model 400 uses metadata related to the event at and / or at the position of the self-attention block 452 to determine the influence of the event within the sequence and / or the relationship between the events.

[0072] The self-attention block 452 can then calculate an attention score representing the influence of each word in the sentence on other words in the sentence (e.g., by taking the dot product between different sets of vectors). The output then proceeds to the normalization layer 454. The normalization layer 454 can normalize the output of the self-attention block 452 (e.g., by applying a softmax function to normalize the scores).

[0073] Similarly, the self-attention block can then be output to the feedforward network block 456, which performs nonlinear transformations to generate new representations of the input and / or relationships between events, sequences, etc. In particular, the feedforward network block 456 can use the calculated and normalized attention scores to calculate the weighted sum of vectors to capture the contextual relationships between words. In some implementations, the normalization layer 454 and / or the self-attention block 452 can perform calculations to generate representations of relationships between words, etc. After the feedforward network block 456, an additional normalization layer 458 can normalize the corresponding output and / or add residual connections to allow the output to move directly to another input. Thus, the model 400 can learn which parts of the input are important (e.g., still prevalent during the normalization process). Depending on the implementation, the model 400 can repeat the process used for the large language training module 450 1 time, 5 times, 10 times, N times, etc. to train the corresponding model.

[0074] Depending on the implementation, the encoder and / or decoder may be trained as described above. In a further implementation, the encoder is trained as described above, and the decoder includes an additional self-attention block (not shown) that also receives the output of the encoder.

[0075] Furthermore, in some implementations, rather than performing the previous four steps only once, the GPT model iterates these steps and performs them in parallel; at each iteration, a new linear projection of the query, key, and value vectors is generated. This iterative, parallel implementation advantageously improves the ability to capture secondary meanings and more complex relationships within the input sequence data.

[0076] Further advantageously, some implementations use supervised, unsupervised, and / or semi-supervised techniques to train and / or tune the model. For example, a model can be trained using labels associated with events to allow the model to identify the "correct" answer in a pattern for a sequence (e.g., for supervised training). Further, the computing system 104 can train the model using reward techniques, reinforcement techniques, and / or any other such techniques adapted to the event data described herein.

[0077] Figure 5 is a flow chart of an example method 500 for generating a customizable model for predicting user behavior. The method 500 may be implemented as instructions stored on one or more computer-readable media and executed by one or more processors in one or more computing devices. For example, the method 500 may be executed by Figure 1The method 500 is implemented by the processor 142 of the computing system 104 in the example embodiment when executing instructions of the prediction generator 150, the scenario generator 152, the training module 154, and / or any other module as described herein. As a further example, the method 500 can be implemented by one or more processors of a computing device communicatively coupled to the computing system 104. It should be understood that any such implementation is exemplary and that the example method 500 can be implemented using additional, fewer, and / or alternative components.

[0078] At box 502 of method 500, the computing system 104 obtains first structured event data representing one or more first events. Additionally, in an implementation of method 500, the first structured event data is in a non-human language and is structured according to the grammatical rules of the non-human language. In some implementations, the first structured event data includes historical event data (e.g., historical event data 202 and / or historical data in the historical data database 174) (e.g., training data) used to train the machine learning model. The first structured event data follows the grammatical rules of the non-human event-focused language and is not translated into a human language (e.g., English, French, Japanese, etc.). Figure 5 In an exemplary implementation of , grammar rules comprise the overall structure for a language. For example, various events may be arranged in sequences or sessions, indicating events that occur within a single predetermined time period (e.g., within a single visit to a website, within a day, within an hour, etc.). Similarly, multiple sessions may be arranged within a user profile corresponding to a user over a period of time, which in turn may be one of a series of user profiles in an event data stream associated with a particular grouping that includes the user (e.g., location-based, device-based, age-based, etc.).

[0079] Depending on the implementation, the first structured event data may be or include data representing one or more interactions with an application, a view of a webpage, a click on a login page for a domain, an interaction with a purchase button, entry of personal information (e.g., credit card information, address information, email information, etc.), receipt of a notification, opening of a notification, and / or any other such interaction with material from a publisher 106, an application developer (not shown), a computing system 104, etc. In further implementations, the first structured event data includes metadata associated with the one or more events. For example, the first structured event data may include relative times of events (e.g., how long between events occurred), a user identity (e.g., a user account ID, a user IP address, a user customer number, etc.), a user location, an indication of a user's event history (e.g., past events associated with the user), one or more device characteristics of a user device (e.g., client device 102), and / or other such metadata.

[0080] At block 504, computing system 104 trains a generative language model (e.g., AI / ML model 162 and / or model 210) using the first structured event data. As described above with respect to block 502, the first structured event data is in a non-human language for events. Thus, computing system 104 trains a generative language model to read and analyze data in a non-human language. More specifically, computing system 104 trains a generative language model on an event sequence (e.g., from an analysis server such as Figure 1 The language model is trained on the analysis server 108 of the first structured event data. In implementations where the first structured event data includes metadata, the computing system 104 additionally uses the metadata to train the language model. For example, the computing system 104 can train the language model based on the metadata to understand the relative time between events and determine whether user behavior is related to the relative time between events.

[0081] Thus, in some implementations, the computing system 104 trains a single model to enable it to directly answer a wide variety of prediction questions without having to adapt and fine-tune it for a specific task. For example, the computing system 104 can feed the generative language model with an event stream of a new type of user and prompt the model to generate the most likely events for a given window (e.g., 3 days, 7 days, 28 days, etc.) based on the input. The computing system 104 can then search the generated stream for key future events generated by the model (e.g., whether the user will convert (e.g., purchase an item), what the user's lifetime value will be, whether the user will churn (stop interacting with the content), etc.).

[0082] In a further alternative implementation, the computing system 104 initially trains a generative language model and fine-tunes the generative language model to perform various tasks and / or predict various operations without having to train a completely new model. For example, the computing system 104 trains a large generative language model with respect to an initial goal (e.g., next event prediction). After completing the base model, the computing system 104 fine-tunes individual task-oriented models (e.g., the probability of converting (e.g., purchasing an item) within the next week) using the encoded output of the base model as input. Thus, the computing system 104 can develop new task-specific models faster than by generating each model separately. In a further implementation, the computing system 104 trains a single model to perform various tasks and fine-tunes the task-specific models based on publisher data sharing settings (e.g., whether the publisher (e.g., publisher 106) does not agree to share data with other publishers).

[0083] At block 506, computing system 104 obtains second structured event data representing one or more second events performed by the user. The second structured event data utilizes the same non-human event language as the first structured event data and is therefore structured according to similar grammatical rules. Further, depending on the implementation, the second structured event data includes information similar to the first structured event data but targeted at a specific user. In further implementations, the second structured event data includes event data for a user group that includes the user (e.g., users accessing a webpage from the same IP address, users accessing an application on the same device, users purchasing items that meet an age range, location, or other demographic information, etc.).

[0084] At block 508, computing system 104 uses the generative language model to predict third structured event data representing behavior associated with the user based on one or more customizable analysis outputs. Similar to the first structured event data and the second structured event data, the third structured event data is in a non-human language and is structured according to grammatical rules based on the non-human language. Therefore, the generative language model does not convert the third structured event data into a human language (such as English, French, Japanese, etc.) for analysis. The third structured event data may be or include predicted user behavior, predicted outcomes for hypothetical scenarios, predicted outcomes for altered historical scenarios (e.g., where computing system 104 adds, modifies, replaces, or otherwise adjusts past event data), etc. In some implementations where the second structured event data includes stream data (e.g., stream data for multiple users) from at least one client device representing one or more sessions of user events, predicting the user behavior includes extracting user events for a specific user from the stream data (e.g., via a user ID, one or more user characteristics, event data ID, etc.). Computing system 104 then uses the trained generative language model to predict at least one future event based on the extracted user events. In some implementations, predicting at least one future event includes determining, by a trained generative language model, a likelihood of occurrence of the at least one future event to occur.

[0085] In some implementations, one or more customizable analysis outputs are or include various metrics for customizing the output analysis. For example, the customizable analysis outputs may be or include a time window for analysis, a specific type of analysis (e.g., detecting the likelihood of user retention, predicting future purchases and / or revenue forecasts associated with a user, determining the potential for a user to refer others to a site, etc.), user identity (e.g., when the second structured event data includes data for a stream of user profiles), and / or any other such metrics. In further implementations, some of the metrics and analyses are determined and performed at a post-processing step (e.g., as described below), and the customizable analysis output is a stop word for indicating where generation of the third structured event data will end.

[0086] The second structured event data (e.g., including user data) includes an initial set of events to prompt the generative language model to generate third structured event data. In some implementations, the second structured event data continues to be automatically fed into the model, causing the generative language model to generate the third structured event data in real time or near real time. In further implementations, the model receives the second structured event data in discrete packets and generates the third structured event data in response to receiving the packets and / or receiving instructions from publishers and / or other content owners. The model can generate third structured event data of any length until a customizable point (e.g., 3 days after expected behavior, 7 days after expected behavior, 28 days after expected behavior, etc.) is reached, indicated by stop words and / or other such instructions to stop generating data in the second structured event data. The computing system 104 or analysis server 108 can perform post-processing steps to adjust the time window in which relevant data will be displayed and / or used to answer the prompt. In some implementations, the post-processing step may be or include pre-existing and / or conventional analysis techniques (e.g., computing system 104 or analysis server 108 may operate on the predicted event sequence using techniques used for conventional historical event data). In further implementations, the post-processing step may be or include analysis techniques customized for the third structured event data and / or a combination of customized and conventional techniques. As an example, computing system 104 may generate predictions for user behavior over a period of one month or longer, but the time window may be three days. In some implementations, the window is based on multiple events, whether a particular event occurred, a period of time, and / or any other such window.

[0087] In some implementations, the computing system 104 uses context data to insert, replace, modify, and / or otherwise adjust event data. Depending on the implementation, the context data can be theoretical and forward-looking, or branched and retrospective. For example, the context data can be or include a theoretical scenario in which the computing system 104 determines the likelihood of a user purchasing an item if the user visits a webpage sometime within the next three days. As another example, the context data can be or include a branched scenario in which the computing system 104 determines the likelihood of a user purchasing an item if the user interacted with an advertisement two days ago. Thus, the computing system 104 can dynamically ask and answer any questions related to user data without having to train a new model. Depending on the implementation, the computing system 104 can receive prompts from publishers (e.g., analytics server 108) and / or other owners of content displayed to users (e.g., webpages, applications, media, etc.) and generate event data representing the event to be inserted into the generative language model as context data.

[0088] In some implementations, computing system 104 uses the same trained generative language model to determine the alternative outputs without modifying the model itself. In particular, computing system 104 can predict fourth event data representing behavior associated with the user (e.g., different from the third structured event data) based on a second set of customizable analysis outputs that is different from the first set of customizable analysis outputs, at least in part by applying the second structured event data as input to the trained generative language model. Depending on the implementation, the second set of customizable analysis outputs can be of a different type (e.g., a time window compared to the type of analysis) or the same type.

[0089] In some implementations, the computing system 104 receives one or more event or indicator prompts from publishers and / or other owners of content displayed to users (e.g., web pages, applications, media, etc.) for use in predicting user behavior or some other outcome. In further implementations, the computing system 104 includes an additional model to convert input from publishers and / or other owners of content into event prompts and / or indicators. Similarly, the computing system 104 may include a separate model to convert the third structured event data into a human language (e.g., English, French, Japanese, etc.) in a post-processing step.

[0090] In a further implementation, computing system 104 generates a confidence score representing the potential accuracy of the third structured event data. Depending on the implementation, computing system 104 may generate the confidence score based on similarity to historical event data (e.g., the first structured event data used to train the generated language model). For example, computing system 104 may determine a full confidence score (e.g., 100% or close to 100%) if the prediction occurs every time and has occurred multiple times in the historical event data, and determine a confidence score of only 50% if the prediction occurs 50% of the time. Similarly, computing system 104 may generate confidence scores in other ways based on other factors (such as, frequency of occurrence in historical event data, similarity between user data and historical event data, user's past knowledge, etc.).

[0091] In some implementations, and as described above, the technology disclosed herein uses artificial intelligence to facilitate the efficient generation of streams of event data (e.g., representing predicted outcomes). Artificial intelligence (AI) is a branch of computer science that focuses on creating models that can perform tasks with little to no human intervention. For example, artificial intelligence systems can utilize machine learning and computer vision. Machine learning and its subsets (such as deep learning) focus on developing models that can infer outputs from data. Outputs can include, for example, predictions and / or classifications. Computer vision focuses on analyzing and interpreting images and videos. Artificial intelligence systems can include generative models that generate new content in response to input prompts and / or based on other information.

[0092] Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine learning models can utilize attention mechanisms, such as self-attention. For example, some machine learning models can include multi-head self-attention models (e.g., transformer models).

[0093] The model can be trained using various training or learning techniques. Training can implement supervised learning, unsupervised learning, reinforcement learning, etc. Training can use techniques such as, for example, back propagation of errors. For example, a loss function can be back propagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions can be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques can be used to iteratively update parameters over multiple training iterations. Various generalization techniques (e.g., weight decay, dropout) can be used to improve the generalization ability of the model being trained.

[0094] The model can be pre-trained before domain-specific alignment. For example, the model can be pre-trained on a general training data corpus and fine-tuned on a more targeted training data corpus. The model can be aligned using prompts designed to elicit domain-specific outputs. The prompts can be designed to include learned prompt values ​​(e.g., soft prompts). The trained model can be validated using input data other than the training data before its use, and can be further updated or refined based on additional feedback / input during its use.

[0095] In some implementations, the computing system 104 can use any one or more of the above-described machine learning models to perform any one or more of the operations discussed herein in conjunction with machine learning. For example, the computing system 104 can use one or more of these machine learning models to generate a stream of event data, as discussed above.

[0096] Although the foregoing text sets forth a detailed description of many different aspects and implementations of the invention, it should be understood that the scope of this patent is defined by the words of the claims set forth at the end of this patent.The detailed description should be construed as exemplary only.

[0097] The following additional considerations apply to the foregoing discussion. Throughout this specification, multiple instances can implement components, operations, or structures described as single instances. Although the individual operations of one or more methods are shown and described as separate operations, one or more of the individual operations can be performed simultaneously, and it is not required to perform the operations in the order shown. The structure and function presented as separate components in the example configuration can be implemented as a combined structure or component. Similarly, the structure and function presented as a single component can be implemented as a separate component. These and other variations, modifications, additions, and improvements all fall within the scope of the subject matter of this disclosure.

[0098] Unless expressly stated otherwise, discussions in this disclosure using words such as "process," "calculate," "calculate," "determine," "present," "display," etc. may refer to the action or process of a machine (e.g., a computer) manipulating or transforming data represented as physical (e.g., electronic, magnetic, or optical) quantities within one or more memories (e.g., volatile memory, non-volatile memory, or a combination thereof), registers, or other machine components that receive, store, transmit, or display information.

[0099] As used in this disclosure, any reference to "one implementation" or "an implementation" means that a particular element, feature, structure, or characteristic described in connection with the implementation is included in at least one implementation or implementations. The appearance of the phrase "in one implementation" in various places in this specification does not necessarily refer to the same implementation.

[0100] As used in this disclosure, the terms "comprises," "comprising," "includes," "including," "has," "having," or any other variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Further, unless expressly stated to the contrary, "or" refers to an inclusive or and not an exclusive or. For example, a condition A or B is satisfied by any of: A being true (or present) and B being false (or absent), A being false (or absent) and B being true (or present), and both A and B being true (or present).

[0101] After reading this disclosure, those skilled in the art will understand, through the principles described herein, additional alternative structural and functional designs for predicting user behavior through customizable event modeling and using non-human language. Therefore, although specific implementations and applications have been shown and described, it should be understood that the disclosed implementations are not limited to the precise structures and components disclosed in this disclosure. Various modifications, changes and variations may be made to the arrangement, operation and details of the methods and apparatus disclosed in this disclosure without departing from the spirit and scope defined in the appended claims, and the modifications, changes and variations will be apparent to those skilled in the art.

Claims

1. A method for predicting user behavior using customizable event modeling, the method comprising: Obtaining, by a computing system, first structured event data representing a first event performed by one or more users; The computing system uses the first structured event data to train a language model; obtaining, by the computing system, second structured event data representing one or more second events performed by the user; as well as Third structured event data representative of behavior associated with the user is predicted by the computing system based on one or more customizable analysis outputs and at least in part by applying the second structured event data as input to a trained generative language model. 2 . The method of claim 1 , wherein the second structured event data comprises metadata associated with the one or more second events, and wherein predicting the behavior associated with the user is further based on the metadata.

3. The method of claim 2, wherein the one or more second events include a plurality of events, and wherein the metadata includes metadata representing a relative timing of a first event in the plurality of events compared to a second event in the plurality of events.

4. A method as claimed in claim 2 or 3, wherein the metadata includes at least one of: (i) user identity; (ii) user location; (iii) an indication of user event history; or (iv) one or more device characteristics.

5. The method according to any one of claims 2 to 4, wherein predicting the third structured event data comprises: One or more embeddings of features associated with the one or more second events and the metadata are encoded by the computing system.

6. A method as claimed in claim 5, wherein each of the one or more embeddings represents a corresponding one of the one or more second events and all metadata in the metadata associated with the corresponding one of the one or more second events.

7. The method according to any one of claims 2 to 6, wherein predicting the third structured event data comprises: generating, by the computing system, a first portion of the third structured event data based on the metadata; as well as A second portion of the third structured event data is generated by the computing system based on the first portion of the third structured event data.

8. The method of any one of the preceding claims, wherein training the generative language model comprises training the generative language model based on one or more long-range dependencies.

9. The method of any one of the preceding claims, wherein the second structured event data comprises stream data representing one or more sessions of user events for the user, and predicting the behavior associated with the user comprises: extracting the user event from the stream data; as well as At least one future event is predicted based on the extracted user events by the trained generative language model.

10. The method according to any one of the preceding claims, further comprising: applying an indication of a theoretical future event as part of said input to the trained generative language model; Wherein predicting the behavior associated with the user is further based on the theoretical future event.

11. The method of any one of the preceding claims, wherein the second structured event data comprises historical event data associated with the user.

12. The method of any preceding claim, wherein the second structured event data comprises theoretical branch event data associated with the user.

13. A method as described in any of the preceding claims, wherein the first event and the one or more second events include at least one of the following: (i) a web page view; (ii) a login page click; (iii) a click associated with an interactive element of a web page; (iv) a click associated with an interactive element of an application; (v) receiving a notification; or (vi) opening a notification.

14. The method of any one of the preceding claims, wherein the one or more customizable analysis outputs is a first set of customizable analysis outputs, and wherein the method further comprises: Predicting, by the computing system, fourth event data representing behavior associated with the user based on a second set of customizable analysis outputs different from the first set and at least in part by applying the second structured event data as input to the generative language model, wherein the fourth event data is different from the third structured event data.

15. A computing system comprising: One or more processors, and a memory storing instructions causing the computing system to perform the method of any preceding claim.

Images