Predicting user behavior through generative modeling of event sequences

Abstract

Systems and methods are provided for generating customizable models for predicting user behavior, including obtaining first structured event data representing first events performed by one or more users, training a generative language model using the first structured event data, obtaining second structured event data representing one or more second events performed by the users, and predicting third structured event data representing behavior associated with the users according to one or more customizable analysis outputs, at least in part by applying the second structured event data as input to the trained generative language model.

Description

[Technical Field]

[0001] The present disclosure relates to generating models, and more particularly, to generating language models that utilize non-human event language to predict user outcomes associated with events. [Background technology]

[0002] The background discussion provided herein is intended to broadly present the context for the disclosure. To the extent provided in this Background section, the work of the inventor(s) named herein, and aspects of the present disclosure that may not otherwise qualify as prior art at the time of filing, are not admitted expressly or impliedly as prior art to the present disclosure.

[0003] In various use cases, it is desirable to generate models that can predict user outcomes, including behavior, actions, results, and / or other user-focused decisions (e.g., for digital advertising, content design, layout optimization, etc.) While some models have such predictive capabilities, they are generally inflexible in that the models require retraining for specific desired outputs and / or timeframes.

[0004] For example, in digital advertising, advertisers may want to determine whether a user is likely to perform a particular action and / or trigger a particular event. In conventional systems, models can be trained to perform such analyses. However, these techniques require that a unique model be trained for each time frame of interest (e.g., a 7-day model, a 28-day model, etc.), for each particular output of interest (e.g., a particular predicted user behavior), etc. Summary of the Invention

[0005] In one exemplary embodiment, a method for generating a customizable model for predicting user behavior includes: (i) obtaining, by a computing system, first structured event data representing first events performed by one or more users; (ii) training, by the computing system, a generative language model using the first structured event data; (iii) obtaining, by the computing system, second structured event data representing one or more second events performed by the users; and (iv) predicting, by the computing system, third structured event data representing behavior associated with the users according to one or more customizable analysis outputs, at least in part by applying the second structured event data as input to the trained generative language model.

[0006] In another exemplary implementation, a computing system includes one or more processors and one or more non-transitory tangible memories that store instructions that, when executed by the one or more processors, cause the computing system to (i) obtain first structured event data representing first events performed by one or more users, (ii) train a generative language model using the first structured event data, (iii) obtain second structured event data representing one or more second events performed by the users, and (iv) predict third structured event data representing behavior associated with the users according to one or more customizable analysis outputs, at least in part by applying the second structured event data as input to the trained generative language model. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram of an example system in which techniques for efficiently generating and training models using an event language may be implemented. [Figure 2] 1 illustrates an exemplary block diagram for training and outputting predictions using a generative language model. [Figure 3A] 1 illustrates an exemplary process for embedding input features into vectors in a dimensional space to compare and determine relationships between events. [Figure 3B] 1 illustrates an exemplary process for running and / or training a sequential text generation model in which output predictions are iteratively used as the next input value. [Figure 4] 1 illustrates an exemplary large-scale language model implemented on a generating device as described with respect to FIG. [Figure 5] FIG. 1 is a flow diagram of an example method for using an event language to train a model to generate predictions about user behavior. DETAILED DESCRIPTION OF THE INVENTION

[0008] In the disclosed invention, a generative language model (e.g., a large-scale language model (LLM)) is trained on event data tokens rather than human language. The event data is structured according to the grammatical rules of a non-human language, allowing the generative language model to interpret and predict the event data using techniques similar in some respects to how existing generative language models interpret and predict human language text. For example, a generative language model may process a stream of characteristic user experiences in a manner similar to a collection of documents, process a user profile containing multiple sessions in a manner similar to a single document, process a session (e.g., an ordered grouping of closely related events) in a manner similar to a sentence or paragraph, and process an event in a manner similar to a word or token. In some embodiments, the disclosed invention uses long-distance dependency training techniques to train the generative language model.

[0009] The disclosed invention can train a generative language model to use data indicative of specific events, and possibly associated metadata (e.g., data indicative of the relative timing of the events), to understand the event data generally and predict additional / subsequent event data in a customizable manner. For example, the generative language model can receive event data associated with a user (e.g., via Google Analytics) to generate a prediction regarding whether that user will exhibit a specific behavior (e.g., whether they will take a specific action, such as interacting with a website or advertisement in a specific way) within a specific time window (e.g., the next hour, the next day, the next week, etc.), both the behavior and the time window being customizable without retraining the model. Thus, the disclosed invention provides a flexible approach that can be relatively quickly and easily modified (e.g., by an advertiser, an analytics team, etc.) without incurring the delays and additional costs associated with training a new model or retraining an existing model.

[0010] Specifically, the disclosed invention utilizes a generative language model rather than a conventional model to process structured event data, thereby avoiding the need to retrain an existing model for a specific desired output or the need to train an entirely new model for a specific desired output. As referred to herein, "structured event data" may refer to event data structured according to one or more grammatical rules of a non-human language. Because generative language models are typically used to process human languages, their use to process structured event data is counterintuitive. Advantageously, however, a generative language model can be trained to understand a sequence of input events and predict / output any future sequence of events of any length (i.e., number of predicted events), without requiring new training or retraining for that specific output. For example, while a conventional model may need to be specifically trained to predict whether event A will occur (e.g., its likelihood) after a series of past events, a generative language model can be trained more generally, for example, to predict whether event A will occur, whether event B will occur, whether event C will be followed by event D, etc. The predicted future events or event sequences then provide a rich source of information that can be processed in any desired manner (e.g., using various known analytical techniques) to predict or calculate any particular outcome or metric. For example, if a generative language model predicts that a user who performed events A and B will next perform events X, Y, and Z, post-processing can be applied to determine, among other things, that event XZ (i.e., the combination of X and Z) is likely to occur, or that event C is likely not to occur, etc.

[0011] Event data can be structured and generated to contain additional information beyond the information contained in human language traditionally used to train standard generative language models, such as event times or relative timing between events. By training a generative language model with the additional information contained in the structure of the event data, the model can take advantage of additional dimensions of information not possible with traditional uses of language models (i.e., when processing human language).

[0012] However, structural differences between the event language for event data and traditional languages ​​for training standard language models pose unique challenges. Specifically, (i) the length of the event sequences in the event data may differ from the length of the training data for traditional language models, and (ii) the event structure and lexical cardinality of the event data may differ from those of the training data for traditional language models. Training data for traditional language models may include related human text documents, which may be or contain hundreds or thousands of related word tokens. However, event data may be only a few event tokens long. For example, a user may load a news site, read an article, subscribe to a newsletter, and never return to the site. Therefore, compared to traditional systems, there may be less event data for training a language model. Similarly, event data includes rich metadata such as page URLs, parameters (e.g., purchase amount), time elapsed since a previous event, and type of computing device. Training data for traditional language models does not include such rich metadata, and thus training the model must take such things into account.

[0013] As noted above, these differences can present challenges. For example, as sequence length decreases, the amount of available context a model can use in training decreases. This is problematic because contextual token disambiguation is important for training many types of models (e.g., generative language models). Furthermore, deep event metadata structures can introduce errors in vocabulary space. For example, a traditional English LLM may contain 30,000 unique subword tokens. For event data, multiplying the cardinality of each metadata dimension (e.g., URL, time gap, floating-point parameter value, etc.) can result in a nearly infinite number of unique tokens, each of which may occur too infrequently in a corpus of historical event data for effective learning. Furthermore, event metadata is not context that can be discarded after analysis and / or training. Instead, a model can generate predictions that include new metadata for the event linguistic output. This hierarchical generation is not part of traditional model architectures.

[0014] The techniques described herein overcome such challenges by repairing and exploiting these structural differences. For example, to address vocabulary cardinality, a computing system can generate embeddings of event metadata components (e.g., as described herein). The encoded feature vectors become part of what a language model trains and predicts. Thus, training a language model on data using the structure of event data described herein allows the dimensionality of each metadata component to become more tractable for such vocabulary.

[0015] As a further example, to address challenges related to short sequence lengths and enable language models to generate complete events with all the appropriate metadata, computing systems can further utilize metadata structures. By encoding and representing each component separately, the model has access to all context-relevant metadata tokens within a single event, allowing for reasonable predictive power even for short sequences.

[0016] When generating output, some categories of metadata are easier to predict than others due to differences in dimensionality. Specifically, metadata has an implicit hierarchy. The computing system divides the generation of output events by generating each portion of the metadata in turn, with later components being able to leverage already predicted portions. Output portions that are more difficult to predict (e.g., URL embedding or event name) can leverage earlier predictions of easier-to-generate, lower-cardinality components for the currently generated event. In other words, the language model can avoid generating the event and associated metadata all at once. In some such implementations, the language model instead takes as input both the previous fully complete event token and the partially complete output token, with a subset of the metadata now populated. The computing system then iterates through all remaining metadata categories, adding a new prediction for one element of the metadata during each iteration.

[0017] In some embodiments, the disclosed invention can further simulate theoretical scenarios to generate predicted scenario outcomes. For example, a computing system can propose to a trained generative language model a scenario in which a user returns to a site 24 hours after their last known / actual visit, and then use the generative language model to predict the likelihood that the user will purchase something within the next three days if that return visit actually occurs. Depending on the embodiment, the simulated / theoretical scenario can be future (e.g., future events that the user could take) or can be diverging historical paths (e.g., hypothetical changes to past data).

[0018] FIG. 1 illustrates an exemplary system 100 in which one or more techniques for generating customizable models for predicting user behavior may be implemented. The exemplary system 100 includes a client device 102, a computing system 104, a publisher 106, an analytics server 108, and a network 110. In some implementations, the computing system 104 is remote from the client device 102 and / or the analytics server 108 and is communicatively coupled to the client device 102 and / or the analytics server 108 via the network 110. The communication / network connections illustrated in FIG. 1 for the publisher 106 represent communication / network connections with computing devices or systems associated with the publisher 106. It will be understood that the exemplary system 100 is illustrative and that another system may include additional, fewer, or alternative components. Similarly, the arrangement of the components of the system 100 may be modified. For example, some elements of the system 100 may be combined, divided, swapped, etc.

[0019] Network 110 may be a single communications network (e.g., the Internet) or, in some implementations, may also include one or more additional networks. As just one specific example, network 110 may include a cellular network, the Internet, and a server-side local area network (LAN). While Figure 1 shows only a single client device 102, publisher 106, and analytics server 108, it will be understood that system 100 may include any suitable number of similar client devices, publishers, and / or content sponsors operating according to the principles disclosed herein.

[0020] In general, the client device 102 can access one or more information resources provided or published by the publisher 106, and the computing system 104 receives event data about the user at the client device 102 and then generates and trains a machine learning model to generate predictions about the user's future behavior according to the data provided by the client device 102.

[0021] In some implementations, the information resources may be web pages of a website hosted by the publisher 106, and the analytics server 108 may analyze and / or store event data (e.g., from the client device 102) associated with user interactions with the information resources. Alternatively, the computing system 104 may include the analytics server 108 and / or store historical data associated with users in addition to or instead of the analytics server 108.

[0022] In some embodiments and / or scenarios, computing system 104 (or other computing systems not shown in FIG. 1 ) trains a machine learning model using techniques as described herein. For example, the machine learning model may be a large-scale language model (LLM) configured to receive event data in a non-human language directed to events. In some embodiments, the event data may consist of one or more tokens that indicate a particular interaction (e.g., an event) by a user with an information resource provided by publisher 106.

[0023] As described herein, a non-human language may refer to a language not used for human communication or that cannot be naturally read by humans. For example, the non-human language may describe events and their relative timing, and the non-human language may use events as individual tokens representing interactions. Furthermore, the events may be structured according to one or more grammatical rules (e.g., similar to the grammatical rules of a human language). For example, various events may be arranged in a sequence or session, indicating events occurring in a single, predetermined time period (e.g., a single website access, a day, an hour, etc.). Similarly, multiple sessions may be arranged into a user profile corresponding to a user over a period of time, and the user profile may then be one of a series of user profiles in a stream of event data associated with a particular grouping that includes users (e.g., based on location, device, age, etc.). Furthermore, the non-human language may include metadata for the events (e.g., indicating a time signature of the events, relative times between events, a device ID associated with the events, a user ID associated with the events, a location associated with the events, past user history associated with users that caused the events, etc.). Similarly, non-human languages ​​may include indications of event types as part of their grammar rules.

[0024] In some implementations, the information resource with which the user interacts may be a screen / user interface / page of an application (e.g., a mobile app) provided for installation by a publisher 106 or other entity to the client device 102, where the screen / user interface / page includes content slots that are populated (e.g., by the computing system 104) with content (e.g., image content, video content, text content, etc.). As another example, the information resource may be a video played by a video player on the client device 102, where the content slots may be distributed in time throughout the video.

[0025] Client device 102 may be or include any fixed, mobile, or portable computing device with wired and / or wireless communication capabilities (e.g., a smartphone, a tablet computer, a laptop computer, a desktop computer, a smart wearable device such as smart glasses or a smart watch, a vehicle head unit computer, etc.). In the exemplary implementation of FIG. 1, client device 102 includes a network interface 120, a processor 122, a memory 124, and a display 126. Processor 122 may be a single processor (e.g., a central processing unit (CPU)) or may include a set of processors (e.g., multiple CPUs, or one or more CPUs and one or more graphics processing units (GPUs)).

[0026] Memory 124 includes one or more computer-readable, non-transitory storage units or devices, which may include persistent (e.g., hard disk) and / or non-persistent memory components. Memory 124 stores instructions executable by processing unit 122 to perform various operations, including instructions for various software applications and data generated and / or used by such applications. In the exemplary implementation of FIG. 1, memory 124 stores at least application 130, which may be, for example, a web browser application, a mobile application downloaded from an application store, or a video player application.

[0027] Generally, application 130 is executed by processor 122 and presents information resources to a user of client device 102 via display 126 (and possibly one or more speakers of client device 102, not shown in FIG. 1 ). In further embodiments, at least one of the information resources includes one or more spatial and / or temporal content slots for dynamically presenting video content, image content, text content, and / or any other such information resource. In embodiments in which application 130 is a web browsing application, for example, the information resource may be a web page hosted by publisher 106, and the web browser causes client device 102 to download hypertext markup language (HTML), scripts, and / or other code for the web page for presentation to the user via display 126.

[0028] Display 126 includes hardware, firmware, and / or software configured to allow a user to view the visual output of client device 102 and may use any suitable display technology (e.g., LED, OLED, LCD, etc.). In some implementations, display 126 is incorporated into a touchscreen that has both display and manual input capabilities. Furthermore, in some implementations in which client device 102 is a wearable device, display 126 is a transparent viewing component (e.g., a smart glass lens) with integrated electronic components. For example, display 126 may include microLED or OLED electronics embedded in a smart glass lens.

[0029] Network interface 120 includes hardware, firmware, and / or software configured to enable client device 102 to exchange electronic data with computing system 104 over network 110. For example, network interface 120 may include a cellular communication transceiver, a WiFi transceiver, and / or a transceiver for one or more other wired and / or wireless communication technologies.

[0030] 1 depicts client device 102 as a single component that communicates directly with computing system 104 (i.e., via network 110), in some implementations, the subcomponents of client device 102 depicted in FIG. 1 are instead split into two or more user-side devices. As just one example, smart glasses may include processor 122, memory 124, and display 126, while a smartphone may include other processing units, other memories, other displays, and network interface 120. The smart glasses (or smart helmet, etc.) may then communicate with the smartphone (e.g., via Bluetooth) as needed to enable the operations described herein.

[0031] Computing system 104 includes a network interface 140, a processor 142, and memory 144. Network interface 140 includes hardware, firmware, and / or software configured to enable computing system 104 to exchange electronic data with other client devices, similar to client device 102, over network 110. For example, network interface 140 may include a wired or wireless router and a modem. Processor 142 may be a single processor or may include two or more processors. Computing system 104 may include, for example, one or more servers, which may reside in a single location or in multiple locations.

[0032] Memory 144 is a computer-readable, non-transitory storage unit or device, or collection of units / devices, which may include persistent and / or non-persistent memory components. Memory 144 stores instructions for prediction generator 150, scenario generator 152, and training module 154, each of which may be executed by processor 142. Prediction generator 150 includes prompt generator 160 and generative artificial intelligence (AI) and / or machine learning (ML) model 162. In some implementations, some of the software modules / units shown in FIG. 1 are omitted. For example, the prediction generator may omit prompt generator 160, or scenario generator 152 may be omitted entirely.

[0033] The forecast generator 150, the scenario generator 152, and the training module 154 are software modules that include instructions executed by the processor 142 to train, configure, and / or otherwise facilitate the generation of an AI and / or ML model 162 (e.g., an LLM). The AI / ML model 162 is trained to enable the AI / ML model 162 to read a customizable amount of data associated with a user and generate predictions for a similarly customizable future time period. For example, the computing system 104 may generate and / or train the AI / ML model 162 to read data associated with a user from the past three days, the past seven days, the past 14 days, the past 28 days, the last month, etc. As another example, the computing system 104 may generate and / or train the AI / ML model 162 to predict user behavior over the next three days, the next seven days, the next 14 days, the next 28 days, the next month, etc.

[0034] Generally, prediction generator 150 generates predictions about user behavior based on past behavior using AI / ML models 162, as described herein. Additionally, scenario generator 152 uses scenario data 164 (e.g., received from publisher 106, analytics server 108, client device 102, and / or other computing devices (not shown)) to implement additional scenarios in generating potential predictions. For example, scenario generator 152 can receive scenario data 164 indicating a scenario in which a user returns to view an information resource 24 hours from the current time. Prediction generator 150 then modifies the predictions accordingly.

[0035] Additionally, training module 154 uses training data 168 to train AI and / or ML models 162 (e.g., including LLMs) as described herein. In some embodiments, the training data is or includes data associated with a user (e.g., historical data in historical data database 174). In further embodiments, the training data is or includes data associated with another user (e.g., historical data in historical data database 174). In still further embodiments, the training data is or includes data provided by publisher 106 (e.g., artificially generated historical data).

[0036] The operation of the forecast generator 150, scenario generator 152, and training module 154, as well as their component parts, are described in further detail below in connection with various exemplary implementations.

[0037] 1 , analytic server 108 is communicatively coupled to historical data database 174. In some embodiments, computing system 104 includes and / or is communicatively coupled to databases (not shown) for storing scenario data 164, training data 168, and / or another related type of data. Depending on the embodiment, each of the databases (e.g., historical data database 174 and / or databases for scenario data 164, training data 168, etc.) may be stored in local memory (e.g., memory 144) or in memory remote from the coupled device / system.

[0038] The historical data database 174 includes, for example, event data generated, collected, created, and / or otherwise stored by the analytic server 108. In some implementations, the analytic server 108 interfaces with the publishers 106, the computing systems 104, and / or the client devices 102 to retrieve and / or generate the event data. In another implementation, the analytic server 108 provides remote access to the computing systems 104, which generate and / or modify the historical data before sending the data to the analytic server 108 for storage in the historical data database 174. In yet another implementation, the historical data database 174 is stored / maintained by the computing systems 104 rather than the analytic server 108. Thus, in some such implementations, the computing systems 104 may perform the functions of the analytic server 108.

[0039] In some implementations, publishers (including publisher 106) maintain accounts associated with services provided by computing system 104. For example, a publisher may create such an account in order to monetize information resources that it publishes or otherwise makes available (e.g., by selling advertisements in content slots on the publisher's hosted web pages). In these implementations, information associated with a publisher account may be stored in an account database (not shown in FIG. 1 ). The account database may be stored in memory 144 or, for example, in one or more memories remote from computing system 104. The account information may include information such as entity name, subscription level, entity preferences (e.g., brand control preferences), etc. In some implementations, the account information includes selection parameters (e.g., bid amount or maximum bid amount) associated with different content sponsors that are used by computing system 104 or different computing systems in selecting content for inclusion in content slots of the publisher's information resource.

[0040] 2 shows an example process 200 for generating an output, such as an output prediction, regarding user behavior by training and / or using a machine learning model (e.g., AI / ML model 162 of FIG. 1). Process 200 may be performed, for example, by computing system 104 of FIG. 1 (e.g., by processor 142 when executing instructions stored in memory 144).

[0041] A model 210 (e.g., an AI / ML model 162) receives historical event data 202 (e.g., from historical data database 174 of FIG. 1). In some implementations, the model 210 receives the historical event data 202 via an application programming interface (API) for the model 210 (e.g., executing on a computing system 104, a client device 102, an analytics server 108, etc.).

[0042] In some implementations, the model 210 is trained (e.g., by the computing system 104) using historical event data 202. Specifically, the historical event data 202 is structured to conform to the grammatical rules of a non-human language, such as the event language described herein. In other words, the computing system 104 can use the historical event data 202 without converting the data into a human-readable language (e.g., English, French, Japanese, etc.), but instead analyze the event data in the non-human language (e.g., its native format). Depending on the implementation, the historical event data 202 may include labeled data (e.g., data with labels indicating the next event in a sequence, the overall session / sequence context, etc.), unlabeled data, and / or any other data representing past events. Accordingly, the computing system 104 may train the model 210 according to supervised techniques (e.g., using labeled data), unsupervised techniques (e.g., using unlabeled data), semi-supervised techniques, etc. Training the model 210 is described in further detail below with respect to FIGS. 3A-3B.

[0043] Although the historical event data 202 has not been converted into human-readable language, the computing system 104 can use the historical event data 202 to train the model 210 in a manner similar to training a traditional LLM. For example, the model 210 can process a stream of characteristic user experiences in a manner similar to a collection of documents, process a user profile containing multiple sessions in a manner similar to a single document, process a session (e.g., an ordered grouping of closely related events) in a manner similar to a sentence or paragraph, and process an event in a manner similar to a word or token.

[0044] Additionally, the historical event data 202 includes metadata associated with the events. The metadata includes information that does not exist in human language, such as the relative time of the events (e.g., the time between events occurring), user identification (e.g., a user account ID, a user IP address (or, e.g., a pseudonymous identifier that serves as a substitute for such), the user's customer number, etc.), the user's location, an indication of the user's event history (e.g., past events associated with the user), one or more device characteristics of the user device (e.g., client device 102), and / or other such metadata. Thus, conventional models trained on human language data may be poorly or unable to utilize such information.

[0045] Depending on the implementation, model 210 may be or include a deep neural network, and more specifically, may be or include a large-scale language model (LLM). An LLM may have a Transformer model architecture with an encoder and decoder and may tokenize input / text, as described below with respect to FIGS. 3A and 3B. A Transformer model may incorporate a self-attention mechanism to facilitate faster learning / training and / or more accurate output. In some implementations, an LLM includes many layers of a neural network, possibly including several embedding layers, several feedforward layers, and several recurrent layers. In some implementations, model 210 includes a generative model other than an LLM. For example, model 210 may instead include a less complex neural network.

[0046] The model 210 may have been trained by the computing system 104 or another computing system using supervised or semi-supervised learning and with training data (e.g., historical event data 202) of an appropriate modality (i.e., event data). The model 210 may be a general-purpose model (e.g., a model trained on a wide array of publicly available datasets of event data) or a domain-specific model (e.g., a model trained on a custom and / or proprietary dataset of event data).

[0047] Differences in the format of the event language for historical event data 202 and techniques for training conventional models pose unique challenges in training a model 210 on historical event data 202. Specifically, (i) the length of event sequences in historical event data 202 may differ from the length of training data for conventional models, and (ii) the event structure and lexical cardinality of historical event data 202 differ from that of training data for conventional models. Training data for conventional language models may include related human text documents, which may be or contain hundreds or thousands of related word tokens. However, historical event data 202 may be only a few event tokens long. For example, a user may load a news site, read an article, subscribe to a newsletter, and never return to the site. Thus, the computing system 104 may have less event data for training a model 210 compared to conventional systems. Similarly, event data includes rich metadata, such as page URLs, parameters (e.g., purchase amount), time elapsed since a previous event, and type of computing device. The training data of conventional systems does not contain such rich metadata, and thus training the model 210 must take such into account.

[0048] As noted above, these differences can present challenges. For example, as sequence length decreases, the amount of available context that model 210 can use in training decreases. This is problematic because contextual token disambiguation is important for training many types of models (e.g., LLMs). Furthermore, deep event metadata structures can introduce errors in vocabulary space. For example, a traditional English LLM may contain 30,000 unique subword tokens. For event data, multiplying the cardinality of each metadata dimension (e.g., URL, time gap, floating-point parameter value, etc.) can result in a nearly infinite number of unique tokens, each of which may occur too infrequently in a corpus of historical event data for effective learning. Furthermore, event metadata is not context that can be discarded after analysis and / or training. Instead, model 210 can generate predictions that include new metadata for the event linguistic output. This hierarchical generation is not part of traditional language model architectures.

[0049] The techniques described herein overcome such challenges by repairing and exploiting these differences. For example, to address vocabulary cardinality, the computing system 104 can generate embeddings of event metadata components (e.g., as described below with respect to FIGS. 3A and 3B). The encoded feature vectors become part of what the model trains and predicts. Thus, the dimensionality of each metadata component can be more manageable for such vocabulary.

[0050] As a further example, to address challenges with short sequence lengths and enable the model 210 to generate complete events with all the appropriate metadata, the computing system 104 can take advantage of metadata structures. By encoding and representing each component separately, the model has access to all context-relevant metadata tokens within a single event, allowing for reasonable predictive power even for short sequences.

[0051] When generating output, some categories of metadata are easier to predict than others due to differences in dimensionality. Specifically, metadata has an implicit hierarchy. Computing system 104 divides the generation of output events by generating each portion of the metadata in turn, with later components being able to leverage already predicted portions. More difficult output portions (e.g., URL embedding or event name) can leverage earlier predictions of easier-to-generate, lower-cardinality components for the currently generated event. In other words, model 210 can avoid generating the event and associated metadata all at once. In some such implementations, model 210 instead takes as input both a previous fully complete event token and a partially complete output token, with a subset of the metadata now populated. Computing system 104 then iterates through all remaining metadata categories, adding a new prediction for one element of the metadata during each iteration.

[0052] By using embeddings to refine the vocabulary space and iteratively generating component parts of an event's metadata, the computing system 104 provides model-based generation of event sequences.

[0053] In some embodiments, model 210 is AI / ML model 162 stored on computing system 104 (e.g., as shown in FIG. 1). In further embodiments, model 210 is another model stored on a remote server and / or another component of system 100 (not shown), but accessed by computing system 104. For example, model 210 may be an LLM that a remote server makes available to computing systems (including computing system 104) via an application programming interface (API).

[0054] Model 210 also receives user data 204 and / or user metadata 206. Depending on the implementation, user data 204 is data associated with a particular user (e.g., using client device 102). User metadata 206 may include the metadata described above with respect to the metadata of historical event data 202. User data 204 and user metadata 206 are similarly in the same non-human language as historical event data 202 and follow the same grammatical structure.

[0055] The model 210 then uses the user data 204 and the user metadata 206 to perform various tasks related to the user, such as generating output predictions 220 (e.g., including output scenarios). For example, the model 210 can utilize the user data 204 over a particular time period to determine output predictions 220 for user behavior within the window. The output of the model 210 can be customized based on custom parameters 208 received by the model 210. Depending on the implementation, the custom parameters 208 can indicate the type of prediction or another behavior (e.g., predicting the likelihood that a user will perform a specified event or action, predicting the likelihood that a user will trigger an event, determining the expected number of users within a time period, determining a particular profit prediction associated with a user, etc.), the window within which the user is predicted to perform the behavior (e.g., 3 days, 7 days, 28 days, 1 month, etc.), and / or any other such parameters.

[0056] In conventional systems, a computing system may generate a model to perform such an analysis. However, by generating a model according to conventional techniques, a unique model typically must be generated and trained for each time frame (e.g., a 7-day model, a 28-day model, etc.), metric (e.g., predicted user behavior, branching scenarios of user history, etc.), and / or other such desired outcome. By training the model 210 with event data as described herein, the computing system 104 can generate, train, and otherwise provide a single model with customizable capabilities.

[0057] In some implementations, computing system 104 further simulates theoretical scenarios to generate output scenarios as part of output 220. Specifically, model 210 uses scenario data 215 to modify, extend, replace, and / or otherwise adjust user data 204 for model 210 to generate the output scenarios. For example, computing system 104 can provide scenario data 215 representing a scenario in which a user returns to a site 24 hours after their last known or actual visit, and computing system 104 (e.g., via model 210) can then predict an output scenario representing the likelihood that the user will purchase something within the next three days if that return visit actually occurs. Depending on the implementation, the output scenarios can be based on scenario data 215 representing the future (e.g., future events that the user could take) or diverging past paths (e.g., hypothetical changes to past data).

[0058] 3A illustrates a process 300A for embedding input data while training a machine learning model, according to one embodiment. Depending on the embodiment, process 300A may be performed, for example, by computing system 104 of FIG. 1 (e.g., by processor 142 when executing instructions stored in memory 144).

[0059] To generate an embedding of input data, computing system 104 receives one or more input features 335 (e.g., historical data and / or user data as described herein). Computing system 104 then generates one or more vector encodings 330 that represent the input features. Computing system 104 generates a D-dimensional (e.g., 2D, 3D, 4D, etc.) space that indicates the similarity between the input features 335 and the vector encodings 330. Computing system 104 plots the vector encodings 330 in the D-dimensional space to generate embedding 320. In some implementations, computing system 104 also generates encodings for additional features 340 of the event data. For example, the additional features 340 may include metadata associated with one or more of the input features 335 (e.g., location of a device associated with the user, time of the event, account ID associated with the user, time of day, day of the week, observable device information (e.g., mobile or desktop, browser, specific device type, device language, etc.), user origin (search engine, direct link, ad interaction, etc.), user demographic information, user demographics (e.g., loyalty rewards program), user membership start date, user lifetime value, user past behavior, device type of the device, device screen size, device browser type, device location (e.g., country or region), URL of the web page, page title or screen name of the web page, content on the web page (e.g., products shown), known metadata about the product (e.g., price or category), type of business associated with the publisher, name or type of the event, time of the event (e.g., day, week, month, year), parameters associated with the event, and / or any other such feature). In another embodiment, the computing system refrains from generating an encoding of the additional features 340. Depending on the implementation, the embedding 320 represents one or more long-distance dependencies of tokens in the event language.The computing system 104 may utilize the embeddings 320 (and thus the long-distance dependencies) in training the model, as described herein with respect to FIGS. 4A and 4B.

[0060] In some implementations, the computing system 104 further generates a log-probability layer 310 that represents a probability distribution of likelihoods for various next outcomes (e.g., the next event determined in the sequence). The computing system 104 determines the probability output 305 based on the embedding layer(s). In further implementations, the computing system 104 determines the probability output 305 using one or more labels 325 associated with the input features to teach the model to embed the input data.

[0061] FIG. 3B shows an example process 300B for running and / or training a sequential text generation model in which an output prediction is iteratively utilized as the next input value. Specifically, model 350 receives an initial input / output value (e.g., I / O 360) and makes a prediction 352 to generate I / O 362. I / O 362 is fed to model 350, and another prediction 354 is made to generate I / O 364. The process repeats iteratively, with each I / O being fed to model 350, which makes a prediction (e.g., predictions 352, 354, 356, 358, etc.), and outputs an I / O value (e.g., I / O 362, 364, 366, 368, etc.), which is then used as an input to model 350. In this manner, the model is capable of iteratively predicting a larger series of events, as described herein. Depending on the implementation, the prediction may be a multinomial distribution process and / or other classification process, as described with respect to FIG. 4 below.

[0062] Depending on the implementation, model 350 may be or include a neural network, such as an artificial neural network (ANN), a convolutional neural network (CNN), or a recurrent neural network (RNN). In further implementations, model 350 may incorporate, include, be, and / or otherwise use techniques reminiscent of language modeling techniques (e.g., large-scale language models (LLMs), bag-of-word models, etc.). For example, model 350 may utilize events such that model 350 processes a stream of characteristic user experiences in a manner similar to a collection of documents, processes a user profile including multiple sessions in a manner similar to a single document, processes a session (e.g., an ordered grouping of closely related events) in a manner similar to a sentence or paragraph, and processes events in a manner similar to a word or token. Similarly, model 350 may incorporate, include, be, and / or otherwise use a Transformer architecture to utilize appropriate language modeling techniques, as described below with respect to FIG. 4.

[0063] FIG. 4 illustrates an exemplary large-scale language model 400 trained as a generative model, as described herein. Specifically, a generator model 410 receives an input vector 405 and outputs an output 460. In some implementations, the input vector 405 may be a fixed-length random vector. In further implementations, the input vector 405 may be randomly drawn from a Gaussian distribution, such that a point in a vector space corresponding to the input vector 405 may correspond to a point in a problem domain representing a data distribution (e.g., corresponding to a point as described with respect to FIGS. 3A and 3B). Depending on the implementation, the vector space corresponding to the input vector 405 may include one or more hidden variables (e.g., variables that are not directly observable). In some implementations, the input vector 405 may be used to seed a generation process.

[0064] In some implementations, the generative AI and / or ML model may be based on an LLM trained to predict an event in a sequence of events. For example, the LLM may be trained to predict the next event following a given sequence of events. For example, a generative AI and / or ML model may be given the sequence "(i) visit a web page, (ii) wait three days, (iii) visit the web page, and (iv) purchase an item from the web page," and the generative AI and / or ML model may predict the next events, such as "return to the web page in three days," "purchase another item," or "do not return to the web page."

[0065] Advantageously, some embodiments use a Transformer in training the model (e.g., by using a Generative Pre-Trained Transformer (GPT) model). More specifically, some embodiments use a GPT model that includes (i) an encoder that processes input sequences and (ii) a decoder that generates output sequences. Both the encoder and decoder may include a multi-head self-attention mechanism that enables the GPT model to differentially weight portions of the input sequences and infer meaning and context (e.g., using historical data and / or metadata in the training data).

[0066] Specifically, input vector 405 may be a vector representing relationships between events, sequences, etc. in the input. Large-scale language training module 450 may include a self-attention block 452 component (e.g., referred to as a multi-self-attention block, a multi-head attention block, a multi-head self-attention block, a masked multi-self-attention block, a masked multi-head attention block, a masked multi-head self-attention block, etc.) that simultaneously or near-simultaneously attends to different portions of the input to capture relationships and / or dependencies between different portions of the input. Specifically, self-attention block 452 associates different positions in a sequence to compute a representation of the sequence. Thus, self-attention block 452 may weight the influence of different events in a sequence when sequencing. Thus, model 400 learns to emphasize different portions of input vector 405. In some implementations, model 400 uses metadata associated with events instead of and / or with self-attention block 452 to determine the influence and / or relationships between events in a sequence.

[0067] Self-attention block 452 may then calculate an attention score that represents the influence of each word in the sentence relative to other words in the sentence (e.g., by taking the dot product between different sets of vectors). The output then goes to normalization layer 454, which may normalize the output of self-attention block 452 (e.g., by applying a softmax function to normalize the scores).

[0068] Similarly, the self-attention block may then output to a feedforward network block 456, which performs nonlinear transformations to generate new representations of the input and / or relationships between events, sequences, etc. Specifically, the feedforward network block 456 may use the calculated and normalized attention scores to calculate a weighted sum of vectors to capture contextual relationships between words. In some implementations, the normalization layer 454 and / or the self-attention block 452 may perform calculations to generate representations of relationships between words, etc. After the feedforward network block 456, an additional normalization layer 458 may normalize each output and / or add remaining connection(s) to allow the output to flow directly to other inputs. Thus, the model 400 may learn which portions of the input are important (e.g., remain dominant through the normalization process). Depending on the implementation, the model 400 may repeat the process of the large-scale language training module 450 one, five, ten, N, etc. times to train each model(s).

[0069] Depending on the implementation, the encoder and / or decoder may be trained as described above. In a further implementation, the encoder is trained as described above, and the decoder also includes an additional self-attention block (not shown) that receives the output of the encoder.

[0070] Furthermore, in some implementations, the GPT model does not perform the previous four steps just once, but rather iterates the steps and performs them in parallel. At each iteration, a new linear projection of the query, key, and value vectors is generated. Such iterative, parallel implementations advantageously improve the capture of local meanings and more complex relationships within the input sequence data.

[0071] Further advantageously, some embodiments train and / or tune the model using supervised, unsupervised, and / or semi-supervised techniques. For example, the model may be trained using labels associated with events, so that the model can recognize the "correct" responses within the pattern of the sequence (e.g., for supervised training). Additionally, the computing system 104 may train the model using reward techniques, reinforcement techniques, and / or any other such techniques adapted to the event data described herein.

[0072] FIG. 5 is a flow diagram of an example method 500 for generating a customizable model for predicting user behavior. Method 500 may be implemented as instructions stored on one or more computer-readable media and executed by one or more processors of one or more computing devices. For example, method 500 may be performed by processor 142 of computing system 104 of FIG. 1 when executing instructions of prediction generator 150, scenario generator 152, training module 154, and / or any other such modules described herein. As a further example, method 500 may be implemented by one or more processors of a computing device communicatively coupled to computing system 104. It will be understood that any such implementation is exemplary, and that additional, fewer, and / or alternative components may be used to implement example method 500.

[0073] At block 502 of method 500, computing system 104 obtains first structured event data representing one or more first events. Furthermore, in embodiments of method 500, the first structured event data is in a non-human language and is structured according to the grammar rules of the non-human language. In some embodiments, the first structured event data includes historical event data (e.g., historical event data 202 and / or historical data in historical data database 174) used to train a machine learning model (e.g., training data). The first structured event data follows the grammar rules of a language focused on non-human events and is not converted into a human language (e.g., English, French, Japanese, etc.). In the exemplary embodiment of FIG. 5 , the grammar rules include the overall structure of the language. For example, various events may be arranged in a sequence or session, representing events occurring in a single, predetermined time period (e.g., a single website access, a day, an hour, etc.). Similarly, multiple sessions may be placed into a user profile corresponding to the user over a period of time, which may then be one of a series of user profiles in the stream of event data associated with a particular grouping that includes the user (e.g., based on location, based on device, based on age, etc.).

[0074] Depending on the implementation, the first structured event data may be or include data representing one or more interactions with an application, a view of a web page, a hit on a landing page of a domain, an interaction with a buy button, an input of personal information (e.g., credit card information, address information, email information, etc.), receiving a notification, opening a notification, and / or any other such interaction with material from a publisher 106, an application developer (not shown), computing system 104, etc. In further implementations, the first structured event data includes metadata associated with one or more events. For example, the first structured event data may include a relative time of the event (e.g., the time between events occurring), a user identification (e.g., a user account ID, a user IP address, a user customer number, etc.), a user location, an indication of a user event history (e.g., past events associated with the user), one or more device characteristics of a user device (e.g., client device 102), and / or another such metadata.

[0075] At block 504, computing system 104 trains a generative language model (e.g., AI / ML model 162 and / or model 210) using the first structured event data. As described above with respect to block 502, the first structured event data is in a non-human language that targets events. Thus, computing system 104 trains the generative language model to read and analyze data in the non-human language. More specifically, computing system 104 trains the generative language model on event sequences (e.g., from an analytics server, such as analytics server 108 of FIG. 1). In embodiments in which the first structured event data includes metadata, computing system 104 further trains the generative language model using the metadata. For example, computing system 104 can train the generative language model to understand relative times between events and, based on the metadata, determine whether user behavior correlates with the relative times between events.

[0076] Thus, in some implementations, the computing system 104 trains a single model so that it can directly answer a variety of predictive questions without having to be fine-tuned and adapted for a specific task. For example, the computing system 104 may feed a new user's event stream into a generative language model and prompt the model to generate the most likely events for a given window (e.g., 3 days, 7 days, 28 days, etc.) based on the input. The computing system 104 may then search the generated stream for future key events generated by the model (e.g., will the user convert (e.g., purchase an item), what will the user's lifetime value be, will the user churn (stop interacting with content), etc.).

[0077] In a further alternative embodiment, the computing system 104 first trains a generative language model and fine-tunes it to perform various tasks and / or predict various actions without training an entirely new model. For example, the computing system 104 trains a large generative language model with an initial goal (e.g., predicting the next event). After the base model is completed, the computing system 104 fine-tunes a separate, individual task-oriented model (e.g., probability of conversion (e.g., purchasing an item) in the next week) using the encoded output of the base model as input. Thus, the computing system 104 can develop new task-specific models at a faster rate than by generating each model individually. In a further embodiment, the computing system 104 trains a single model to perform various tasks and fine-tunes the task-specific model based on the publisher's data sharing settings (e.g., if the publisher (e.g., publisher 106) has not agreed to share data with another publisher).

[0078] At block 506, the computing system 104 obtains second structured event data representing one or more second events performed by the user. The second structured event data is in the same non-human event language as the first structured event data and is therefore structured according to similar grammatical rules. Furthermore, depending on the implementation, the second structured event data includes similar information as the first structured event data, but includes information about a particular user. In further implementations, the second structured event data includes event data for a user group that includes the user (e.g., users accessing web pages from the same IP address, users accessing applications on the same device, users fitting an age range, location, or other demographic information purchasing items, etc.).

[0079] At block 508, the computing system 104 uses the generative language model to predict third structured event data representing behavior associated with the user according to one or more customizable analysis outputs. The third structured event data, like the first structured event data and the second structured event data, is in a non-human language and is structured according to grammar rules based on a non-human language. Thus, the generative language model does not convert the third structured event data into a human language (e.g., English, French, Japanese, etc.) for analysis. The third structured event data may be or may include predicted user behavior, predictions for hypothetical scenarios, predictions for modified historical scenarios (e.g., in which the computing system 104 adds, modifies, replaces, or otherwise adjusts past event data), etc. In some implementations in which the second structured event data includes stream data of at least one client device representing one or more sessions of user events (e.g., stream data of multiple users), predicting the user behavior includes extracting user events for a particular user from the stream data (e.g., via a user ID, one or more user characteristics, event data ID, etc.). The computing system 104 then predicts at least one future event based on the extracted user events using the trained generative language model. In some implementations, predicting the at least one future event includes determining, by the trained generative language model, a likelihood of occurrence of the at least one future event occurring.

[0080] In some embodiments, the one or more customizable analysis outputs are or include various metrics for customizing the output analysis. For example, the customizable analysis output can be or include a time window for the analysis, a particular type of analysis (e.g., detecting the likelihood of user retention, predicting future purchases and / or revenue associated with a user, determining the likelihood that a user will refer others to the site, etc.), user identification (e.g., when the second structured event data includes data from a stream of user profiles), and / or any other such metric. In further embodiments, some of the metrics and analysis are determined and performed in a post-processing step (e.g., a step as described below), and the customizable analysis output is a stop token indicating where generation of the third structured event data should end.

[0081] The second structured event data (e.g., including user data) comprises an initial set of events that prompts the generative language model to generate third structured event data. In some implementations, the second structured event data continues to feed the model automatically, causing the generative language model to generate the third structured event data in real time or near real time. In further implementations, the model receives the second structured event data in separate packets and generates the third structured event data in response to receiving the packet(s) and / or in response to receiving an indication from the publisher and / or another content owner. The model may generate the third structured event data to any length until a customizable point is reached (e.g., 3 days after expected behavior, 7 days after expected behavior, 28 days after expected behavior, etc.), as indicated by a stop token and / or another indication to stop generating data in the second structured event data. The computing system 104 or the analytics server 108 may perform post-processing steps to adjust the time window in which relevant data is displayed and / or used in response to a prompt. In some implementations, the post-processing steps may be or include existing and / or traditional analytical techniques (e.g., computing system 104 or analytics server 108 may operate on the predicted event sequences using traditional techniques for historical event data). In further implementations, the post-processing steps may be or include analytical techniques customized for the third structured event data, and / or a combination of customized and traditional techniques. As an example, computing system 104 may generate predictions regarding user behavior over a period of one month or more, while the time window may be three days. In some implementations, the window is based on the number of events, whether a particular event occurs, a time period, and / or any other such window.

[0082] In some implementations, the computing system 104 uses scenario data to insert, replace, modify, and / or otherwise adjust event data. Depending on the implementation, the scenario data can be theoretical and future-oriented, or branching and past-oriented. For example, the scenario data can be or include a theoretical scenario in which the computing system 104 determines the likelihood of a user purchasing an item if the user visits a web page anytime within the next three days. As another example, the scenario data can be or include a branching scenario in which the computing system 104 determines what the likelihood of a user purchasing an item was if the user interacted with an advertisement two days ago. Thus, the computing system 104 can dynamically ask and answer any question related to user data without training a new model. Depending on the implementation, the computing system 104 can receive prompts from a publisher (e.g., the analytics server 108) and / or another owner of content (e.g., web pages, applications, media, etc.) displayed to the user and generate event data representing events to insert as scenario data into the generative language model.

[0083] In some implementations, the computing system 104 uses the same trained generative language model to determine alternative outputs without modifying the model itself. Specifically, the computing system 104 can predict fourth event data (e.g., different from the third structured event data) that represent behavior associated with the user according to a second set of customizable analysis outputs that are different from the first set of customizable analysis outputs, at least in part by applying the second structured event data as input to the trained generative language model. Depending on the implementation, the second set of customizable analysis outputs can be of a different type (e.g., time window compared to type of analysis) or the same.

[0084] In some implementations, computing system 104 receives one or more event or metric prompts from a publisher and / or other owner of content (e.g., web pages, applications, media, etc.) displayed to a user to predict user behavior or other outcomes. In further implementations, computing system 104 includes additional models for converting input from the publisher and / or other owner of the content into event prompts and / or metrics. Similarly, computing system 104 may include a separate model for converting the third structured event data into a human language (e.g., English, French, Japanese, etc.) in a post-processing step.

[0085] In further embodiments, the computing system 104 generates a confidence score that represents the potential accuracy of the third structured event data. Depending on the embodiment, the computing system 104 may generate the confidence score based on similarity to historical event data (e.g., the first structured event data used to train the generative language model). For example, the computing system 104 may determine a full confidence score (e.g., 100% or approximately 100%) if the prediction happens every time and occurs multiple times in the historical event data, or a confidence score of only 50% if the prediction occurs 50% of the time. Similarly, the computing system 104 may otherwise generate the confidence score based on other factors, such as frequency of occurrence in the historical event data, similarity between the user data and the historical event data, the user's past knowledge, etc.

[0086] In some embodiments, as noted above, the techniques disclosed herein use artificial intelligence to facilitate efficient generation of streams of event data (e.g., representing predicted outcomes). Artificial intelligence (AI) is a segment of computer science that focuses on creating models that can perform tasks with little or no human intervention. Artificial intelligence systems can utilize, for example, machine learning and computer vision. Machine learning and subsets thereof, such as deep learning, focus on developing models that can infer outputs from data. Outputs may include, for example, predictions and / or classifications. Computer vision focuses on analyzing and interpreting images and videos. Artificial intelligence systems may include generative models that generate new content in response to input prompts and / or based on other information.

[0087] Exemplary machine-learned models include neural networks or other multi-layer nonlinear models. Exemplary neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some exemplary machine-learned models can utilize attention mechanisms such as self-attention. For example, some machine-learned models may include multi-head self-attention models (e.g., Transformer models).

[0088] The model(s) can be trained using various training or learning techniques. Training can be performed using supervised learning, unsupervised learning, reinforcement learning, etc. Training can use techniques such as backpropagation of errors. For example, a loss function can be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on the gradient of the loss function). Various loss functions can be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent can be used to iteratively update parameters over several training iterations. Several generalization techniques (e.g., weight decay, dropout) can be used to improve the generalization ability of the trained model.

[0089] Prior to domain-specific alignment, the model(s) can be pre-trained. For example, the model can be pre-trained on a general corpus of training data and fine-tuned to a more targeted corpus of training data. The model can be aligned using prompts designed to elicit domain-specific outputs. The prompts can be designed to include learned prompt values ​​(e.g., soft prompts). The trained model(s) can be validated before use using input data other than the training data and can be further updated or refined during its use based on additional feedback / input.

[0090] In some implementations, the computing system 104 may use any one or more of the above machine learning models to perform any one or more of the operations described herein in connection with machine learning. For example, the computing system 104 may use such one or more machine learning models to generate a stream of event data, as described above.

[0091] Although the foregoing text has detailed many different aspects and embodiments of the invention, it should be understood that the scope of the patent is defined by the language of the claims set forth at the end of this patent. The detailed description is to be construed as exemplary only.

[0092] The following additional considerations apply to the above description: Throughout this specification, multiple examples may implement components, operations, or structures that are described as a single example. While individual operations of one or more methods are illustrated and described as separate operations, one or more of the individual operations may occur simultaneously and the operations need not occur in the order shown. Structures and functions depicted as separate components in an illustrative configuration may be implemented as a combined structure or component. Similarly, structures and functions presented as a single component may be implemented as separate components. These and other variations, modifications, additions, and improvements are within the scope of the subject matter of this disclosure.

[0093] Unless otherwise expressly stated, discussions in this disclosure using words such as "processing," "calculating," "computing," "determining," "presenting," "displaying," or similar expressions may refer to machine (e.g., computer) actions or processing that manipulate or transform data represented as physical quantities (e.g., electrical, magnetic, or optical quantities) within one or more memories (e.g., volatile memory, non-volatile memory, or a combination thereof), registers, or another machine component that receives, stores, transmits, or displays information.

[0094] As used in this disclosure, a reference to either "one implementation" or "an implementation" means that a particular element, feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment. The appearances of the phrase "in one embodiment" in various places in this specification do not necessarily all refer to the same embodiment.

[0095] As used in this disclosure, "comprises," "comprising," "includes," "including," "has," "having," or any other variation thereof, is intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that includes a list of elements is not necessarily limited to only those elements and may include other elements not expressly listed or inherent in such process, method, article, or apparatus. Furthermore, unless expressly stated to the contrary, "or" refers to an inclusive "or," not an exclusive "or." For example, condition A or B can be satisfied by any one of the following: A is true (or present) and B is false (or absent); A is false (or absent) and B is true (or present); and both A and B are true (or present).

[0096] Upon reading this disclosure, those skilled in the art will recognize still additional structural and functional design alternatives for predicting user behavior using customizable event modeling and using non-human languages ​​through the principles described herein. Accordingly, while specific implementations and applications have been illustrated and described, it should be understood that the disclosed implementations are not limited to the precise structure and components disclosed in this disclosure. Various modifications, changes, and variations apparent to those skilled in the art may be made in the arrangement, operation, and details of the methods and apparatus disclosed in this disclosure without departing from the spirit and scope, as defined by the appended claims.

Claims

1. 1. A method for predicting user behavior using customizable event modeling, comprising: obtaining, by a computing system, first structured event data representing first events performed by one or more users; training, by the computing system, a generative language model using the first structured event data; obtaining, by the computing system, second structured event data representing one or more second events performed by a user; predicting, by the computing system and at least in part by applying the second structured event data as input to the trained generative language model, third structured event data representative of behavior associated with the user according to one or more customizable analysis outputs; A method comprising:

2. 2. The method of claim 1 , wherein the second structured event data includes metadata associated with the one or more second events, and predicting the behavior associated with the user is further based on the metadata.

3. 3. The method of claim 2, wherein the one or more second events comprise a plurality of events, and the metadata comprises metadata representing a relative timing of a first event of the plurality of events with respect to a second event of the plurality of events.

4. 4. The method of claim 2 or 3, wherein the metadata includes at least one of: (i) a user identification; (ii) a user location; (iii) an indication of a user event history; or (iv) one or more device characteristics.

5. Predicting the third structured event data includes: encoding, by the computing system, one or more embeddings of features associated with the one or more second events and the metadata. The method according to any one of claims 2 to 4, comprising:

6. 6. The method of claim 5, wherein each of the one or more embeddings represents a respective one of the one or more second events and all of the metadata associated with the respective one of the one or more second events.

7. Predicting the third structured event data includes: generating, by the computing system, a first portion of the third structured event data based on the metadata; and generating, by the computing system, a second portion of the third structured event data based on the first portion of the third structured event data; The method according to any one of claims 2 to 6, comprising:

8. The method of any one of claims 1 to 7, wherein training the generative language model comprises training the generative language model based on one or more long-distance dependencies.

9. The second structured event data includes stream data of the user representing one or more sessions of user events, and predicting the behavior associated with the user includes: extracting the user events from the stream data; predicting at least one future event based on the extracted user events by the trained generative language model; and The method according to any one of claims 1 to 8, comprising:

10. applying an indication of a theoretical future event as part of the input to the trained generative language model; The method of any one of claims 1 to 9, wherein predicting the behavior associated with the user is further based on the theoretical future events.

11. The method of any preceding claim, wherein the second structured event data comprises historical event data associated with the user.

12. The method of any preceding claim, wherein the second structured event data comprises theoretical branching event data associated with the user.

13. 13. The method of claim 1, wherein the first event and the one or more second events comprise at least one of: (i) a web page view; (ii) a hit on a landing page; (iii) a hit associated with an interactive element on a web page; (iv) a hit associated with an interactive element on an application; (v) receiving a notification; or (vi) opening a notification.

14. The one or more customizable analysis outputs are a first set of customizable analysis outputs, and the method further comprises:

14. The method of claim 1, comprising predicting, by the computing system and at least in part by applying the second structured event data as input to the generative language model, fourth event data representing behavior associated with the user according to a second set of customizable analysis outputs that differ from the first set, wherein the fourth event data is different from the third structured event data.

15. A computing system comprising one or more processors and a memory storing instructions causing said computing system to perform the method of any one of claims 1 to 14.

Images