Analyzing personalization framework

Abstract

The present disclosure provides an analytic personalization framework. Methods, systems, and computer readable media for personalized analysis of user interfaces. The method includes generating a training dataset from received user interaction data, inputting the training dataset to a machine learning model to train the model, generating a set of user interest scores for a particular user, where each user interest score indicates a user's interest to access information corresponding to a UI element of an application, and generating a user interest score for the particular user. The method includes determining a user interest score, determining that the user is interested in a particular UI element that is not included in the initial UI and has at least a threshold score according to the user interest score, dynamically modifying the initial UI to include the particular UI element, presenting an updated UI, monitoring further user interactions, updating a model based on the further user interactions, and modifying the updated UI based on the updated model.

Description

[0001] This application is a divisional application of the Chinese invention patent application with application date of October 23, 2019, application number 201911011979.1, and invention name “Analysis Personalization Framework”. Technical Field

[0002] This document is concerned with providing a personalization framework within an analytics environment. Background Art

[0003] This document relates to providing a personalization framework within an analytics environment. Users interacting with content on the Internet generate interaction data that can be viewed and analyzed through different user interface (UI) elements. The user interface and the content presented by the user interface (such as analytics data) can be customized through the automation framework. Summary of the invention

[0004] An embodiment of the present disclosure provides a method executed by a computing device, the method comprising: generating a set of user interest scores for a user using a trained machine learning model, wherein each user interest score indicates the user's interest in accessing information corresponding to a user interface UI element of an application, wherein the trained machine learning model is trained using a training data set, the training data set comprising a set of features related to the user's interaction with one or more UI elements of the application, the features being selected for training the trained machine learning model based on corresponding durations of UI elements corresponding to the features being presented to one or more users; determining that at least one user interest score in the set of user interest scores positively indicates the user's interest in accessing information corresponding to an unpackaged UI element; The invention relates to a method of dynamically modifying the initial UI by incorporating the specific UI element into the initial UI to generate an updated UI including the specific UI element that was not included in the initial UI; presenting the updated UI including the specific UI element to the user; after presenting the updated UI to the user, monitoring further user interaction with the updated UI; updating the trained machine learning model based on further user interaction; and selecting an updated set of UI elements to present in the updated UI based on the updated machine learning model.

[0005] An embodiment of the present disclosure provides a system, comprising: one or more processors; and a memory communicatively coupled to the one or more processors, the memory storing instructions, which, when executed by the one or more processors, cause the one or more processors to perform operations including the following: generating a set of user interest scores for a user using a trained machine learning model, wherein each user interest score indicates a user's interest in accessing information corresponding to a user interface (UI) element of an application, wherein the trained machine learning model is trained using a training data set, the training data set comprising a set of features related to a user's interaction with one or more UI elements of an application, the features being selected for training the trained machine learning model based on corresponding durations during which UI elements corresponding to the features are presented to one or more users; determining at least one user interest score in the set of user interest scores positively indicates a user's interest in accessing information corresponding to a specific UI element that is not included in an initial UI presented to the user; in response to determining that at least one user interest score in the set of user interest scores positively indicates a user's interest in accessing information corresponding to the specific UI element, dynamically modifying the initial UI, including merging the specific UI element into the initial UI to generate an updated UI including the specific UI element that is not included in the initial UI; presenting the updated UI including the specific UI element to the user; after presenting the updated UI to the user, monitoring further user interaction with the updated UI; updating the trained machine learning model based on further user interaction; and selecting an updated set of UI elements to present in the updated UI based on the updated machine learning model.

[0006] An embodiment of the present disclosure provides a non-transitory computer-readable storage device storing instructions, which, when executed by one or more processors, causes the one or more processors to perform operations, the operations comprising: generating a set of user interest scores for a user using a trained machine learning model, wherein each user interest score indicates a user's interest in accessing information corresponding to a user interface (UI) element of an application, wherein the trained machine learning model is trained using a training data set, the training data set comprising a set of features related to user interactions with one or more UI elements of an application, the features being selected for training the trained machine learning model based on corresponding durations during which UI elements corresponding to the features are presented to one or more users; and determining at least one of the user interest score sets; The user interest score positively indicates the user's interest in accessing information corresponding to a specific UI element that is not included in an initial UI presented to the user; in response to determining that at least one user interest score in the set of user interest scores positively indicates the user's interest in accessing information corresponding to the specific UI element, dynamically modifying the initial UI, including merging the specific UI element into the initial UI to generate an updated UI including the specific UI element that is not included in the initial UI; presenting the updated UI including the specific UI element to the user; after presenting the updated UI to the user, monitoring further user interaction with the updated UI; updating the trained machine learning model based on further user interaction; and selecting an updated set of UI elements to present in the updated UI based on the updated machine learning model.

[0007] In general, an innovative aspect of the subject matter described in this specification can be embodied in a method, the method including receiving, by a computing device, user interaction data indicating an interaction of a first user with one or more user interface (UI) elements of an initial UI of an analysis report application presented to the first user. The method includes generating a training data set based on the received user interaction data, and inputting the training data set into a machine learning model. In response to the training data set, the method includes training the machine learning model by the computing device, and generating a set of user interest scores for a second user using the trained machine learning model, wherein each of the user interest scores indicates the second user's interest in accessing information corresponding to the UI element of the analysis report application. The method includes determining, based on a threshold score associated with a specific UI element not included in the initial UI, that at least one of the set of user interest scores positively indicates the second user's interest in accessing information corresponding to the specific UI element, and in response to determining that at least one of the set of user interest scores positively indicates the second user's interest in accessing information corresponding to the specific UI element, dynamically modifying the initial UI, including merging the specific UI element into the initial UI to obtain an updated UI including the specific UI element not included in the initial UI. The method includes presenting an updated UI including the particular UI to the second user, after presenting the updated UI to the second user, monitoring further user interactions with the updated UI, and updating the machine learning model based on the further user interactions. The method includes selecting an updated set of UI elements to present in the UI based on the updated machine learning model.

[0008] Each of these and other embodiments can optionally include one or more of the following features. In some embodiments, the machine learning model is a logistic regression model. In some embodiments, the machine learning model performs deep and wide learning. In some embodiments, the machine learning model is a long short-term memory model.

[0009] The method may also include determining that a score in the user interest score set associated with another UI element is less than a threshold score associated with the other UI element; wherein based on the determination, not displaying the other UI element to the second user. The method may also include classifying the second user based on the second user's interactions and characteristics.

[0010] In some implementations, the received user interaction data relates to two or more different topics, and a different weight is assigned to each of the two or more different topics based on the importance of the corresponding topic to the user determined according to the user interest score for the topic.

[0011] The method may also include receiving a user interaction dataset indicating interactions of a group of multiple different users with one or more UI elements, generating a user group training dataset based on the received user interaction dataset, inputting the user group training dataset into the machine learning model, and training the machine learning model by a computing device in response to the user group training dataset.

[0012] In some embodiments, the first user and the second user are the same user. In some embodiments, the first user and the second user are different users.

[0013] Other embodiments of this aspect include corresponding systems, apparatus, and computer programs configured to perform the actions of the methods encoded on computer storage devices.

[0014] Specific embodiments of the subject matter described in this document may be implemented to achieve one or more of the following advantages. In some environments, there was previously no way to provide a "warm start" to new users, or an interface that was immediately personalized to provide relevant content or UI elements. Users who are unfamiliar with a particular application or interface, especially one that is complex or provides multiple functions or content, may be intimidated or frustrated by the UI or their experience, and are unlikely to continue using the application. For example, in some environments, a user may begin using a new application to analyze data, and because the application is too complex, provides the user with too much content to sort through before finding content that is useful or interesting to the user, or is difficult to navigate, the user may give up and return to using their existing application, or continue to use the application inefficiently. By customizing content to meet the needs of new users who are unfamiliar with the application or interface, so as to make the user experience easier, faster, and more efficient, the system increases the likelihood that the user will return to use the application again, and increases the value that the user obtains from the content presented to the user. Based on the user's interaction, the system can determine, select, and present the content that is most relevant to the user, and in some cases, can avoid presenting content that is not interesting to the user or is useless to the user. By avoiding the presentation of content that is not of interest to the user, effective personalization of content can reduce the computer resources used (e.g., processing power, memory, etc.) within the context of an analysis that adjusts the user interface to display data relevant to the user and not display data irrelevant to the user. In other words, it requires fewer interactions (e.g., menu selections, network requests, etc.) for the user to be directed to the content they need, thereby providing a more efficient and effective user interface relative to conventional systems and user interfaces. In addition, the presentation of data that the user will not use can be avoided, and data transmission, such as from a remote server, can also be avoided, thereby reducing unnecessary bandwidth utilization.

[0015] As the amount / type of data collected and the number of different available reports grow, the ability to reduce the number of interactions to reach a desired data set and to customize user interfaces (and especially data analysis user interfaces) becomes increasingly important, as this growth in data and available reports makes it difficult for users to reach the data they are looking for. Additionally, users in one particular role may only be interested in a subset of the data and / or reports, while users in different roles may only be interested in different subsets of the data, such that a standard user interface will not be useful for all groups of users. The techniques described in this document enable automatic customization of the user interface presented to each user such that information relevant to that user is presented in the user interface. This customization can be performed by utilizing machine learning models based on the user's interactions with various user interface elements.

[0016] The system can automatically organize content and modify the presentation of content (e.g., through visual or auditory effects, such as physical placement on a page, visual prominence, visual effects, sound effects, etc.) to adjust content for a specific user by mining user data to generate training data for a user interest model, where the user interest model learns user interests based on previous user interaction data with content or UI elements. The previous user interaction data can indicate the previous interactions of the specific user himself and is used to train a user interest model specific to the specific user. In some embodiments, the previous user interaction data can indicate the previous interactions of other users and is used to train a user interest model specific to the specific user-other users can have a threshold similarity with the specific user in several user characteristics, and the previous user interaction data can include the specific user. In some embodiments, the previous user interaction data can indicate the previous interactions of other users and is used to train a user interest model for users with specific characteristics. Therefore, the user interface can be an adaptive user interface that adapts to the user, for example, based on the attributes of the user or based on input received from a user associated with the user interface.

[0017] Furthermore, utilizing this method of selecting content for a user allows new users to experience similar utility and comfort when interacting with a particular application or environment as users who are already familiar with the particular application or environment. The method allows for optimization of content presented to a user and allows for content to be provided to a user based on the user's everyday interactions. Using the system, the user does not have to do anything different than what they already do to receive more relevant content - the method uses fewer processing resources because the user does not have to individually provide feedback on what content the user finds most relevant. In some applications or environments, the user does not have any customization options. Thus, the described technology provides more relevant content to a user in an easy to use format while requiring less input and less computing resources than currently available methods.

[0018] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a block diagram of an example environment in which user interactions with digital components distributed for presentation with electronic documents are used to generate training data for a user interest model that dynamically customizes a user interface that presents analysis of the interaction data.

[0020] Figure 2 is an example data flow for customizing content in an application based on previous user interaction data.

[0021] Figure 3 is an example application user interface that can be customized based on previous user interaction data.

[0022] Figure 4 is a flow diagram of an example process for determining not to submit an opportunity to send a digital component to a user.

[0023] Figure 5 is a block diagram of an example computing system.

[0024] The same reference numbers and designations in different drawings indicate the same elements. DETAILED DESCRIPTION

[0025] This document describes methods, systems and devices for improving content relevance and ease of use and browsing for users of a specific application or environment, and allows content providers to provide users with more useful data. In some cases, when a digital component is presented to a user, the user interacts with the digital component. For example, the user can interact, for example, by selecting a digital component, by scrolling past the digital component, by closing the digital component, by selecting to view more content related to the digital component, etc. Some user interactions with digital components can be classified as positive interactions. Positive interactions are indications that a specific user (or a user who shares a threshold amount of characteristics with a specific user) is interested in a digital component or content similar to a digital component. Some user interactions can be classified as negative interactions. The interaction of a user with a digital component classified as negative interactions indicates that a specific user or a user who shares a threshold amount of characteristics with a specific user is not interested in a digital component or content similar to a digital component. Based on the collected user interaction data, the system can generate training data for a user interest model that determines whether a specific user - or a user with similar characteristics - may be interested in a specific type or presentation of content. The system can perform this personalization process using two steps: feature extraction, where user interaction data is processed to determine relevant features within the user interaction data; and user interest scoring or prediction. The system then uses the user interest scoring to personalize UI elements by presenting the UI elements that are most relevant to a particular user (e.g., from a large number of possible UI elements that can be presented). The system provides a similar optimized experience for each user in a different user group, even if some users have less interaction with the system or less experience.

[0026] The server can perform feature extraction by examining interaction data generated when a user interacts with a "digital component." The digital component can be presented to the user, for example, through a user device. In this example, the distribution system receives a request for a digital component to be presented to a specific user through a specific user device associated with the user. When the request is received, the distribution system communicates with the server to determine the digital component that the user is most likely to be interested in. The server receives the user interaction data and generates a training data set based on the received user interaction data by extracting features from the user interaction data to generate training examples provided to the machine learning model. The training data can include, for example, training examples indicating a set of conditions and whether those conditions lead to positive or negative results. The server then trains the user interest model by, for example, inputting the generated training data set into the machine learning model.

[0027] Multiple levels of personalization may occur.User interaction data may include both user identifiers and profile identifiers - the system's UI and content personalization may be applied on a user-specific basis, or characteristics that may be shared across multiple users may be generalized to a profile level.

[0028] The system uses the captured user interactions with UI elements to infer the user's interest in specific features. As discussed in more detail below, features may include, for example, metrics, dimensions, filters, reports, etc.

[0029] Each of the user's interactions with a particular feature or feature set can be used to generate a training data set for the user's interest model. For example, if the user frequently generates reports with default metrics and selects a particular dimension of the data, the system can determine that the user is interested in the particular dimension of the data, and thus the user interaction of generating a report with the particular dimension of the data can be a positive example of the user's interest in the particular dimension of the data. For brevity, throughout this document, the dimension of the data is referred to as a dimension.

[0030] In some embodiments, the system acquires data from the environment. A web page or application has attributes that can indicate that a user is interested in certain metrics, dimensions, filters, etc. For example, a user generating a report for an application that delivers groceries within Wisconsin may be interested in the dimension Wisconsin and a metric indicating the number of new users who have started using the application. These features of the application can be extracted and input as training features to inform the personalization of future user experiences. The system can extract features from data such as the URL of a web page, persistent identifier interaction logs, etc.

[0031] Note that the techniques described in this document can also be implemented in situations where a user is browsing available applications (e.g., in an online analysis application or web browser) or in other environments (e.g., on a publisher web page). For brevity, much of the description below will refer to the data analysis environment.

[0032] As used throughout this document, the phrase "digital component" refers to a discrete unit of digital content or digital information (e.g., a video clip, an audio clip, a multimedia clip, an image, text, or another unit of content). A digital component can be electronically stored in a physical memory device as a single file or as a collection of files, and a digital component can take the form of a video file, an audio file, a multimedia file, an image file, or a text file, and include advertising information, such that an advertisement is a type of digital component. Typically, a digital component is defined (or provided) by a single provider or source (e.g., an advertiser, publisher, or other content provider), but can be a combination of content from multiple sources. Digital components from multiple different sources can be combined into a single electronic document (e.g., a collection of various different digital components), and portions of various digital components from different sources can be combined with information extracted from search results (or other portions of content) into a single digital component.

[0033] Figure 1 1 is a block diagram of an example environment 100 in which digital components are distributed for presentation with electronic documents. User interactions with these digital components are collected and used to generate training data. The generated data is used to train a user interest model for dynamically modifying an analysis user interface. The example environment 100 includes a network 102, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. The network 102 connects an electronic document server 120, a user device 104, a third-party device 106, and a digital component distribution system (DCDS) 112 (also referred to as a distribution system). The example environment 100 may include many different electronic document servers 120, different types of user devices 104, and third-party devices 106.

[0034] The user device 104 or third-party device 106 is an electronic device capable of requesting and receiving resources over the network 102. Example user devices 104 or third-party devices 106 include personal computers, mobile communication devices, and other devices capable of sending and receiving data over the network 102. The user device 104 or third-party device 106 typically includes a user application such as a web browser to facilitate sending and receiving data over the network 102, although local applications executed by the user device 104 or third-party device 106 may also facilitate sending and receiving data over the network 102.

[0035] An electronic document is data that presents a set of content on a user device 104 or a third-party device 106. Examples of electronic documents include analytical reports, web pages, word processing documents, portable document format (PDF) documents, images, videos, search result pages, and feed sources. Local applications (e.g., "apps") such as applications installed on mobile, tablet, or desktop computing devices are also examples of electronic documents. Electronic documents 121 ("Electronic Docs") can be provided to user devices 104 or third-party devices 106 by electronic document servers 120 ("Electronic Doc Servers"). For example, electronic document servers 120 may include servers that host publisher websites. In this example, a user device 104 or a third-party device 106 may initiate a request for a resource such as a given publisher web page, and the electronic document server 120 that hosts the given publisher web page may respond to the request by sending machine executable instructions that initiate the presentation of a given web page at the user device 104 or the third-party device 106.

[0036] In another example, electronic document server 120 may include an application server from which user device 104 or third-party device 106 can download applications. In this example, user device 104 or third-party device 106 can request resources such as files needed to install an application, download files, and then execute the downloaded application locally.

[0037] The electronic document 121 may include a variety of content. For example, the electronic document 121 may include static content (e.g., text or other specified content) that is within the electronic document itself and / or does not change over time. The electronic document 121 may also include dynamic content that may change over time or based on each request. For example, the publisher of a given electronic document 121 may maintain a data source for populating a portion of the electronic document 121. In this example, a given electronic document 121 may include one or more tags or scripts that cause the user device 104 or the third-party device 106 to request content from the data source when the given electronic document 121 is processed (e.g., rendered or executed) by the user device 104 or the third-party device 106. The user device 104 or the third-party device 106 integrates the content obtained from the data source into the given electronic document 121 to create a composite electronic document including the content obtained from the data source. The electronic document may also include a dynamically generated report that processes data from other sources. For example, a given electronic document 121 may be a dynamically generated report that analyzes data from multiple other data sources, such as the interaction of a user with other electronic documents 121.

[0038] In some cases, a given electronic document 121 may include one or more digital component tags or digital component scripts that reference the DCDS 112. In these cases, when the given electronic document 121 is processed by the user device 104 or the third-party device 106, the digital component tags or digital component scripts are executed by the user device 104. Execution of the digital component tags or digital component scripts configures the user device 104 to generate a request 114 for a resource including one or more digital components (referred to as a "component request"), which is transmitted to the DCDS 112 via the network 102. For example, the digital component tags or digital component scripts may enable the user device 104 or the third-party device 106 to generate a packetized data request including a header and payload data. The component request 114 may include event data that specifies characteristics such as the name (or network location) of a server from which the digital component is requested, the name (or network location) of the requesting device (e.g., the user device 104 or the third-party device 106), and / or information that the DCDS 112 may use to select one or more digital components to provide in response to the request. The component request 114 is transmitted by the user device 104 or the third party device 106 to a server of the DCDS 112 via the network 102 (eg, a telecommunications network).

[0039] The component request 114 may include event data specifying other event features such as the characteristics of the electronic document being requested and the location of the electronic document where the digital component can be presented. For example, event data may be provided to the DCDS 112, which event data specifies a reference (e.g., URL) to the electronic document (e.g., web page) in which the digital component will be presented, an available location of the electronic document that can be used to present the digital component, the size of the available location, and / or the media type that is eligible to be presented in the location. Similarly, event data specifying keywords associated with the electronic document ("document keywords") or entities (e.g., people, places, or things) referenced by the electronic document may also be included in the component request 114 (e.g., as payload data) and provided to the DCDS 112 to facilitate the identification of digital components that are eligible to be presented with the electronic document. The event data may also include a search query submitted from the user device 104 or the third-party device 106 to obtain a search results page, and / or data specifying the search results and / or the text, auditory, or other visual content included in the search results.

[0040] The component request 114 may also include event data related to other information, such as information that has been provided by a user of the client device, geographic information indicating the state or region from which the component request is submitted, or other information that provides context for the environment in which the digital component will be displayed (e.g., the time of day of the component request, the day of the week of the component request, the type of device (such as a mobile device or tablet device) that will display the digital component. The component request 114 may be transmitted, for example, over a packetized network, and the component request 114 itself may be formatted as packetized data having a header and payload data. The header may specify a destination for the packet, and the payload data may include any of the information discussed above.

[0041] The DCDS 112 selects the digital components to be presented with a given electronic document in response to receiving the component request 114 and / or using the information included in the component request 114. In some embodiments, the digital components are selected in less than one second (using the techniques described herein) to avoid errors that may be caused by delayed selection of the digital components. For example, a delay in providing the digital components in response to the component request 114 may result in page loading errors at the user device 104, or may result in other portions of the electronic document not being filled even after a portion of the electronic document is presented at the user device 104 or the third-party device 106. In addition, as the delay in providing the digital components to the user device 104 or the third-party device 106 increases, when the digital components are delivered to the user device 104 or the third-party device 106, the electronic document is more likely to no longer be presented at the user device 104, thereby negatively affecting the user's experience of the electronic document. In addition, for example, if the electronic document 121 is no longer presented at the user device 104 or the third-party device 106 when the digital components are provided, the delay in providing the digital components may result in a failure in the delivery of the digital components. This delay is reduced when the digital component is not provided with the content presented to the user. The user can obtain the content faster without the digital component and there is less data transferred and no delay between the database retrieving the component and the server providing the component, improving the user's experience.

[0042] The system uses data generated based on user interactions with digital components, such as electronic documents 121 , to personalize the selection of electronic documents 121 presented to the user device 104 .

[0043] The training data generator 108 collects user interaction data and generates one or more training data sets for a machine learning model to determine user interest in specific topics, content types, content presentation forms (e.g., types of UI elements), and various other features of the electronic document 121. The training data generator 108 can be implemented as one or more servers and includes a feature extractor 109.

[0044] The feature extractor 109 analyzes the user interaction data to extract features of the interaction, including user characteristics and electronic document characteristics. The user interaction data may be included, for example, in a request 112 for an electronic document 121. In some embodiments, when a user interacts with an electronic document served by a user device 104 or a third-party device 106, the interaction generates separate data indicating features of the interaction. The user interaction may include, for example, clicking on a UI element within an electronic document 121, viewing a specific portion of an electronic document 121, clicking to view more data related to a specific topic or portion of an electronic document 121, and the like.

[0045] The feature extractor 109 collects user interaction data by, for example, accessing stored user interaction data or intercepting communications between the user device 104 or the third-party device 106 and the electronic document server 120. In some embodiments, the user interaction data may be transmitted directly from the user device 104 or the third-party device 106 to the feature extractor 109. The user interaction data may be provided in various formats and, depending on the format, may indicate different features. The user interaction data in the form of an access request URL may indicate whether the user has entered a term, used a filter to narrow their search, selected to view data with a specific dimension value, etc. For example, the access request URL "http: / / example.com / data analyticsrreport / report / visitors-demographics gender / drill=analytics.gender:women,analytics.age:25-34 / " may indicate that the user has interacted with a report on example.com / dataanalyticsreport and has drilled down two dimensions (gender and age) to view analysis for women aged 25-34. The user interaction data may appear in other forms, such as data stored in association with a persistent identifier (e.g., a cookie), event data (e.g., application listeners tracking events such as mouse clicks, scrolling, zooming, touch gestures, etc.), or other types of interaction data that may be collected from a device such as a user device 104 or a third-party device 106, an electronic document server 120, a DCDS 112, or collected through a communication interface such as a network 102.

[0046] Once the user interaction data is collected and features are extracted, the system analyzes the extracted features to predict the user's interest in additional content, information, or analysis reports. In order to analyze the extracted features, the system generates training data to input into the model. For example, the system can generate training examples for the neural network. The trained neural network can then predict the user's interest in various features and provide recommendation data to DCDS112. The recommendation data can indicate the features of interest to a group of users of a specific user or a specific profile that meets the user's characteristics. The trained neural network can also provide recommendation data in the form of user interface generation data. For example, the user interface generation data can be an instruction that presents content to the user in a specific format when executed. For example, if a user conventionally selects a graphical view of an analysis report instead of a table view, the system can determine that this user - and other users who meet the specific profile of the user characteristics that matches the specific profile of the user characteristics of this user - are interested in viewing the analysis report in a graphical view based on the user interaction data of this user. Similarly, if a user historically requests presentation of a specific analysis report after presentation of different analysis reports, the system can modify the user interface to present a link to the specific analysis report when presenting the different analysis reports, even if the link to the specific analysis report was not predefined in the presentation of the different analysis reports and even if the specific analysis report is in an unlaunched state.

[0047] The training data generator 108 receives the extracted features of the user interaction data and generates one or more training data sets for the machine learning model. The training data generator 108 can use the user interactions and the extracted features to generate training examples. For example, the training data generator 108 can generate negative training examples for the neural network based on user interactions, where a company user, for example, dismisses a suggested query including a metric indicating the number of new users in Michigan by selecting a UI element that causes the suggested query to be dismissed. The generated training examples can additionally include data indicating one or more attributes of a user associated with the user interaction.

[0048] A user interest model 110 (UIM 110) that receives training data from a training data generator 108 trains a machine learning model and predicts user interests. The UIM 110 can be, for example, a neural network that receives training examples and user or profile information as input and outputs predictions of content that a user (or a user who fits the profile information) may be interested in. The UIM 110 can be applied to a specific user whose user-specific data has been received. For example, the UIM 110 can be used to predict whether the ElectronicsSalesManager 12 is interested in a metric such as the number of new users within several dimension values ​​(female, 18-24 years old, Spain). If the confidence level in the personalized recommendation generated by the UIM 110 is at least a confidence threshold level, the UIM 110 can provide the recommendation to the DCDS 112 or the user device 104 to generate a personalized UI or present personalized content, such as a UI element that is predicted to be of most interest to a specific user.

[0049] UIM 110 can also be applied to specific users for whom only profile level data is available. For example, a new user may not interact with a specific application or environment to a level sufficient to enable the system to generate personalized content or content presentation formats. Then, the system can provide profile level personalized recommendations for content or content presentation formats. For example, if a new (or rare) user is the sales operations director of a home and garden magazine located in New York, the system can determine a profile for the user that indicates that she is the operations director, that her department is the sales department, that her company is located in New York, and that she is interested in the metrics and dimensions associated with sales growth.

[0050] In this particular example, the system does not collect enough user-specific data for the new user through the user device 104. UIM 110 can receive user interaction data from the third-party devices 106 of other users, and UIM 110 can analyze the user interaction data that matches at least the threshold amount of the profile of the new user. For example, the system can analyze the user interaction data that matches at least three of the five profile characteristics of the new user. In some embodiments, the specific profile characteristics must match (for example, the user must have the same job title, the user must be in the same state, etc.). Once the system determines and selects the user interaction data from the third-party device 106 that matches the threshold amount of the profile of the new user, UIM 110 can use the training examples generated by the training data generator 108 according to the selected user interaction data to train the machine learning model. The trained UIM 110 can then be used to predict or select the content or content presentation format that the new user may be interested in at the profile level. This enables the generation of a customized user interface for the user, even if the historical interaction data available to the user is seldom or not available. In this way, the user can be directed to the data of interest more quickly, for example, by modifying the user interface to incorporate UI elements that launch reports for the data of interest when they are not launched, and without requiring the user to navigate through a large number of user interfaces or menus to launch reports for the data of interest.

[0051] In some implementations, if the confidence level in the personalized content or content presentation format recommendation at the user level or profile level does not meet the confidence threshold level, the system does not provide personalized recommendations.

[0052] For situations where the systems discussed herein collect personal information about users or may make use of personal information, users may be provided with the opportunity to control whether a program or feature collects personal information (e.g., information about a user's social network, social actions or activities, occupation, a user's preferences, or a user's current location), or to control whether and / or how content that may be more relevant to the user is received from a content server. In addition, certain data may be anonymized in one or more ways before it is stored or used, so that personally identifiable information is removed. For example, a user's identity may be anonymized so that personally identifiable information cannot be determined for the user, or a user's geographic location may be summarized where location information is obtained (e.g., city, ZIP code, or state level) so that the user's specific location cannot be determined. Thus, a user may control how a server collects and uses information about him or her.

[0053] Figure 2 is used in Figure 11. The operations of data flow 200 are performed by UIM or server 110 in communication with user device 104 or third-party device 106. In some implementations, the stages of flow 200 are performed within a network environment such as environment 100.

[0054] The system collects user interaction data and extracts features of these user interactions. The user interaction data may be stored in a user interaction database 206. Such user interaction data may include user profile data that may be stored in a user profile 208. Once the system has collected features of the user interactions, the system generates a training data set to train a user interest model that predicts content or content presentation formats that a particular user may be interested in. The training data is provided to a machine learning model, and the model provides an output vector of values ​​that predict content that may be of interest to the user.

[0055] In this particular example, the communications performed as stages of flow 200 occur over network 102, which may be the Internet.

[0056] The user device 104 includes a display that presents content to the user of the user device 104 within a UI. The visible portion of the content is within a viewport 202 of the display. In some embodiments, the viewport 202 is the visible portion of a particular application or web page that the user of the user device 104 is accessing. For example, if a digital component is presented within an application that does not cover the entire visible area of ​​the display of the user device 104, the viewport 202 is determined to be the visible portion of the application. In some applications, there may be more content than can be displayed at one time within the viewport 202. Such content may be on the same portion of the application as the content within the viewport 202 of the user device 104, but is not actually visible to the user and is within the invisible portion 203 of the user device 104. For example, content 204a and content 204b are visible within the viewport 202, but content 204c is within the portion 203 that is not visible to the user of the user device 104. Content 204 (i.e., content 204a, content 204b, and content 204c) may be as described above with reference to Figure 1 The various types of digital components described above. The analytical application supports a large number of options for presenting data, and not all reports, statistics, and available types of metrics can be presented in the visual portion of the analytical UI at one time. UIM 110 collects user interaction data to determine and predict user information that a particular user is most likely to be interested in. UIM 110 then modifies the analytical UI to present links to related reports, UI elements that display specific metrics, etc., to provide the most relevant information to a particular user when the available options cannot be displayed due to display limitations and may be overwhelming.

[0057] The user of the user device 104 interacts with the content 204 displayed by the user device 104, and generates and collects user interaction data indicating characteristics of these interactions. The user interaction details will be described below and with reference to Figure 3 Detailed description.

[0058] User interaction data can be collected in various forms, and the feature extractor 109 can parse and process all of these user interaction data. Entities that can have user interest scores include metrics, dimensions, dimension filters, reports, audiences / segments, goals, etc. Reports can include analytical data about specific data sets, and specific data sets can be aggregated or analyzed. The target audience or segment indicates the type of user that the user is interested in viewing the analysis. The user's goals can indicate the target metrics that the user wants to see when monitoring the analysis.

[0059] Metrics measure performance, behavior, and / or activity. For example, performance metrics for a particular application may include newUsers, a metric that indicates the number of new users in a particular time period. Other metrics may indicate, for example, the percentage of sessions that were newly created, the number of unique page views, the average amount of time spent on a web page, the number of unique social interactions that occurred on a web page, the average server connection time, the average page load time, etc. Different environments may have different metrics - an application may have a metric that indicates the amount of time spent on a particular screen of the application, a site may have a metric that tracks events, a search engine may have a metric that indicates the number of clicks a particular result received in a predetermined time period, and other environments may have a variety of other metrics including user-defined custom metrics.

[0060] Custom or user-defined user metrics may be given more weight than features that are not set by default or selected directly by the user. For example, if a user creates a custom metric to be included in a report or report template, the system may apply a higher weight to the custom metric within the training examples provided to the user interest model.

[0061] Dimensions represent specific categories or characteristics of user interaction data. For example, the dimensions of a user interaction dataset may include the type of device on which the user interaction was performed, the country in which the user interaction was performed, the type of interaction performed, etc. Different environments may have different available dimensions, and users may define custom dimensions to focus their analysis on or categorize the data into dimensions that interest them.

[0062] Filters can be specific characteristics according to which data is classified. For example, filters can be used to show only data from the last 30 days, or all data from Georgia, Nebraska, and New York, or only data from a specific profile of user characteristics. Different environments can have different filters, and users can define custom filters based on their interest in the specific data being analyzed.

[0063] In some examples, user device 104 is a mobile device such as a cellular phone, a virtual reality device (e.g., implemented in a headset or other device such as a combination of a speaker and a display), a smart phone, a personal digital assistant (e.g., implemented in a desktop speaker or other device such as a combination of a speaker and a display), or a tablet computer, and communicates via a wireless network. Training data generator 108 and UIM 110 can communicate via a wired network that is independent of the wireless network through which user device 104 communicates. For example, user device 104 may be a smart phone that communicates via a wireless cellular network, while training data generator 108 and UIM 110 may be remote servers that communicate via a wired network.

[0064] UIM 110 receives user interaction data from user device 104 via network 102, and predicts the user's interest in digital components. UIM 110 then modifies UI elements using user interest scores for presentation at user device 104. For example, UIM 110 may determine that user device 104 has interacted with a digital component provided by a digital component provider via a digital component server, which indicates a positive interest in a report on the number of users who make more than one purchase per day at a clothing store. UIM 110 identifies a unique identifier associated with the interaction data. The unique identifier may be, for example, a user or device identifier. In some embodiments, the unique identifier may be a user identifier carried across platforms and devices. For example, the unique identifier may be a user account used across browsers and devices, and may be used regardless of whether the user is using a software application or a web application.

[0065] The unique identifier can be protected when sent to a third party. For example, the unique identifier can be encrypted when sent in a forwarded opportunity to present content. In some examples, the forwarded opportunity and the unique identifier can be transmitted via a Secure Sockets Layer (SSL). In some examples, the unique identifier associated with a user of user device 104 can be reset by various methods, including via a factory reset of user device 104. The user can also choose not to send their unique identifier at all in any content request.

[0066] In some embodiments, the environment in which the system operates is a web application, and the user interaction data can be extracted from the URL of the web application. For example, the URL can indicate a hit page count and the number of page requests. Each hit has a page request URL that indicates the user-selected portion of the page, the metric the user is selecting, the UI element the user has selected to drill down to, etc., and the page request URL also indicates the name of the profile and / or user identifier or the network location where the request was made.

[0067] The feature extractor 109 can determine features at a user-specific level and a profile level. Exemplary profile-level fields include quality of service (QOS) level (e.g., whether the user has premium service), time when the profile was created (e.g., 20 Dec 2013 13:20:12), network location of the requested page, name of the page (e.g., Birds are Cool - website and store), time zone, or the most recent time the profile was seen. Such profile data can be stored, for example, as a user profile 208.

[0068] User interaction data can be collected from data logs collected from the user interface or from user interaction events. In some examples, the front end of the application can create custom events to track these interactions.

[0069] Dimensions can be determined by explicit user interactions or by default settings. In some embodiments, the user has explicitly selected dimensions, and these user explicit operations (e.g., selecting dimensions from a drop-down menu, drilling down to dimensions, etc.) can be recorded. Dimensions can also be determined according to default settings, such as default dimensions selected when generating content such as reports. For example, if a user in Italy generates a weekly report on the number of active (at least one user interaction) users aged 12-17, the feature extractor 109 can determine the default dimensions (e.g., Italy country) as well as the explicit dimensions (e.g., users aged 12-17, active users, etc.). In some embodiments, the report may be too general and include too many dimensions to be analyzed for user interests. For example, in an overview report, default dimensions may not be generated because that would include every dimension available to the system.

[0070] The feature extractor 109 can also determine features based on, for example, a snapshot of what a particular user was viewing when the data was collected. If a user is viewing a particular portion of a report about monthly sessions, the feature extractor 109 can record the configuration and associated features of the report as the user views the report.

[0071] In some analytical environments, content referred to as "insights" may be provided for display to users. Insights provide analytical insights into user interaction data that may be of interest to the user. For example, a pop-up window (or card) within a monthly review tab of a report may be presented as an insight, which indicates a year-long snapshot of the monthly progress of the metrics that the user is most interested in. Metrics for insights may include whether the insight is provided, how many times the insight is provided, whether the insight is viewed, when the insight is viewed, whether the insight is bookmarked for future reference, when the insight is bookmarked, whether the insight is discarded, when the insight is discarded, how many times the insight is shared, whether the insight is marked as helpful, how many times the insight is marked as helpful within a predetermined time period (e.g., 30 days), how many times the insight is marked as unhelpful within a predetermined time period, or any action that can be used to interact with the insight. Depending on the context of the content, these and other metrics can be applied to other applications or content. Each metric can have a default value. In some embodiments, the default value can be specified by the user.

[0072] Users can interact with each other or with the environment through, for example, a question-and-answer interface. The dimensions, metrics, and filters used by users when they ask or select questions related to their interests are recorded. For example, if a user asks "How many new customers viewed landing page 5 before making at least one purchase?" the system can determine that the user is interested in metrics indicating first-time customers, customers who purchased at least once, and the dimension landing page 5.

[0073] Users can also create custom alerts for data. For example, a regional strategist may be interested in monitoring user traffic levels at each of multiple regional product pages, and can create a custom alert when the user traffic level increases (positive or negative) by a threshold amount to determine when a region is popular. In another example, a user can be identified as a new user responsible for the global promotion of a series of products, and a custom alert can be created when a threshold number of customers purchase products from the series at an aggregate level. UIM 110 can use, for example, machine learning techniques to determine custom reports, metrics, dimensions, alerts, etc. that users will be interested in.

[0074] For example, user interaction data may be stored by linking user identifiers and profile identifiers with the extracted features. The user identifier is linked to a specific user, while the profile identifier is linked to a profile of user characteristics that may be associated with multiple users having the same interests, having the same role in an organization, etc. When generating training data, additional data, such as the frequency of use of each dimension, metric, filter, etc., may be stored and analyzed to determine which metrics are most useful. In some embodiments, the system may perform such analysis to learn the optimal allocation of computing resources to provide the best service to its user base.

[0075] User interaction data, user interest scores, frequency of use data, etc. can be stored in different formats for ease of use depending on the environment and the needs of the user. For example, the system can store user interaction data in a column-oriented format, so that the value of each field is stored separately, and the computational overhead is proportional to the number of fields actually read from the storage device. The system can only access relevant fields, and thus avoids reading the entire record to access, for example, a single field; therefore, the system can use resources in this format more efficiently than other storage formats. Other formats that allow different efficiency levels for other environments include storing data in spanner tables, in matrix decompositions, and the like. In some embodiments, the system stores different types of data in different formats depending on the environment in which the data set or data type is most frequently used.

[0076] The data may be purged at fixed intervals, and in some cases, the data is only stored for a predetermined period of time to maintain the user's privacy. The system may determine a minimum period of time to maintain a level of accuracy in predicting the user's interests, and may store the user interest data for the determined minimum period of time. In some implementations, the user may indicate a period of time for which they will allow the system to store their data. For example, a user interested in a more personalized experience may allow the system to store their data for a longer period of time. The system may store the user interest score in the user's profile, while purging the user interaction data to maintain the user's level of personalization.

[0077] Once the feature extractor 109 has processed the user interaction data and extracted relevant features, the training data generator 108 generates training examples. For example, the training data generator 108 can format the user interaction data features into input and expected output pairs. These input-output pairs can be provided to the UIM 110 to train the model.

[0078] In this particular example, the training data generator 108 samples the user interaction data to generate training data that is input to the neural network 111a. The training data generator 108 may select random samples to obtain negative examples to train, for example, a logistic regression model. The training data generator 108 may also obtain samples of the most frequent unviewed metrics to maximize margin. The training data generator 108 may obtain multiple samples of the user interaction data to generate one or more training data sets.

[0079] When generating the training data set, the training data generator 108 selects training features to use in the training data set from the user's historical interactions with the report. The training data generator 108 determines the training example weights in one of several ways. In some embodiments, the training data generator 108 may weight each training example equally and only sample data points within a predetermined time period. In some embodiments, the training data generator 108 may use a decay function to weight the training examples differently depending on how recent the training examples are.

[0080] The system can split different percentages of modeling time and resources into training, validation, and generating test data. The system can also cross-validate the output of the model.

[0081] UIM 110 receives training data and trains a machine learning model to select digital components and / or content presentation formats. The machine learning model can use any of a variety of techniques, such as decision trees, linear regression models, logistic regression models, neural networks, classifiers, support vector machines, inductive logic programming, model combinations (e.g., using techniques such as bagging, boosting, random forests, etc.), genetic algorithms, Bayesian networks, etc., and can be trained using a variety of methods, such as deep learning, perceptrons, association rules, inductive logic, clustering, maximum entropy classification, learning classification, etc. In some examples, the machine learning model uses supervised learning. In some examples, the machine learning model uses unsupervised learning. The machine learning model can also use wide and deep learning, long short-term memory modeling, boosting, matrix decomposition, user embedding, or project embedding.

[0082] Wide and Deep Learning trains a wide linear memory model for memorization and a deep neural network for generalization across many features to combine the strengths of both models and is particularly useful for general, large-scale regression and classification problems with sparse inputs. For example, Wide and Deep Learning is very useful for problems with classification features that have a large number of possible feature values.

[0083] For ease of explanation, the operation of the neural network 111a is described with respect to logistic regression, but it should be understood that the neural network 111a may perform any of the numerous other techniques described above.

[0084] Neural network 111a is used to predict the probability that a particular user / profile is interested in a given feature. For example, given a (user, profile) pair and a metric newUsers, neural network 111a can be trained to predict the probability that (user, profile) is interested in a given metric newUsers. In this example, neural network 111a uses training tags indicating whether the (user, profile) pair is interested in newUsers. For a single entity (e.g., metric, dimension, dimension filter, audience / segment, target, etc.), UIM 11 calculates a user interest score for the entity. When a particular digital component includes two or more entities, UIM 110 can calculate a separate user interest score and confidence score for each entity, and aggregate these scores to generate an overall interest and confidence score.

[0085] The output of the neural network 111a is a prediction of the probability of a specific user, either a user for whom previous user interaction data was collected, or a user who matches the profile data to a certain extent. The UIM 110 can provide personalized digital component recommendations, personalized UI elements and presentation formats, and / or personalized suggestions for user queries, such as search queries, form fields, etc.

[0086] In some implementations, UIM 110 can provide suggestions for digital components, such as specific metrics to be included in reports, dimension values ​​of interest to the user, specific dimension filters to apply, etc. UIM 110 can determine how much content is visible within the UI's viewport 202. In some implementations, if there is more content to be presented in the UI than there is visual real estate in the viewport 202, and the content must be presented in the non-visible portion 203, UIM 110 can assign a visual presentation location to a specific digital component based on the user's interest score for the component.

[0087] In some embodiments, UIM 110 determines the presentation format of the digital components that the user is most likely to be interested in. For example, if the user prefers to generate a report with list information so that she can sort the data herself, UIM 110 can know that the user prefers list information rather than graphical information, and can generate an appropriate user interface - a table - in which the digital components that the user may be interested in are presented. UIM 110 generates UI elements using user interface generator 111b. User interface generator 111b generates instructions that, when executed, cause the selected digital components to be presented in a specific visual format that the user may be interested in.

[0088] In some embodiments, UIM 110 can determine what the user may be interested in for a query. For example, when a user enters a search query "how many repeat customers are there...", UIM 110 can determine that because the user is an account manager for Chinese and Japanese accounts, the user may be interested in the query "how many repeat customers did the Chinese and Japanese branches see this month". Based on previously recorded user interactions, UIM 110 can provide suggestions to the user, such as autofill suggestions for queries, form fields, drill-down paths, etc. When providing suggested questions, UIM 110 evaluates whether a particular user is interested in the metrics and dimensions contained in the suggested questions; suggested questions with higher overall interest scores are more likely to be shown in the visible area of ​​the analysis UI.

[0089] UIM 110 calculates an overall user interest score for a particular feature, which may be a frequency score that decays over time. This overall score may be used most frequently when considering whether to personalize the user experience around a particular metric / dimension. UIM 110 also calculates a confidence score that represents the accuracy of the personalized data for a particular user or user profile. In some embodiments, if the user-specific personalized score does not meet the threshold score, the system may use profile-level personalization instead. Profile-level personalization is less specific, but still useful because a particular user may have similar interests to other users with the same profile, or have a threshold amount of similarity to the profile. For example, in the case of a new user, if this is her first time browsing the analytics application, and the system does not have enough user-specific user interaction data to generate personalization in which the confidence score meets the threshold score, then if the profile-level personalization has a confidence score that meets the threshold score, UIM 110 may provide a personalized report at the profile level.

[0090] The system may also save computing resources by determining not to personalize data / UI for a particular user if there is not enough information, but instead provide default information. For example, if both the user-specific personalization confidence score and the profile-level personalization confidence score of the user do not meet a threshold confidence level. Instead of providing personalized data that the user may not be interested in, the system may provide default information and learn from the user's interaction with the default data to determine what data to provide next time / for a similar user with a similar profile.

[0091] For example, for each user, a potential digital component in the form of an insight may be scored by the UIM 110 based on each metric and dimension in the component. The UIM 110 may then utilize the user interest score to more accurately rank and score the insight, and then rank it higher or lower for the user. In some embodiments, if there is a limit to the number of digital components that can be presented within the viewport 202 of the UI, the UIM 110 may determine the visual presentation position of the digital components so that components that may be of interest to the user are presented within the viewport 202 of the UI, and components that may be less of interest to the user are presented toward the non-viewable portion 203 of the UI. For example, if an insight for the user profile jane1234 includes the metric BounceRate and the dimension Japan, the UIM 110 calculates scores for the metric and the dimension, and determines the interest level of jane1234 for each of these entities, so as to factor the interest scores for the individual entities into the overall interest score for the insight.

[0092] An app can have a diverse user base in terms of interests and skill levels; in order to serve as many user bases as possible, many apps are developed to be powerful and feature-rich. As the app feature set expands, the complexity and variety of options become overwhelming for many users. For example, while a user may access an app and see hundreds of reports, segments, toggles, and actions, most of her interest lies in a few metrics (e.g., number of target users and publisher revenue within segment 5A), a set of dimensions (e.g., device category and country), and a value (e.g., mobile, China, and Japan). Without information about the topics that the user is interested in, the app cannot be optimized for her needs to provide a useful, simple UI.

[0093] By providing user interest scores for entities in the analytical context, the system allows personalization to be used in the scoring algorithm for scoring metrics that are of interest to the user higher than metrics that are not of interest to the user. The user interest score can be used to identify the user's goals and intent (e.g., get more campaign revenue, optimize product sales), and categorize metrics by intent to present growth opportunities to the right user. The user interest score can be used to generate suggested queries or content for the user to search for - for example, suggestions for specific metrics / dimensions / filters that can be included in the query can be suggested based on the user's context.

[0094] The context includes the entity the user is looking at (e.g., report, data point, insight, etc.) or the question the user previously submitted to the system. When suggesting entities that are not within the user's current context (such as metrics, dimensions, or dimension filters), the system can select metrics, dimensions, or dimension filters with high user interest scores based on the user's previous interactions. For example, if the user has previously asked for "revenue growth last year," a subsequent suggestion could be "revenue growth in the United States last year."

[0095] By understanding the interests of users at multiple levels, the system can develop and adjust features for users and other users who share features with these users. For example, the application of the system can include serving corporate customers or enterprise customers with specific roles in the organization. By understanding the metrics, dimensions and values ​​(e.g., e-commerce revenue, specific activities, data collected from Alabama, etc.) that users care about, the system can present relevant digital components to users in a way that users can easily understand. This improved machine learning efficient UI generation system reduces the computing resources used to determine and select digital components and generate UI elements, without wasting resources on content or formats that users are unlikely to be interested in. Understanding the topics that users care about or the roles of users in the enterprise can be used as input or context of user interest learning models. For example, if a user is in an email marketing position, the user may be interested in content delivered by email and e-commerce revenue. Understanding the business intention or goal of the user can be used as input of the user interest learning model. For example, if the user's intention is to obtain at least $200,000 in revenue from mobile content (mobile content) next month, the system can determine that the user is interested in mobile devices and marketing revenue. UIM 110 can then weight these metrics more heavily in its user interest scoring model.

[0096] Figure 3200 is an example UI 300 for user interaction with a UI presenting a digital component with specific features. The process shown in flow 200 collects data about the interaction between the user and the digital component being presented, and populates a user interaction database 206, by retaining a record of user interaction, extracting features from the user interaction data, generating training examples based on the extracted features, and using the training examples to train the user interest model to improve the personalization and selection of digital components and digital component presentation formats. In some embodiments, a resource (e.g., a web page or application) may contain more content than is displayed at one time in the viewport of a client device. For example, an analysis report may contain several large graphs and tables, and if the user does not scroll down, the user cannot view all the content. If the content is provided outside the viewport of the UI (e.g., the non-viewable portion 203), the user device 104 can avoid presenting data, thereby saving processing and memory resources. Therefore, the UIM 110 can reduce and / or optimize the use of computing resources by modifying the analysis report and UI elements to show the UI elements that the user is most likely to be interested in in the viewport of the UI. Flow 200 involves training data generator 108 , UIM 110 , and user device 104 .

[0097] Each of the user actions takes into account the user interest score. For example, when a user views a browser report, whether she is viewing the default primary dimensions shown in the report or whether she has selected custom metrics and dimensions is taken into account when training UIM 110. In addition, the visual aspects of the report (e.g., which metrics are shown, which metrics are scrolled to, which metrics the user spends the most time looking at) are taken into account in the training examples.

[0098] The system can determine whether to use specific user interaction data and which features of the selected user interaction data set to use in generating training examples for the user interest model. In one example, when a user views an overview report, the report may include too many metrics to be used for relevant training examples, the system can record the user's view of the report and analyze only the portion of the report that was visible to the user and that the user spent a threshold amount of time viewing. In another example, when the user is viewing a real-time report, the report is dynamically updated and it is difficult to determine which values ​​to record. The system can take a snapshot of the report at a specific time when the user views the report and extract features from the snapshot.

[0099] There are many default values ​​for entities in reports and analytical content presented to users. For example, an overview report may include all available metrics and dimensions, and a monthly sales summary report may include default metrics related to sales and revenue, and there are metrics, dimensions, filters, etc. that the user may not be interested in, but are included by default. Typically, all dimensions, metrics, filters, and other entities indicated in user interaction data are recorded, even if it is just viewing an overview report. Therefore, it is advantageous to weight user actions and interactions more heavily, because these interactions are explicit actions of the user's body parts, and these interactions can be more confidently interpreted as intentional expressions of interest.

[0100] For example, drilling down to see more specific information can be recorded as a user's interest in more specific information, as well as interest in the metric being drilled down. Other actions that can be interpreted as intentional expressions of interest or disinterest can include changing, removing or adding a metric, sorting by a metric, changing a primary dimension, changing, removing or adding a secondary dimension, changing, removing or adding a filter, interacting with insights beyond scrolling through digital components, viewing and creating custom reports, manually selecting and manipulating data within reports, tabs, UI elements, manual selections (e.g., selecting a graph, selecting advanced filters in a table, selecting a specific target), applying a market segment - custom or default, performing advanced analysis.

[0101] Meaningful feature interactions by users in the context of events indicate help the system improve analytics and the user's application experience. Using previously collected user interaction data, the system can recommend new features and personalize the user's reporting experience (e.g., by suggesting reports to view, providing filtered views of existing reports, creating audiences, creating sessions, adding new users to the administrator, etc.). When an action is associated with some data (e.g., the set of metrics / dimensions associated with the action), the system's record keeping is intended to maintain the link and the time and order in which the user took the action.

[0102] The interaction of the user with the UI element presenting the content can provide positive or negative training examples for the machine learning model.For example, in viewport 202, content item 204a includes UI element 302. UI element 302 represents that the user utilizes it to cancel, refuse or generally provide negative feedback to content or UI element.For example, if the user selects UI element 302 to cancel content item 204a, user interaction data can indicate that content item 204a is cancelled, and the user is not interested in content item 204a.Like this, the system allows the user to provide feedback through her daily interaction with application, without additional interaction.

[0103] In some implementations, UIM 110 may modify the analytics UI immediately based on user interaction data. For example, once a user cancels a particular UI element that contains aggregated statistics at the state level but not at the national level, UIM 110 may determine that the user may not be interested in statewide statistics and will not present those statistics within the UI's viewport.

[0104] In some embodiments, UI 300 can present feedback UI element, and through feedback UI element, the user can provide further details about why she chooses to cancel content item 204a.For example, the user can select or input free response from the predetermined options.If the user indicates that she is not interested in content item 204a, because she is not interested in the presentation mode of content item 204.By providing additional feedback, the user can receive better, more relevant and more understandable content.UIM 110 can analyze feedback to determine, for example, whether the user is temporarily not interested in analytical data but wants to see it again at a different time, whether the user is interested in analytical data but wants to see it in different formats, whether the user has turned to different roles in her organization and has different statistical report requirements, etc.

[0105] In viewport 202, content item 204b includes UI element 304. UI element 304 represents UI element, and with this UI element, the user who wants to see the wider range of content related to content item 204b can see additional content, and provides positive indication of interest to content item 204b. For example, if the user selects UI element 304 to see "other content in subsection B", user interaction data can indicate that content item 204b is cancelled, and the user is interested in seeing different or more content related to content item 204b. The user can be viewing the UI element of user statistics data of Bergen County in presenting subsection B---New Jersey, and select UI element 304 to see other content in the New Jersey subsection, such as Hudson County. This user interaction can produce user interaction data, to indicate that the user is interested in the content in dimension "New Jersey", but is not interested in Bergen County.

[0106] In viewport 202, content item 204a includes UI element 306. UI element 306 represents UI element, and with this UI element, the user who wants to narrow the content scope related to content item 204a can see additional content, and provides positive indication of interest to content item 204a. For example, if the user selects UI element to go deep down and see more content in the content of category 3. In some embodiments, UI element 306 allows the user to select different categories in the classification where content item 204a is located. The user can be viewing the UI element presenting the statistics of the user who owns a vehicle, and select UI element 306 to go deep down to see the user who owns a minivan. This user interaction can generate user interaction data, to indicate that the user is interested in the dimension of the user who owns a vehicle, and further interested in the dimension of a minivan. UIM 110 can use this user interaction data to predict, for example, the user may be interested in the dimension of the user who owns a family of more than two people.

[0107] In hidden portion 203, content item 204c is presented, but may not be rendered. For example, if a threshold portion of content item 204c is not visible from the current position of the user in UI 300, system 100 can avoid rendering content item 204c to save computing resources. In some embodiments, if the user scrolls down to view content item 204c, the action of scrolling that makes content item 204c reach at least a predetermined time period in viewport 202 can be considered as a positive indication of interest in content item 204c. UIM 110 can then generate positive training examples from the scrolling action. The user can see the top of content item 204c and determine that content item 204c is of interest to him; if the user scrolls so that content item 204c reaches at least 20 seconds in viewport 202, such user interaction data can indicate to UIM 110 that the user is interested in content item 204c. However, if the user scrolls so that content item 204c is within viewport 202 and determines that content item 204c is not interesting, he may scroll away; user interaction data indicating that content item 204c is not within viewport 202 for at least 20 seconds may be interpreted by UIM 110 as a negative training example.

[0108] Other user interactions such as zooming in on a portion of a page or a particular UI element, marking a content item or presentation format as helpful, spending at least a predetermined period of time viewing or interacting with a content item or UI element, querying for specific content, etc. may also serve as positive indications of interest.

[0109] Figure 44 is a flow chart of an example process 400 for personalizing an analysis environment and user experience based on previously collected user interaction data. In some implementations, process 400 may be implemented by one or more servers. For example, process 400 may be implemented by Figure 1-Figure 2 The training data generator 108 and UIM 110 of the system may be implemented. In some implementations, the process 400 may be implemented as instructions stored on a non-transitory computer-readable medium, and when the instructions are executed by one or more servers, the instructions may cause the one or more servers to perform the operations of the process 400.

[0110] The system receives user interaction data indicating a particular user's interaction with one or more UI elements of an initial UI of an analytics reporting application presented to the particular user (402). For example, training data generator 108 may collect the user interaction data, and feature extractor 109 may extract features of the recorded user interactions. As described above, these interactions may include, for example, clicking on a UI element to indicate interest, canceling a UI element, viewing other UI elements related to a UI element, and the like.

[0111] The system can build a database that tracks user interactions. The user interaction data can represent, for example, features of content that a particular user is interested in. The data is processed through a pipeline of data processing tasks, for example, the system processes the data to extract features of the data, such as metrics, dimensions, filters, etc. For example, the database can track how often and how recently a particular user has interacted with a particular metric, dimension, dimension filter, etc. In this particular example, the database can be the user interaction database 206.

[0112] The system converts the received user interaction data into a training data set (404). Once the system has extracted features of the user interaction data, the system generates training examples to provide to the machine learning model to predict user interests. In this particular example, the feature extractor 109 provides the extracted features to the training data generator 108, which uses the recorded user interaction data and the extracted features to generate training examples. The training examples may include pairs of inputs and expected outputs. The training examples may be influenced by the user interaction data and used by the UIM 110 to identify analysis report items and UI elements that may be of interest to the user.

[0113] The system inputs the training data set to the machine learning model (406). In this particular example, the training data generator 108 provides training examples to the UIM 110, and in particular to the neural network 111a. As described above, other types of machine learning models can be used.

[0114] In response to the training data set, the system trains the machine learning model (408). In this particular example, UIM 110 trains neural network 111a using training examples from training data generator 108. In some implementations, UIM 110 undergoes multiple iterations of training, validation, and feature extraction. For example, UIM 110 may continue to train neural network 111a until a threshold confidence score has been reached for a particular feature for a particular user or profile.

[0115] The system generates a set of user interest scores for a particular user using the trained machine learning model, wherein each of the user interest scores indicates the user's interest in accessing information corresponding to a UI element of the application (410). In this particular example, UIM 110 generates a user interest score for each entity referenced by the UI element using neural network 111a to provide an overall interest score for the UI element.

[0116] The system may then determine that at least one of the set of user interest scores positively indicates a user interest in accessing information corresponding to a particular UI element that is not included in the initial UI and has at least a threshold score 412. In this particular example, UIM 110 may determine that at least one of the set of user interest scores indicates that the user is interested in content corresponding to the UI element (e.g., including different content, or including the same content in a different presentation format).

[0117] In response to determining that the user interest score in the specific UI element is at least the threshold score, the system dynamically modifies the initial UI, including merging the specific UI element into the initial UI to obtain an updated UI (414) that includes the specific UI element that was not included in the initial UI. In this particular example, if UIM 110 determines that the user is interested in a specific content presentation format, user interface generator 111b can generate UI instructions that, when executed, cause content to be presented in the specific content presentation format. If UIM 110 determines that the user is interested in content that was not previously presented, UIM 110 uses the user interest score to select content that the user is most likely to be interested in. UIM 110 can rank or select content and modify the analysis UI to present the content that the user is most likely to be interested in within the viewport of the UI, thereby improving the user experience and helping the user navigate the large number of options and UI elements available within the analysis UI.

[0118] The system presents an updated UI including a specific UI element to the user (416). In this particular example, the user interface generator 111b provides UI instructions to the user device 104 to cause the presentation of the specific UI element. The UIM 110 modifies the specific UI element based on the generated user interest score to present the content that the user is most likely to be interested in.

[0119] The system monitors further user interactions with the updated UI after presenting the updated UI to the user (418). In this particular example, the training data generator 108, and more specifically, the feature extractor 109, continuously collects user interaction data and extracts features, records the user interaction data in the user interaction database 206, and records the user profile data in the user profile 208.

[0120] The system updates the machine learning model based on further user interactions (420). In this particular example, the training data generator 108 generates updated training examples and provides the updated training examples to the UIM 110. The UIM 110 can continuously retrain the neural network 111a based on the current user interaction data.

[0121] The system selects an updated set of UI elements to present in the UI based on the updated machine learning model (422). In this particular example, user interface generator 111b can continuously update UI instructions provided to user device 104, and neural network 111a can continuously update user interest scores that UIM 110 uses to dynamically modify the user's environment and analytical experience based on recorded user interaction data.

[0122] Figure 5 5 is a block diagram of an example computer system 500 that can be used to perform the above operations. System 500 includes a processor 510, a memory 520, a storage device 530, and an input / output device 540. Each of components 510, 520, 530, and 540 can be interconnected, for example, using a system bus 550. Processor 510 is capable of processing instructions for execution within system 500. In one embodiment, processor 510 is a single-threaded processor. In another embodiment, processor 510 is a multi-threaded processor. Processor 510 is capable of processing instructions stored in memory 520 or storage device 530.

[0123] Memory 520 stores information within system 500. In one embodiment, memory 520 is a computer readable medium. In one embodiment, memory 520 is a volatile memory unit. In another embodiment, memory 520 is a non-volatile memory unit.

[0124] The storage device 530 can provide mass storage for the system 500. In one embodiment, the storage device 530 is a computer-readable medium. In various embodiments, the storage device 530 may include, for example, a hard disk device, an optical disk device, a storage device shared by multiple computing devices (e.g., a cloud storage device) over a network, or some other mass storage device.

[0125] The input / output device 540 provides input / output operations for the system 500. In one embodiment, the input / output device 540 may include one or more network interface devices, such as an Ethernet card, a serial communication device, such as an RS-232 port, and / or a wireless interface device, such as an 802.11 card. In another embodiment, the input / output device may include a drive device configured to receive input data and send output data to other input / output devices, such as a keyboard, a printer, and a display device 560. However, other embodiments may also be used, such as a mobile computing device, a mobile communication device, a set-top TV client device, etc.

[0126] Although already Figure 5 An example processing system is described in the specification, but the subject matter and implementation of the functional operations described in this specification may be implemented in other types of digital electronic circuits, or in computer software, firmware or hardware including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of them.

[0127] An electronic document (referred to simply as a document for brevity) does not necessarily correspond to a file. A document may be stored in a portion of a file that holds other documents, in a single file dedicated to the document in question, or in multiple coordinated files.

[0128] The subject matter and the embodiments of the operation described in this specification may be implemented in digital electronic circuits, or in computer software, firmware or hardware including the structure disclosed in this specification and its structural equivalent, or in a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, that is, one or more modules of computer program instructions, encoded on one or more computer storage media for execution by a data processing device or for controlling the operation of a data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device. The computer storage medium may be or be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. In addition, although the computer storage medium is not a propagation signal, the computer storage medium may be the source or destination of the computer program instructions encoded in the artificially generated propagation signal. The computer storage medium may also be or be included in one or more separate physical components or media (e.g., multiple CDs, disks or other storage devices).

[0129] The operations described in this specification may be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0130] The term "data processing apparatus" includes all types of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer, a system on a chip, or a plurality or combination of the foregoing. The apparatus may include a dedicated logic circuit, for example, an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the device may also include code that creates an execution environment for the computer program in question, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment may implement a variety of different computing model infrastructures, such as network services, distributed computing, and grid computing infrastructures.

[0131] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to be executed on one or more computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0132] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, and the device can also be implemented as special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits).

[0133] For example, processors suitable for executing computer programs include general and special microprocessors. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for performing actions according to instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, or be operably coupled to the mass storage device to receive data from the one or more mass storage devices or transfer data to the mass storage device or both. However, a computer does not require such a device. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (such as a universal serial bus (USB) flash drive), just to name a few examples. Devices suitable for storing computer program instructions and data include all forms of nonvolatile memory, media and storage devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROMs and DVD-ROMs. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0134] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) display screen, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0135] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component, such as a data server, or that includes a middleware component, such as an application server, or that includes a front-end component, such as a client computer with a graphical user interface or a web browser, through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), inter-networks (such as the Internet), and peer-to-peer networks (such as ad hoc peer-to-peer networks).

[0136] A computing system may include a client and a server. The client and the server are usually remote from each other and typically interact through a communication network. The relationship between the client and the server is due to the computer programs running on the respective computers, and there is a client-server relationship between each other. In some embodiments, the server transmits data (e.g., an HTML page) to a client device (e.g., in order to display data to a user interacting with the client device and receive user input from a user interacting with the client device). Data generated at the client device (e.g., as a result of user interaction) can be received from the client device at the server.

[0137] Although this specification contains many specific implementation details, these should not be interpreted as limitations on any invention or the scope that may be claimed, but rather as descriptions of features of specific embodiments of specific inventions. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although the above-mentioned features may be described as working in certain combinations, and even initially claimed as such, in some cases, one or more features in the claimed combination may be deleted from the combination, and the claimed combination may point to a sub-combination or a variant of a sub-combination.

[0138] Similarly, although operations are described in a particular order in the accompanying drawings, this should not be understood as requiring that the operations be performed in the particular order or sequence shown, or that all of the operations shown be performed, to achieve the desired results. In some cases, multitasking and parallel processing can be advantageous. In addition, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0139] Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. Furthermore, the processes described in the accompanying drawings do not necessarily require the particular order or sequence shown to achieve the desired results. In certain embodiments, multitasking and parallel processing can be advantageous.

Claims

1. A method performed by a computing device, the method comprising: generating a set of user interest scores for the user using a trained machine learning model, wherein each user interest score indicates an interest of the user in accessing information corresponding to a user interface (UI) element of the application, wherein the trained machine learning model is trained using a training data set, the training data set comprising a set of features related to user interactions with one or more UI elements of the application, the features being selected for training the trained machine learning model based on respective durations of time that the UI elements corresponding to the features are presented to one or more users; determining that at least one user interest score in the set of user interest scores positively indicates a user interest in accessing information corresponding to a particular UI element that was not included in an initial UI presented to the user; In response to determining that at least one user interest score in the set of user interest scores positively indicates a user interest in accessing information corresponding to the specific UI element, dynamically modifying the initial UI, including merging the specific UI element into the initial UI to generate an updated UI that includes the specific UI element that was not included in the initial UI; Presenting an updated UI including the specific UI element to the user; After presenting the updated UI to the user, monitoring further user interactions with the updated UI; updating the trained machine learning model based on further user interactions; and An updated set of UI elements that are presented in the updated UI are selected based on the updated machine learning model.

2. The method of claim 1, wherein the trained machine learning model is one of: (i) a logistic regression model, (ii) a machine learning model that combines a wide linear model for memory with a deep neural network for generalization across multiple features, or (iii) a long short-term memory model.

3. The method according to claim 1 also includes determining that a specific user interest score associated with another UI element in the user interest score set is less than a threshold score, wherein based on determining that the specific user interest score associated with the other UI element is less than the threshold, the other UI element is not displayed to the user in the updated UI.

4. The method of claim 1 , further comprising receiving a user interaction data set indicating interactions of a set of a plurality of different users with the one or more UI elements; Generate a user group training data set according to the user interaction data set; and The trained machine learning model is trained using the user group training data.

5. The method according to claim 4, wherein: The user interaction dataset relates to two or more different topics, and each of the two or more different topics is assigned a different weight based on the importance of the topic to at least one user among the plurality of different users.

6. The method according to claim 1, further comprising: After obtaining a sufficient amount of user interaction data for a user and training a user-specific machine learning model using the user interaction data for the user, determining that the user-specific machine learning model has at least a threshold confidence level for the user; and In response to determining that the user-specific machine learning model has at least a threshold confidence level, switch from the trained machine learning model to the user-specific machine learning model for one or more subsequent updates to the UI for the user.

7. The method of claim 1, wherein at least one of the initial UI or the updated UI is presented to the user by a gaming console.

8. The method of claim 1, wherein the trained machine learning model is trained for the user, the method further comprising: identifying a second user determined to be similar to the user; and Based on the second user being determined to be similar to the user, the trained machine learning model and / or the updated machine learning model is used to select UI elements for presentation to the second user.

9. The method according to claim 1, wherein: The set of features related to user interaction with UI elements of one or more UIs of an application includes one or more of: (i) features related to zooming to view the content of a particular UI element, (ii) features related to marking particular content as helpful, or (iii) features related to submitting a query for additional content.

10. The method according to claim 1, further comprising: Identifying specific UI elements that the user is determined to no longer be interested in; and The particular UI element is prevented from being presented to the user for at least a specified period of time.

11. The method according to claim 1, further comprising: initially training the trained machine learning model based on user interaction data of users; identifying one or more second users that share common user characteristics with the user; and The trained machine learning model is used to select a UI element for presentation to the one or more second users.

12. A system comprising: one or more processors; and a memory communicatively coupled to the one or more processors, the memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including: generating a set of user interest scores for the user using a trained machine learning model, wherein each user interest score indicates an interest of the user in accessing information corresponding to a user interface (UI) element of the application, wherein the trained machine learning model is trained using a training data set, the training data set comprising a set of features related to user interactions with one or more UI elements of the application, the features being selected for training the trained machine learning model based on respective durations of time that the UI elements corresponding to the features are presented to one or more users; determining that at least one user interest score in the set of user interest scores positively indicates a user interest in accessing information corresponding to a particular UI element that was not included in an initial UI presented to the user; In response to determining that at least one user interest score in the set of user interest scores positively indicates a user interest in accessing information corresponding to the specific UI element, dynamically modifying the initial UI, including merging the specific UI element into the initial UI to generate an updated UI that includes the specific UI element that was not included in the initial UI; Presenting an updated UI including the specific UI element to the user; After presenting the updated UI to the user, monitoring further user interactions with the updated UI; updating the trained machine learning model based on further user interactions; and An updated set of UI elements that are presented in the updated UI are selected based on the updated machine learning model.

13. The system of claim 12, wherein the trained machine learning model is one of: (i) a logistic regression model, (ii) a machine learning model that combines a wide linear model for memory with a deep neural network for generalization across multiple features, or (iii) a long short-term memory model.

14. The system of claim 12, the operations further comprising: Determine that a particular user interest score associated with another UI element in the set of user interest scores is less than a threshold score, wherein the other UI element is not displayed to the user in the updated UI based on determining that the particular user interest score associated with the other UI element is less than the threshold.

15. The system of claim 12, the operations further comprising: receiving a user interaction data set indicating interactions of a set of a plurality of different users with one or more UI elements; generating a user group training data set according to the received user interaction data set; and The trained machine learning model is trained using the user group training data.

16. The system of claim 15, wherein: The user interaction dataset relates to two or more different topics, and each of the two or more different topics is assigned a different weight based on the importance of the topic to at least one user among the plurality of different users.

17. The system of claim 12, wherein the operations further comprise: After obtaining a sufficient amount of user interaction data for the user and training the user-specific machine learning model using the user interaction data for the user, determining that the user-specific machine learning model has at least a threshold confidence level for the user; and In response to determining that the user-specific machine learning model has at least a threshold confidence level, switch from the trained machine learning model to the user-specific machine learning model for one or more subsequent updates to the UI for the user.

18. The system of claim 12, wherein: The trained machine learning model is trained for the user, and the operations further include: identifying a second user determined to be similar to the user; and Based on the second user being determined to be similar to the user, the trained machine learning model and / or the updated machine learning model is used to select UI elements for presentation to the second user.

19. The system of claim 12, wherein: The set of features related to user interaction with UI elements of one or more UIs of an application includes one or more of: (i) features related to zooming to view the content of a particular UI element, (ii) features related to marking particular content as helpful, or (iii) features related to submitting a query for additional content.

20. A non-transitory computer-readable storage device storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: generating a set of user interest scores for the user using a trained machine learning model, wherein each user interest score indicates an interest of the user in accessing information corresponding to a user interface (UI) element of the application, wherein the trained machine learning model is trained using a training data set, the training data set comprising a set of features related to user interactions with one or more UI elements of the application, the features being selected for training the trained machine learning model based on respective durations of time that the UI elements corresponding to the features are presented to one or more users; determining that at least one user interest score in the set of user interest scores positively indicates a user interest in accessing information corresponding to a particular UI element that was not included in an initial UI presented to the user; In response to determining that at least one user interest score in the set of user interest scores positively indicates a user interest in accessing information corresponding to the specific UI element, dynamically modifying the initial UI, including merging the specific UI element into the initial UI to generate an updated UI that includes the specific UI element that was not included in the initial UI; Presenting an updated UI including the specific UI element to the user; After presenting the updated UI to the user, monitoring further user interactions with the updated UI; updating the trained machine learning model based on further user interactions; and An updated set of UI elements that are presented in the updated UI are selected based on the updated machine learning model.

Images