Large language model-based response to targeted UI element

Abstract

To provide an electronic device, method and program for generating a response to a targeted UI element.SOLUTION: A method for causing a display device to display a response includes: receiving, by a first content manager, in response to a request for first content, the first content from a first remote web server; detect, by a targeted UI element manager, a targeted UI element within a portion of the first content, and transmit, in response to detecting the targeted UI element, the portion of the first content at least; receive, by a second content manager, second content including a difference based on the type of the targeted UI element, for the portion of the first content, from a generative model; and initiate, by a rendering display manager, displaying the first content with the portion of the first content updated with the second content on a display device.SELECTED DRAWING: Figure 5

Description

[Background technology]

[0001] Some websites have user interface elements that attempt to induce users to perform a desired action. Some such inducements include user interface elements that contain formatting and / or text intended to prompt the desired action. Summary of the Invention

[0002] Embodiments described herein relate to generating responses to target UI elements. Target UI elements may be displayed on a website via email or through any other electronic communication. The purpose of such target UI elements is to induce a user to perform a particular action that the user may not otherwise perform. For example, target UI elements may indicate or imply scarcity, prompting the user to initiate an application installation or perform an action on an object. Such target UI elements pressure the user to perform an action in the browser that would not normally be performed in the absence of the target UI element. Improvements include training generative models, such as large language models (LLMs) and multimodal models (including joint modalities with text and images, video, audio, etc.), to recognize target UI elements and generate responses to such target UI elements. For example, when a user expresses interest in an object by clicking a hyperlink on a website in a browser on a computer, embodiments are configured to detect the target UI element in a first portion of content received from the website as a result of clicking the hyperlink. In some embodiments, the computer then sends the first portion of content to a generative model, such as an LLM. A generative model, such as the LLM, is trained to generate a response to the target UI element. The computer then receives second content from the generative model, including the response to the target UI element, and displays the response. In some implementations, the generative model receives the first content, identifies the target UI element, and generates a response to the identified target UI element. In some implementations, the response includes an annotation to the first content. For example, in response to a prompt to book a hotel room because the hotel room is 90% booked, the second content can indicate, "The information that the room is 90% booked may or may not be correct, but it is a strategy to get you to book faster."In some implementations, the response includes a change to the text of the first content, hi some implementations, the response includes a change in font, color, or emphasis.

[0003] In one general aspect, a computer program product includes a non-transitory storage medium, the computer program product including code that, when executed by a processing circuit of a computing device, causes the processing circuit to perform a method. The method may include receiving the first content from a first remote web server in response to a request for the first content. The method may also include detecting a target user interface (UI) element in a portion of the first content. The method may further include transmitting at least a portion of the first content to a generative model in response to detecting the target UI element. The method may further include receiving second content from the generative model. The second content includes a difference to the portion of the first content, the difference being based on a type of the target UI element. The method may further include initiating displaying the first content, the first portion of which has been updated with the second content, on a display device.

[0004] In another general aspect, a computer program product includes a non-transitory storage medium, the computer program product including code that, when executed by a processing circuit of a computing device, causes the processing circuit to perform a method. The method may include receiving first content from a remote web server via a client device. The client device is configured to detect a target user interface (UI) element in the first content. The method may also include generating second content using the generative model. The second content includes a difference to the first content, the difference being based on the target UI element. The method may further include transmitting the second content to the client device for display on a display device of the client device.

[0005] In another general aspect, a method may include receiving first content from a first remote web server in response to a request for the first content. The method may also include detecting a target user interface (UI) element within a portion of the first content. The method may further include sending at least a portion of the first content to a generative model in response to detecting the target UI element. The method may further include receiving second content from the generative model. The second content includes a difference to the portion of the first content, the difference being based on a type of the target UI element. The method may further include commencing displaying the first content, with the first portion updated by the second content, on a display device.

[0006] In another general aspect, an electronic device includes a memory and a processing circuit coupled to the memory. The processing circuit may be configured to receive the first content from a first remote web server in response to a request for the first content. The processing circuit may also be configured to detect a target user interface (UI) element within a portion of the first content. The processing circuit may be further configured to send at least a portion of the first content to the generative model in response to detecting the target UI element. The processing circuit may be further configured to receive second content from a second remote web server. The second content includes a difference to the portion of the first content, the difference being based on a type of the target UI element. The processing circuit may be further configured to initiate displaying the first content, the first portion of which has been updated with the second content, on a display device.

[0007] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the following description and drawings, and from the claims. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 illustrates an exemplary network environment in which the improvements described herein may be implemented. [Figure 2A] FIG. 2 illustrates an exemplary first content including a target UI element. [Figure 2B] A diagram showing an example first content including a target UI element and a second content including a response. [Figure 2C] A diagram showing an example first content including a target UI element and a second content including a response. [Figure 2D] A diagram showing an example first content including a target UI element and a second content including a response. [Diagram 3] FIG. 1 illustrates an exemplary electronic environment in which the improvements described herein may be implemented. [Figure 4] FIG. 1 illustrates an exemplary electronic environment in which the improvements described herein may be implemented. [Diagram 5] 1 is a flowchart illustrating an exemplary method for generating a response to a target UI element according to disclosed embodiments. [Figure 6] 1 is a flowchart illustrating an exemplary method for generating a response to a target UI element according to disclosed embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] A goal of websites, applications, and generally any e-commerce provider is to encourage users to take action quickly and frequently on the products they offer. Such a goal may be achieved by either or a combination of special user interface (UI) affordances and / or target UI elements that create the illusion of urgency (e.g., "80% of stock is gone"). The mechanisms used may be simple, such as altering the placement of an advertisement or changing colors or fonts, or may be complex, such as using specific words repeatedly to create the illusion of increasing availability.

[0010] A technical challenge with the above is that it is not clear how browsers should mitigate interactions with target UI elements. For example, browser users may not realize that they are being guided by an invisible hand to make a given decision. In that case, users may end up making purchasing decisions based not on facts but on the illusion of urgency caused by the target UI element.

[0011] According to embodiments described herein, technical solutions to the above technical problems include improved browsers, applications, or other interfaces that automatically mitigate interactions with target UI elements. Such interfaces use generative models to modify content so that mechanisms used to guide users to decisions are more transparent to the user and / or minimize perceived stress. Implementations can operate as part of a browser or as a browser extension. Implementations can identify target UI elements in browser content (web pages) and use generative models to modify the target UI elements before presenting the browser content to the user. In some implementations, identifying the target UI elements can be performed by a generative model.

[0012] In one example, when a client computer requests and receives content, the client computer can detect a predefined pattern that represents a target UI element. Upon detecting such an element, the client computer can provide the content to a generative model. The generative model can run on the client device or on a remote server. The generative model then modifies the target UI element. This can include replacing the target UI element with new content. The new content can include, for example, an annotation explaining that the message in the target UI element is designed to get the user to make a purchasing decision more quickly. The modification can also include different colors, different fonts, and / or different highlighting, significant modification or removal of text, images, audio, video, or haptic features. With user permission, the generative model can generate different content for different users based on user history. For example, the generative model can utilize examples of past user behavior to determine the modifications (for the types (categories) of content that the user may engage with) that the user is most likely to respond to. In some implementations, with user permission, an observation period can be used to determine which types of target UI affordances the user is likely to select and which types are not. This can be used to trigger modifications personalized for the user.

[0013] The generative model may be fine-tuned (trained) to recognize which pieces of content represent the target UI elements to be modified, for example, using example content from the internet that is labeled with "modify" or "don't modify" labels.

[0014] A technical advantage of the disclosed embodiments is that users are less likely to make decisions through pressure tactics such as targeting UI elements; that is, the generative model can make it clear to the user which parts of the content use manipulative words, wording, colors, etc. In this way, users become aware that an invisible hand is guiding them towards or away from a decision, and as a result, they are better equipped to make decisions based on factual information.

[0015] 1 illustrates an example environment 100 in which a user can interact with one or more websites controlled from a remote server. In some implementations, the websites offer merchandise that the user can purchase via a user device 106. In some cases, the merchandise may be of limited stock, such as flights, hotel stays, event tickets, etc.

[0016] 1 , the exemplary environment 100 includes a network 102, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof, connecting websites 104, user devices 106, and generative models 120. In some examples, the network 102 may be accessed via wired and / or wireless communication links. For example, a mobile computing device, such as a smartphone, may access the network using a cellular network. The environment 100 may include millions of websites 104 and user devices 106.

[0017] In some examples, a website 104 is associated with a domain name and provided as one or more resources 105 hosted by one or more servers. An exemplary website is a collection of web pages formatted in a suitable machine-readable language (e.g., HyperText Markup Language (HTML)), which may include text, images, multimedia content, and programming elements such as scripts. Each website 104 is maintained by a publisher (e.g., an entity that manages and / or owns the website). Website resources 105 may be static or dynamic.

[0018] In some examples, resources 105 are data provided over network 102 that are associated with a resource address, e.g., a Uniform Resource Locator (URL). In some examples, resources 105 that may be provided by website 104 include, for example, web pages, word processing documents, and portable document format (PDF) documents, images, videos, and feed sources, among other suitable digital content. Resources 105 may include content, such as, for example, words, phrases, images, and sounds, and may also include embedded information (e.g., meta information and hyperlinks) and / or embedded instructions (e.g., scripts).

[0019] In some examples, user devices 106 are electronic devices under the control of a user and capable of requesting and receiving resources 105 over the network 102. Exemplary user devices 106 include personal computers, mobile computing devices, e.g., smartphones, wearable devices, and / or tablet computing devices, capable of sending and receiving data over the network 102. As used throughout this specification, the term mobile computing device ("mobile device") refers to a user device configured to communicate over a mobile communications network. A smartphone, e.g., a phone capable of communicating over the Internet, is an example of a mobile device, as are wearable devices and other smart devices, such as smart speakers. User devices 106 typically include user applications, e.g., a web browser, to facilitate sending and receiving data over the network 102.

[0020] A user device 106 issues a request 109 for content at a URL. In some examples, the user device 106 may include one or more input modalities. Exemplary modalities may include a keyboard, a touch screen, and / or a microphone. For example, a user may enter a URL using a keyboard and / or a touch screen. In some implementations, a user may generate a URL via a search query to a search system.

[0021] In response, the user device 106 receives the first content 110. The first content 110 may include text, images, audio, video, haptic content, etc. The first content 110 may be transmitted to the user device 106 in an encoded format, such as HTML, XML, Javascript, etc. In some implementations, the first content 110 takes the form of a Document Object Model (DOM) of UI elements for a mobile application, for example, if the user device 106 is a mobile device.

[0022] Upon receiving the first content 110, the user device 106 scans the first content 110 to determine whether the target UI element is present in the first content 110. In some implementations, the user device 106 determines whether the target UI element is present using a look-up table (LUT) of known target UI elements. For example, the LUT may include a list of known words, phrases, clauses, sentences, images, color schemes, etc. that are known to promote a user's sense of urgency.

[0023] In some implementations, the user device 106 evaluates the first content using a model trained to identify the target UI element in the content. In some implementations, the model is a neural network. In some implementations, the neural network is supervised and trained using known words, phrases, clauses, sentences, images, color schemes, etc. that are known to promote / mediate a user's sense of urgency. In such implementations, the first content can be sent to the model, and the model can provide a result indicating that the first content includes the target UI element. In some implementations, the result can include an identifier of the portion. The identifier can be an object identifier (e.g., a DOM node identifier, an accessibility tree identifier), the text of the portion or a subset of the text (e.g., a predetermined number of characters from the text), etc.

[0024] In some implementations, at this point, the user device 106 displays the first content 110 in a browser window on a display device. Note that while Figure 1 is directed to a scenario in which the user device accesses the website 104, there are other means by which the first content 110 may be obtained. For example, in some implementations, the first content 110 is obtained via an email program and viewed in an email window.

[0025] When the target UI element 112 is detected in a portion of the first content 122, the user device 106 sends the portion of the first content 122 to the Generative Model 120. As shown in FIGURE 1, the Generative Model 120 runs on a remote server and is accessed over the network 102. However, in some implementations where the Generative Model 120 is small enough to run on the user device 106, the Generative Model is limited to the user device 106.

[0026] In some implementations, the generative model 120 includes a large-scale language model (LLM). An LLM is a probabilistic model of natural language that generates probabilities for a set of words based on a text corpus on which it is trained. Because typical LLMs are trained over the Internet, such LLMs tend to be large enough to reside on one or more remote servers. In some cases, however, a task-specific LLM may be small enough to reside on the user device 106. In some implementations, the generative model 120 may be an LLM specialized for identifying and modifying target UI elements, such as the target UI element 112.

[0027] In some implementations, with the user's permission, the generative model 120 may be fine-tuned to take into account the user's behavior. For example, in addition to recognizing a target UI element, such as the target UI element 112, the generative model 120 may be fine-tuned to generate a response to the target UI element based on the user's past behavior (i.e., user history) in response to such a target UI element. For example, a first user responds to a prompting word in a target UI element by predictably purchasing, while a second user responds to the same prompting word by closing the transaction without purchasing. Thus, the training data for the first user may result in stronger changes to the website display than for the second user. For example, the training data for the first user may result in annotations or substantial changes to the target UI element, while the training data for the second user may result in no changes at all.

[0028] Implementations involving fine-tuned (personalized) modifications to a user may provide the user with control, allowing the user to make choices about both whether and when the environment 100 and / or user device 106 may enable collection of user history and / or whether user history may be utilized in training and / or obtaining responses from the generative model 120. Thus, implementations provide the user with control over what information is collected, how that information is used, and what information is provided to the user.

[0029] When the generative model 120 receives the portion of the first content 122, it generates a response to the target UI element 112 in the portion of the first content 122 as second content 130. The second content 130 includes a difference to the portion of the first content 122. The difference to the first portion of the content is based on a type of the target UI element 112. The type of the target UI element may include a text type, an emphasis type, a color type, a font type, an image type, an audio type, a video type, or a haptic type.

[0030] The secondary content 130, in some implementations, is a replacement for a portion of the first content 122. In some implementations, the secondary content 130 is an addition to a portion of the first content 122. For example, in some implementations, the secondary content 130 includes an annotation to a portion of the first content 122. In some implementations, the secondary content 130 includes a color change, a font change, or an emphasis change to text. In some implementations, the secondary content 130 includes an image, an audio, a video, or a UI affordance.

[0031] Once the second content 130 is generated, the second content 130 is sent to the user device 106 for display as an update to the first portion 110. In some implementations where the generative model 120 resides on a remote server, the remote server sends the second content 130 to the user device 106 over the network 102. In some implementations where the generative model 120 resides locally on the user device 106, the second content 130 is sent to a location in memory where it can be accessed by, for example, a web browser, an email program, etc. In either case, display of the portion of the first content 122 updated by the second content 130 begins once the second content 130 is received by the user device 106.

[0032] 2A is a diagram illustrating exemplary first content 220 including a target UI element 222 and second content 230 including a response. In this example, a browser window 200 includes an address bar 210 that shows a current website address as "travelsite.com." As shown in FIG. 2A, the browser window 200 displays the first content 220 including search results for hotel rooms in Nashville, Tennessee, listing desired dates and a price per night as a search result.

[0033] As shown in FIG. 2A , the first content 220 also includes a target UI element 222, which includes text categorized as conveying a sense of urgency and urging the user to reserve the room (e.g., urging an immediate purchase decision). The target UI element 222 is shown in a bold font. In some implementations, it may be a particular color (e.g., red) designed to grab the user's attention. The target UI element 222 may also flash text to further grab the user's attention. Additionally, the target UI element 222 is located next to a purchase button 224, which indicates the user's request to reserve the room.

[0034] 2B illustrates a response to a first content 220 that includes a target UI element 222, where a generative model (e.g., generative model 120 of FIG. 1) generates a second content 230. As shown in FIG. 2B, the second content 230 takes the form of an annotation to the target UI element 222 and is placed immediately adjacent to or below the target UI element 222.

[0035] In some implementations, the second content 230 is displayed in the browser window 200 after the first content 220 has already been displayed in the browser window 200. This is because it may take some time for the generative model to generate the second content 230, which may be longer than a user is willing to wait for the content to appear in the browser window 200.

[0036] FIG. 2B illustrates second content 230 in the form of annotations to the first content 220 (specifically the target UI element 222). The improvements described herein are not limited to such annotations and may take any of a number of forms. For example, instead of or in addition to the annotations shown in FIG. 2B, FIG. 2C illustrates second content 240 in the form of edits to the target UI element 222. In the case shown in FIG. 2C, the target UI element "90% of the rooms will be filled in 1 hour!" is replaced with "There are likely rooms to be filled in 1 hour."

[0037] 2D illustrates the second content 250 in the form of a change in emphasis (e.g., removing the bold and exclamation point.) Other changes may include a change in placement within the browser window, a change in color, or even the wholesale removal of the target UI element 222.

[0038] 2A-2D also show responses to the target UI element 222 shown in a browser window. Such responses may be generated for any program executing on a user device (e.g., user device 106 of FIG. 1) in which the target UI element may be displayed. In some implementations, the second content 230, 240, or 250 is generated for an email that displays the first content 220 in an email window that includes a preview mode.

[0039] 3 is a diagram illustrating an exemplary electronic device in which the above technical solution may be implemented. A computer 320 is configured to generate a response to a target UI element and display it on a display device.

[0040] The computer 320 includes a network interface 322, one or more processing units 324, a memory 326, and a display interface 328. The network interface 322 includes, for example, an Ethernet adapter, a token ring adapter, etc., for converting electronic and / or optical signals received from a network into an electronic format for use by the computer 320. The set of processing units 324 includes one or more processing chips and / or processing assemblies. The memory 326 includes both volatile memory (e.g., RAM) and non-volatile (non-transitory) memory (e.g., one or more ROMs, disk drives, solid state drives, etc.). The set of processing units 324 and the memory 326 together form a processing circuit, which is configured and arranged to perform various methods and functions described herein. The display interface 328 is configured to provide data to a display device for rendering and display to a user.

[0041] In some implementations, one or more of the components of the computer 320 may be or may include a processor (e.g., processing unit 324) configured to process instructions stored in the memory 326. Examples of such instructions shown in FIG. 3 include a content manager 330, a target UI element manager 340, a second content manager 350, and a rendering and display manager 360. Additionally, as shown in FIG. 3, the memory 326 is configured to store various data. This will be described with respect to each manager that uses such data.

[0042] The first content manager 330 is configured to request the first content from the first remote web server over the network. The first content manager 330 is also configured to receive the first content from the first remote web server over the network as first content data 332. In some implementations, the first content data 332 is in the form of HTML code, which is rendered and displayed within a browser window by the rendering and display manager 360 on the display device.

[0043] The target UI element manager 340 is configured to detect the target UI element in a portion of the first content. The portion of the first content is the target UI element data 342. In some implementations, the target UI element manager 340 uses a look-up table (LUT) to determine whether the target UI element is in the portion of the first content. In some implementations, the target UI element manager 340 evaluates the first content using a model trained to identify target UI elements in content. For example, the model can be a supervised neural network trained using a dataset of known target UI elements.

[0044] The second content manager 350 is configured to send a portion of the first content (e.g., target UI element data 342) to the Generative Model. The Generative Model, in some implementations, resides on a second remote server accessible over a network. In some implementations, the Generative Model resides on the computer 320. In such implementations, because the Generative Model resides on the computer 320, the Generative Model is small and configured for a specialized task.

[0045] The generative model is configured to generate the second content in response to the target UI element data 342. In some implementations, the generative model includes a large-scale language model (LLM). In some implementations, the generative model is trained to generate the second content in response to identifying the target UI element, and to modify the first content by updating the target UI element data 342 using the second content.

[0046] The second content manager 350 is also configured to receive the second content generated by the generative model as second content data 352. The second content data 352 includes differences to the target UI element data 342 (e.g., differences to a portion of the first content). The differences to the first portion of the content are based on a type of the target UI element. For example, if the type of the target UI element is text, then the differences are text.

[0047] The rendering and display manager 360 is configured to initiate displaying on the display device the first content data 332 with the portion of the first content (the target UI element data 342) updated by the second content data 352. The rendering and display data 362 includes the first content data 332 with the target UI element data 342 updated by the second content data 352.

[0048] The components of computer 320 (e.g., modules, processing units 324) may be configured to operate based on one or more platforms (e.g., one or more of the same or different platforms), which may include one or more types of hardware, software, firmware, operating systems, runtime libraries, and / or the like. In some implementations, the components of computer 320 may be configured to operate within a cluster of devices (e.g., a server farm). In such implementations, functionality and processing of the components of computer 320 may be distributed across several devices of the cluster of devices.

[0049] The components of computer 320 may be or include any type of hardware and / or software configured to process attributes. In some implementations, one or more portions of the components depicted in the components of computer 320 of FIG. 3 may be or include hardware-based modules (e.g., digital signal processor (DSP), field programmable gate array (FPGA), memory), firmware modules, and / or software-based modules (e.g., modules of computer code, a set of computer-readable instructions that can be executed by a computer). For example, in some implementations, one or more portions of the components of computer 320 may be or include software modules configured for execution by at least one processor (not shown). In some implementations, the functionality of the components may be included in different modules and / or different components than those depicted in FIG. 3. This involves combining functionality shown as two components into a single component.

[0050] Although not shown, in some implementations, the components of computer 320 (or portions thereof) may be configured to operate in, for example, a data center (e.g., a cloud computing environment), a computer system, one or more server / host devices, and / or the like. In some implementations, the components of computer 320 (or portions thereof) may be configured to operate within a network. Thus, the components of computer 320 (or portions thereof) may be configured to function within various types of network environments that may include one or more devices and / or one or more server devices. For example, the network may be or may include a local area network (LAN), a wide area network (WAN), and / or the like. The network may be or may include a wireless network, and / or a wireless network implemented, for example, using gateway devices, bridges, switches, and / or the like. A network may include one or more segments and / or may have portions based on different protocols, such as the Internet Protocol (IP) and / or a proprietary protocol. The network may include at least a portion of the Internet.

[0051] In some implementations, the memory 326 may be any type of memory, such as random access memory, disk drive memory, flash memory, and / or others. In some implementations, the memory 326 may be implemented as multiple memory components (e.g., multiple RAM components or disk drive memory) associated with components of the computer 320. In some implementations, the memory 326 may be a database memory. In some implementations, the memory 326 may be or may include a non-local memory. For example, the memory 326 may be or may include a memory shared by multiple devices (not shown). In some implementations, the memory 326 may be associated with a server device (not shown) in a network and may be configured to serve components of the computer 320. As shown in FIG. 3, the memory 326 is configured to store various data, including first content data 332, target UI element data 342, second content data 352, and rendering and display data 362.

[0052] 4 is a diagram illustrating an exemplary electronic device in which the above technical solution may be implemented. A computer 420 is configured to generate a response to a target UI element. In some implementations, the computer 420 communicates with a user device via a network. In some implementations, the computer 420 is a user device.

[0053] The computer 420 includes a network interface 422, one or more processing units 424, and a memory 426. The network interface 422 includes, for example, an Ethernet adapter, a token ring adapter, etc., for converting electronic and / or optical signals received from a network into an electronic format for use by the computer 420. The set of processing units 424 includes one or more processing chips and / or processing assemblies. The memory 426 includes both volatile memory (e.g., RAM) and non-volatile (non-transitory) memory (e.g., one or more ROMs, disk drives, solid state drives, etc.). The set of processing units 424 and the memory 426 together form a processing circuit, which is configured and arranged to perform the various methods and functions described herein.

[0054] In some implementations, one or more of the components of computer 420 may be or may include a processor (e.g., processing unit 424) configured to process instructions stored in memory 426. Examples of such instructions shown in FIG. 4 include a target UI element manager 430, a generative model manager 440, and a second content manager 450. Additionally, as shown in FIG. 4, memory 426 is configured to store various data. This will be described with respect to each manager using such data.

[0055] The target UI element manager 430 is configured to receive target UI element data 432 representing at least one target UI element. The target UI element data 432 may also represent additional content in which the at least one target UI element is included. The additional content is received by the user device in response to a request for the first content and constitutes a portion of the first content provided by the website over the network.

[0056] In some implementations, the target UI element data 432 is received over a network, in which the computer 420 on which the generative model executes is remote from the computer that received the target UI elements from the website.

[0057] The generative model manager 440 is configured to generate the second content as generative model data 442 in response to the target UI element in the target UI element data 432. In some implementations, the generative model includes a large language model (LLM). In some implementations, the generative model is trained to generate the second content in response to identifying the target UI element, and to modify the target UI element data 342 by updating the target UI element data 342 with the second content.

[0058] The second content manager 450 is configured to transmit the second content (e.g., the Generative Model data 442) to the computer that received the target UI element from the website. In some implementations where the computer 420 on which the Generative Model executes is remote from the computer that received the target UI element from the website, the second content manager 450 transmits the second content over a network.

[0059] The components of computer 320 (e.g., modules, processing unit 424) may be configured to operate based on one or more platforms (e.g., one or more of the same or different platforms), which may include one or more types of hardware, software, firmware, operating systems, runtime libraries, and / or the like. In some implementations, the components of computer 420 may be configured to operate within a cluster of devices (e.g., a server farm). In such implementations, functionality and processing of the components of computer 420 may be distributed across several devices of the cluster of devices.

[0060] The components of computer 420 may be or include any type of hardware and / or software configured to process attributes. In some implementations, one or more portions of the components depicted in the components of computer 420 of FIG. 4 may be or include hardware-based modules (e.g., digital signal processor (DSP), field programmable gate array (FPGA), memory), firmware modules, and / or software-based modules (e.g., modules of computer code, a set of computer-readable instructions that can be executed by a computer). For example, in some implementations, one or more portions of the components of computer 420 may be or include software modules configured for execution by at least one processor (not shown). In some implementations, the functionality of the components may be included in different modules and / or different components than those depicted in FIG. 4. This involves combining functionality shown as two components into a single component.

[0061] Although not shown, in some implementations, the components of computer 420 (or portions thereof) may be configured to operate in, for example, a data center (e.g., a cloud computing environment), a computer system, one or more server / host devices, and / or the like. In some implementations, the components of computer 420 (or portions thereof) may be configured to operate within a network. Thus, the components of computer 420 (or portions thereof) may be configured to function within various types of network environments that may include one or more devices and / or one or more server devices. For example, the network may be or may include a local area network (LAN), a wide area network (WAN), and / or the like. The network may be or may include a wireless network, and / or a wireless network implemented, for example, using gateway devices, bridges, switches, and / or the like. A network may include one or more segments and / or may have portions based on different protocols, such as the Internet Protocol (IP) and / or a proprietary protocol. The network may include at least a portion of the Internet.

[0062] In some implementations, the memory 426 may be any type of memory, such as random access memory, disk drive memory, flash memory, and / or others. In some implementations, the memory 426 may be implemented as multiple memory components (e.g., multiple RAM components or disk drive memory) associated with components of the computer 420. In some implementations, the memory 426 may be a database memory. In some implementations, the memory 426 may be or include a non-local memory. For example, the memory 426 may be or include a memory shared by multiple devices (not shown). In some implementations, the memory 426 may be associated with a server device (not shown) in a network and may be configured to serve components of the computer 420. As shown in FIG. 4, the memory 426 is configured to store various data, including target UI element data 432 and generative model data 442.

[0063] 5 is a flow chart illustrating an exemplary method 500 for generating and displaying on a display device a response to a target UI element according to the improvements described above. The method 500 may be performed by the software configuration described with respect to FIG. 3. The software configuration resides in the memory 326 of the computer 320 and is executed by the set of processing units 324.

[0064] At 502, the first content manager 330 receives the first content from the first remote web server in response to a request for the first content.

[0065] At 504, the target UI element manager 340 detects the target UI element within the portion of the first content. In some implementations, the detection includes determining that a word conveying urgency is present in the portion of the first content. In some implementations, the detection is performed using a look-up table (LUT). In some implementations, the detection is performed using a model trained to identify the target UI element in the content and evaluate the first content.

[0066] At 506, the target UI element manager 340 sends at least a portion of the first content to the Generative Model in response to detecting the target UI element. In some implementations, the target UI element manager 340 sends at least a portion of the first content to the Generative Model over a network.

[0067] At 508, the second content manager 350 receives second content from the generative model. The second content includes a difference to the portion of the first content, the difference being based on a type of the target UI element. In some implementations, the type of the target UI element is a text element and the difference includes a text element or a removal of the text element. In some implementations, the type of the target UI element is an image element and the difference includes an image element or a removal of the image element. In some implementations, the type of the target UI element is a font, a color, or an emphasis and the difference includes a different font, a different color, or a different emphasis. In some implementations, the second content manager 350 performs the detection instead of the target UI element manager 340.

[0068] At 510, the rendering and display manager 360 begins displaying the first content on the display device, where a portion of the first content has been updated with the second content.

[0069] 6 is a flow chart illustrating an example method 600 for generating a response to a target UI element according to the above refinements. The method 600 may be performed by the software configuration described with respect to FIG. 4. The software configuration resides in the memory 426 of the computer 420 and is executed by the set of processing units 424.

[0070] At 602, the target UI element manager 430 receives a first content via a client device. The client device is configured to detect a target user interface (UI) element in the first content.

[0071] At 604, the generative model manager 440 generates second content using the generative model. The second content includes differences relative to the first content, where the differences are based on the target UI element. In some implementations, the detection performed in step 602 may instead be performed by the generative model manager 440.

[0072] At 606, the second content manager 450 transmits the second content to the client device for display on a display device of the client device.

[0073] Clause 1. A computer program product including a non-transitory storage medium, the computer program product including code that, when executed by a processing circuit, causes the processing circuit to perform a method, the method including: receiving first content from a first remote web server in response to a request for first content; detecting a target user interface (UI) element within a portion of the first content; in response to detecting the target UI element, sending at least the portion of the first content to a generative model; receiving second content from the generative model, the second content including a difference from the portion of the first content, the difference being based on a type of the target UI element; and initiating displaying the first content, with the portion of the first content updated by the second content, on a display device.

[0074] Clause 2. The computer program product of clause 1, wherein at least the portion of the first content is transmitted to a second remote server that contains the generative model.

[0075] Clause 3. The computer program product of any of clauses 1 or 2, wherein detecting the target UI element includes sending the first content to the generative model, the generative model being trained to identify the target UI element.

[0076] Clause 4. The computer program product of clause 3, wherein the generative model is trained to generate the second content in response to identifying the target UI element and to modify the first content by updating the portion of the first content using the second content, and receiving the second content includes receiving the modified first content.

[0077] Clause 5. The computer program product of any of clauses 1-4, wherein detecting the target UI element includes providing the first content to a model trained to identify target UI elements in content, and receiving a result from the model indicating that the portion of the first content includes the target UI element.

[0078] Clause 6. The computer program product of any of clauses 1-5, wherein the second content includes at least one annotation to the portion of the first content.

[0079] Clause 7. A computer program product as described in any of clauses 1 to 6, wherein the portion of the first content includes a first text and the second content includes a second text that is different from the first text.

[0080] Clause 8. The computer program product of any of clauses 1-7, wherein the portion of the first content includes a first color and the second content includes a second color different from the first color.

[0081] Clause 9. A computer program product including a non-transitory storage medium, comprising code that, when executed by a processing circuit, causes the processing circuit to perform a method, the method comprising: receiving first content via a client device, the client device configured to detect a target user interface (UI) element in the first content; generating second content using a generative model, the second content comprising differences to the first content, the differences being based on the target UI element; and transmitting the second content to the client device for display on a display device of the client device.

[0082] Clause 10. The computer program product of clause 9, wherein the second content includes at least one annotation to the first content.

[0083] Clause 11. A computer program product according to any of clauses 9 to 10, wherein the first content includes a first text and the second content includes a second text that is different from the first text.

[0084] Clause 12. The computer program product of any of clauses 9 to 11, wherein the first content includes a first color and the second content includes a second color different from the first color.

[0085] Clause 13. A method comprising: receiving first content from a first remote web server in response to a request for the first content; detecting a target user interface (UI) element within a portion of the first content; in response to detecting the target UI element, sending at least the portion of the first content to a generative model; receiving second content from the generative model, the second content including differences relative to the portion of the first content, the differences being based on a type of the target UI element; and initiating displaying the first content, with the portion of the first content updated by the second content, on a display device.

[0086] Clause 14. The method of clause 13, wherein detecting the target UI element includes sending the first content to the generative model, the generative model being trained to identify the target UI element.

[0087] Clause 15. The method of clause 14, wherein the generative model is trained to generate the second content in response to identifying the target UI element and to modify the first content by updating the portion of the first content using the second content, and receiving the second content includes receiving the modified first content.

[0088] Clause 16. A method according to any of clauses 13 to 15, wherein detecting the target UI element includes providing the first content to a model trained to identify target UI elements in content, and receiving a result from the model indicating that the portion of the first content includes the target UI element.

[0089] Clause 17. The method of any of clauses 13-16, wherein the second content includes at least one annotation to the portion of the first content.

[0090] Clause 18. The method of any of clauses 13 to 17, wherein the portion of the first content includes a first text and the second content includes a second text that is different from the first text.

[0091] Clause 19. The method of any of clauses 13 to 18, wherein the portion of the first content includes a first color and the second content includes a second color different from the first color.

[0092] Clause 20. An electronic device comprising: a memory; and a processing circuit coupled to the memory, the processing circuit configured to: receive first content from a first remote web server in response to a request for the first content; detect a target user interface (UI) element within a portion of the first content; in response to detecting the target UI element, send at least the portion of the first content to a generative model; receive second content from the generative model, the second content including a difference relative to the portion of the first content, the difference being based on a type of the target UI element; and initiate displaying the first content, with the portion of the first content updated by the second content, on a display device.

[0093] Clause 21. The electronic device of clause 20, wherein at least the portion of the first content is transmitted to a second remote server that contains the generative model.

[0094] Clause 22. An electronic device described in any of clauses 20 to 21, wherein the processing circuit configured to detect the target UI element is further configured to send the first content to the generative model, the generative model being trained to identify the target UI element.

[0095] Clause 23. The electronic device of clause 22, wherein the generative model is trained to generate the second content in response to identifying the target UI element and to modify the first content by using the second content to update the portion of the first content, and receiving the second content includes receiving the modified first content.

[0096] Clause 24. The electronic device of any of Clauses 20 to 23, wherein the processing circuitry configured to detect the target UI element is configured to perform the following: providing the first content to a model trained to identify target UI elements in content; and receiving a result from the model indicating that the portion of the first content includes the target UI element.

[0097] Clause 25. The electronic device of any of clauses 20 to 24, wherein the second content includes at least one annotation to the portion of the first content.

[0098] Clause 26. An electronic device according to any of clauses 20 to 25, wherein the portion of the first content includes a first text and the second content includes a second text that is different from the first text.

[0099] Clause 27. The electronic device of any of clauses 20 to 26, wherein the portion of the first content includes a first color and the second content includes a second color different from the first color.

[0100] Various implementations of the systems and techniques described herein may be realized in digital electronic circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor. Such programmable processor may be special purpose or general purpose and may be coupled to send and receive data and instructions from a storage system, at least one input device, and at least one output device.

[0101] These computer programs (also referred to as programs, software, software applications, or code) contain machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "non-transitory machine-readable medium" and "non-transitory computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLS)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to send machine instructions and / or data to a programmable processor.

[0102] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to provide interaction with a user, for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback) and input from the user can be received in any form, such as acoustic, spoken language, or tactile input.

[0103] The systems and techniques described herein can be implemented in a computing system that includes a back-end component (e.g., as a data server), or can be implemented in a computing system that includes a middleware component (e.g., an application server), or can be implemented in a computing system that includes a front-end component (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or a combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0104] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0105] Several implementations have been described. However, it will be appreciated that various modifications can be made without departing from the spirit and scope of the specification.

[0106] It should be understood that when an element is described as being on, connected to, electrically connected to, coupled to, or electrically coupled to another element, the element may be directly on, directly connected to, or coupled to, another element, or there may be one or more intervening elements. On the other hand, when an element is described as being directly on, directly connected to, or directly coupled to, another element, there are no intervening elements. Throughout the detailed description, elements that are shown as being directly on, directly connected to, or directly coupled to may be so referred to, even if the terms "directly on," "directly connected," or "directly coupled" are not used. The claims of this application may be amended to show the exemplary relationships described in the specification or shown in the drawings.

[0107] While certain features of the described embodiments have been illustrated as described herein, many modifications, substitutions, changes, and equivalents will readily occur to those skilled in the art. It is therefore to be understood that the appended claims are intended to cover all such modifications and changes as fall within the scope of the embodiments. As will be apparent, the embodiments are presented by way of example only and not by way of limitation, and various changes in form and details may be made. Any part of the apparatus and / or method described herein may be combined in any combination except in mutually exclusive combinations. The embodiments described herein may include various combinations and / or subcombinations of the functions, components, and / or features of the different embodiments described.

[0108] Additionally, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desired results. Additionally, other steps may be provided in or removed from the depicted flows, and other components may be added to or removed from the depicted systems. Accordingly, other implementations are within the scope of the following claims.

Claims

1. 1. A computer program product including a non-transitory storage medium, the computer program product including code that, when executed by a processing circuit, causes the processing circuit to perform a method, the method comprising: receiving a first content from a first remote web server in response to a request for the first content; detecting a target user interface (UI) element within a portion of the first content; in response to detecting the target UI element, sending at least the portion of the first content to a generative model; receiving second content from the generative model, the second content including a difference to the portion of the first content, the difference being based on a type of the target UI element; commencing displaying the first content on a display device, the first content having the portion of the first content updated with the second content; 2. A computer program product comprising:

2. The computer program product of claim 1 , wherein at least the portion of the first content is transmitted to a second remote server that contains the generative model.

3. 2. The computer program product of claim 1 , wherein detecting the target UI element comprises sending the first content to the generative model, the generative model being trained to identify target UI elements.

4. the generative model is trained to generate the second content in response to identifying the target UI element and to modify the first content by updating the portion of the first content using the second content; The computer program product of claim 3 , wherein receiving the second content comprises receiving the modified first content.

5. Detecting the target UI element includes: providing the first content to a model trained to identify target UI elements in the content; receiving a result from the model indicating that the portion of the first content includes the target UI element; 2. The computer program product of claim 1 , comprising:

6. The computer program product of claim 1 , wherein the second content comprises at least one annotation to the portion of the first content.

7. 2. The computer program product of claim 1, wherein the portion of the first content includes a first text and the second content includes a second text that is different from the first text.

8. The computer program product of claim 1 , wherein the portion of the first content includes a first color and the second content includes a second color that is different from the first color.

9. receiving a first content from a first remote web server in response to a request for the first content; detecting a target user interface (UI) element within a portion of the first content; in response to detecting the target UI element, sending at least the portion of the first content to a Generative Model; and receiving second content from the Generative Model, the second content comprising a difference to the portion of the first content, the difference based on a type of the target UI element; commencing displaying, on a display device, the first content with the portion of the first content updated by the second content; A method comprising:

10. 10. The method of claim 9, wherein detecting the target UI element comprises sending the first content to the generative model, the generative model being trained to identify target UI elements.

11. the generative model is trained to generate the second content in response to identifying the target UI element and to modify the first content by updating the portion of the first content using the second content; The method of claim 10 , wherein receiving the second content comprises receiving the modified first content.

12. Detecting the target UI element includes: providing the first content to a model trained to identify target UI elements in the content; and receiving a result from the model indicating that the portion of the first content includes the target UI element.

13. The method of claim 9 , wherein the second content includes at least one annotation to the portion of the first content.

14. The method of claim 9 , wherein the portion of the first content includes a first text and the second content includes a second text that is different from the first text.

15. The method of claim 9 , wherein the portion of the first content includes a first color and the second content includes a second color that is different from the first color.

16. 1. An electronic device comprising: Memory, a processing circuit coupled to the memory, the processing circuit comprising: receiving a first content from a first remote web server in response to a request for the first content; detecting a target user interface (UI) element within a portion of the first content; in response to detecting the target UI element, sending at least the portion of the first content to a generative model; receiving second content from the generative model, the second content comprising a difference to the portion of the first content, the difference being based on a type of the target UI element, the processing circuitry further comprising: commencing displaying, on a display device, the first content with the portion of the first content updated by the second content; 16. An electronic device configured to:

17. The electronic device of claim 16 , wherein at least the portion of the first content is transmitted to a second remote server that contains the generative model.

18. 17. The electronic device of claim 16, wherein the processing circuitry configured to detect the target UI element is further configured to send the first content to the generative model, the generative model being trained to identify the target UI element.

19. the generative model is trained to generate the second content in response to identifying the target UI element and to modify the first content by updating the portion of the first content using the second content; The electronic device of claim 18 , wherein receiving the second content comprises receiving the modified first content.

20. The processing circuitry configured to detect the target UI element further comprises: providing the first content to a model trained to identify target UI elements in the content; receiving a result from the model indicating that the portion of the first content includes the target UI element; 20. The electronic device of claim 16, configured to execute:

21. The electronic device of claim 16 , wherein the second content includes at least one annotation to the portion of the first content.

22. The electronic device of claim 16 , wherein the portion of the first content includes a first text and the second content includes a second text that is different from the first text.

23. 17. The electronic device of claim 16, wherein the portion of the first content includes a first color and the second content includes a second color that is different from the first color.

Images