Large language model-based response to target UI elements

Abstract

The invention relates to a large language model-based response to a target UI element. The techniques include using a generative model to change the content such that the mechanism for guiding the user to make decisions becomes common and / or minimizes perceived urgency to the user. Implementations may operate as part of a browser or as an extension of the browser. Implementations may identify a target UI element in browser content (web page) and modify the target UI element using a generative model prior to presenting the browser content to a user. In some implementations, identification of a target UI element may be performed by a generative model.

Description

Technical Field

[0001] The present disclosure relates to large language model based responses to target UI elements. Background Art

[0002] Some websites include user interface elements that attempt to induce users to perform desired actions. Some of these inducements include user interface elements that include formatting and / or text for encouraging desired actions. Summary of the invention

[0003] The implementation described herein is related to generating a response to a target UI element. The target UI element may appear in a website, via an email, or through any other electronic communication. The purpose of such a target UI element is to persuade the user to take a specific action that the user may not take without such a target UI element. For example, the target UI element may prompt the user to initiate the installation of the app or take action on the object based on a statement or suggestion of scarcity. Such a target UI element puts pressure on the user, resulting in actions performed in the browser that are usually not performed without the presence of the target UI element. Improvements are related to training a generative model such as a large language model (LLM) and a multimodal model (including joint modalities with text and images, videos, audio, etc.) to identify the target UI element and generate a response to such a target UI element. For example, when a user expresses interest in an object by clicking a hyperlink in a website in a browser on a computer, the implementation is configured to detect a target UI element in a portion of the first content received from the website as a result of clicking the hyperlink. In some implementations, the computer then sends the first portion of the content to a generative model such as an LLM, which is trained to generate a response to the target UI element. The computer then receives a second content including a response to the target UI element from the generative model and displays the response. In some implementations, the generative model receives the first content, identifies the target UI element, and generates a response to the identified target UI element. In some implementations, the response includes an annotation to the first content. For example, in response to prompting the purchase of such a room because 90% of hotel rooms are booked, the second content may state "statements of a room being almost booked may or may not be true but is a tactic to get you to purchase more quickly (a statement that the room is almost booked may or may not be true, but a strategy to get you to purchase more quickly)." In some implementations, the response includes a change to the text of the first content. In some implementations, the response includes a change in font, color, or emphasis.

[0004] In one general aspect, a computer program product includes a non-transitory storage medium, the computer program product including code that, when executed by a processing circuit system of a computing device, causes the processing circuit system to perform a method. The method may include receiving the first content from a first remote web server in response to a request for the first content. The method may also include detecting a target user interface (UI) element within a portion of the first content. The method may also include: in response to detecting the target UI element, sending at least a portion of the first content to a generation model. The method may also include receiving second content from the generation model, the second content including a difference from a portion of the first content, the difference being based on the type of the target UI element. The method may also include initiating display of the first content on a display device, wherein the first portion is updated with the second content.

[0005] In another general aspect, a computer program product includes a non-transitory storage medium, the computer program product including code, which, when executed by a processing circuit system of a computing device, causes the processing circuit system to perform a method. The method may include receiving first content from a remote web server via a client device, the client device being configured to detect a target user interface (UI) element in the first content. The method may also include generating second content using a generative model, the second content including a difference from the first content, the difference being based on the target UI element. The method may also include sending the second content to the client device for display on a display device of the client device.

[0006] In another general aspect, the method may include receiving the first content from a first remote web server in response to a request for the first content. The method may also include detecting a target user interface (UI) element within a portion of the first content. The method may also include sending at least a portion of the first content to a generation model in response to detecting the target UI element. The method may also include receiving second content from the generation model, the second content including a difference from a portion of the first content, the difference based on a type of the target UI element. The method may also include initiating display of the first content on a display device, wherein the first portion is updated with the second content.

[0007] In another general aspect, an electronic device includes a memory and a processing circuit system coupled to the memory. The processing circuit system can be configured to receive the first content from a first remote web server in response to a request for the first content. The processing circuit system can also be configured to detect a target user interface (UI) element within a portion of the first content. The processing circuit system can also be configured to send at least a portion of the first content to a generation model in response to detecting the target UI element. The processing circuit system can also be configured to receive a second content from a second remote web server, the second content including a difference from a portion of the first content, the difference being based on the type of the target UI element. The processing circuit system can also be configured to initiate display of the first content on a display device, wherein the first portion is updated with the second content.

[0008] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a diagram illustrating an example network environment in which the improved techniques described herein may be implemented.

[0010] Figure 2A is a diagram illustrating an example first content having a target UI element.

[0011] Figure 2B is a diagram illustrating example first content with a target UI element and second content with a response.

[0012] Figure 2C is a diagram illustrating example first content with a target UI element and second content with a response.

[0013] Figure 2D is a diagram illustrating example first content with a target UI element and second content with a response.

[0014] Figure 3 is a diagram illustrating an example electronic environment in which the improved techniques described herein may be implemented.

[0015] Figure 4 is a diagram illustrating an example electronic environment in which the improved techniques described herein may be implemented.

[0016] Figure 5 is a flow chart of an example method of generating a response to a target UI element according to an example of the disclosed implementation.

[0017] Figure 6 is a flow chart of an example method of generating a response to a target UI element according to an example of the disclosed implementation. DETAILED DESCRIPTION

[0018] Websites, apps, and in general any e-commerce provider have the goal of persuading users to act quickly and frequently on the products being offered. Such a goal can be achieved through any or a combination of special user interface (UI) affordances or targeted UI elements that create an illusion of urgency (e.g., "80% of our inventory is gone!"). The mechanisms employed can be as simple as modifying the positioning of an ad or changing the color or font, or can be as complex as repeatedly using specific wording to create a false sense of limited availability.

[0019] The above technical problem is that it is unclear how the browser should mitigate the interaction with the target UI element. For example, the user of the browser may not realize that they are being guided by an invisible hand to make a predetermined decision. In this case, the user may eventually make a purchase decision not based on facts but on the illusion of urgency created by the target UI element.

[0020] According to the implementation described herein, the technical solution to the above technical problem includes an improved browser, app or other interface that automatically mitigates the interaction with the target UI element. Such an interface uses a generative model to change the content so that the mechanism for guiding the user to make a decision becomes common and / or minimizes the perceived urgency for the user. The implementation can be operated as part of a browser or an extension of a browser. The implementation can identify the target UI element in the browser content (web page) and use the generative model to modify the target UI element before presenting the browser content to the user. In some implementations, the identification of the target UI element can be performed by the generative model.

[0021] In one example, when a client computer requests and receives content, the client computer can detect a predetermined pattern representing a target UI element. Once such an element is detected, the client computer can provide the content to a generation model. The generation model can be executed on a client device or a remote server. Then, the generation model modifies the target UI element. This can include replacing the target UI element with new content. The new content can include annotations, which, for example, explain that the message in the target UI element is designed to allow users to make purchase decisions faster. Modifications can also include different colors, fonts and / or emphasis, comprehensive changes or deletions to text, graphics, audio, video or tactile functions. With user permission, the generation model can generate different content for different users based on user history. For example, the generation model can use examples of past user behavior to determine the changes that users are most likely to respond to, and what types (categories) of content users may participate in. In some implementations, an observation period can be used to determine which types of target UI revelations users may choose, and which types of target UI revelations users will not choose, with user permission. This can be used to trigger modifications personalized for users.

[0022] The generative model can be fine-tuned (trained) to recognize which parts of the content represent target UI elements that need to be modified. For example, this can be achieved using sample content from the Internet labeled with "modify" or "don't modify".

[0023] A technical advantage of the disclosed implementation is that users are less likely to make decisions based on pressure strategies such as targeted UI elements. That is, the generative model can make it clear to the user which parts of the content are using manipulative words, expressions, colors, etc. In this way, users become aware of the invisible hand guiding them toward or away from decisions, and are therefore better able to make decisions based on factual information.

[0024] Figure 1 An example environment 100 is depicted in which a user may interact with one or more websites controlled from a remote server. In some implementations, the website offers products that may be purchased by the user via a user device 106. In some cases, the product may have limited inventory, such as flights, hotel rooms, event tickets, etc.

[0025] Continue to refer Figure 1, the example environment 100 includes a network 102, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof, which connects websites 104, user devices 106, and a generation model 120. In some examples, the network 102 can be accessed through a wired and / or wireless communication link. For example, a mobile computing device such as a smart phone can access the network using a cellular network. The environment 100 may include millions of websites 104 and user devices 106.

[0026] In some examples, website 104 is provided as one or more resources 105 associated with a domain name and hosted by one or more servers. An example website is a collection of web pages formatted in a suitable machine-readable language, such as Hypertext Markup Language (HTML), which may contain text, images, multimedia content, and programming elements such as scripts. Each website 104 is maintained by a publisher, such as an entity that manages and / or owns the website. Website resources 105 may be static or dynamic.

[0027] In some examples, resource 105 is data provided by network 102 and associated with a resource address, such as a uniform resource locator (URL). In some examples, resources 105 that may be provided by website 104 include web pages, word processing documents, and portable document format (PDF) documents, images, videos, and feeds, and other appropriate digital content. Resource 105 may include content such as words, phrases, images, and sounds, and may include embedded information such as meta information and hyperlinks and / or embedded instructions such as scripts.

[0028] In some examples, user device 106 is an electronic device under the control of a user and capable of requesting and receiving resources 105 through network 102. Example user devices 106 include personal computers, mobile computing devices, such as smart phones, wearable devices, and / or tablet computing devices that can send and receive data through network 102. As used throughout this document, the term mobile computing device ("mobile device") refers to a user device configured to communicate through a mobile communication network. Smart phones (e.g., phones capable of communicating through the Internet) are examples of mobile devices, and wearable devices and other smart devices such as smart speakers are also examples of mobile devices. User device 106 typically includes a user application (e.g., a web browser) to facilitate sending and receiving data through network 102.

[0029] The user device 106 submits a request 109 for the content at the URL. In some examples, the user device 106 may include one or more input modalities. Example modalities may include a keyboard, a touch screen, and / or a microphone. For example, a user may use a keyboard and / or a touch screen to type in a URL. In some implementations, a user may generate a URL via a search query to a search system.

[0030] In response, the user device 106 receives the first content 110. The first content 110 may include text, graphics, audio, video, tactile content, etc. The first content 110 may be transmitted to the user device 106 in a compiled format such as HTML, XML, javascript, etc. In some implementations, when the user device 106 is a mobile device, the first content 110 takes the form of a document object model (DOM) of a UI element of a mobile app, for example.

[0031] After receiving the first content 110, the user device 106 scans the first content 110 to determine whether there are any target UI elements in the first content 110. In some implementations, the user device 106 uses a lookup table (LUT) of known target UI elements to determine whether there are any target UI elements. For example, the LUT may include a list of known words, phrases, clauses, sentences, graphics, color schemes, etc. that are known to promote a user's sense of urgency.

[0032] In some implementations, the user device 106 evaluates the first content using a model trained to identify a target UI element in the content. In some implementations, the model is a neural network. In some implementations, the neural network is supervised and trained using known words, phrases, clauses, sentences, graphics, color schemes, etc. that are known to promote / convey a user's sense of urgency. In such an implementation, the first content can be provided to the model, and the model can provide a result indicating that the first content includes a target UI element. In some implementations, the result can include an identification of the portion. The identification can be an object identifier (e.g., a DOM node identifier, an accessibility tree identifier), text in the portion or a subset of the text (e.g., a predetermined number of characters from the text), etc.

[0033] In some implementations, at this time, the user device 106 displays the first content 110 in a browser window on the display device. It is worth noting that although Figure 1 The scenario in which the user device visits the website 104 is targeted, but there are other means of obtaining the first content 110. For example, in some implementations, the first content 110 is obtained via an email program and viewed in an email window.

[0034] If the target UI element 112 is detected in a portion of the first content 122, the user device 106 sends the portion of the first content 122 to the generation model 120. Figure 1 As shown, the generative model 120 runs on a remote server and is accessed via the network 102. However, in some implementations where the generative model 120 is small enough to run on the user device 106, the generative model is local to the user device 106.

[0035] In some implementations, the generation model 120 includes a large language model (LLM). An LLM is a probabilistic model of natural language that generates the probability of a series of words based on a text corpus on which it is trained. Since typical LLMs are trained on the Internet, such LLMs are often large enough that they reside on a separate server or servers. However, in some cases, an LLM dedicated to a specific task may be small enough to reside on the user device 106. In some implementations, the generation model 120 may be an LLM dedicated to identifying and modifying a target UI element such as the target UI element 112.

[0036] In some implementations, with the user's permission, the generation model 120 can be fine-tuned to take into account the user's behavior. For example, in addition to identifying target UI elements such as target UI element 112, the generation model 120 can be fine-tuned to generate responses to target UI elements based on the user's past behavior (i.e., user history) in response to such target UI elements. For example, while a first user reacts to urgent language in a target UI element by predictably making a purchase, a second user reacts to the same urgent language by ending the transaction without making a purchase. Therefore, the training data of the first user may involve stronger changes to the website display than the training data of the second user. For example, the training data of the first user may involve annotations or comprehensive changes to the target UI element, while the training data of the second user may not involve changes at all.

[0037] In implementations that include modifications for user fine-tuning (personalization), controls may be provided to the user, allowing the user to make choices about whether and when the environment 100 and / or user device 106 may enable collection of user history and / or whether the user history may be used in training the generative model 120 and / or obtaining responses from the generative model 120. Thus, implementations provide the user with control over what information is collected, how that information is used, and what information is provided to the user.

[0038] The generation model 120, upon receiving the portion of the first content 122, generates a response to the target UI element 112 in the portion of the first content 122 as the second content 130. The second content 130 includes a difference from the portion of the first content 122. The difference from the first portion of the content is based on the type of the target UI element 112. The type of the target UI element may include a text type, an emphasis type, a color type, a font type, a graphic type, an audio type, a video type, or a tactile type.

[0039] In some implementations, the second content 130 is a replacement for a portion of the first content 122. In some implementations, the second content 130 is an addition to a portion of the first content 122. For example, in some implementations, the second content 130 includes annotations to a portion of the first content 122. In some implementations, the second content 130 includes a change in color, font, or emphasis of text. In some implementations, the second content 130 includes graphics, audio, video, or UI affordances.

[0040] Once the second content 130 has been generated, the second content 130 is sent to the user device 106 for display as an update to the first portion 110. In some implementations where the generation model 120 resides on a remote server, the remote server sends the second content 130 to the user device 106 via the network 102. In some implementations where the generation model 120 resides locally on the user device 106, the second content 130 is sent to a location in memory where it can be accessed, for example, by a web browser, an email program, etc. In either case, once the second content 130 is received by the user device 106, display of the portion of the first content 122 updated with the second content 130 is initiated.

[0041] Figure 2A 2 is a diagram illustrating an example first content 220 with a target UI element 222 and a second content 230 with a response. In this example, the browser window 200 includes an address bar 210 indicating that the current website address is "travelsite.com". Figure 2A As shown, the browser window 200 displays first content 220, which includes search results for hotel rooms located in Nashville TN, listing the requested dates and resulting prices per night.

[0042] like Figure 2AAs shown, the first content 220 also includes a target UI element 222, which includes text that is classified as conveying urgency, pleading the user to book a room as soon as possible--for example, to make a purchase decision. The target UI element 222 is shown in bold font. In some implementations, it can also be a specific color (for example, red) designed to attract the user's attention. The target UI element 222 can also make the text flash, to further attract the user's attention. In addition, the target UI element 222 is located next to the purchase button 224 that will indicate the user's expectation of booking a room.

[0043] Figure 2B In response to the first content 220 including the target UI element 222, a model (eg, Figure 1 The generation model 120) generates the second content 230. Figure 2B As shown, the second content 230 takes the form of an annotation to the target UI element 222 , and is placed next to or below the target UI element 222 .

[0044] In some implementations, the second content 230 is displayed in the browser window 200 after the first content 220 has been displayed in the browser window 200. The reason for this is that the generation model may take some time to generate the second content 230, which may be longer than the user is willing to wait for the content to be displayed in the browser window 200.

[0045] Figure 2B The second content 230 is shown in the form of an annotation to the first content 220, in particular the target UI element 222. The improvements described herein are not limited to such annotations and may take any of a variety of forms. For example, instead of or in addition to Figure 2B In addition to the notes shown in Figure 2C The second content 240 is shown in the form of an edit to the target UI element 222. Figure 2C In the case shown, the target UI element "90% of the rooms will be taken in thenext hour!" is replaced with "Some rooms may be taken in the next hour."

[0046] Figure 2D The second content 250 is shown in the form of a change to emphasis (e.g., removal of bold and exclamation mark). Other changes may include a placement change within the browser window, a color change, or a complete deletion of the target UI element 222.

[0047] FIG. 2A to FIG. 2DAlso shown is a response to a target UI element 222 being displayed within the browser window. The target UI element may be displayed in a user device (e.g., Figure 1 Any program running on the user device 106) generates such a response. In some implementations, the second content 230, 240 or 250 is generated for the email displaying the first content 220 in the email window, including a preview mode.

[0048] Figure 3 3 is a diagram showing an example electronic device in which the above technical solution can be implemented. The computer 320 is configured to generate a response to a target UI element and display the response on a display device.

[0049] The computer 320 includes a network interface 322, one or more processing units 324, a memory 326, and a display interface 328. The network interface 322 includes, for example, an Ethernet adapter, a token ring adapter, etc., for converting electronic and / or optical signals received from the network into electronic form for use by the computer 320. The set of processing units 324 includes one or more processing chips and / or assemblies. The memory 326 includes both volatile memory (e.g., RAM) and non-volatile (non-temporary) memory such as one or more ROMs, disk drives, solid-state drives, etc. The set of processing units 324 and the memory 326 together form a processing circuit system that is configured and arranged to perform the various methods and functions described herein. The display interface 328 is configured to provide data to a display device for rendering and display to a user.

[0050] In some implementations, one or more of the components of computer 320 may be or may include a processor (e.g., processing unit 324) configured to process instructions stored in memory 326. Figure 3 Examples of such instructions depicted in include content manager 330, target UI element manager 340, second content manager 350, and rendering and display manager 360. In addition, as Figure 3 As shown, the memory 326 is configured to store various data that are described with respect to the corresponding managers using such data.

[0051] The first content manager 330 is configured to request the first content from the first remote web server over the network. The first content manager 330 is also configured to receive the first content from the first remote web server over the network as first content data 332. In some implementations, the first content data 332 is in the form of HTML code, which is rendered and displayed by the rendering and display manager 360 within a browser window on a display device.

[0052] The target UI element manager 340 is configured to detect a target UI element within a portion of the first content, the portion of the first content being the target UI element data 342. In some implementations, the target UI element manager 340 uses a lookup table (LUT) to determine whether a target UI element exists within the portion of the first content. In some implementations, the target UI element manager 340 evaluates the first content using a model trained to identify target UI elements in content. For example, the model can be a supervised neural network trained using a data set of known target UI elements.

[0053] The second content manager 350 is configured to send a portion of the first content, such as the target UI element data 342, to the generative model. In some implementations, the generative model resides on a second remote server accessible via a network. In some implementations, the generative model resides on the computer 320. In such an implementation, for the generative model that resides on the computer 320, the generative model is small and configured for a dedicated task.

[0054] The generation model is configured to generate the second content in response to the target UI element data 342. In some implementations, the generation model includes a large language model (LLM). In some implementations, the generation model is trained to generate the second content, and in response to identifying the target UI element, the first content is modified by updating the target UI element data 342 with the second content.

[0055] The second content manager 350 is also configured to receive the second content generated by the generation model as second content data 352. The second content data 352 includes differences from the target UI element data 342, such as differences from a portion of the first content. The difference from the first portion of the content is based on the type of the target UI element. For example, when the type of the target UI element is text, the difference will be text.

[0056] The rendering and display manager 360 is configured to initiate display of the first content data 332, wherein a portion of the first content, the target UI element data 342, is updated with the second content data 352 on the display device. The rendering and display data 362 includes the first content data 332 and the target UI element data 342 updated with the second content data 352.

[0057] The components (e.g., modules, processing unit 324) of computer 320 can be configured to operate based on one or more platforms (e.g., one or more similar or different platforms), which can include one or more types of hardware, software, firmware, operating systems, runtime libraries, etc. In some implementations, the components of computer 320 can be configured to operate within a device cluster (e.g., a server farm). In such implementations, the functionality and processing of the components of computer 320 can be distributed to several devices of the device cluster.

[0058] The components of computer 320 may be or include any type of hardware and / or software configured to process attributes. Figure 3 One or more portions of the components shown in the components of computer 320 in FIG. 3 may be or may include hardware-based modules (e.g., digital signal processors (DSPs), field programmable gate arrays (FPGAs), memory), firmware modules, and / or software-based modules (e.g., computer code modules, a set of computer-readable instructions that can be executed at a computer). For example, in some implementations, one or more portions of the components of computer 320 may be or may include software modules configured to be executed by at least one processor (not shown). In some implementations, the functionality of the components may be included in conjunction with Figure 3 The different modules and / or different components shown in the drawings include combining the functionality illustrated as two components into a single component.

[0059] Although not shown, in some implementations, the components (or portions thereof) of the computer 320 may be configured to operate within, for example, a data center (e.g., a cloud computing environment), a computer system, one or more server / host devices, etc. In some implementations, the components (or portions thereof) of the computer 320 may be configured to operate within a network. Therefore, the components (or portions thereof) of the computer 320 may be configured to function within various types of network environments that may include one or more devices and / or one or more server devices. For example, the network may be or may include a local area network (LAN), a wide area network (WAN), etc. The network may be or may include a wireless network and / or a wireless network implemented using, for example, a gateway device, a bridge, a switch, etc. The network may include one or more segments and / or may have portions based on various protocols such as the Internet Protocol (IP) and / or a proprietary protocol. The network may include at least a portion of the Internet.

[0060] In some implementations, memory 326 can be any type of memory, such as random access memory, disk drive memory, flash memory, etc. In some implementations, memory 326 can be implemented as more than one memory component associated with a component of computer 320 (e.g., more than one RAM component or disk drive memory). In some implementations, memory 326 can be a database memory. In some implementations, memory 326 can be or include non-local memory. For example, memory 326 can be or include memory shared by multiple devices (not shown). In some implementations, memory 326 can be associated with a server device (not shown) within a network and configured to serve components of computer 320. Figure 3 As shown, the memory 326 is configured to store various data, including first content data 332 , target UI element data 342 , second content data 352 , and rendering and display data 362 .

[0061] Figure 4 4 is a diagram showing an example electronic device in which the above technical solution can be implemented. Computer 420 is configured to generate a response to a target UI element. In some implementations, computer 420 communicates with a user device via a network. In some implementations, computer 420 is a user device.

[0062] The computer 420 includes a network interface 422, one or more processing units 424, and a memory 426. The network interface 422 includes, for example, an Ethernet adapter, a token ring adapter, etc., for converting electronic and / or optical signals received from the network into electronic form for use by the computer 420. The set of processing units 424 includes one or more processing chips and / or assemblies. The memory 426 includes both volatile memory (e.g., RAM) and non-volatile (non-temporary) memory such as one or more ROMs, disk drives, solid-state drives, etc. The set of processing units 424 and the memory 426 together form a processing circuit system that is configured and arranged to perform the various methods and functions described herein.

[0063] In some implementations, one or more of the components of computer 420 may be or may include a processor (e.g., processing unit 424) configured to process instructions stored in memory 426. Figure 4 Examples of such instructions depicted include a target UI element manager 430, a generation model manager 440, and a second content manager 450. Figure 4 As shown, the memory 426 is configured to store various data that are described with respect to the corresponding managers using such data.

[0064] The target UI element manager 430 is configured to receive target UI element data 432 representing at least one target UI element. The target UI element data 432 may also represent additional content including at least one target UI element. The additional content constitutes a part of the first content received by the user device in response to a request for the first content and provided by the website through the network.

[0065] In some implementations, the target UI element data 432 is received over a network. In such implementations, the computer 420 on which the generation model runs is remote from the computer that receives the target UI element from the website.

[0066] The generation model manager 440 is configured to generate the second content as the generation model data 442 in response to the target UI element in the target UI element data 432. In some implementations, the generation model includes a large language model (LLM). In some implementations, the generation model is trained to generate the second content, and in response to identifying the target UI element, the target UI element data 342 is modified by updating the target UI element data 342 with the second content.

[0067] The second content manager 450 is configured to send the second content, such as the generated model data 442, to the computer that receives the target UI element from the website. In some implementations, where the computer 420 on which the generated model runs is remote from the computer that receives the target UI element from the website, the second content manager 450 sends the second content over the network.

[0068] The components of the computer 320 (e.g., modules, processing unit 424) can be configured to operate based on one or more platforms (e.g., one or more similar or different platforms), which can include one or more types of hardware, software, firmware, operating systems, runtime libraries, etc. In some implementations, the components of the computer 420 can be configured to operate within a device cluster (e.g., a server farm). In such an implementation, the functionality and processing of the components of the computer 420 can be distributed to several devices of the device cluster.

[0069] The components of computer 420 may be or include any type of hardware and / or software configured to process attributes. Figure 4One or more portions of the components shown in the components of computer 420 in FIG. 4 may be or may include hardware-based modules (e.g., digital signal processors (DSPs), field programmable gate arrays (FPGAs), memory), firmware modules, and / or software-based modules (e.g., computer code modules, a set of computer-readable instructions that can be executed at a computer). For example, in some implementations, one or more portions of the components of computer 420 may be or may include software modules configured to be executed by at least one processor (not shown). In some implementations, the functionality of the components may be included in conjunction with Figure 4 Among those different modules and / or different components shown in the drawings, the functionality illustrated as two components is combined into a single component.

[0070] Although not shown, in some implementations, the components (or portions thereof) of the computer 420 may be configured to operate within, for example, a data center (e.g., a cloud computing environment), a computer system, one or more server / host devices, etc. In some implementations, the components (or portions thereof) of the computer 420 may be configured to operate within a network. Therefore, the components (or portions thereof) of the computer 420 may be configured to function within various types of network environments that may include one or more devices and / or one or more server devices. For example, the network may be or may include a local area network (LAN), a wide area network (WAN), etc. The network may be or may include a wireless network and / or a wireless network implemented using, for example, a gateway device, a bridge, a switch, etc. The network may include one or more segments and / or may have portions based on various protocols such as the Internet Protocol (IP) and / or a proprietary protocol. The network may include at least a portion of the Internet.

[0071] In some implementations, memory 426 can be any type of memory, such as random access memory, disk drive memory, flash memory, etc. In some implementations, memory 426 can be implemented as more than one memory component associated with a component of computer 420 (e.g., more than one RAM component or disk drive memory). In some implementations, memory 426 can be a database memory. In some implementations, memory 426 can be or include non-local memory. For example, memory 426 can be or include memory shared by multiple devices (not shown). In some implementations, memory 426 can be associated with a server device (not shown) within a network and configured to serve components of computer 420. Figure 4 As shown, the memory 426 is configured to store various data, including target UI element data 432 and generation model data 442 .

[0072] Figure 5is a flowchart depicting an example method 500 for generating a response to a target UI element and displaying the response on a display device according to the improved technique described above. The method 500 may be combined with Figure 3 The described software structure is executed by the computer 320, which resides in the memory 326 and is executed by the set of processing units 324.

[0073] At 502, the first content manager 330 receives first content from a first remote web server in response to a request for first content.

[0074] At 504, the target UI element manager 340 detects a target UI element within a portion of the first content. In some implementations, the detection includes determining that there are words conveying urgency in the portion of the first content. In some implementations, the detection is performed using a lookup table (LUT). In some implementations, the detection is performed using a model trained to identify target UI elements in content to evaluate the first content.

[0075] At 506, target UI element manager 340 sends at least a portion of the first content to the generation model in response to detecting the target UI element. In some implementations, target UI element manager 340 sends at least a portion of the first content to the generation model over a network.

[0076] At 508, the second content manager 350 receives the second content from the generation model, and the second content includes a difference with the part of the first content, and the difference is based on the type of the target UI element. In some implementations, the type of the target UI element is a text element, and the difference includes the text element or the deletion of the text element. In some implementations, the type of the target UI element is a graphic element, and the difference includes the graphic element or the deletion of the graphic element. In some implementations, the type of the target UI element is a font, color or emphasis, and the difference includes different fonts, colors or emphasis. In some implementations, the second content manager 350 performs detection instead of the target UI element manager 340.

[0077] At 510 , the rendering and display manager 360 initiates display of the first content on the display device, wherein a portion of the first content is updated with the second content.

[0078] Figure 6 6 is a flowchart depicting an example method 600 for generating a response to a target UI element according to the improved technique described above. The method 600 may be combined with Figure 4 The described software structure is executed by the computer 420, which resides in the memory 426 and is executed by the set of processing units 424.

[0079] At 602 , the target UI element manager 430 receives first content via a client device, the client device being configured to detect a target user interface (UI) element in the first content.

[0080] At 604 , generative model manager 440 generates second content using the generative model, the second content including a difference from the first content, the difference being based on the target UI element. In some implementations, the detection performed in step 602 may be performed by generative model manager 440 instead.

[0081] At 606 , the second content manager 450 sends the second content to the client device for display on a display device of the client device.

[0082] Item 1. A computer program product comprising a non-transitory storage medium, the computer program product comprising code, which, when executed by a processing circuit system, causes the processing circuit system to perform a method, the method comprising: receiving first content from a first remote web server in response to a request for first content; detecting a target user interface (UI) element within a portion of the first content; sending at least a portion of the first content to a generation model in response to detecting the target UI element; receiving second content from the generation model, the second content including differences from a portion of the first content, the differences being based on a type of the target UI element; and initiating display of the first content on a display device, wherein the portion of the first content is updated with the second content.

[0083] Clause 2. The computer program product of Clause 1, wherein at least a portion of the first content is sent to a second remote server comprising a generated model.

[0084] Clause 3. The computer program product of any of Clauses 1 or 2, wherein detecting the target UI element comprises sending the first content to a generative model that is trained to identify the target UI element.

[0085] Clause 4. A computer program product as described in Clause 3, wherein the generation model is trained to generate second content, and in response to identifying the target UI element, the first content is modified by updating a portion of the first content using the second content, wherein receiving the second content includes receiving the modified first content.

[0086] Clause 5. A computer program product as described in any of Clauses 1 to 4, wherein detecting the target UI element comprises: providing the first content to a model trained to identify the target UI element in the content; and receiving a result from the model indicating that a portion of the first content includes the target UI element.

[0087] Clause 6. The computer program product of any of Clauses 1 to 5, wherein the second content includes at least one annotation to a portion of the first content.

[0088] Clause 7. The computer program product of any of Clauses 1 to 6, wherein the portion of the first content comprises a first text and the second content comprises a second text different from the first text.

[0089] Clause 8. The computer program product of any of Clauses 1 to 7, wherein the portion of the first content comprises a first color and the second content comprises a second color different from the first color.

[0090] Clause 9. A computer program product comprising a non-temporary storage medium, the computer program product comprising code, which, when executed by a processing circuit system, causes the processing circuit system to perform a method, the method comprising: receiving first content via a client device, the client device being configured to detect a target user interface (UI) element in the first content; generating second content using a generative model, the second content comprising differences from the first content, the differences being based on the target UI element; and sending the second content to the client device for display on a display device of the client device.

[0091] Clause 10. The computer program product of Clause 9, wherein the second content includes at least one annotation to the first content.

[0092] Clause 11. The computer program product of any of Clauses 9 to 10, wherein the first content comprises a first text and the second content comprises a second text different from the first text.

[0093] Clause 12. The computer program product of any of Clauses 9 to 11, wherein the first content comprises a first color and the second content comprises a second color different from the first color.

[0094] Clause 13. A method comprising: receiving first content from a first remote web server in response to a request for first content; detecting a target user interface (UI) element within a portion of the first content; sending at least a portion of the first content to a generative model in response to detecting the target UI element; receiving second content from the generative model, the second content including differences from a portion of the first content, the differences being based on a type of the target UI element; and initiating display of the first content on a display device, wherein the portion of the first content is updated with the second content.

[0095] Clause 14. The method of Clause 13, wherein detecting the target UI element comprises sending the first content to a generative model that is trained to identify the target UI element.

[0096] Clause 15. The method of clause 14, wherein the generation model is trained to generate second content, and in response to identifying the target UI element, the first content is modified by updating a portion of the first content with the second content, wherein receiving the second content comprises receiving the modified first content.

[0097] Clause 16. A method as described in any of Clauses 13 to 15, wherein detecting the target UI element comprises: providing the first content to a model trained to identify the target UI element in the content; and receiving a result from the model indicating that a portion of the first content includes the target UI element.

[0098] Clause 17. The method of any of Clauses 13 to 16, wherein the second content includes at least one annotation to a portion of the first content.

[0099] Clause 18. The method of any of Clauses 13 to 17, wherein the portion of the first content includes a first text and the second content includes a second text different from the first text.

[0100] Clause 19. The method of any of Clauses 13 to 18, wherein the portion of the first content comprises a first color and the second content comprises a second color different from the first color.

[0101] Item 20. An electronic device comprising: a memory; and a processing circuit system coupled to the memory, the processing circuit being configured to: receive the first content from a first remote web server in response to a request for the first content; detect a target user interface (UI) element within a portion of the first content; send at least a portion of the first content to a generation model in response to detecting the target UI element; receive second content from the generation model, the second content including a difference from a portion of the first content, the difference being based on a type of the target UI element; and initiate display of the first content on a display device, wherein the portion of the first content is updated with the second content.

[0102] Clause 21. The electronic device of Clause 20, wherein at least a portion of the first content is sent to a second remote server comprising a generated model.

[0103] Clause 22. The electronic device of any of Clauses 20 to 21, wherein the processing circuitry configured to detect the target UI element is further configured to send the first content to a generative model that is trained to identify the target UI element.

[0104] Clause 23. An electronic device as described in Clause 22, wherein the generation model is trained to generate second content, and in response to identifying the target UI element, the first content is modified by updating a portion of the first content using the second content, wherein receiving the second content includes receiving the modified first content.

[0105] Clause 24. An electronic device as described in any of Clauses 20 to 23, wherein the processing circuit system configured to detect a target UI element is also configured to: provide the first content to a model trained to identify target UI elements in the content; and receive a result from the model indicating that a portion of the first content includes the target UI element.

[0106] Clause 25. The electronic device of any of Clauses 20 to 24, wherein the second content includes at least one annotation to a portion of the first content.

[0107] Clause 26. The electronic device of any of Clauses 20 to 25, wherein the portion of the first content includes a first text, and the second content includes a second text different from the first text.

[0108] Clause 27. The electronic device of any of Clauses 20 to 26, wherein the portion of the first content comprises a first color and the second content comprises a second color different from the first color.

[0109] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system that includes at least one programmable processor that can be either special purpose or general purpose and can be coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0110] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "non-transitory machine-readable medium," "non-transitory computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0111] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, verbal, or tactile input.

[0112] The systems and techniques described herein may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a client can interact with an implementation of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0113] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises from computer programs running on the respective computers and having a client-server relationship to each other.

[0114] A number of implementations have been described, however, it will be appreciated that various modifications can be made without departing from the spirit and scope of the specification.

[0115] It should also be understood that when an element is referred to as being on another element, connected to another element, electrically connected to another element, coupled to another element, or electrically coupled to another element, the element may be directly on another element, connected or coupled to another element, or one or more intermediate elements may be present. On the contrary, when an element is referred to as being directly on another element, directly connected to another element, or directly coupled to another element, there is no intermediate element. Although the terms "directly on...", "directly connected to..." or "directly coupled to..." may not be used throughout the detailed description, the element shown as being directly on...", directly connected or directly coupled may be referred to as such. The claims of the present application may be modified to narrate the example relationships described in the specification or shown in the accompanying drawings.

[0116] Although certain features of the described implementations have been illustrated as described herein, many modifications, substitutions, changes and equivalents will now occur to those skilled in the art. Therefore, it should be understood that the appended claims are intended to cover all such modifications and changes that fall within the scope of the implementations. It should be understood that they are presented only by way of example and not limitation, and various changes in form and detail may be made. In addition to mutually exclusive combinations, any part of the devices and / or methods described herein may be combined in any combination. The implementations described herein may include various combinations and / or sub-combinations of the functions, components and / or features of the different implementations described.

[0117] Furthermore, the logic flows depicted in the figures do not require the particular order shown or sequential order to achieve the desired results. In addition, other steps may be provided, or steps may be excluded from the described flows, and other components may be added to or removed from the described systems. Therefore, other implementations are within the scope of the following claims.

Claims

1. A computer program product comprising a non-transitory storage medium, the computer program product comprising code, which when executed by a processing circuit system causes the processing circuit system to perform a method, the method comprising: In response to a request for first content, receiving the first content from a first remote web server; Detecting a target user interface (Ul) element within a portion of the first content; In response to detecting the target UI element, sending at least the portion of the first content to a generation model; receiving second content from the generative model, the second content comprising a difference from the portion of the first content, the difference being based on a type of the target UI element; as well as Display of the first content is initiated on a display device, wherein the portion of the first content is updated with the second content.

2. The computer program product of claim 1, wherein: At least the portion of the first content is sent to a second remote server comprising the generative model.

3. The computer program product of claim 1, wherein: Detecting the target UI element includes sending the first content to the generative model, the generative model being trained to identify target UI elements.

4. The computer program product of claim 3, wherein: The generation model is trained to generate the second content, and in response to identifying the target UI element, the first content is modified by updating the portion of the first content using the second content, wherein receiving the second content includes receiving the modified first content.

5. The computer program product of claim 1, wherein: Detecting the target UI element includes: providing the first content to a specific model trained to identify one or more target UI elements in the content; and A result is received from the specific model indicating that the portion of the first content includes the target UI element.

6. The computer program product of claim 1, wherein: The second content includes at least one annotation to the portion of the first content.

7. The computer program product of claim 1, wherein: The portion of the first content includes a first text, and the second content includes a second text different from the first text.

8. The computer program product of claim 1, wherein: The portion of the first content includes a first color, and the second content includes a second color different from the first color.

9. A method comprising: In response to a request for first content, receiving the first content from a first remote web server; Detecting a target user interface (Ul) element within a portion of the first content; In response to detecting the target UI element, sending at least the portion of the first content to a generation model; receiving second content from the generative model, the second content comprising a difference from the portion of the first content, the difference being based on a type of the target UI element; as well as Display of the first content is initiated on a display device, wherein the portion of the first content is updated with the second content.

10. The method of claim 9, wherein: Detecting the target UI element includes sending the first content to the generative model, the generative model being trained to identify target UI elements.

11. The method of claim 10, wherein: The generation model is trained to generate the second content, and in response to identifying the target UI element, the first content is modified by updating the portion of the first content using the second content, wherein receiving the second content includes receiving the modified first content.

12. The method of claim 9, wherein: Detecting the target UI element includes: providing the first content to a specific model trained to identify one or more target UI elements in the content; and A result is received from the specific model indicating that the portion of the first content includes the target UI element.

13. The method of claim 9, wherein: The second content includes at least one annotation to the portion of the first content.

14. The method of claim 9, wherein: The portion of the first content includes a first text, and the second content includes a second text different from the first text.

15. The method of claim 9, wherein: The portion of the first content includes a first color, and the second content includes a second color different from the first color.

16. An electronic device, comprising: Memory; as well as processing circuitry coupled to the memory, the processing circuitry being configured to perform a method comprising: In response to a request for first content, receiving the first content from a first remote web server; Detecting a target user interface (Ul) element within a portion of the first content; In response to detecting the target UI element, sending at least the portion of the first content to a generation model; receiving second content from the generative model, the second content including a difference from the portion of the first content, the difference being based on a type of the target UI element; and Display of the first content is initiated on a display device, wherein the portion of the first content is updated with the second content.

17. The electronic device according to claim 16, wherein: At least the portion of the first content is sent to a second remote server comprising the generative model.

18. The electronic device according to claim 16, wherein: Detecting the target UI element includes sending the first content to the generative model, the generative model being trained to identify target UI elements.

19. The electronic device according to claim 18, wherein: The generation model is trained to generate the second content, and in response to identifying the target UI element, the first content is modified by updating the portion of the first content using the second content, wherein receiving the second content includes receiving the modified first content.

20. The electronic device according to claim 16, wherein: Detecting the target UI element includes: providing the first content to a specific model trained to identify one or more target UI elements in the content; and A result is received from the specific model indicating that the portion of the first content includes the target UI element.

21. The electronic device according to claim 16, wherein: The second content includes at least one annotation to the portion of the first content.

22. The electronic device according to claim 16, wherein: The portion of the first content includes a first text, and the second content includes a second text different from the first text.

23. The electronic device according to claim 16, wherein: The portion of the first content includes a first color, and the second content includes a second color different from the first color.

Images