Extended UI generated via natural language queries and semantic data connections

Abstract

Techniques for changing the user interface associated with a browser application include enabling a user to change the web page displayed by a web browser in ways that the creator of the web page elements did not anticipate, without the need to understand the internal structure of the web page. For example, an improved technique implemented as part of a technical solution enables participants in a web conference to highlight the names of other participants with the same manager, without knowing anything about the architecture or programming code associated with the web conference application.

Description

Technical Field

[0001] Cross - reference to Related Applications This application is a continuation of U.S. Application No. 17 / 814,066, filed on July 21, 2022, claims priority thereto, and incorporates by reference in its entirety the disclosure of this U.S. application herein.

Background Art

[0002] A browser is an application that enables a user to navigate to various websites and view the content or web pages generated by the websites. Such content can include web page elements (e.g., markup language elements, Document Object Model (DOM) objects) that are rendered by the browser to generate the presentation of the web page for display.

Summary of the Invention

[0003] The embodiments described herein relate to enabling a user to modify web page elements displayed by a web browser in a way that the creator of the web page elements did not anticipate and without the need to understand the internal structure of the web page. In particular, the disclosed embodiments respond to natural language requests to modify the presentation of the current web page being displayed by the browser, identify data elements (such as markup language elements, DOM elements) within the web page, and modify the appearance of the response data elements according to the above requests. In some embodiments, the proposed system can access data in a data store to determine which elements to modify.

[0004] In one general aspect, a computer-based method can include receiving, from a user of a web browser, query data representing a request to change a presentation of a web page displayed by the web browser, where the web page includes a plurality of structured data elements and the request identifies a criterion. The computer-based method can also include analyzing the web page to identify structured data elements among the plurality of structured data elements that meet the criterion. The computer-based method can further include applying a change to the structured data elements that meet the criterion to generate a changed presentation of the web page displayed by the web browser.

[0005] In other general aspects, a computer program product includes a non-transitory storage medium, and the computer program product includes code that, when executed by a processing circuit of a computing device, causes the processing circuit to perform a method. The method can include receiving, from a user of a web browser, query data representing a request to change a presentation of a web page displayed by the web browser, where the web page includes a plurality of structured data elements and the request identifies a criterion. The method can also include analyzing the web page to identify structured data elements among the plurality of structured data elements that meet the criterion. The method can further include applying a change to the structured data elements that meet the criterion to generate a changed presentation of the web page displayed by the web browser.

[0006] In another general aspect, an electronic device includes a memory and a control circuit coupled to the memory. The control circuit can be configured to receive query data representing a request from a user of a web browser to change the presentation of a web page displayed by the web browser, the web page including a plurality of structured data elements and the request identifying a criterion. The control circuit can also be configured to analyze the web page to identify structured data elements among the plurality of structured data elements that meet the criterion. The control circuit can further be configured to change the structured data elements that meet the criterion to generate a changed presentation of the web page displayed by the web browser.

[0007] Details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims.

Brief Description of the Drawings

[0008]

Figure 1A

Figure 1B

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

DETAILED DESCRIPTION OF THE INVENTION

[0009] Users of web pages often desire to change the web page to retrieve additional information beyond what is presented on the web page. For example, consider a web conferencing application that places identifiers of conference participants (e.g., names, avatars) in a column beside the window displaying the live stream of a conference. In a conventional approach to changing a web page, the user was only able to change the interface implemented by the application developer. If a user desired to highlight (e.g., highlight, bold, blink) the identifiers of participants who report to the same manager as the user, the user was unable to do so because the web developer did not permit the user to do so. The disclosed embodiments can fulfill such requests by enabling the user to change the web page even when the web developer did not design the web page to incorporate such changes. The usability may take the form of a natural language query that can be interpreted by a natural language processor in conjunction with semantic information within the web page.

[0010] A query represents a request to change web page elements according to a criterion. The criterion can identify the target of an emphasis operation (e.g., highlighting) based on structured data elements representing the target (e.g., semantic markup elements, table elements, lists, etc.). The web browser determines which structured data elements can be related to the target of the emphasis operation. In that case, a natural language processor is configured to determine the criterion based on the query and semantic information. For example, in the case of a query "Highlight the participants in the meeting who have a relationship with the same manager as me", the criterion is "the same manager as me".

[0011] On the other hand, in order to identify the target of the emphasis operation, the disclosed embodiments may attempt to access an external data set. Along this line, there may be a relevance that enables determining the relevance between a structured data element and an external data set via a natural language query including a criterion. For example, in the above query, the disclosed embodiments may include identifying, searching / sorting a human resources database to find the manager of the participants and determine the participants represented by the structured data elements that meet the criterion. Thereby, since the data is in a machine-readable format via semantic metadata, the user is provided with the ability to change any web page using semantic information that allows the web browser to interpret the data within the web page.

[0012] Some approaches to changing the user interface of a browser application include browser extensions that allow users to manipulate web elements of the user interface. However, these approaches are vulnerable to changes in the interface made by the creator of each browser application and require some programming experience (understanding of how web applications are structured and understanding of browser scripts such as JavaScript (registered trademark)). Therefore, most users cannot change the browser application view to be executable, and for users who can change it, for example, if the creator or provider of a web page changes or redesigns the structure and layout of the elements included in the web page without prior notice, the change may be interrupted and the user may need to redo the change.

[0013] The technical problem with the above conventional approaches is that the creator of a web page may provide some options to the user to customize the user interface of a web application, but those customizations are limited and static. Conventional third - party approaches to changing the user interface of a web application have controls that allow users to manipulate elements of the user interface. Conventional approaches to changing the user interface of a web application (e.g., using extensions) are vulnerable to changes in the interface made by the creator of each browser application. Therefore, it is not possible for a user of a conventional browser application to extend the view of the web application without knowing detailed information about how the web application is structured and without manually creating a vulnerable custom browser extension (in browser scripts such as JavaScript (registered trademark)) to change web page elements.

[0014] According to the embodiments described in this specification, the technical solution to the above technical problem includes enabling a web page displayed by a web browser to be modified by a user in a way not expected by the creator of the web application, for example, beyond the scope of any options provided to the user, without the need to understand the internal structure of the web page. For example, an improved technique executed as part of the technical solution enables, as shown in FIG. 1A, highlighting the names of participants with the same manager even if the user knows nothing about the architecture or programming code related to the web conferencing application. The disclosed system includes a processing circuit that enables a user to submit a query that references criteria satisfied by structured data elements within the web page.

[0015] In the improved technique, the browser parses a natural language query, identifies the criteria in the query, analyzes structured data elements on the web page, and potentially one or more known external data sets to evaluate the criteria and identify the structured data elements to be modified, thereby enabling the user to modify (e.g., extend) the presentation of the web page displayed by the browser. The browser achieves this by receiving a natural language query containing the criteria provided by the user.

[0016] Natural language queries are processed using a natural language processor. Natural language processing relates to the interaction between a computer and human language, particularly the way of programming a computer to process and analyze large amounts of natural language data. The goal is to have a computer that can "understand" the content of a document, including the contextual nuances in the language within the document. A natural language processor can accurately extract the information and insights contained in a document and can also classify and organize the document itself. When a query is described in natural language, using a natural language processor to process the query is advantageous because the processor can interpret the user's intent much more accurately than a conventional query parser in combination with the markup language information within a web page.

[0017] A query represents, for example, a request to modify structured data elements that meet certain criteria. The criteria can identify an emphasis operation and the target of the emphasis operation. The browser analyzes web page elements to identify elements that include data structured elements (such as semantic markup elements, table elements, lists, etc.). The browser determines which structured data elements may be related to the target of the emphasis operation. For example, some web page elements may be used to identify other web page elements to which an emphasis operation is applied. These elements (the elements to which the target operation is applied) may be referred to as web elements having structured data elements that meet the criteria. The browser can also identify and obtain one or more known external data sets based on the query criteria, and this external data set (including the open browser tab and / or a data set associated with the user, such as a shopping list or a list of interests) may be stored locally or on a server other than the server hosting the web page. The browser uses the information from one or more external data sets together with semantic information to change the presentation of the web page as the user intended.

[0018] The technical advantages of the disclosed embodiments are that improved techniques enable a user to change the presentation of a web page without the user having to recognize the internal structure and elements of the web page and / or without the user having to create their own dedicated browser extension, for example, using natural language commands. Specifically, the embodiments can identify structured data elements within a web page identified by semantic information and can cross-reference the structured data elements with an external data store to identify the structured data elements in response to a request. The embodiments can change these response data elements within the current web page to alter the appearance of these elements. Further, to ensure privacy, data from the external data store remains local to the computer running the browser and is not sent to the server associated with the web application. Thus, changes to the presentation of the web page can be achieved without sending data to the server hosting the web page.

[0019] FIG. 1A is a diagram showing an example of changing the presentation of a web page, for example, user interface 111, in response to a query from a user. This example is not intended to be limiting and is presented for ease of explanation. The user interface 111 shown in FIG. 1 displays information for participants in a web conference. The user interface 111 displays web page elements 150 and 160. The web page element 150 displays the avatars or video streams of the participants arranged in an array. The web page element 160 displays a list of participant names. For example, as shown in FIG. 1A, the name "User 2" is in bold. This may be the name of a participant or the default setting of the web conference application. On the other hand, when a participant submits a query "Show participants who have the same manager as me" to a processing circuit 120 (FIGS. 1B and 2) configured to process queries, the processing circuit 120 processes the query, identifies the criteria and changes in the query, identifies an external data set (e.g., an organizational chart) in which information about managers is provided, and applies the criteria from the query to the web page elements in consideration of the external data set to determine the participants who have the same manager as the participant, thereby implementing an improved technique. As shown in FIG. 1A, those participants are User 7 and User 8. As a result, the processing circuit 120 displays the list elements of User 7 and User 8 in italics. Note that the word "highlight" from the query has been changed by the processing circuit and is interpreted in this case as an italic display operation. The result is a change 162 of the web element, i.e., a changed presentation of the web page (i.e., a changed user interface). In some embodiments, the word "highlight" is used for other changes, such as highlighting, bold, blinking, etc. In some embodiments, other words in the query are interpreted to provide a change.

[0020] In some embodiments, the query is input by the user via keyboard input such as text box input. In some embodiments, the query is input by the user via voice, i.e., spoken words from the user input to an application configured to capture the voice and parse it into query components (e.g., criteria, target, change). In some embodiments, the voice is displayed to the user via a voice-to-text conversion application.

[0021] In some embodiments, processing circuit 120 processes the query using a natural language processor. A browser application that receives a natural language query from a participant may use existing natural language processing functions (such as part-of-speech tagging, entity identification, etc.) related to semantic representation, language understanding, and question answering for the search application. Improved techniques utilize such functions to understand the intent of natural language requests and convert the query into instructions for fetching data or extending the web application interface.

[0022] A web browser is application software that runs on a processing circuit (e.g., processing circuit 120) for accessing web pages. When a user requests a web page from a website, the web browser retrieves the content from the web server and renders and displays the content within a browser window on the user's device. A web application is identified by the uniform resource locator (URL) entered by the user. A web browser can display multiple web pages within a single user interface. Each such web page is activated using a tab, e.g., at the top of the user interface. A user of the web browser can activate another web page by clicking on the tab of that web page on the user interface.

[0023] A web application is an application developed by an application creator, stored on a web server, and executed within a web browser. A web application defines web elements to be rendered by a browser to generate a user interface. When receiving a request to access the browser application from a user device (i.e., via a URL), the web server sends code that is interpreted (rendered) and executed within the web browser on the user device.

[0024] Web page elements are objects defined by a web application. These include visible elements (e.g., a participant list and participant streaming images in a web conferencing application) and non-visible elements (e.g., JavaScript). For example, in the above web conferencing application, web page elements 150 and 160 include visual information regarding the participants and participant names of the meeting. In a browser application, some web page elements include application data (e.g., identifiers, avatars, and photos of meeting participants) and other forms of data including semantic information (e.g., person, movie, director).

[0025] Structured data elements are data objects stored in a data structure that provides a hierarchical description of information within a web element and are organized according to a data model. In some embodiments, structured data elements follow a tabular form having relationships between different rows and columns. In this way, different rows and columns can be sorted or searched according to a query. An example of a structured data element in the context of changing a web page element will be described with respect to FIG. 3.

[0026] Semantic markup information is not simply to define its presentation or appearance, but is markup that enhances the semantics or meaning of information within web pages and browser applications. Semantic markup information is processed not only by conventional web browsers, but also by many other user agents. Further, other more detailed semantic markup information, including Resource Description Framework (RDF), Web Ontology Language (OWL), and Extensible Markup Language (XML), is provided in the Semantic Web. HTML describes documents and the links between documents. In contrast, RDF, OWL, and XML can describe anything such as people, meetings, or cooking ingredients.

[0027] In some embodiments, the semantic information within a web page is expressed according to the W3C Consortium standards of the Semantic Web. The Semantic Web enables web pages to encode web page elements using semantics (i.e., meaning). For example, in FIG. 1A, data representing a column of meeting participants (i.e., web page element 160) can include metadata for identifying that this is a list of participants, and the data elements within the column can be, for example, <div itemscope itemtype=""https: / / schema.org / Person”">can be clearly identified, such as by identifying data elements as individuals, names, surnames, LDAP / email addresses, etc. As a result, the data becomes machine-readable via semantic metadata, enabling the browser to interpret the data within the web page. This example is not limiting, and any label field (e.g., Or It is also possible to identify data elements in the "label" field of HTML markup tags).

[0028] In this context, a change in the presentation of a web page displayed by a web browser is a change to one or more web elements in the browser that changes the appearance of the content displayed within the web browser. In some embodiments, the change is an extension to structured data elements within the web browser, such as annotations like underlines, asterisks, etc. In some embodiments, the change is an emphasis such as italic text, a change in text color, a change in text font, a change in text font size, a change in background color, bold text, underlined text, highlighting, etc. This change is sometimes referred to as a visual pointer because it visually distinguishes the changed element. Thus, the change can add to the browser a web page element that, when rendered by the browser, causes a change in the appearance of a web page element. In some embodiments, the change is a removal or deletion of display data in the user interface or web element. In some embodiments, when the structured data element meeting the criteria includes a table, the change represents a sorting operation where the table rows within the table are rearranged to generate a changed sort order when the change is applied. In some embodiments, when the structured data element meeting the criteria from a subsequent query includes the table rows of a list element, the change includes a re-sorting (rearrangement) of the table rows.

[0029] In some embodiments, the change may be selected from a pre-specified change, i.e., a library of pre-defined emphasis operations. Such a library may include a change identifier and, for each change identifier, instructions for making the change. For example, a change identifier for bold may markup an element (e.g., ) can be associated with instructions for applying, and the highlighting change identifier can be associated with instructions for applying markup elements for changing the background color of web page elements that meet the criteria. The change identifier may be text (e.g., "bold"), or any string, number, or combination thereof corresponding to text that may appear within the query. This library can also be used when the query is ambiguous regarding the change. For example, if the query states "show participants with the same manager as me", the processing circuit can randomly select a change from the library and execute that instruction. In some embodiments, the improved technique includes default changes when the query renders the change as uncertain. Exemplary instructions include adding markup tags for executing a pre-specified change. In some embodiments, the instructions can be customized by the user (e.g., enabling the user to select a highlight color, default change, etc.).

[0030] In some embodiments, the processing circuit may be configured to provide an interface that enables the user to select a change from two or more changes (e.g., selectable changes). For example, a query including a highlight may correspond to two or more changes within the library, such as highlighting in blue or highlighting in yellow. In such embodiments, the user is enabled to select a change from two or more changes (also referred to as selectable changes or change candidates). For example, the change candidates may be presented to the user in a scrollable list, and the user may click / select the desired change.

[0031] In some embodiments, the user may be enabled to select default changes that are made when the user does not specify a change. Such default changes are, in some embodiments, stored in a profile associated with the user and / or device, such that the default changes are effective for the user when using any web application within the web browser. Further, the user may save their most recent changes to a buffer (e.g., cache memory) or other storage for application to subsequent web pages.

[0032] In some embodiments, the change may take the form of de-emphasizing web page elements that do not meet the criteria. For example, de-emphasis can take the form of lightening or fading the text color, returning bold text to normal text, removing emphasis, reducing the size of the font text, etc.

[0033] Query data represents a query, or a request to change the user interface or web element. Query data in this context includes the identification of data (e.g., web page elements) within the web page to be changed, the criteria for identifying those structured data elements to be changed, and the changes to be made to the presentation of the web page. In some embodiments, the query data is sent to a natural language processor for identification of the data, criteria, and changes. For example, the natural language processor may identify the changes in the query data. The natural language processor may then send an identifier representing the changes to the processor executed by the web browser.

[0034] A criterion is a condition that is satisfied by the data being changed and not satisfied by the data that is not changed. In some embodiments, the reverse occurs. In the example of the web conferencing application above, the criterion is "same manager as me". This means that the processing circuit identifies an external dataset that includes information about the participants' managers and compares the manager of each participant to the participant who made the query. For example, the processing circuit may identify all participants listed in a database in the external dataset who have the same manager as the participant. In other examples, the processing circuit may identify all participants listed in a database who do not have the same manager as the participant.

[0035] FIG. 1B is a diagram showing an exemplary electronic environment 100 in which an improved technique for changing a web page of a browser application described herein may be implemented. An example of a web conferencing application has already been described. Other exemplary web pages that may be changed according to the improved technique include pages for activities such as cooking, banking transactions, news sites, watching sports events, and statistical surveys. In short, any web page that includes structured data elements and semantic markup information can be changed using the improved technique.

[0036] As shown in FIG. 1B, the electronic environment 100 includes a user device 106 and a network 102, and the user device accesses a browser application from a web page server 105 (e.g., a browser application server) via the network 102. The network 102 may take the form of, for example, a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. In some examples, the network 102 can be accessed via wired and / or wireless communication links. For example, a mobile computing device such as a smartphone can access the network using a cellular network. There may be multiple user devices 106 and web page servers 105 within the electronic environment 100.

[0037] In some examples, the web page of the browser application is provided as a web page server 105 associated with a domain name. An exemplary website can include a collection of web pages formatted in a suitable machine-readable language, such as a markup language, such as Hypertext Markup Language (HTML), that can include text, images, multimedia content, and programming elements, such as scripts. Text, images, multimedia content, and programming elements are examples of web page elements. Each website is maintained by a publisher, such as an entity that manages and / or owns the website. The web page server 105 can be static or dynamic.

[0038] In some examples, the web page server 105 provides data via the network 102 and is associated with a resource address, such as a Uniform Resource Locator (URL). In some examples, the web page server 105 can provide a website that includes digital content, such as web pages, word processing documents, and Portable Document Format (PDF) documents, images, videos, and feed sources, among other suitable digital content. The web page server 105 can store content, such as words, phrases, images, and audio, and can include embedded information, such as meta information and hyperlinks, and / or embedded instructions, such as scripts.

[0039] In some examples, user device 106 is an electronic device under the control of a user that can request and receive resources from web page server 105 via network 102. Examples of user device 106 include a personal computer capable of sending and receiving data via network 102, a mobile computing device, such as a smartphone, a wearable device, a smart television, and / or a tablet computing device. As used throughout this specification, the term mobile computing device ("mobile device") refers to a user device configured to communicate via a mobile communication network. A smartphone, for example, a phone capable of communicating via the Internet, is an example of a mobile device, as are other smart devices such as wearable devices and smart speakers. User device 106 typically includes a user application, such as a web browser, to facilitate sending and receiving data via network 102.

[0040] In some embodiments where multiple user devices exist within network 102, each of the multiple user devices is associated with a respective user. In such cases, changes to a web page initiated by a first user device associated with a first user can be shared with a second user device associated with a second user. For example, at least some or all of the participants in the web conference shown in FIG. 1A may be able to view changes to the web page in a manner initiated by one of the participants. Specifically, that participant can select the participants with whom to share the change to the web page and exclude other participants from being able to view the change to the web page. Such sharing may occur, for example, within the same channel as a chat channel provided within a web conference application. The ability to share changes to a web page is not limited to this example of a web conference application, and any change to a web page may be shared with any other user device within network 102.

[0041] In some embodiments where multiple user devices exist within network 102, each of the multiple devices may change a web page along the lines disclosed herein. As disclosed above, each of the multiple users on each user device may select other users who share the changes to the web page. The change data generated by processor 120 can be sent to other user devices within network 102. In this case, when a change to the web page is made within the browser extension, the browser extension running on the browser of the other user device receives the change data and changes the web page according to the received change data.

[0042] As shown in FIG. 1B, user device 106 includes processing circuitry 120. Processing circuitry 120 is configured to execute a web browser and display a browser window within a user interface in the display of user device 106. User interface 111 is also configured to receive a user query, process the received user query to identify web elements to be changed, structured data elements that identify data within the web elements to be changed, and to identify the changes to be applied. The processing circuitry may be further configured to identify an external data set against which a criterion is evaluated (e.g., external data set 122 and / or a data set from external server 121), and web element data selected for change. Further details regarding processing circuitry 120 are described with respect to FIG. 2.

[0043] As shown in FIG. 1B, the user device 106 displays a user interface 111 of a web page within a web browser via a processing circuit 120. In the example of FIG. 1B, the user interface 111 is shown as a single browser tab that displays a web page including various web page elements 112, 113, 114. However, in some embodiments, other web pages may be displayed by clicking on a tab at the top of the user interface 111. Also, in FIG. 1B, the user interface 111 is shown as an entity separate from the processing circuit 120. This is one possible configuration, but in some embodiments shown in FIG. 2, elements of the user interface 111 are integrated within the processing circuit 120.

[0044] The web page elements 112, 113, 114 are objects that are displayed by the web browser in the presentation of the web page 111. The web page elements 112, 113, 114 each contain data that is presented to the user, as discussed with respect to FIG. 1A and other examples. For example, a restaurant ordering application or a cooking site can display the ingredients of a dish provided in the web page element 112, the description of the dish provided in the web element 113, and / or instructions for preparing the dish in the web page element 114.

[0045] As shown in FIG. 1B, each of the web page elements 112, 113, 114 includes respective structured data elements 115, 116, 117. The structured data element 115 functions as a descriptor of the data. For example, a table or a list is a structured data element arranged to be easy to view and sort. For example, in a restaurant ordering application, the ingredients listed in the web page element 112 may include flour, sugar, eggs, water, milk, baking powder, salt, vanilla, chocolate chips, and bananas.

[0046] In some embodiments, the web page elements 112, 113, 114 are associated with a document model (e.g., a document object model) that includes a set of nodes representing the structured data elements 115, 116, 117. In the above example, the item may be a list element, e.g., It can be identified by HTML tags. In the above example, an item may be in a separate table row, for example, where each item is in a column labeled "component" and appears along with other columns such as measurement type, measured quantity, and / or calories. In the above example, an item may have semantic markup, for example, metadata tags indicating that each item is an ingredient of a recipe.

[0047] In one example, a user may submit a query 108 such as "mark the ingredients in the shopping list sold at a nearby grocery store". The processing circuit 120 performs natural language processing operations on the query and, in conjunction with the semantic data on the web page, determines that the user's intention is to access food information for the user, for example, by associating it with the user profile or the user's account. Note that some of the shopping list data described here may be private and generally not shared. Thus, the processing circuit 120 should be able to change the user interface without, for example, sending private data or any other data to the web page server 105.

[0048] Furthermore, the criteria specified by the user in query 108 is that the food identifiers listed in the web page element 112 match the food identifiers in the grocery store database that identify that the food is on sale. Thus, the processing circuit accesses the structured data element 115 to obtain the food identifiers of the structured data element 115 and compares these food identifiers with the food identifiers in the grocery store database. The processing circuit identifies a match between the food identifier and the food identifier as shown by the structured data element 118 identified in the example of FIG. 1B.

[0049] Query data also specifies certain changes. The changes can be specific, such as "highlight" or "make blue". The changes can also be general, i.e., "mark" of exemplary query 108, or "show me". The system can include a plurality of pre-defined change identifiers corresponding to text strings in the query data (e.g., "highlight", "italicize", "bold", "underline", etc.). These pre-defined changes can include templates with fixed and variable parts, for example, the variable part can include a corresponding "turn *" that matches what follows the fixed part. The variable part may represent a set of pre-defined values such as colors. Thus, "make red" or "make blue" can be change identifiers corresponding to pre-defined changes. Some pre-defined terms in the query can be converted into change identifiers. For example, the processing circuit 120 can recognize "mark" in the query as a non-specific change request and convert "mark" (non-specific change request) into "highlight" (change identifier). This conversion can be determined by randomly selecting a change identifier from among the pre-defined change identifiers. This conversion can be determined by selecting a default change identifier. This conversion can be determined by rules or by a trained classifier that performs a machine learning-based classification operation to identify the "mark" that is most likely to refer to highlighting. The change identifier for highlighting may be associated with an instruction to apply a background color markup tag to a web page element (s) that meet the criteria. Thus, the processing circuit 120 sends an instruction in the form of change data 109 to highlight the structured data element 115 whose ingredient identifier matches the food identifier in the food database. At this time, the user interface 111 shows the user which materials are sold in nearby grocery stores.

[0050] In some embodiments, the processing circuit 120 obtains an external data set from an external data set that is available to the user device 106 and not on the public network. For example, the external data set 122 may be local storage on the user device 106. Alternatively, the external data set 122 may be an external hard drive directly connected to the user device 106 or connected via a wireless LAN.

[0051] In some embodiments, the processing circuit 120 obtains an external data set from an external server 121 via the network 103. In some embodiments, the network 103 is not distinct from the network 102. In some embodiments, the network 103 is separate from the network 102. In some embodiments, the network 103 includes a virtual private network (VPN) for accessing private data.

[0052] FIG. 2 is a diagram showing an exemplary processing circuit 120 (FIG. 1B) capable of implementing the above technical solution. The processing circuit 120 is configured as follows.

[0053] The processing circuit 120 includes a network interface 222, one or more processing units 224, a memory 226, and a display interface 228. The network interface 222 includes, for example, an Ethernet (registered trademark) adapter, a token ring adapter, etc., to convert electronic signals and / or optical signals received from a network into an electronic format for use by the processing circuit 120. The set of processing units 224 includes one or more processing chips and / or processing assemblies. The memory 226 includes both volatile memory (e.g., RAM) and non-volatile (non-transitory) memory such as one or more ROMs, disk drives, solid state drives, etc. The set of processing units 224 and the memory 226 together form a control circuit that is configured and arranged to perform various methods and functions as described herein. The display interface 228 is configured to provide data to a display device for rendering and display to a user.

[0054] In some embodiments, one or more of the components of the processing circuit 120 can be or include a processor (e.g., a processing unit 224) configured to process instructions stored in the memory 226. Examples of instructions as shown in FIG. 2 include a query manager 230, a web browser manager 240, an external data set manager 250, and a web page element change manager 260. Further, as shown in FIG. 2, the memory 226 is configured to store various data, and this data is described with respect to each manager that uses such data.

[0055] The query manager 230 is configured to receive query data 231. In some embodiments, the query manager 230 receives the query data 231 via the network interface 222, i.e., via the network. In some embodiments, the query manager 230 receives the query data 231 directly as user input, e.g., from the user device 106. As shown in FIG. 2, the query manager 230 includes a natural language processing (NLP) manager 232.

[0056] The query data 231 represents a query (i.e., a request) to change user interface web page elements according to a criterion. In some embodiments, the query data 231 is in the form of a character string. In some embodiments, the query data 231 takes an alternative form, such as a graphic, an image, an audio clip, a video, etc. In some embodiments, the query data 231 is described in a form suitable for natural language processing.

[0057] The NLP manager 232 is configured to process the query data 231 using natural language processing algorithms to generate NLP data 233. The NLP algorithms mainly function on the syntax (i.e., structure) of the query represented by the query data 231 and the semantic data, e.g., the semantics (i.e., meaning) represented by the semantic data 244. On the syntax side, the NLP algorithms may be configured to classify and organize words and associate them with their basic forms, e.g., part-of-speech analysis. In some embodiments, the NLP algorithms include decomposing sentences, inducing grammatical structures, and segmenting words. On the semantics side, the NLP algorithms may perform analysis and recognition of entities and entity attributes (including entity names and synonyms).

[0058] The NLP data 233 represents the output of executing the NLP algorithm of the NLP manager 232. The NLP data 233 is in a form interpretable by the processing circuit 120 for identifying structured data elements for which the user interface 111 (FIG. 1B) should be changed and for changing the user interface 111 based on the identified structured data elements. As shown in FIG. 2, the NLP data 233 includes reference data 234, structured data element identifier data 235, change data 236, and / or external data set identifier data 237, and this data represents the output from a natural language processing algorithm.

[0059] The reference data 234 represents a reference against which structured data elements for which the user interface 111 should be changed are identified. For example, in the case of a request such as "Show participants who have the same manager as me", the NLP manager 232 may presume that the reference represented by the reference data 234 identifies participant names within the external data set related to the request as those having the same manager as the user. In another example, in the case of a request such as "Mark ingredients sold at nearby grocery stores", the NLP manager 232 may presume that the reference represented by the reference data 234 identifies food identifiers within the grocery database that are identified as being on sale. Generally, the reference data 234 indicates aspects of the external data set that are compared to fields within the structured data elements.

[0060] The structured data element identifier data 235 represents a reference to a structured data element (e.g., the structured data element 115 in FIG. 1B) of a web page element (e.g., the web page element 112) indicated by a criterion. For example, in FIG. 1A, the structured data element identifier data 235 represents user identifiers "User 1", "User 2", ···, "User 12". Next, the query manager 230 is further configured to match the structured data element identifier data 235 with corresponding data (e.g., the external data set data 251) in an external data set. For example, the query manager 230 compares the user identifiers of the structured data element identifier data 235 with the user identifiers in a human resources database that provides manager information for each identifier. Next, together with the reference data 234, the query manager 230 identifies structured data element identifiers that meet the criterion, i.e., user identifiers that have a specific manager. Further details regarding the structured data element will be described with respect to FIG. 3.

[0061] In some embodiments, the query manager 230 identifies structured data elements by identifying table elements, list elements, or semantic markup elements. For example, HTML table elements that can be identified include tags such as (defining a table row), (defining a header), and (defining a table cell). HTML list elements that can be identified include unordered lists (bulleted lists) , of the ordered list and There are tags such as (definition list). The semantic markup items of HTML that define semantic markup information include descriptors of what each element represents. For example, the HTML tag <div itemtype=""http: / / schema.org / Restaurant”"> <span property=""rating”">4 <span property=""telephone”">867-5309 notifies the natural language processor that the value of the first field of the structured data element represents a restaurant defined in the schema.org hierarchy. The rating of the restaurant is 4 and the phone number is 867 - 5309. Thus, the semantic markup information positions the rating and phone number in the context of the restaurant. When a structured data element is identified via these table elements, list elements, or semantic markup elements, the query manager 230 compares it against a reference as described with respect to FIGS. 4 and 6.

[0062] FIG. 3 is a diagram showing exemplary web elements and structured data elements within web page 300. In this context, web page 300 represents a web conferencing application as in the above example. The user interface of web page 300 in this case may be similar to user interface 111 shown in FIG. 1A. FIG. 3 represents the hierarchical composition of the web conferencing application. In some embodiments, FIG. 3 represents a simplified document object model.

[0063] Web page 300 has a top - level page 302 that represents the entire web application and is displayed in the user interface (i.e., user interface 111). In the case shown in FIG. 3, the web application is a web conference. This top - level page 302 has two web elements, namely, a first web element that displays a chat section 304 and a second web element that displays a participant list 306.

[0064] The chat section 304 is configured to enable participants to send private or public messages to other participants during a web conference. As shown in FIG. 3, the chat section 304 includes a message list 308 that represents public messages and / or private messages sent among conference participants.

[0065] The message list 308 includes structured data elements 310(1) and 310(2), each representing data (e.g., metadata) that describes each message. As shown in FIG. 3, the structured data element 310(1) includes fields having a message identifier of "Message 1", a sender identifier of "Judy@", and a representation of the message body of "Hi, Andy". Note that the representation of the message body may include the entire message, a part of the message, or the first few words of the message. The structured data element 310(2) has field values of "Message 2", "Jason@", and "Let’s meet in Austin", respectively.

[0066] The participant list 306 is configured to include all participants in the web conference. In some embodiments, the participant list 306 may be changed in real time according to which participants join or leave the web conference while the web conference is being conducted. In some embodiments, the participant list 306 is static. The participant list 306 includes structured data elements 312(1), 312(2), 312(3), and 312(4), each representing data (e.g., metadata) that describes each participant. As shown in FIG. 3, the structured data element 312(1) includes fields having a name identifier of "Judy", a surname identifier of "Archibald", and an email address identifier of "Judy@" (which matches the sender identifier of the structured data element 310(1)). The structured data element 312(2) includes a name identifier of "Andy", a surname identifier of "Lewis", and an email address identifier of "Andy@". The structured data element 312(3) includes a name identifier of "Jason", a surname identifier of "Caleb", and an email address identifier of "Jason@". The structured data element 312(4) includes a name identifier of "Austin", a surname identifier of "Joshua", and an email address identifier of "Austin@". In some embodiments, "Austin" in the message body of the structured data element 310(2) is rendered as different from the name identifier "Austin" of the structured data element 312(4) by an NLP algorithm. The former context identifies a location, and the latter context identifies a person.

[0067] Note that the structured data elements of the web page elements are represented by a set of nodes in a tree structure when the document model associated with the web page elements is in a tree structure. That is, the document model associated with the web page elements describes the interrelationships of the structured data elements.

[0068] Returning to FIG. 2, the change data 236 represents changes applied to the structured data element identifiers of the web elements of the user interface that meet the criteria. For example, the NLP algorithm identifies changes based on the word "show" and processes the above query to identify semantic information such as semantic data 244. In some embodiments, the query manager 230 interprets this change as indicating an emphasis operation on the selected structured data element identifier. In some embodiments, the change is selected from a finite list of specified changes.

[0069] The external data set identifier data 237 represents the external data set identified by the query (e.g., external data set 122 in FIG. 1B). For example, the NLP algorithm processes the above query to identify the external data set based on the word "manager" and semantic information such as semantic data 244. Based on the semantic data, the NLP algorithm identifies the word "manager" as having relevance to the human resources database. In this case, the external data set identifier data 237 includes data representing the identifier of the human resources database.

[0070] In some embodiments, in the above example, the human resources database is stored locally and the identifier provides the location of the human resources database in storage. In some embodiments, the human resources database is accessed via a network (e.g., network 103) at an external server (e.g., external server 121). In some embodiments, the human resources database is non - public and may be accessible only from the external server using various data protection schemes (e.g., passwords, two - factor authentication, knowledge - based authentication, etc.). Further details regarding the external data set are described with respect to FIGS. 4 and 5.

[0071] In some embodiments, the external dataset is associated with the user's profile. In some embodiments, the profile is associated with a username / password. In some embodiments, the user's profile is stored along with web browser information and can be accessed via web browser preferences. For example, the stored user profile may include browsing history, a set of bookmarks, browser themes, avatars, security options, etc. Based on such a profile, the external dataset manager 250 may select a particular external dataset. For example, if the user has a bookmark that designates a grocery store website, the external dataset manager 250 can choose to extract data from that website.

[0072] In some embodiments, the external dataset is the search results returned from a search engine index and can search a document repository and / or a knowledge graph (entity / fact repository). In some embodiments, the processing circuit may submit a query (e.g., a second query or a search engine query) including terms and / or phrases from the request and / or from structured data elements to the search engine. For example, a user viewing a web page with a recipe may provide a request to "highlight ingredients that contain gluten." After identifying the structured elements of the ingredients within the web page elements, the system can send queries for each element, such as "milk has gluten," "cornmeal has gluten," etc., to the search engine. The system can receive the search results and analyze the results to determine whether the ingredient needs to be modified (highlighted). Some of the search results may include the result of one box or short response for such a query of fact. Thus, the external dataset can be selected based on its presence in the search engine index.

[0073] In some embodiments, the external data set includes data transmitted from other open browser tabs. Such data is technically external data because it is external to the browser tab of interest.

[0074] FIG. 4 is a diagram showing an exemplary external data set 400 for changing web elements of a web page. The exemplary external data set 400 represents an organizational chart stored in a human resources database accessible to the user. For example, an organization to which the user belongs (e.g., an employee of a company or corporation) may send the organizational chart to the user in a format readable by the query manager 230 and / or the external data set manager 250.

[0075] The exemplary external data set 400 is represented in FIG. 4 as a tree where arrow connectors represent reporting relationships. The reference numbers shown in FIG. 4 are arranged according to the manager. Thus, the member represented by the database element 410 is the manager of the member represented by the database element 420. The member represented by the database element 420 is the manager of the members represented by the database elements 430, 432, and 434. The member represented by the database element 430 is the manager of the members represented by the database element 440 and the database element 442. The member represented by the database element 432 is the manager of the member represented by the database element 450.

[0076] Consider the case where the member represented by database element 432 is the user who forms the above query or request to "display participants with the same manager as me". Query manager 230 has determined, via an NLP algorithm, that the intent of the query indicates the human resources database as an external dataset. Query manager 230 determines the participants within the participant list who report to the same participant as the member represented by database element 432. As shown in FIG. 4, those members are represented in external dataset 400 as database elements 430 and database element 434. The members represented by database elements 430 and 434 can be found by matching the identifiers from structured data elements 312(1 - 4) with the identifiers within external dataset 400. For example, query manager 230 can determine that the value of the email field within structured data elements 312(1 - 4) corresponds to the value of the email address within external dataset 400. From those matches, query manager 230 can identify the structured data elements pointed to by the same structured data elements pointed to by database element 420. Thus, query manager 230 returns a modification command that references the email identifiers of database element 430 and database element 434.

[0077] FIG. 5 is a diagram showing an exemplary external dataset 500 for changing web elements of a web page. In this case, the external dataset is related to a food item list within a database that stores information about food items in a grocery store and has database elements shown to the user. An example of the use of external dataset 500 in this context includes queries from a cooking application such as "highlight the food ingredients sold in nearby grocery stores".

[0078] As shown in FIG. 5, the external dataset 500 shows database elements arranged in a tree format. In contrast to the external dataset 400 shown in FIG. 4, the database elements have different information at each level. At the topmost level of the database element 510, there is information identifying a nearby grocery store, and the fields include a store identifier ("Ggrocery STORE G") and an address ("123 Main St.00000"). Note that the address can be in any format and is not limited to the format shown in FIG. 5 and postal codes.

[0079] As shown in FIG. 5, the database element 510 is the parent database element for the database elements 520 and 530, and the database elements 520 and 530 represent the departments of the grocery store. The database element 520 has a field value of "produce". The database element 530 has a field value of "bread-making materials", and the data is arranged as database elements 532, 534, 536, and 538. The database elements 532, 534, 536, and 538 represent various products sold in the "bread-making materials" department. The database element 532 represents the material wheat flour, and an additional field identifies that the wheat flour has the brand "Brand A" and specifies that it is not for sale. The database element 534 has the field values "wheat flour", "Brand B", and "for sale". The database element 536 has the field values "sugar", "Brand C", and "out of stock", and the database element 538 has the field values "vanilla", "Brand D", and "for sale".

[0080] Note that the external data sets 400 and 500 discussed with respect to FIGS. 4 and 5 may each contain personal data. FIGS. 4 and 5 may each contain private data. For example, the organization chart of the external data set 400 may be confidential data for the organization. Thus, such private data used for the changes to the user interface described herein should not be shared with other entities, namely, the web server providing the web conferencing application or the restaurant ordering application. Accordingly, the external data set manager 250 is configured to prevent the transmission of private data to other entities, whether electronic or not.

[0081] In some embodiments, changes to a web page, web page element, or structured data element are not applied based on the identification of external data sets such as external data sets 400 and 500. In such embodiments, the changes are based on data local to the user device 106 (FIG. 1B). For example, in this case, the processor 120 (FIG. 1B) may be configured to use the elements of the displayed web page to determine that the criteria are met. In other words, if the processor 120 can determine that the criteria are met by inspecting the structured elements of the web page, the requested changes can be applied without referring to an external data set. In one example, the web page can be information within a search results page based on a user query. The processor 120 may be able to identify search result(s) that meet the criteria and apply the changes to the result(s). For example, if the web page is a search result for "nearby restaurants", the user may submit a query such as "highlight inexpensive restaurants", and the processor 120 can identify the web page elements representing the cost associated with each search result (e.g., from the labels of DIV tags or SPAN tags) and use this information to determine which search results to apply the highlight to. In another example, the external data set may be the result of a query to a search engine. For example, the processor 120 can obtain data from the search engine result page corresponding to the element and use the data from the search engine result page to identify the structured data element(s) that meet the criteria. For example, instead of accessing the external data set 500, the user device may generate such data from a search query for nearby grocery stores with ingredients and a search query for whether such ingredients are being sold. In some embodiments, the processor 120 may store the search results in local storage, i.e., directly on the user device or on a disk connected via a wireless network. In some embodiments, the processor 120 can assign metadata to the search results based on the search query.In some embodiments, the metadata includes a markup language, such as HTML DIV or SPAN tags derived from a search query. Based on the markup language in the metadata, the processor 120 can determine that the reference includes an element corresponding to the data.

[0082] Returning to FIG. 2, the web browser manager 240 is configured to execute a web browser whose user interface (e.g., user interface 111) receives changes according to a user query. Recall that the web browser is application software executed on a processing circuit (e.g., processing circuit 120) for accessing a web page. When a user requests a web page from a website, the web browser retrieves web browser data 241 from the web server and displays content within the page in a browser window on the user's device in response to a resource locator input by the user. The web browser can display multiple web pages within a single user interface. Each such web page is activated using a tab, e.g., a tab at the top of the user interface. A user of the web browser can activate another web page by clicking on the tab of the other web page on the user interface.

[0083] The web browser data 241 represents data used and / or displayed by a web browser that can be accessed for the purpose of changing the user interface of the web browser. As shown in FIG. 2, the web browser data includes web page element data 242(1··M), where M is the number of web page elements managed by the web browser and displayed on the web page. The web browser data also includes semantic data 244.

[0084] Each of the web page elements 242(1··M), for example 242(1), is an object displayed by a web browser in a user interface within a browser window based on data received from a web page server (for example, web page server 105). For example, in the above web conferencing application, the web page elements 150 and 160 include visual information regarding the participants and names of the participants. In a browser application, some web page elements include application data (for example, identifiers, avatars, and photos of conference participants) and other forms of data including semantic information (for example, first name, last name, email address). In some embodiments, the web page elements 150 and 160 are received from the web page server in the form of data and instructions for display.

[0085] As shown in FIG. 2, each web page element, for example web page element 242(1), includes structured data element data 243(1,1··N_1), where N_1 is the number of structured data elements within web page element 242(1). (N_k is the number of structured data elements within web page element 242(1,k).) Each of the structured data elements, for example, structured data element data 243(1,1), is a data object stored in a data structure and organized according to a data model. In some embodiments, the structured data elements follow a tabular form having relationships between different rows and columns. In this way, different rows and columns can be sorted or searched according to a query. For example, FIG. 3 shows various structured data elements that can be stored in a web page and whose field values can be compared with the field values of an external data set.

[0086] Semantic data 244 represents semantic information embedded in web pages that includes semantic metadata. To enable encoding semantic information in data, technologies such as the Resource Description Framework (RDF) and the Web Ontology Language (OWL) are used. These technologies are used to formally represent metadata. For example, an ontology can describe concepts, relationships between entities, and categories of things. These built-in semantics provide important advantages such as reasoning about data and operating on heterogeneous data sources.

[0087] For example, in the above example of the web conferencing application, the data representing the list of conference participants (e.g., the participant list 306 in FIG. 3) can include metadata that identifies this as a list of users, and all the data in the table can be clearly identified, such as by name, surname, LDAP / email address, etc. As a result, the data within the web page is in a machine-readable form via the semantic metadata of the semantic data 244, enabling improved technology understanding. This also enables improved technology for connecting that data within one web page to other data, thereby allowing changes to the user interface, such as the user interface 111.

[0088] The external data set manager 250 is configured to identify and obtain external data set data 251 from various sources. In some embodiments, the external data set manager 250 is configured to access information sources from local sources, i.e., connected or internal hard drives, or remote sources via the network interface 222, before any user query is received by the query manager 230. In some embodiments, the external data set manager 250 assigns a unique identifier to each external data set so accessed and obtained.

[0089] In some embodiments, the external dataset manager 250 is configured to access external dataset data 251 in response to the query manager 230 receiving query data, such as query 108 of FIG. 1B. For example, in the above example of the web conferencing application, in response to the query manager 230 receiving and processing a query via an NLP algorithm using semantic data 244, the external dataset manager 250 is configured to obtain data via a network interface 222 and a remote source based on the semantic data 244.

[0090] The external dataset data 251 includes information related to the user's query, such as the organizational chart of FIG. 4 or the food product data of FIG. 5, each of which is modeled as a tree structure. As shown in FIG. 2, the external dataset data 251 can include public data 252 and private data 253. The public data 252 is data that can be viewed by other users. The private data 253 is data that can be viewed only by authorized individuals or groups. In some embodiments, the external dataset manager 250 is configured to prevent the private data 253 from being sent to the remote server hosting the browser application.

[0091] The web page element change manager 260 is configured to perform changes to the web page elements of a web page according to web page element change data 271, as determined from query data received by the query manager 230. In some embodiments, the web page element change manager 260 is implemented as part of a browser or as a browser extension capable of changing the context of the web page when the web page is displayed. In some embodiments, the web page element change manager 260 can change the appearance of the pixels that display the web page elements to be changed. In some embodiments, the web page element change manager 260 may generate other effects on the web page elements, for example, increase the brightness of the pixels within the area to be changed, or generate animation effects such as blinking or simulating waves on the area. In some embodiments, the web page element change manager 260 changes the web page such that the web page elements are rearranged, for example, rearranging or repositioning the rows of a table or list elements. In some embodiments, the web page element change manager 260 removes an object from the display as a change.

[0092] In some embodiments, the page element change manager 260 selects specific changes from a library or repository of changes based on a query, as represented by the library data 272 in FIG. 2. Such a library may contain instructions for applying the changes to a web page. This may include markup tags or labels corresponding to specific changes. The library (repository) can also improve the performance of the changes when the query is unclear about the changes. For example, if the query states "show participants with the same manager as me", the processing circuit can randomly select a change from the library. In some embodiments, the improved technique includes default changes when the query renders the changes as uncertain. In some embodiments, the library data 272 includes descriptors of changes with respect to a markup language. In some embodiments, the library data 272 includes instructions for applying an action to a web element or user interface. For example, if it is determined that the change to a web page element is an italicization action from the library, the web page element change manager 260 can add a markup (e.g., HTML) tag or label to the web page to change the element. In some embodiments, the changes may be applied to several related web page elements. For example, if a cell within a row of a table meets the criteria, the change may be applied to the entire row. Similarly, individual search results may be changed if the information of 1 item within the result meets the criteria. In some embodiments, the structured elements determined to meet the criteria are changed.

[0093] The components of user device 106 (e.g., modules, processing unit 224) can be configured to operate based on one or more platforms (e.g., one or more similar or different platforms) that can include one or more types of hardware, software, firmware, operating system, runtime library, and / or others. In some embodiments, the components of processing circuit 120 can be configured to operate within a cluster of devices (e.g., a server farm). In such embodiments, the functions and processing of the components of processing circuit 120 can be distributed across several devices in the cluster of devices.

[0094] The components of processing circuit 120 can be or include any type of hardware and / or software configured to process attributes. In some embodiments, one or more portions of the components shown for processing circuit 120 in FIG. 2 can be or include hardware-based modules (e.g., digital signal processor (DSP), field programmable gate array (FPGA), memory), firmware modules, and / or software-based modules (e.g., modules of computer code, sets of computer-executable computer-readable instructions). For example, in some embodiments, one or more portions of the components of processing circuit 120 can be or include software modules configured for execution by at least one processor (not shown). In some embodiments, the functionality of the components can be included in different modules and / or different components than those shown in FIG. 2, including combining the functionality shown as two components into a single component.

[0095] Although not shown, in some embodiments, components (or multiple parts thereof) of the processing circuit 120 may be configured to operate, for example, within a data center (such as a cloud computing environment), a computer system, one or more servers / host devices, and / or the like. In some embodiments, components (or multiple parts thereof) of the processing circuit 120 can be configured to operate within a network. Thus, components (or multiple parts thereof) of the processing circuit 120 can be configured to function within various types of network environments including one or more devices and / or one or more server devices. For example, the network can be or can include a local area network (LAN), a wide area network (WAN), etc. The network can be or can include a wireless network and / or a wired network implemented using, for example, a gateway device, a bridge, a switch, etc. The network can include one or more segments and / or can include portions based on various protocols such as the Internet Protocol (IP) and / or a proprietary protocol. The network can include at least a portion of the Internet.

[0096] In some embodiments, one or more of the components of the processing circuit 120 can be or can include a processor configured to process instructions stored in a memory. For example, the query manager 230 (and / or a part thereof), the web browser manager 240 (and / or a part thereof), the external dataset manager 250 (and / or a part thereof), and the web page element manager 260 (and / or a part thereof).

[0097] In some embodiments, memory 226 can be any type of memory such as random access memory, disk drive memory, flash memory, and / or the like. In some embodiments, memory 226 can be implemented as a plurality of memory components (e.g., a plurality of RAM components or disk drive memory) associated with components of processing circuit 120. In some embodiments, memory 226 can be database memory. In some embodiments, memory 226 can be or can include non-local memory. For example, memory 226 can be or can include memory shared by a plurality of devices (not shown). In some embodiments, memory 226 can be associated with a server device (not shown) within a network and can be configured to provide services to components of processing circuit 120. As shown in FIG. 2, memory 226 is configured to store various data including query data 231, web browser data 241, and external dataset data 251.

[0098] FIG. 6 is a flowchart showing an exemplary method 600 according to the improved technique above for generating a modified user interface to be displayed by a web browser based on a query from a user. Method 600 may be executed by the software constructs described in connection with FIG. 2. These software constructs are present within memory 226 of processing circuit 120 and are executed by a set of processing units 224.

[0099] At 602, query manager 230 receives query data (e.g., query 108) representing a request from a user of the web browser to change the presentation of a web page (e.g., web page element 112) displayed by the web browser in a user interface (e.g., user interface 111). The web page element includes a plurality of structured data elements (e.g., structured data element 115), and the request identifies a criterion (e.g., criterion data 234). For example, query manager 230 may receive a user query such as "Highlight the ingredients sold at nearby grocery stores." Using an NLP algorithm, query manager 230 determines that the user intends to look up grocery price data and identify the items for sale in the ingredient list.

[0100] At 604, query manager 230 analyzes the web page to identify structured data elements (e.g., structured data element 118) that meet the criterion among the plurality of structured data elements. For example, the criterion determined from the query in the above example is "ingredient identifier corresponding to (matching) the food identifier in the food list of the food items on sale." Next, query manager 230 identifies and accesses an external data set managed by external data set manager 250 and performs a comparison. If there is an ingredient identifier that matches the food identifier in the food list, query manager 230 identifies the structured data element having that ingredient identifier as a field value. After applying NLP and semantic markup information to the query, query manager 230 identifies the criterion as "Product identifier == food identifier AND food identifier.isOnSale = TRUE" ("Prtoduct identifiers == foodIdentifiers AND foodIdentifiers.isOnSale = TRUE.”). NLP is <div itemscope itemtype=""https: / / schema.org / SomeProducts”" itemid=""#product”">Identify this standard as such using semantic markup information such as this.

[0101] At 606, the web page change manager 260 applies a change (e.g., change data 236) to a structured data element that meets the criteria to generate a changed presentation of the web page (e.g., changed web page element 160) displayed by the web browser. For example, when a structured data element and a change that meet the criteria are identified, the web page element change manager 260 performs a predetermined change (e.g., an emphasis operation), such as highlighting, checking, bolding, italicizing, blinking, sorting, etc., on the displayed representation of the structured data element.

[0102] In some embodiments, the page change manager 260 applies a change selected from a library of changes. For example, such a library can enable selection of changes such as highlighting, checking, bolding, italicizing, blinking, etc., as listed above.

[0103] FIG. 7 shows examples of a general computing device 700 and a general mobile computing device 750 that can be used with the techniques described herein. The computer device 700 is an exemplary configuration of the user device 106 of FIG. 1A and / or the processing circuit 120 of FIG. 2.

[0104] As shown in FIG. 7, computing device 700 is intended to represent various forms of digital computers, such as a laptop, desktop, workstation, personal digital assistant, server, blade server, mainframe, and other appropriate computers. Computing device 750 is intended to represent various forms of mobile devices, examples of which include personal digital assistants, cellular phones, smartphones, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are intended only as examples and are not intended to limit the embodiments of the invention described and / or claimed herein.

[0105] Computing device 700 includes a processor 702, a memory 704, a storage device 706, a high-speed interface 708 that connects to memory 704 and high-speed expansion port 710, and a low-speed interface 712 that connects to low-speed bus 714 and storage device 706. Each of the components 702, 704, 706, 708, 710, and 712 is interconnected using various buses and may be attached to a common motherboard or in other manners as required. Processor 702 processes instructions for execution within computing device 700, including instructions stored in memory 704 or storage device 706, and can display graphical information for a GUI on an external input / output device such as display 716 connected to high-speed interface 708. In other embodiments, multiple processors and / or multiple buses may be used, along with multiple memories and multiple types of memory, as required. Also, multiple computing devices 700 may be connected and each device may provide a plurality of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

[0106] Memory 704 stores information within computing device 700. In one embodiment, memory 704 is a volatile memory unit(s). In other embodiments, memory 704 is a non-volatile memory unit(s). Memory 704 may also be other forms of computer-readable media, such as a magnetic disk or an optical disk.

[0107] Storage device 706 can provide large-capacity storage to computing device 700. In one embodiment, storage device 706 is a computer-readable medium, or can include such a medium, such as a floppy (registered trademark) disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configured device, including numerous devices. A computer program product can be tangibly embodied in an information carrier. The computer program product may also include instructions that execute one or more of the methods as described above when executed. The information carrier is a computer-readable medium or a machine-readable medium, examples of which include memory 704, storage device 706, or memory on processor 702.

[0108] The high-speed controller 708 manages the bandwidth-intensive operations of the computing device 700, and the low-speed controller 712 manages the low-bandwidth-intensive operations. Such an assignment of functions is merely an example. In one embodiment, the high-speed controller 708 is coupled to a high-speed expansion port 710 that can receive the memory 704, the display 716 (e.g., via a graphics processor or accelerator), and various expansion cards (not shown). In this embodiment, the low-speed controller 712 is coupled to the storage device 706 and the low-speed expansion port 714. The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth®, Ethernet®, wireless Ethernet®), may be coupled to one or more input / output devices such as a keyboard, a pointing device, a scanner, etc., or may be coupled to a network device such as a switch or a router, e.g., via a network adapter.

[0109] The computing device 700 may be implemented in a plurality of various forms as shown in the figure. For example, it may be implemented as a standard server 720, or may be implemented multiple times in a group of such servers. Also, the computing device may be implemented as part of a rack server system 724. Additionally, the computing device may be implemented in a personal computer such as a laptop computer 722. Alternatively, the components of the computing device 700 may be combined with other components within a mobile device (not shown) such as the device 750. Each of such devices may include one or more of the computing devices 700, 750, and the entire system may be composed of a plurality of computing devices 700, 750 that communicate with each other.

[0110] Computing device 750 includes, among other components, a processor 752, a memory 764, input / output devices such as a display 754, a communication interface 766, and a transceiver 768. Additionally, the device 750 may be provided with a storage device such as a microdrive or other device to provide additional storage. Each component 750, 752, 764, 754, 766, and 768 is interconnected using various buses, and some components may be attached to a common motherboard or in other manners as required.

[0111] Processor 752 can execute instructions within computing device 750, including instructions stored in memory 764. The processor may be implemented as a chipset of chips including multiple separate analog and digital processors. The processor may, for example, provide coordination of other components of device 750, examples of which include control of the user interface, applications executed by device 750, and wireless communication by device 750.

[0112] Processor 752 may communicate with the user via a control interface 758 and a display interface 756 coupled to display 754. Display 754 may be, for example, a thin film transistor liquid crystal display (TFT LCD) or an organic light emitting diode (OLED) display, or other suitable display technology. Display interface 756 may include appropriate circuitry to drive display 754 to present graphic and other information to the user. Control interface 758 may receive commands from the user and convert them for submission to processor 752. Additionally, an external interface 760 may be provided to communicate with processor 752 to enable short-range communication between device 750 and other devices. External interface 760 may, for example, provide wired communication in some implementations, or wireless communication in other implementations, and may use multiple interfaces.

[0113] Memory 764 stores information within computing device 750. Memory 764 can be implemented as one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. An expansion memory 774 may also be provided and connected to device 750 via expansion interface 772. Expansion interface 772 may include, for example, a single in-line memory module (SIMM) card interface. Such expansion memory 774 may provide additional storage space for device 750, or may store applications or other information for device 750. Specifically, expansion memory 774 may include instructions for performing or supplementing the above-described processes, and may also include secure information. Thus, for example, expansion memory 774 may be provided as a security module for device 750 and programmed with instructions that enable secure use of device 750. Additionally, secure applications may be provided via the SIMM card along with additional information, such as placing identification information in a non-hackable manner on the SIMM card.

[0114] The memory may include, for example, flash memory and / or NVRAM memory, as described below. In one embodiment, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods as described above. The information carrier is a computer-readable medium or a machine-readable medium, examples of which include memory 764, expansion memory 774, or memory on processor 752. This information carrier may be received, for example, via transceiver 768 or external interface 762.

[0115] Device 750 may perform wireless communication via communication interface 766, which may include digital signal processing circuitry if necessary. Communication interface 766 may provide communication in various modes or protocols, examples of which include, among others, GSM (registered trademark) voice calls, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA (registered trademark), CDMA2000 or GPRS. Such communication may be performed, for example, via radio frequency transceiver 768. Additionally, short-range communication may be performed, such as using Bluetooth (registered trademark), WiFi (registered trademark), or other such transceivers (not shown). Additionally, a global positioning system (GPS) receiver module 770 may provide additional wireless data related to navigation and location to device 750, which may be used as needed by applications running on device 750.

[0116] Device 750 may also perform voice communication using audio codec 760. This audio codec 760 may receive voice information from the user and convert it into usable digital information. Similarly, audio codec 760 may generate sounds audible to the user, such as through a speaker (e.g., within the handset of device 750). Such sounds may include sounds from a voice telephone call, recorded sounds (e.g., voice messages, music files, etc.), or sounds generated by an application running on device 750.

[0117] Computing device 750 may be implemented in a plurality of various forms, as shown in the figures. For example, it may be implemented as a mobile phone 780. Additionally, it may be implemented as part of a smartphone 782, a personal digital assistant, or other similar mobile device.

[0118] FIG. 8 shows an example of a general computer device 800 that may be the processing circuit 120 of FIG. 2, which may be used with the techniques described herein. Computing device 800 is intended to represent various exemplary forms of large-scale data processing devices, such as servers, blade servers, data centers, mainframes, and other large-scale computing devices. Computing device 800 may be a distributed system, which may optionally include network-connected storage nodes and have multiple processors interconnected by one or more communication networks. The components shown herein, their connections and relationships, and their functions are intended to be illustrative only and not intended to limit the embodiments of the invention described and / or claimed herein.

[0119] The distributed computing system 800 may include any number of computing devices 880. Computing devices 880 may include servers or rack servers, mainframes, etc. that communicate via a local or wide area network, a dedicated optical link, a modem, a bridge, a router, a switch, a wired or wireless network, etc.

[0120] In some embodiments, each computing device may include a plurality of racks. For example, computing device 880a includes a plurality of racks 858a-858n. Each rack may include one or more processors such as processors 852a-852n and 862a-862n. Processors may include data processors, network-connected storage devices, and other computer control devices. In some embodiments, one processor may operate as a master processor and control scheduling tasks and data distribution tasks. Processors may be interconnected via one or more rack switches 858, and one or more racks may be connected via a switch 878. Switch 878 may handle communication between multiple connected computing devices 800.

[0121] Each rack may include memories such as memories 854 and 864, and storage such as 856 and 866. Storage 856 and 866 may provide mass storage and may include volatile or non-volatile storage, such as a network-connected disk, a floppy (registered trademark) disk, a hard disk, an optical disk, a tape, a flash memory, or other similar solid-state memory device, or an array of devices including a device within a storage area network or other configuration. Storage 856 or 866 may be shared among multiple processors, multiple racks, or multiple computing devices and may include a computer-readable medium storing instructions executable by one or more of the processors. Memories 854 and 864 can include, for example, volatile memory unit(s), non-volatile memory unit(s), and / or other forms of computer-readable media such as magnetic or optical disks, flash memory, cache, random access memory (RAM), read only memory (ROM), and combinations thereof. Memories such as memory 854 may also be shared among processors 852a - 852n. A data structure such as an index may be stored, for example, across storage 856 and memory 854. Computing device 800 may include other components (not shown) such as a controller, a bus, input / output devices, a communication module, etc.

[0122] An entire system, such as electronic environment 100, may be composed of multiple computing devices 800 that communicate with each other. For example, device 880a can communicate with devices 880b, 880c, and 880d, and together they may be collectively referred to as electronic environment 100. As another example, processing circuit 120 of FIG. 2 may include one or more computing devices 800. Some of the computing devices may be located geographically close to each other, while others may be located geographically far apart. The layout of system 800 is only an example, and the system can take other layouts or configurations.

[0123] Example 1: The improved technique can include receiving query data from a user of a web browser that represents a request to change the presentation of a web page displayed by the web browser, where the web page includes a plurality of structured data elements and the request identifies criteria. The improved technique can also include analyzing the web page to identify structured data elements that meet the criteria among the plurality of structured data elements. The improved technique can further include applying a change to the structured data elements that meet the criteria to generate a changed presentation of the web page to be displayed by the web browser.

[0124] Example 2: According to an exemplary embodiment of Example 1, the request includes a natural language query, the plurality of structured data elements of the web page elements include semantic markup information, and analyzing the web page includes determining terms within the natural language query based on the semantic markup information.

[0125] Example 3: According to an exemplary embodiment described in Example 1 or 2, the change is obtained from a library of available changes for the user. In some embodiments, the library of changes includes highlight display changes.

[0126] Example 4: According to an exemplary embodiment of Examples 1 - 3, the improved technique includes determining that the criteria include elements corresponding to information in an external data set, obtaining data corresponding to the elements from the external data set, and using the data from the external data set to identify structured data elements that meet the criteria.

[0127] Example 5: According to an exemplary embodiment of Example 4, the external data set is obtained from a server different from the server that sends the web page elements to the web browser.

[0128] Example 6: According to the exemplary embodiments of Examples 4 to 5, the external dataset is associated with the user's profile.

[0129] Example 7: According to the exemplary embodiments of Examples 4 to 6, the external dataset is associated with the search engine index.

[0130] Example 8: According to the exemplary embodiments of Examples 4 to 7, the external dataset includes the open browser tabs within the web browser.

[0131] Example 9: According to the exemplary embodiments of Examples 4 to 8, the external dataset includes private data resulting from private data resources.

[0132] Example 10: According to the exemplary embodiments of Examples 4 to 9, the web page elements are received from the web page server, and the improved technology includes preventing private data from being sent to the web page server.

[0133] Example 11: According to the exemplary embodiments of Examples 1 to 10, analyzing a web page includes identifying a plurality of structured data elements from the web page elements by identifying table elements, list elements, semantic markup elements, or accessibility data, and comparing the plurality of structured data elements to a reference.

[0134] Example 12: According to the exemplary embodiment of Example 11, the request further identifies an emphasis operation, and the modified user interface applies the emphasis operation to the structured data elements that meet the reference.

[0135] Example 13: According to the exemplary embodiments of Examples 11 to 12, the emphasis operation is mapped to one of a plurality of predefined emphasis operations.

[0136] Example 14: According to the exemplary embodiments of Examples 11 to 13, a web page element includes a table having table elements, a structured data element that meets the criteria is within the area of the table, and generating a modified user interface further includes changing the appearance of the area.

[0137] Example 15: According to the exemplary embodiments of Examples 11 to 14, a web page element is associated with a document model including a set of nodes representing a plurality of structured data elements of the web page element, and changing a structured data element includes editing a node of the document model represented by a structured data element that meets the criteria to change the appearance of the structured data element.

[0138] Example 16: According to the exemplary embodiment of Example 4, obtaining data from an external data set corresponding to an element includes identifying a uniform resource locator (URL) associated with the element and accessing the data from a web resource determined from the URL.

[0139] Example 17: According to the exemplary embodiments of Examples 4 and 16, the user belongs to an organization, and obtaining data from an external data set corresponding to an element includes receiving data from a resource related to the organization at a time point before receiving query data.

[0140] The various embodiments of the systems and techniques described herein can be implemented in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can be either special or general purpose, and can include one or more programmable processors, at least one input device, and at least one output device coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, and executable and / or interpretable in a programmable system including the one or more programmable processors, at least one input device, and at least one output device. Embodiments can include those in one or more computer programs executable and / or interpretable in a programmable system including the one or more programmable processors, at least one input device, and at least one output device coupled to receive data and instructions from, and to transmit data and instructions to, a storage system.

[0141] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages, and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor including a machine-readable medium that receives the machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0142] To provide interaction with a user, the systems and techniques described herein are implemented on a computer having a display device (such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (such as a mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user. For example, the feedback provided to the user can be any form of sensory feedback (such as visual feedback, auditory feedback, or tactile feedback), and the input from the user can be received in any form including acoustic, speech language, or tactile input.

[0143] The systems and techniques described herein can be implemented in a computing system that includes back-end components (such as a data server), or includes middleware components (such as an application server), or includes front-end components (such as a client computer having a graphical user interface or a web browser through which a user can interact with an embodiment of the systems and techniques described herein), or in a computing device that includes any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by digital data communication in any form or medium (such as a communication network). Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), and the Internet.

[0144] A computing system can include a client and a server. The client and the server are generally far apart from each other and usually interact through a communication network. The relationship between the client and the server is caused by computer programs that operate on respective computers and have a client-server relationship with each other.

[0145] Numerous embodiments have been described. Nevertheless, it will be understood that various modifications can be made without departing from the spirit and scope of this specification.

[0146] Also, when an element is referred to as being on, connected to, electrically connected to, coupled to, or electrically coupled to another element, it can be directly on, directly connected to, or directly coupled to the other element, or one or more intervening elements may be present. In contrast, when an element is referred to as being directly on, directly connected to, or directly coupled to another element, no intervening element is present. Throughout the description of the embodiments for carrying out the invention, even if the terms directly on, directly connected to, or directly coupled to are not used, an element shown as being directly on, directly connected to, or directly coupled to can be referred to as such. The claims of this application can be amended to list the exemplary relationships described in the specification or shown in the drawings.

[0147] As described herein, while certain features of the described embodiments have been illustrated, numerous modifications, substitutions, alterations, and equivalents will occur to those skilled in the art. Thus, it is to be understood that the appended claims are intended to cover all such modifications and changes that fall within the scope of the embodiments. They are presented by way of example only and not of limitation, and it should be understood that various changes in form and detail may be made. Any part of the apparatus and / or method described herein can be combined in any combination, except mutually exclusive combinations. The embodiments described herein may include various combinations and / or sub-combinations of the functions, components, and / or features of the different embodiments described.

[0148] Furthermore, the logic flows shown in the figures do not require the particular order shown, i.e., the sequential order, to achieve the desired result. Additionally, other steps may be provided to the flows described, or steps may be eliminated from the flows described, and other components may be added to the systems described, or other components may be removed from the systems described. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A computer-based method comprising receiving, from a user of a web browser, query data representing a request to change the presentation of a web page displayed by the web browser, the web page including a plurality of structured data elements, the request identifying a criterion, and the computer-based method further comprising: analyzing the web page to identify structured data elements among the plurality of structured data elements that meet the criterion; and applying a change to the structured data elements that meet the criterion to generate a changed presentation of the web page displayed by the web browser. A computer-based method as claimed in claim 1, wherein the request includes a natural language query and the plurality of structured data elements of the web page include semantic markup information, and analyzing the web page includes determining terms within the natural language query that correspond to the semantic markup information.

2. The computer-based method as claimed in claim 1, further comprising analyzing the request to determine a change identifier and determining, based on the change identifier, instructions for applying the change to the structured data elements.

3. The computer-based method as claimed in claim 3, wherein the structured data elements include a table, the change identifier represents a sorting order operation, and applying the change to the structured data elements that meet the criterion includes rearranging table rows of the table to generate a changed sorting order of the table.

4. The computer-based method as claimed in claim 3, wherein the change identifier corresponds to one of a plurality of predefined highlighting operations.

5. The computer-based method as claimed in claim 1, wherein the web page is associated with a document model including a set of nodes representing the plurality of structured data elements of the web page, and applying the change to the structured data elements includes editing nodes of the document model represented by the structured data elements that meet the criterion to change an appearance of the structured data elements.

6.

7. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ The computer-based method according to claim 1, wherein the change is retrieved from a change repository, and the repository contains descriptors of the change with respect to the markup language. **Claim 8** The computer-based method according to claim 7, wherein the change repository further includes instructions for applying the change to the structured data element that meets the criteria, the instructions including adding the markup language to the web page, and the markup language includes HTML markup tags. **Claim 9** Determining that the criteria includes elements corresponding to information in an external dataset; Obtaining data corresponding to the elements from the external dataset; Using the data from the external dataset to identify the structured data element that meets the criteria; The computer-based method according to claim 1, further including. **Claim 10** The computer-based method according to claim 9, wherein the external dataset is obtained from a server different from the server that sends the web page to the web browser. **Claim 11** The computer-based method according to claim 9, wherein the external dataset is associated with any of the user's profile, search results, or open browser tabs within the web browser. **Claim 12** The web page is received from a web page server, The external dataset includes private data resulting from a private data resource, The computer-based method according to claim 9, wherein the analyzing and the applying are performed without sending the private data to the web page server. **Claim 13** The computer-based method further includes Determining that the criteria includes elements corresponding to a dataset local to a user device configured to execute the web browser, the dataset including metadata for identifying the dataset, the determining being based on the metadata, and the computer-based method further includes Obtaining data corresponding to the elements from the dataset; Using the data from the dataset to identify the structured data element that meets the criteria; The computer-based method according to claim 1, including. **Claim 14** The computer-based method according to claim 13, wherein the data set includes search engine results based on queries submitted to the search engine.

15. A computer program product comprising a non-transitory storage medium, the computer program product including code that, when executed by a processing circuit, causes the processing circuit to perform a method, the method comprising: Receiving query data from a user of a web browser, the query data representing a request to change the presentation of a web page displayed by the web browser, the web page including a plurality of structured data elements, the request identifying criteria, and the method further comprising: Analyzing the web page to identify structured data elements among the plurality of structured data elements that meet the criteria; Changing the structured data elements that meet the criteria to generate a changed presentation of the web page displayed by the web browser; A computer program product comprising the above.

16. Analyzing the web page comprises: Identifying the plurality of structured data elements of the web page by identifying table elements, list elements, accessibility elements, or semantic markup elements; Comparing the plurality of structured data elements with the criteria; The computer program product according to claim 15, comprising the above.

17. The request further identifies one of a plurality of predefined highlighting operations, and changing the web page comprises applying the highlighting operation to the structured data elements that meet the criteria. The computer program product according to claim 15.

18. The method comprises: The computer program product according to claim 17, further comprising performing a classification operation on the query data to identify the one highlighting operation from the plurality of predefined highlighting operations.

19. The web page includes a table with table elements, and the structured data elements that meet the criteria are within the area of the table. Generating the changed presentation of the web page comprises: The computer program product according to claim 15, further comprising changing the appearance of the area.

20. An electronic device, the electronic device comprising: A memory; A processing circuit coupled to the memory; comprising, wherein the processing circuit is configured to receive query data representing a request from a user of a web browser to change the presentation of a web page displayed by the web browser, the web page including a plurality of structured data elements, the request identifying a criterion, and the processing circuit further analyze the web page to identify structured data elements among the plurality of structured data elements that meet the criterion; and modify the structured data elements that meet the criterion to generate a modified presentation of the web page to be displayed by the web browser. An electronic device configured to perform the above. **Claim 21** The processing circuit is further configured to determine that the criterion includes an element corresponding to information in an external data set; obtain data corresponding to the element from the external data set; and use the data from the external data set to identify the structured data elements that meet the criterion. The electronic device according to claim 20, configured to perform the above. **Claim 22** The processing circuit is further configured to identify a Uniform Resource Locator (URL) associated with the element; and access the data with a web resource determined from the URL. The electronic device according to claim 21, configured to perform the above. **Claim 23** The user belongs to an organization, and the processing circuit is further configured to receive the data from a resource associated with the organization prior to receiving the query data. **Claim 24** The processing circuit is further configured to send to the second user device change data including a structured data element identifier that identifies the structured data element that meets the criterion and a change applied to the structured data element corresponding to the structured data element identifier on a web page executed on a browser of the second user device. **Claim 25** The processing circuit configured to modify the structured data elements that meet the criterion is further configured to remove the highlighting of the structured data elements among the plurality of structured data elements that do not meet the criterion. **Claim 26** The processing circuit configured to modify the structured data element that meets the criterion is further configured to provide an interface that enables the user to select from two or more modifications, receive the selected modification from the interface, use the selected modification to modify the presentation of the web page displayed by the web browser, The electronic device according to claim 20, which is configured to perform the above.

Images