Remembering content associated with a resource locator

Abstract

The method may be performed by a browser and may include presenting at least a portion of a resource address and presenting a rendered web page in a browser content window, the rendered web page being associated with the resource address, presenting an annotation area, the annotation area being outside the browser content window, receiving an annotation in the annotation area in association with a user account, and storing the annotation in association with the user account and the resource address.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Application No. 63 / 266,291, filed December 31, 2021, and entitled "BROWSER-BASED ANNOTATION INTERFACE FOR INTERNET DOCUMENTS," the disclosure of which is incorporated herein by reference. [Background technology]

[0002] Websites provide useful information or functionality to users, and many users use the Internet to research products, places, companies, services, etc. The web pages associated with a website are typically stored on a remote server. Summary of the Invention

[0003] Web sites provide useful information or functionality to users, and many users use the Internet to research products, places, companies, services, and the like. Web pages associated with the web sites are typically stored on remote servers. Research across various sites and various web pages within those sites can be complex, often requiring integration, comparison, and decision making. Users can view the web pages within a browser running on a local computing device. In embodiments, browser-integrated tools are provided to assist with these and other tasks. The tools include novel user interfaces and novel browser features, such as storing information associated with the web pages and / or web sites. Stored information associated with the web pages and / or web sites is also associated with a user account. Stored information associated with the web pages and / or web sites is generated by a user who owns and / or manages the user account. Stored information is not accessible by the server that maintains the web site with which the user-generated information is associated. The novel user interface, called the annotation user interface, assists users in organizing, understanding, and interacting with Internet-based documents without coordination and / or communication with the servers and / or entities that maintain the web sites. The annotation user interface provides a highly integrated information generation (e.g., note-taking) experience that solves the decision-making funnel problems. In particular, the annotation user interface can include intelligence that enables a user to extract the most important details from a website or collection of websites, compare information across websites, and return to the original content (web page) without having to bookmark the content and without having to provide the information to the servers and / or entities that maintain the websites. The annotation user interface can extract rich content, provide note and / or image suggestions, and / or organize annotations around the annotation target.Annotation targets can represent web pages (source documents), topics, tab groups, etc. The annotation user interface allows users to integrate user-created annotations with suggested content, track annotations across websites (e.g., by associating annotations with topics or tab groups), and compare notes across sites and web pages. The user interface can be viewed in multiple ways relative to a user account and can be presented at various levels of detail (e.g., in a side panel, across multiple tables, or as a standalone document). Integrating the annotation interface into a browser allows the browser to separate the annotation data from the rendering process. Such separation allows the interface to appear integrated, but the annotations themselves are not exposed to the rendering process, and information about the annotations cannot be obtained by the server and / or administrators maintaining the website.

[0004] A method may be performed in a browser, the method including presenting at least a portion of a resource address and presenting a rendered web page in a browser content window, the rendered web page associated with the resource address, the method further including presenting an annotation area, the annotation area outside the browser content window, the method further including receiving an annotation in the annotation area in association with a user account, and storing the annotation in association with the user account and the resource address.

[0005] A non-transitory computer-readable storage medium including instructions stored in the non-transitory computer-readable storage medium and configured, when executed by at least one processor, to cause a computing system to present at least a portion of a resource address and to present a rendered web page in a browser content window, the rendered web page being associated with the resource address, the instructions, when executed by the at least one processor, to cause the computing system to present an annotation area, the annotation area being outside the browser content window, and the instructions, when executed by the at least one processor, to further cause the computing system to receive an annotation in the annotation area in association with a user account, and store the annotation in association with the user account and the resource address.

[0006] The computing system may include at least one processor and a non-transitory computer-readable storage medium that may include instructions stored on the non-transitory computer-readable storage medium and that, when executed by the at least one processor, cause the computing system to present at least a portion of the resource address and present a rendered web page in a browser content window, the rendered web page being associated with the resource address, the instructions, when executed by the at least one processor, cause the computing system to present an annotation region, the annotation region being outside of the browser content window, and the instructions, when executed by the at least one processor, are configured to cause the computing system to receive an annotation in the annotation region in association with a user account and to store the annotation in association with the user account and the resource address.

[0007] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and the drawings. [Brief description of the drawings]

[0008] [Figure 1A] 1 illustrates an example technique for displaying a browser annotation interface, according to some aspects. [Figure 1B] 1 illustrates an example technique for displaying a browser annotation interface, according to some aspects. [Figure 1C] 1 illustrates an example technique for displaying a browser annotation interface, according to some aspects. [Figure 1D] 1 illustrates an example technique for displaying a browser annotation interface, according to some aspects. [Figure 2A] FIG. 2 illustrates another example technique for displaying a browser annotation interface, according to one aspect. [Figure 2B] FIG. 2 illustrates another example technique for displaying a browser annotation interface, according to one aspect. [Diagram 3] 3A and 3B are diagrams illustrating an example browser annotation interface for a computing device with limited display area, according to one aspect. [Figure 4A] 1 illustrates an example browser annotation interface with suggested annotation content, according to some aspects. [Figure 4B] 1 illustrates an example browser annotation interface with suggested annotation content, according to some aspects. [Figure 4C] 1 illustrates an example browser annotation interface with suggested annotation content, according to some aspects. [Figure 4D] 1 illustrates an example browser annotation interface with suggested annotation content, according to some aspects. [Figure 4E]1 illustrates an example browser annotation interface with suggested annotation content, according to some aspects. [Figure 4F] 1 illustrates an example browser annotation interface with suggested annotation content, according to some aspects. [Figure 5A] 1 illustrates an example browser annotation interface with grouped annotation content, according to some aspects. [Figure 5B] 1 illustrates an example browser annotation interface with grouped annotation content, according to some aspects. [Figure 5C] 1 illustrates an example browser annotation interface with grouped annotation content, according to some aspects. [Figure 5D] 1 illustrates an example browser annotation interface with grouped annotation content, according to some aspects. [Figure 6] 1 illustrates an exemplary annotation rendering, according to some aspects. [Figure 7] FIG. 1 illustrates a system for providing a browser annotation interface, according to one aspect. [Figure 8] 1 is a block diagram of components for a browser application that implements a system for providing a browser annotation interface, according to one aspect. [Figure 9] 4 is a timing diagram illustrating an example operation for loading an annotated Internet document, according to one aspect. [Figure 10] 4 is a timing diagram illustrating an example operation for navigating using annotations, according to one aspect. [Figure 11] FIG. 1 illustrates an example of an exemplary computing device, according to one aspect. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Like reference numbers refer to like elements. Embodiments relate to providing a browser-integrated annotation system for annotating and storing annotations on Internet-based documents, such as websites, web pages, and the like. Integrating the annotation system into the browser can improve usability, facilitate integration across web pages, tabs, and search sessions, and allow annotations to be associated with and stored in a user account. The browser-based annotation system can also facilitate organization and navigation and provide automated assistance, such as automatically generated annotation suggestions. In some embodiments, such suggestions may be based on machine learning models, document structure, and / or historical information (e.g., analysis of anonymized search records). Because the annotation interface is integrated into the browser, embodiments can provide a variety of techniques for displaying the annotation interface and can serve as a navigation aid.

[0010] FIG. 1A illustrates an exemplary browser user interface (UI) 100 according to implementations described herein. Generally, the UI 100 is generated and rendered by a browser application (e.g., browser application 718 of FIG. 7) executing on an operating system (e.g., operating system 710) of a computing device. The UI 100 may include a tab strip 108 associated with the browser application. In this example, the UI 100 includes five browser tabs 105, although any number of browser tabs or tab groups may be opened by the browser application. Each browser tab, e.g., browser tab 105, may be associated with web content (web page) that is displayed in a corresponding browser content window 120 of the browser application. As used herein, a web page refers to any content that is at least partially generated / served by / from a remote server hosting a website. A web page may be associated with a resource address, such as a uniform resource locator (URL) or a uniform resource indicator (URI). A web page may include hypertext markup language (HTML) content. A browser application may request a web page from a remote server hosting a website based on the resource address (e.g., URL). A browser application can request a web page, for example, by sending a HyperText Transfer Protocol (HTTP) GET request to a remote server hosting a website. A web page can refer to a progressive web application as well as a traditional web page. Additionally, as used herein, a source document refers to the web page currently being rendered in the browser content window 120 and any other data (metadata) used to render the document (e.g., markup and scripts that are not actually displayed).

[0011] The UI 100 may also include an address bar area 110. An address of a web page displayed in the browser content window 120 may be displayed in the address bar area 110 (e.g., address input area 115). The address of the web page may be represented as a resource address. The resource address included in the address bar area 110 may be associated with the web page displayed in the browser content window 120. In some examples, the address bar area 110 may display at least a portion of the resource address. Other controls, icons, and the like may be included in the address bar area 110. For example, the address bar area 110 may include a user icon 125 (shown in FIG. 1B). The user icon 125 may provide an indication of a user account associated with the browser session. The user icon 125 may be an image, text, or other representation representing the user account. In some examples, the user account is associated with a user, and the user may be required to enter a unique username, as well as a passcode associated with the username, to join a browser session associated with the user account. In some examples, no login is required to join a browser session, and the user account may be considered to be associated with the local computing device on which the browser application is running. Other controls / icons included in the address bar area 110 may include a forward control, a back control, a refresh control, a home control, an advanced features control, and / or a bookmarks control. The address bar area 110 may be controlled by and / or associated with a browser (e.g., browser application 718). While the content of the address bar area 110 may be controlled by the browser application, the content of the browser content window 120 is controlled by the web page being displayed and / or the provider of the web page.Thus, the appearance and user interface elements displayed in the address bar area 110 are controlled by the browser application.

[0012] The UI 100 includes an annotation user interface shown as annotation area 130. In the example of FIG. 1A, annotation (or note) area 130 is located within or implemented as a browser sidebar. The browser sidebar may be a contiguous area, for example a combined area representing address bar area 110 and annotation area 130. Thus, the content displayed (rendered) in the sidebar is controlled (determined) locally by the browser and not by a server hosting the website displayed (rendered) in the browser content window 120. Although the sidebar is shown as a contiguous area with address bar area 110, it may be a split window, a pop-up window, or some other part of the browser UI 100. Annotations entered in annotation area 130 may be stored locally on the computing device on which the browser is running or may be stored remotely in association with a user account. Annotations entered in annotation area 130 may not be accessible by the server hosting the website. Notification of the annotation is not sent to the server hosting the website. In other words, the annotations are associated locally with the website without notifying or interacting with a server associated with the website. Storing annotations locally on a computing device and / or remotely in association with a user account allows a user to securely generate and store annotations without the knowledge of the server hosting the website and / or the website administrators.

[0013] Annotations and / or notes may include content entered by a user into annotation area 130. In some examples, annotations may include text entered by a user into annotation area 130. In some examples, annotations may include changes to the appearance and / or formatting of a portion of text included in a rendered web page. Changes to the appearance and / or formatting of a portion of text may include highlighting or typeface changes. In some examples, annotations may include resource addresses (e.g., URLs) of other web pages. In some examples, annotations may include files uploaded by a user.

[0014] Because the annotation area 130 may be integrated as part of the browser user interface 100 (e.g., address bar area 110), the annotation area 130 cannot be spoofed (imitated), for example, by a third party or an owner of the content displayed in the browser content window 120. Because the annotation area 130 is part of the application of the browser application, the integration of the annotation area 130 is difficult to imitate. In some implementations, the annotation area 130 may be displayed (invoked, rendered) in response to a menu option (e.g., menu option 122). The menu option 122 may be an option provided in a menu that is displayed in response to a command to display the tab menu (e.g., right-clicking on the tab 105).

[0015] In the example of FIG. 1A, tab 105 does not currently have any notes. Thus, annotation area 130 includes new note UI element 132. New note UI element 132 can be any UI element into which a user can input text (e.g., keyboard, speech-to-text, stylus, etc.). Annotation area 130 can include a display of the tab / website with which the note is associated. Annotation area 130 can include controls for adding an annotation (note) to be associated with the webpage currently displayed in browser content window 120. The controls are configured to perform a predefined task / operation when actuated by the user. The controls can include text control 133 for adding a text-based note. Text control 133 can open a new note UI element, such as UI element 132. The controls can include to-do control 134. The controls can include image control 135. Image control 135 allows a user to upload, take, and / or search for an image to add as a note. The controls can include link control 136. Link control 136 allows a user to add a link (such as an address, such as a URL) to another Internet-based document. The controls may include auto-suggestion control 137, which may cause a browser application to analyze the annotation target (e.g., both the visible and invisible content of browser content window 120) and provide one or more suggestions for information items to include in the annotation target's notes. Examples of auto-suggestions are shown in Figures 4A-4F.

[0016] In some implementations, annotation area 130 may be displayed as shown in FIG. 1B. In FIG. 1B, annotation area 130 is displayed in response to menu option 123. Menu option 123 may be provided in a menu displayed in response to a command to display a selected text menu. Thus, for example, a user may select content displayed in browser content window 120 and right-click within the selection to display a menu including menu option 123. In FIGS. 1B, 2A, 2B, 3A, and 3B, text with a shaded background may be considered highlighted and is an example of a text change. The highlighted text may be considered content selected by the user when the user right-clicked a mouse or otherwise entered a human interface device (HID) to display a menu including menu option 123. Selecting menu option 123 may cause annotation area 130 of FIG. 1B to be rendered by a browser application, such as by a rendering process of the browser application. In the example of Figure 1B, the annotation target already has one existing note (which may be considered a first annotation associated with the web page), so annotation area 130 includes note 138 (which may be considered a second annotation associated with the web page). Additionally, in response to selection of menu option 123, an additional new note UI element 132 is included in annotation area 130. The technique for displaying annotation area 130 shown in Figure 1B may be combined with the technique for displaying annotation area 130 shown in Figure 1A. In other words, a browser may be configured to use either or both techniques.

[0017] In some implementations, such as the example shown in FIG. 1C, the annotation area 130 may include a control 139 configured to expand the tab strip 108. The expanded tab strip 108' may display a preview area 140 for each tab. Each tab may be associated with a different annotation target, i.e., a resource such as a web page and / or a website (e.g., a first resource, a second resource, and a third resource). The preview area 140 may include any existing notes for the annotation target associated with the tab, but may lack controls (e.g., controls 133-137). Thus, the preview area 140 is a scaled-down version of the annotation area 130 for the tab. The preview area 140 may display portions of multiple rendered web pages and associated annotations, such as a portion of a first rendered web page and a first annotation associated with the first rendered web page, a portion of a second rendered web page and a second annotation associated with the second rendered web page, and a portion of a third rendered web page and a third annotation associated with the third rendered web page. Thus, the expanded tab strip 108' provides a novel way to quickly and conveniently compare notes associated with various annotation targets, e.g., between URLs, web pages and / or websites, tabs, tab groups, etc. The expanded tab strip 108' reduces the area of ​​the browser content window 120, and some previously viewable content may be off-screen and / or hidden from the browser. The technique for displaying the preview area 140 shown in FIG. 1C can be combined with the technique for displaying the annotation area 130 shown in FIG. 1A and FIG. 1B. In other words, the browser can be configured to use one, two, or all three techniques.

[0018] In some examples (not shown), the browser can display a preview area 140 for a tab in response to a user performing a pre-selection action for the tab, such as hovering a cursor over the tab or sliding a finger over the tab. The browser can display a web page associated with the tab in response to a user performing a second type of selection for the tab, such as clicking or tapping on the tab. In some implementations, the preview area 140 can include a selectable control that, when selected, causes the browser to display a web page associated with a resource address (e.g., a URL) and simultaneously display the annotation area 130.

[0019] In some implementations, such as the example shown in FIG. 1D, the annotation area (e.g., annotation area 130) may be displayed in response to menu option 124. Menu option 124 may be provided in a menu that is displayed in response to selection of a control that displays a browser menu, for example as shown in FIG. 1D. Although not shown in FIG. 1D, selection of menu option 124 may cause annotation area 130 of FIG. 1A or 1B to be rendered (depending on whether the source document of the annotation target (e.g., current tab) has existing notes (e.g., 1B) or does not have existing notes (e.g., 1A)). The technique for displaying annotation area 130 shown in FIG. 1D may be combined with the techniques described in FIGS. 1A-1C. In other words, a browser may be configured to use one, two, three, or all four techniques.

[0020] In some implementations, the annotation area 130 may be displayed as shown in FIG. 2A. In the example of FIG. 2A, the control 126 is provided by a browser application. In some implementations, the control 126 may be provided in the address bar area 110. In some implementations (not shown), the control 126 may be located before the address entry area 115. In some implementations, the control 126 may be located in the address entry area 115. In some implementations (not shown), the control 126 may be located after the address entry area 115. The control 126 may be provided when an annotation target (e.g., a document currently displayed in the browser content window 120) has a note (e.g., an annotation) associated therewith. In some implementations, the control 126 may be provided regardless of whether there is a note and / or annotation associated with the annotation target. In some implementations, the appearance of the control 126 may change depending on whether there is a note and / or annotation associated with the annotation target. For example, the text, color, shading of the control 126, or an icon included in the control 126 may differ depending on whether there is a note and / or annotation associated with the annotation target. In some implementations, the text may indicate the number (quantity) of existing notes and / or annotations for the annotation target. In the example shown in Figure 2A, the shaded background represents a text highlight of the text portion associated with the notes and / or annotations for the annotation target.

[0021] When a user activates (selects) the control 126, such as by tapping and / or clicking the control 126, the browser can render an annotation region 130 including a note regarding the annotation target, as shown in FIG. 2B. Existing notes and / or annotations (e.g., 238a, 238b, etc.) can be configured to navigate within a source document associated with the annotation target. For example, when a user activates (selects, clicks on) an existing note 238b and / or annotation, the browser can scroll the source document to a location associated with the existing note 238b and / or annotation. The interface of FIG. 2B also illustrates that the content of the notes and / or annotations can have a full highlight and / or a partial highlight. For example, the content corresponding to the selected note 238a and / or annotation is fully highlighted (shown with a shaded background), while the text associated with the other notes and / or annotations is partially highlighted (no shading is shown in FIG. 2B). If the user selects note 238b, the browser may fully highlight the content associated with note 238b (with a shaded background) and change the content associated with note 238a (if it is still displayed in the browser content window 120) to a partial highlight.

[0022] 3A and 3B show an exemplary preview area 340 for a device with limited display area, such as a mobile device, tablet, wearable, etc. Preview area 340 is similar to preview area 140 of FIG. 1C. Thus, preview area 340 displays first notes and / or annotations associated with a source document, which may extend into annotation area 330, as shown in FIG. 3B. Annotation area 330 is similar to and / or includes similar features and / or functionality as annotation area 130 of FIGS. 1A-2B.

[0023] 4A-4F illustrate an exemplary browser annotation interface with suggested annotation content, according to some aspects. In the example of FIG. 4A, a user navigates to a web page that is now at least partially displayed in browser content window 120 and opens annotation region 130. In some implementations, annotation region 130 can include a description 431 of the annotation target, in this case the title of the web page displayed in browser content window 120. In some implementations, the title is stored as a data element with the annotation record of the annotation (e.g., the annotation record stored in local notes storage 726 of FIG. 7). In the example of FIG. 4A, a user activates auto-suggest control 137. When auto-suggest control 137 is activated, annotation region 130 is updated with a suggested text-based content element 432. The suggested text-based content element 432 is a content element selected from the content of the annotation target. Auto-suggestion can be based on the schema markup of the annotation target. Auto-suggestion can be based on historical search records. Auto-suggestion can be based on the document structure of the annotation target. The user may select one or more of the suggested text-based content elements 432. In the example of FIG. 4B, the user has selected four of the suggested text-based content elements 432, as indicated by the dotted background of the four suggested text-based content elements 432 selected by the user. The dotted background may represent a color change or highlighting of the suggested text-based content elements 432. FIG. 4B shows four notes 433 added in response to the selection of the four suggested text-based content elements 432.

[0024] In the example of FIG. 4C, the user scrolls to another portion of the annotation target and changes the appearance of text 421. In this example, the change in appearance of text 421 includes highlighting text 421. Selecting auto-suggest control 137 may display auto-suggested element 434, including selected highlighted text 421 and image 422. In response to the user selecting the proposed text of auto-suggested element 434, FIG. 4D shows that the browser application adds a new note 435 and / or annotation to annotation region 130. This new note 435 has the same annotation target as note 433 and / or annotation. Although not shown in the figure, selecting one of notes 433 and / or annotations causes the annotation target to scroll, such that content corresponding to the selected one of notes 433 and / or annotations is displayed in browser content window 120, as shown, for example, in FIG. 4E.

[0025] In the example of FIGURE 4E, upon selection of auto-suggestion control 137, suggested image elements 436 may be added to the annotation of the annotation target (e.g., displayed in browser content window 120). The user may select one or more of the images, which may be added to the annotation of the annotation target as image notes 437 and / or image annotations, as shown in FIGURE 4F. In some implementations, suggested text-based content elements 432 and auto-suggested elements 434 are selectable elements. These selectable elements may be configured to add suggested annotation content as annotations, as described above, upon selection.

[0026] 5A-5D illustrate an exemplary browser annotation interface with grouped annotation content, according to some aspects. In the example of FIG. 5A, a user has two tabs 105 (e.g., 105a and 105b) open. The user has annotation area 130 open, which displays two notes for an annotation target of a website corresponding to tab 105b, as can be seen from annotation target description 531 included in annotation area 130. The user can select tab 105a, which causes the browser application to display the interface of FIG. 5B. In response to the selection of tab 105a, the browser displays annotations in annotation area 130 for the annotation target that corresponds to the resource address (e.g., URL) associated with the selected tab 105a. The annotation target now corresponds to the website associated with tab 105a, and annotation area 130 indicates this change by changing description 531. In the example of FIG. 5B, the current annotation target does not have any existing annotations and / or notes, but the user can create an annotation and / or note that is an image included in the web page by dragging image 521 to new note UI element 132, as indicated by the dashed arrow in FIG. 5B. In addition to being a text-based UI element (e.g., configured to receive text), new note UI element 132 can be configured as a drop location. Thus, when content, such as an image, copied text, or some other content, is dragged (in whole or in part) onto new note UI element 132 included in annotation area 130 and released, the browser application can consider the drag and / or drop of the content as a request to store the content as a new annotation and / or note in the annotation target (e.g., the annotation target corresponding to description 531). FIG. 5C illustrates the result of this request, e.g., the browser application has added the new image as an annotation in the annotation target. If the annotation is an image, the image is downloaded to the user's device, allowing the user to access the note even without a network connection.

[0027] In some implementations, as shown in FIG. 5D, a user may create a tab group by grouping two tabs 105a and 105b into a tab group 505. The user may have given the tab group 505 a description of "Mountain climbing." Tab groups may also be annotation targets. In some implementations, the creation of the tab group 505 may cause the browser application to group existing annotations for annotation targets contained in the tab group and also associate the tab group as an annotation target. This may be done directly (e.g., adding an index entry that associates annotation data with an identifier for the tab group) or indirectly (e.g., adding an index entry that associates a web page with the tab group). The annotation region 130 may be updated to reflect the new association, e.g., description 531 of the annotation region 130 may be changed to a description of the tab group 505. Annotation targets that represent a group of annotation targets may include additional information, such as the number of annotation targets (two tabs in this example) and / or the number of notes and / or annotations for the group of annotation targets. In some implementations, annotation region 130 can add a group control 535, which allows a user to select between annotation targets in a group upon actuation of control 535. The annotation region 130 of an annotation target that represents multiple annotation targets (e.g., tab group 505) can display annotations associated with one or more of the annotation targets. In the example of FIG. 5D, annotation data for both tab 105a and tab 105b is displayed (rendered) in annotation region 130.

[0028] FIG. 6 illustrates exemplary variations of annotation rendering, according to some aspects. These exemplary variations may be applied to any note (annotation) displayed in annotation area 130 of any of the previous figures. The shaded background represents a highlighted area and / or highlighting of the text. Exemplary annotation 602 includes text, but does not include a title. Exemplary annotation 604 includes a title ("Note Title") and text. Exemplary annotation 606 includes text and a date / time stamp when annotation 606 was entered by a user. Exemplary annotation 608 includes text with a modified appearance, such as highlighted text ("Highlight Hint Text") and additional text. Exemplary annotation 610 includes text with a modified appearance, such as highlighted text ("Highlight Hint Text"), a title, and additional text. Exemplary annotation 612 includes text with a modified appearance, such as highlighted text ("Highlight Hint Text"), a title, and additional text. Exemplary annotation 614 includes a title. Exemplary annotation 616 includes a title and text. Exemplary annotation 618 includes a date and time (timestamp) and text. Exemplary annotation 620 includes a title and includes text that has a changed appearance, such as highlighted text. Exemplary annotation 622 includes a title, unaltered text, and text that has a changed appearance, such as highlighted text. Exemplary annotation 624 includes a date and time (timestamp), text, and text that has a changed appearance, such as highlighted text.

[0029] 7 illustrates a system 700 for providing a browser annotation interface, according to one embodiment. The browser annotation interface may include the browser user interface 100 and / or the annotation area 130.

[0030] The system 700 may include a computing system 702. The computing system 702 may include a local computing device and / or a client device, such as, but not limited to, a smartphone, a tablet, a mobile computing system, a laptop or notebook computer, or a desktop computer.

[0031] Computing system 702 may include memory 704. Memory 704 may include a non-transitory computer-readable storage medium that includes instructions that, when stored on the non-transitory computer-readable storage medium and executed by at least one processor (e.g., CPU / GPU 732 included in computing system 702), are configured to cause computing system 702 to perform any combination of the methods, functions, and / or techniques described herein.

[0032] The memory 704 may include instructions for executing an operating system (O / S) 710. The operating system 710 may control which applications have access to various components of the computing system 702, such as the memory 704, the output device(s) 724, the input device(s) 730, the CPU / GPU 732, and / or the camera(s) 734.

[0033] The operating system 710 enables multiple application(s) 712, such as a browser application 718, to run on the computing system 702. The browser application 718 can perform any combination of the methods, functions, and / or techniques described herein. For each application 712, the operating system 710 can maintain and / or store application information 715 and / or application layers that enable the application 712 to communicate with other applications running on the computing system 702. For each application 712, the operating system 710 can maintain and / or store session data 717 that enables sessions between application processes to be initiated, terminated, and managed.

[0034] The memory 704 may include a local note storage 726. The local note storage 726 may store notes and / or annotations, such as those described above and shown in the previous figures. The local note storage 726 may store notes and / or annotations in annotation records. In some implementations, the annotation records may include a user identifier. In some implementations, the location (directory) where the annotation records are stored reflects an association with a particular user. Each annotation record may include a metadata portion, a body portion, and a target portion. The metadata portion of the annotation record may include an identifier, a creation date, a modification date, etc. The identifier allows the browser to uniquely identify each annotation record. The ID may be unique per user. The ID may be unique per client device. In some implementations, the body of the annotation record may include a note type indicating the type of annotation. In some implementations, the metadata portion of the annotation record may include a note type. The note type may identify the type of annotation, metadata such as text, image, rich text, handwritten, audio, video, location, geolocation, and / or file. The target portion includes an annotation target identifier. The annotation target identifier identifies the resource (e.g., a web page, a website, a document, a file, a tab group, etc.) to which the annotation applies. The annotation target identifier can be a resource address, such as a URL. The annotation target identifier can be a group identifier, such as an identifier for a tab group. The body portion of the annotation record can include text. In some examples, the body portion of the annotation record can include an address, such as a URL, that links to and / or identifies an external information source and / or data source. The text represents the text selected / provided by the user as the annotation. The body portion of the annotation record can include an image identifier. For annotations that are tied to a specific element of the annotation target (e.g., specific text, specific image, etc.), the annotation target element can be included in the metadata portion of the annotation record.In some examples, an annotation target element may be included in the annotation record. The annotation target element may be a selector (e.g., a CSS selector) that identifies specific content within the annotation target. In some examples, the annotation target element may include a portion of text contained in a web page. In some examples, the annotation target may identify a file (e.g., an image) and / or the annotation target identifier may include a filename and / or a hash value associated with the file. The hash value may be determined based on any hash algorithm, such as Message Digest 5 (MD5) or Secure Hash Algorithm (SHA).

[0035] The local note storage 726 can store notes and / or annotations in association with addresses and / or groups, as well as in association with a user account, such as a user account associated with a user who created the note and / or annotation. In some examples, a note and / or annotation can be considered to be stored in association with a user account if the note and / or annotation is stored in a directory associated with the user, such as a local directory accessible to the user. The note and / or annotation can be stored locally in memory 704 of the computing system 702 on which the browser application is running. The note and / or annotation can be stored locally in long-term storage, such as non-volatile memory that is accessible after the computing system is powered off and back on. In some examples, the rendered web page is not stored in long-term memory, and the note and / or annotation can be retrieved and displayed when the rendered web page is retrieved again from the service(s) / website(s) 770 and rendered in the browser application 718. Storing the note and / or annotation locally can prevent the server hosting the website associated with the note and / or annotation from recognizing the note and / or annotation.

[0036] The memory 704 can store preferences 728. The preferences 728 can be received from a user and can include, but are not limited to, web browser appearance, favorite sites or bookmarks, and / or the first application to launch.

[0037] The computing system 702 may include one or more camera(s) 734. The camera(s) 734 can capture images, such as images of users for videoconferencing.

[0038] Computing system 702 may include at least one processor, such as a central processing unit (CPU) and / or a graphics processing unit (GPU) 732. The CPU / GPU can execute instructions, such as instructions stored in memory 704, to perform any combination of the methods, functions, and / or techniques described herein.

[0039] The computing system 702 may include one or more input device(s) 730. The input device(s) may include, but are not limited to, a keyboard, a mouse, a microphone, or a human interface device such as a touch screen.

[0040] The computing system 702 may include one or more output device(s) 724. The output device(s) may include, but are not limited to, output devices such as a display, speakers, or a printer.

[0041] The input device(s) 730 and / or output device(s) 724 may also include interfaces for communicating with other computing devices. The interface(s) for communicating with other computing devices may receive and / or transmit data to a server and / or may receive input from a user and provide output to a user. The input and output functions of the interface(s) may be combined into a single node or may be split into separate input and output nodes. An input / output node may include one or more wired or wireless interfaces for communicating with other computing devices.

[0042] The system 700 may include a server computing system 750. The server computing system 750 may store and / or maintain user accounts 760. The user accounts 760 stored and / or maintained by the server computing system 750 may be associated with and / or owned by users and may be associated with a unique identifier, such as an email address. The server computing system 750 may store notes 756 and / or annotations in association with each user account. The notes 756 and / or annotations may have the functionality described above and may be stored in association with the user account 760 and an annotation target, such as a URL. The server computing system 750 may prevent servers that host and / or maintain resources associated with the annotation target, such as server(s) / website(s) 770, from accessing the notes 756 and / or annotations.

[0043] The server computing system 750 can store session data 762 in association with the user account 760, allowing for initiation, termination, and management of sessions while a user is logged in using the user's user account 760. The server computing system 750 can store a profile 766 in association with the user account 760, the profile including browsing patterns, bookmarks, and / or appearance preferences.

[0044] Server computing system 750 can include at least one processor(s) 752. The at least one processor(s) 752 can execute instructions, such as instructions stored in at least one memory device(s) 754, to cause server computing system 750 to perform any combination of the methods, functions, and / or techniques described herein.

[0045] The server computing system 750 can include at least one memory device(s) 754. The at least one memory device(s) 754 can include a non-transitory computer-readable storage medium. The at least one memory device(s) 754 can store data and instructions that, when executed by at least one processor, such as the processor(s) 752, are configured to cause the server computing system 750 to perform any combination of the methods, functions, and / or techniques described herein. Thus, in any embodiment described herein (even if not explicitly mentioned in connection with a particular embodiment), the software (e.g., processing modules, stored instructions) and / or hardware (e.g., processors, memory devices, etc.) associated with or included in the server computing system 750 can be configured to perform any combination of the methods, functions, and / or techniques described herein, either alone or in combination with the computing system 702.

[0046] The system 700 can include service(s) / website(s) 770. The service(s) / website(s) 770 can be maintained and / or hosted by one or more remote servers. The service(s) / website(s) 770 can be addressed by a resource address (e.g., a URL) and / or store files addressed by the resource addresses (e.g., a URL). The computing system 702, the server computing system 750, and / or the service(s) / website(s) 770 can communicate with each other over a network 740, such as the Internet.

[0047] FIG. 8 is a block diagram of components for a browser application implementing a system for providing a browser annotation interface, such as the browser application 718 of FIG. 7, according to one aspect. The browser application 718 includes an annotation UI coordinator 805, which is a platform-specific component of a browser process 850. In other words, the annotation UI coordinator 805 is configured for a particular operating system and device type. The annotation UI coordinator 805 may have two main components: a note creation UI 810 and a note rendering UI 815. The note creation UI 810 includes components configured to add new notes and / or annotations 837 and associate the new notes and / or annotations 837 with annotation targets 836 (e.g., source documents or tab groups). Components of an exemplary note creation UI 810 include 132-137 of FIG. 1A and / or suggested elements 432, 434, 436 of FIG. 4A-4F, etc. The note rendering UI 815 is configured to render an annotation area (e.g., annotation area 130) and / or a preview area (e.g., preview area 140) in which the note and / or annotation 837 is displayed. In some implementations, the annotation UI coordinator 805 may also include a note sharing coordinator 820. The annotation UI coordinator 805 may communicate with the annotation renderer coordinator 835, for example, sending an identifier of the annotation 837 to the annotation renderer coordinator 835. Upon receiving the identifier of the annotation 837 from the annotation UI coordinator 805, the annotation renderer coordinator 835 may identify the content of the annotation source associated with the annotation identifier and display (render) that portion of the annotation target 836 in the browser content window 120. In some implementations, messages may be passed through the annotation browser coordinator 825.

[0048] The browser application 718 can include an annotation browser coordinator 825. The annotation browser coordinator 825 is a cross-platform component (i.e., an operating system independent component) of the browser process 850 that contains the business logic of the application. The annotation browser coordinator 825 also maintains the annotation data in a private storage (e.g., local to the computing device on which the browser application 718 is running). In some examples, the annotation data itself may be excluded from the renderer process 855. In other words, the annotation data is rendered in a rendering process different from the renderer process 855, and the annotation data is not provided to the annotation renderer coordinator 835 or another renderer process. This separation of the rendering process provides security and guarantees privacy. For example, the renderer process 855 (e.g., the annotation renderer coordinator 835) can receive information about the annotation's ID and the annotation target 836 (e.g., what to highlight fully, what to highlight partially, etc.). This mitigates the risk of a misused renderer leaking sensitive user data within the annotation itself. In some implementations, upon a user request, the annotations 837 may be synchronized with the user account, e.g., so that if a user uses multiple computing devices, the user's annotation data can be accessed from any computing device when the user is logged into the user account. The user can stop synchronization at any time.

[0049] In some implementations, the locally stored annotation data (which may represent the annotation 837) may have a format based on the Web Annotations W3C Recommended Web Annotation Data Model. The annotation data may consist of three main parts: a metadata part, a body part, and a target part. The metadata part may include information such as ID, creation date, modification date, etc. The ID may be used to communicate about a particular annotation between the annotation UI coordinator 805, the annotation renderer coordinator 835, and the annotation browser coordinator 825. The body part of the annotation 837 includes the text, images, rich text, links, etc. of the annotation 837. The body part of the annotation 837 is not shared with the rendering process. The annotation target 836 (target part) defines what the annotation 837 is attached to. The annotation target 836 may be a web page element, a web page, a tab group, a topic (user-defined group), etc. In some implementations, in addition to a web page as the annotation target 836, an annotation target element is associated with the annotation 837. The annotation target element may be identified via a selector. The selector can be a text fragment selector and / or a fragment finder 838. The selector and / or the fragment finder 838 can be a CSS selector. Because the annotations 837 can be linked to elements of the web page that may change, it is possible that the annotation renderer coordinator 835 may not be able to find a particular annotation target element. In some implementations, these "orphaned" notes and / or annotations may continue to be displayed in the annotation area 130 of the web page. In some implementations, such orphaned annotations may be displayed with an additional message indicating that the target element cannot be found. If the web page associated with a uniform resource locator (URL) is inaccessible, the annotation 837 for that annotation target 836 may be displayed as an orphaned note and / or annotation in the annotation area 130. For example, a notes menu option, such as menu option 124, allows a user to select notes and / or annotations for annotation targets 836 that are not associated with an open tab.

[0050] The annotation browser coordinator 825 can retrieve associated annotations 837 for the current page from local storage, as described in more detail in Figure 9. The annotation browser coordinator 825 can interact with the annotation UI coordinator 805 and the annotation renderer coordinator 835 to properly display the notes and / or annotations. The annotation browser coordinator 825 can determine which tasks to send to the annotation UI coordinator 805 and which tasks to send to the annotation renderer coordinator 835.

[0051] The annotation browser coordinator 825 can include a private note coordinator 830. The private note coordinator 830 can be configured to store and retrieve annotation data (notes and / or annotations) from a local storage 831. The local storage 831 can be included in the memory 704 of the computing system 702 on which the browser is executing. The private note coordinator 830 can associate annotations with annotation targets (e.g., source documents or tab groups) to which the notes are tied. The note coordinator 830 can index the annotation data so that it can be retrieved, for example, by annotation target, tab group, topic, etc. The note coordinator 830 can also be configured to sync 832 the annotation data so that it can be accessed on other devices if the user is signed in and synced.

[0052] The annotation browser coordinator 825 can include an inline note coordinator 833. The inline note coordinator 833 can receive, store, retrieve, and / or present notes and / or annotations within annotation targets 836 and / or within web pages other than within the annotation region 130.

[0053] The annotation browser coordinator 825 can include a remote note coordinator 834. The remote note coordinator 834 can receive, store, retrieve, and / or present notes and / or annotations that are stored remotely in association with a user account, such as notes 745 on the server computing system 750.

[0054] The annotation renderer coordinator 835 can be configured to search for and highlight selectors to which various annotations are attached. To do this, the annotation renderer coordinator 835 can hold a subset of annotation data, such as ID and selector information. The annotation browser coordinator 825 can coordinate the annotation renderer coordinator 835 and annotation UI coordinator 805 to smoothly link annotations 837 to their target selectors using annotation IDs (GUIDs) and synchronize operations. Specifically, the annotation UI coordinator 805 is responsible for annotation bodies (which hold all the data) and the annotation renderer coordinator 835 is responsible for annotation targets 836 (what gets rendered in the browser content window 120). In some implementations, the annotations 837 may not be rendered next to the highlighted text. In some implementations, the annotations 837 may be rendered next to the highlighted text. In some implementations, when a user clicks on an annotation 837 in the side panel (e.g., annotation area 130), the annotation browser coordinator 825 can cause the annotation renderer coordinator 835 to increase the highlight of the annotation target 836 and scroll the highlight of the annotation target 836 into view. However, when a user clicks on a lightly highlighted (partially highlighted) annotation target 836 (e.g., in the browser content window 120), the annotation browser coordinator 825 can cause the annotation UI coordinator 805 to focus on the associated annotation in the side panel (annotation area 130). To accomplish this, messages between processes can include the ID of the selected annotation so that the other process can activate the appropriate annotation.

[0055] 9 is a timing diagram illustrating exemplary operations for loading an annotated Internet document, according to one aspect. A user 900 selects a web page by entering a resource address, such as a URL, into the address entry field 115 and / or selecting or opening a hyperlink displayed in the browser content window 120 (902). The browser application 718 responds to the selection of the web page by performing a URL load operation (904), (906). The URL load operation (904), (906) may include generating and sending a request, such as an HTTP GET request, for the web page identified and / or addressed by the URL to a server, such as a server maintaining the service(s) / website(s) 770. The browser application 718, in conjunction with the renderer 835, may render and display the web page received from the server.

[0056] After performing a URL load operation (904), (906), such as receiving and rendering a web page, the browser application 718 may retrieve (908) an annotation 837 related to and / or associated with the URL. The browser application 718 may retrieve the annotation 837, for example, from local note storage 726, 831, or from notes 756 stored on a remote server computing system in association with a user account associated with the user 900. The browser application 718 may send (910) a current page annotation target 836, which may include a portion of the rendered web page, to the renderer 835.

[0057] The browser application 718 may display the annotation region 130 (e.g., a side panel or an overlay) with the annotation 837 obtained at (908). In instances where the annotation 837 includes a textual change in a webpage, such as lightly highlighting a portion of the text contained in the webpage (the annotation target 836 may include a webpage associated with a URL), the renderer 835 may lightly highlight the annotation target (912).

[0058] 10 is a timing diagram illustrating example operations for navigating using annotations, according to one aspect. In some examples, the operations shown and described with respect to FIG. 10 may be performed after the operations shown and described with respect to FIG.

[0059] The user 900 may select (1002), such as by clicking, a first note (e.g., note A) on the annotation region 130 (e.g., in a side panel or overlay). The browser application 718 may respond to the selection of note A by focusing (1004) annotation A in the annotation region 130, such as by making annotation A visible in the annotation region and / or side panel.

[0060] The browser application 718 may also respond to the selection of annotation A by sending a request (1006) to the renderer 835 to highlight and scroll to annotation A. The renderer 835 may respond to the request (1006) by scrolling to and highlighting (1008) the annotation target. The annotation target may include a portion of the web page (e.g., a first portion of text) highlighted at (912), and the annotation may include modifying the first portion of text by highlighting the portion of text. Highlighting the annotation target (1008) may be an example of performing an action on the web page based on the stored annotation, including modifying the web page.

[0061] The user 900 can scroll and click (1010) on a second annotation target, which can be considered annotation B. The annotation of the annotation target can be a first modification to a second portion of text contained in the web page associated with the URL. The first modification can include lightly highlighting the second portion of text. In one example, the annotation of annotation target B can include a second modification of the second portion of text, and the renderer 835 can respond to the user's click (1010) on the annotation B target by performing (1012) the second modification of the annotation B target. The second modification can include, but is not limited to, a full highlighting of the second portion of text and / or a modification that is visually noticeable to the user, such as making it darker, less transparent, bolder, or more saturated in color. The renderer 835 can send a request to the browser application 718 to focus annotation B (1014), and / or the browser application 718 can receive the request to focus annotation B (1014) from the renderer 835. The browser application 718 can then focus on annotation B in the annotation region 130 and / or side panel (1016), such as by displaying annotation B in the side panel and / or annotation region 130.

[0062] FIG. 11 illustrates an example of an exemplary computing device, according to one aspect. In some implementations, the computing device 1500 is an example of the computing system 702. In some implementations, the computing device 1500 is an example of the server computing system 750. In some implementations, the mobile computing device 1550 is an example of the computing system 702. The computing device 1500 is intended to represent various forms of digital computers, such as laptops, desktops, tablets, workstations, personal digital assistants, televisions, servers, blade servers, mainframes, and other suitable computing devices. The computing device 1550 is intended to represent various forms of mobile devices, such as personal digital assistants, mobile phones, smartphones, and other similar computing devices. The components illustrated herein, their connections and relationships, and their functions are intended to be illustrative only and are not intended to limit the implementation of the embodiments of the invention described and / or claimed herein.

[0063] The computing device 1500 includes a processor 1502, a memory 1504, a storage device 1506, a high-speed interface 1508 connecting to the memory 1504 and a high-speed expansion port 1510, and a low-speed interface 1512 connecting to a low-speed bus 1514 and the storage device 1506. The processor 1502 may be a semiconductor-based processor. The memory 1504 may be a semiconductor-based memory. The components 1502, 1504, 1506, 1508, 1510, and 1512 are interconnected using various buses and may be mounted on a common motherboard or otherwise mounted as needed. The processor 1502 may process instructions for execution within the computing device 1500, including instructions stored in the memory 1504 or the storage device 1506, to display graphical information for a GUI on an external input / output device, such as a display 1516 connected to the high-speed interface 1508. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and multiple types of memories, where appropriate. Also, multiple computing devices 1500 may be connected, each performing some of the required operations (eg, as a bank of servers, a group of blade servers, or a multi-processor system).

[0064] The memory 1504 stores information within the computing device 1500. In one implementation, the memory 1504 is a volatile memory unit(s). In another implementation, the memory 1504 is a non-volatile memory unit(s). The memory 1504 may be another form of computer-readable medium, such as a magnetic disk or an optical disk.

[0065] The storage device 1506 can provide mass storage for the computing device 1500. In one embodiment, the storage device 1506 can be or include a computer-readable medium such as a floppy disk device, a hard disk device, an optical disk device, or an array of devices including a tape device, a flash memory or other similar solid-state memory device, or a device in a storage area network or other configuration. The computer program product can be tangibly embodied in an information medium. The computer program product can also include instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or a machine-readable medium, such as the memory 1504, the storage device 1506, or a memory on the processor 1502.

[0066] The high-speed controller 1508 manages bandwidth-intensive operations of the computing device 1500, while the low-speed controller 1512 manages less bandwidth-intensive operations. This allocation of functionality is merely exemplary. In some implementations, the high-speed controller 1508 is coupled to the memory 1504, a display 1516 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 1510 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 1512 is coupled to the storage device 1506 and the low-speed expansion port 1514. The low-speed expansion port, which may include various communication ports (e.g., USB, BLUETOOTH, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a network device, such as a switch or router, for example, via a network adapter.

[0067] The computing device 1500 can be implemented in many different forms, as shown in the figure. For example, it can be implemented as a standard server 1520, or multiple times within a group of such servers. It can also be implemented as part of a rack server system 1524. In addition, it can be implemented in a personal computer, such as a laptop computer 1522. Alternatively, components from the computing device 1500 can be combined with other components in a mobile device (not shown), such as device 1550. Each such device can include one or more of the computing devices 1500, 1550, and the entire system can be made up of multiple computing devices 1500, 1550 that communicate with each other.

[0068] Computing device 1550 includes components such as a processor 1552, memory 1564, input / output devices such as a display 1554, a communication interface 1566, and a transceiver 1568, among other components. Device 1550 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the components 1550, 1552, 1564, 1554, 1566, and 1568 are interconnected using various buses, and some of the components may be mounted on a common motherboard or otherwise mounted as desired.

[0069] The processor 1552 can execute instructions within the computing device 1550, including instructions stored in the memory 1564. The processor can be implemented as a chipset of chips including separate analog and digital processors. The processor can provide coordination of other components of the device 1550, such as control of a user interface, applications run by the device 1550, and wireless communications by the device 1550.

[0070] The processor 1552 can communicate with a user via a control interface 1558 and a display interface 1556 connected to a display 1554. The display 1554 can be, for example, a TFT LCD (thin film transistor liquid crystal display) or an OLED (organic light emitting diode) display, or other suitable display technology. The display interface 1556 can include suitable circuitry for driving the display 1554 to present graphical and other information to the user. The control interface 1558 can receive commands from the user and convert them for transmission to the processor 1552. Additionally, an external interface 1562 can be provided in communication with the processor 1552, allowing for short-range communication between the device 1550 and other devices. The external interface 1562 can, for example, provide for wired communication in some implementations and wireless communication in other implementations, and multiple interfaces can be used.

[0071] The memory 1564 stores information within the computing device 1550. The memory 1564 can be implemented as one or more of a computer-readable medium(s), a volatile memory unit(s), or a non-volatile memory unit(s). An expansion memory 1574 is also provided and is connected to the device 1550 via an expansion interface 1572, which can include, for example, a SIMM (single in-line memory module) card interface. Such expansion memory 1574 can provide additional storage space for the device 1550 or can store applications or other information for the device 1550. In particular, the expansion memory 1574 can include instructions for performing or supplementing the processes described above and can also include secure information. Thus, for example, the expansion memory 1574 can be provided as a security module for the device 1550 and can be programmed with instructions that enable secure use of the device 1550. Additionally, secure applications may be provided via the SIMM card along with additional information, such as by placing identifying information on the SIMM card in an unhackable manner.

[0072] The memory may include, for example, flash memory and / or NVRAM memory, as described below. In one embodiment, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as memory 1564, expansion memory 1574, or memory on processor 1552, which may be received, for example, via transceiver 1568 or external interface 1562.

[0073] The device 1550 can communicate wirelessly via a communication interface 1566, which can include digital signal processing circuitry as needed. The communication interface 1566 can provide communication in a variety of modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication can occur, for example, via a radio frequency transceiver 1568. In addition, short-range communication can occur using BLUETOOTH, Wi-Fi, or other such transceivers (not shown), etc. In addition, a Global Positioning System (GPS) receiver module 1570 can provide additional navigation and location related wireless data to the device 1550, which can be used as appropriate by applications executing on the device 1550.

[0074] Device 1550 may also provide voice communications using audio codec 1560, which receives voice information from a user and converts it into usable digital information. Audio codec 1560 may similarly generate sounds that are audible to a user, such as through a speaker in a handset or the like of device 1550. Such sounds may include voice telephone calls, recorded sounds (e.g., voice messages, music files, etc.), and sounds generated by applications running on device 1550.

[0075] The computing device 1550 can be implemented in many different forms, as shown in the figure, for example, it can be implemented as a mobile phone 1580, or it can be implemented as part of a smartphone 1582, a personal digital assistant, a tablet, a wearable 1590, or another similar mobile device.

[0076] In some implementations, the illustrated computing device may include sensors that interface with a wearable (e.g., AR headset / HMD) device 1590 to generate an augmented environment for displaying inserted content in a physical space. For example, one or more sensors included in the illustrated computing device 1550 or other computing devices may provide input to the AR headset 1590 or generally to the AR space. The sensors may include, but are not limited to, a touch screen, an accelerometer, a gyroscope, a pressure sensor, a biometric sensor, a temperature sensor, a humidity sensor, and an ambient light sensor. The computing device 1550 may use the sensors to determine the absolute position and / or detected rotation of the computing device in the AR space and use it as input to the AR space. For example, the computing device 1550 may be incorporated into the AR space as a virtual object such as a controller, a laser pointer, a keyboard, a weapon, etc. The user's positioning when incorporating the computing device / virtual object into the AR space allows the user to position the computing device to display the virtual object in a particular way in the AR space. For example, if the virtual object represents a laser pointer, the user can manipulate the computing device as if it were a real laser pointer. The user can move the computing device left and right, up and down, in circles, etc., and use the device in a manner similar to using a laser pointer. In some implementations, the user can aim the virtual laser pointer at a target location.

[0077] In some implementations, one or more input devices included in or connected to the computing device 1550 can be used as input to the AR space. The input devices can include, but are not limited to, a touch screen, a keyboard, one or more buttons, a trackpad, a touchpad, a pointing device, a mouse, a trackball, a joystick, a camera, a microphone, an earphone or earbuds with input capabilities, a game controller, or other connectable input devices. When the computing device is integrated into the AR space, a user manipulating an input device included in the computing device 1550 can cause certain actions to occur in the AR space.

[0078] In some implementations, the touch screen of the computing device 1550 can be rendered as a touch pad in the AR space. A user can manipulate the touch screen of the computing device 1550. The manipulations are rendered, for example in the AR headset 1590, as movements on the rendered touch pad in the AR space. The rendered movements can control virtual objects in the AR space.

[0079] In some implementations, one or more output devices included in the computing device 1550 can provide output and / or feedback to a user of the AR headset 1590 within the AR space. The output and feedback can be visual, tactical, or audio. The output and / or feedback can include, but is not limited to, vibration, turning one or more lights or strobes on and off or blinking and / or flashing, sounding an alarm, playing a chime, playing a song, playing an audio file, and the like. Output devices include, but are not limited to, vibration motors, vibration coils, piezoelectric devices, electrostatic devices, light emitting diodes (LEDs), strobes, speakers, and the like.

[0080] In addition to the above, a user may be provided with controls that allow the user to make choices about both whether and when the systems, programs, or features described herein may enable collection of user information (e.g., information about the user's social network, social actions, or activities, occupation, user preferences, or the user's current location), and whether content or communications are sent from the server to the user. In addition, certain data may be processed in one or more ways such that personally identifiable information is removed before it is stored or used. For example, the user's identity may be processed such that the user's personally identifiable information cannot be identified, or if location information is obtained (such as to the city, zip code, or state level), the user's geographic location may be generalized such that the user's specific location cannot be identified. Thus, the user may control what information is collected about the user, how that information is used, and what information is provided to the user.

[0081] Various implementations of the systems and techniques described herein may be realized in digital electronic circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, which may be special-purpose or general-purpose, coupled to receive data and instructions from the storage system, and to transmit data and instructions to the storage system.

[0082] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine language. As used herein, "machine-readable medium", "computer-readable medium" refers to any computer program product, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic circuit (PLD)) used to provide machine instructions and / or data to a programmable processor including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0083] To provide for interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (e.g., a CRT or LCD monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices may also be used to provide for interaction with a user. For example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user may be received in any form, such as acoustic, spoken language, or tactile input.

[0084] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or that includes middleware components (e.g., an application server), or that includes front-end components (e.g., a client computer having a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or in a combination of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0085] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0086] In this specification and the appended claims, the singular forms "a," "an," and "the" do not exclude plural referents unless the context clearly dictates otherwise. Furthermore, conjunctions such as "and," "or," and "and / or" are inclusive unless the context clearly dictates otherwise. For example, "A and / or B" includes A only, B only, and A and B. Furthermore, the connecting lines or connectors shown in the various illustrated figures are intended to illustrate examples of functional relationships and / or physical or logical couplings between the various elements. In an actual device, many alternative or additional functional relationships, physical connections, or logical connections may exist. Furthermore, no item or component is essential to the implementation of the embodiments disclosed herein unless an element is specifically described as "essential" or "critical."

[0087] Without limitation, terms such as "approximately," "substantially," and "generally" are used herein to indicate that an exact value or range thereof is not required and need not be specified. As used herein, the above terms have meanings that are immediately understood by one of ordinary skill in the art.

[0088] Additionally, terms such as top, bottom, upper, lower, side, end, front, back, etc., used herein are used with reference to the orientation currently considered or illustrated, and it should be understood that when these are considered from another orientation, such terms should be modified accordingly.

[0089] Although certain exemplary computer-implemented methods, apparatus, and articles of manufacture have been described herein, the scope of this patent is not limited thereto. It is to be understood that the terminology used herein is for the purpose of describing particular aspects and is not intended to be limiting. To the contrary, this patent covers all methods, apparatus, and articles of manufacture fairly falling within the scope of the claims of this patent.

[0090] In one aspect, a non-transitory computer-readable medium stores instructions that, when executed by a processor on a receiving computing device, cause the receiving computing device to perform any of the methods disclosed herein.

[0091] In one aspect, a computing device may be configured with at least one processor and a memory storing instructions that, when executed by the at least one processor, perform any of the methods disclosed herein.

Claims

1. 1. A method implemented by a browser, comprising: presenting a web page in a browser content window, the web page being associated with a first resource address, the method further comprising: presenting an annotation area, the annotation area being outside the browser content window, the method further comprising: receiving an annotation for a tab group in the annotation area in association with a user account, the tab group including the first resource address and a second resource address, the method further comprising: storing the annotations in association with the user account and the tab group.

2. storing the annotations includes storing the annotations locally in long-term storage in association with the user account and the tab group; The method of claim 1 , wherein the web pages are not stored locally in the long-term storage.

3. The method of claim 1 , wherein the annotation causes a change in appearance of a portion of text contained in the web page.

4. The method of claim 3 , wherein the modification is performed using a rendering process different from a rendering process that rendered the web page.

5. The method of claim 1 , wherein storing the annotation occurs without notifying a server associated with the first resource address.

6. The method of claim 1 , wherein the annotations are stored on a local computing device on which the browser is running.

7. The method of claim 1 , wherein the annotations are stored in annotation records comprising a metadata portion, a body portion, and a target portion.

8. the web pages include a first web page in a first tab of the tab group; The annotations include a first annotation; the tab group includes the first tab and a second tab, the second tab being associated with the second resource address, and the method further comprises: Detecting a preselection action on a third resource address of a third tab; and 2. The method of claim 1, further comprising: in response to detecting the pre-selection action, presenting a preview area of ​​the third resource address, the preview area including a second annotation of the third resource address.

9. receiving a request to display the web page associated with the first resource address; retrieving the annotations in response to receiving the request to display the web page; in response to retrieving the annotations, adding a notification to an address bar area indicating that the tab group has at least one annotation; receiving a selection of the notification; The method of claim 1 , wherein presenting the annotation region occurs in response to receiving the selection of the notification.

10. receiving a selection of the annotation; 10. The method of claim 1, further comprising, in response to receiving the selection of the annotation, scrolling to a portion of the web page associated with the annotation, the annotation being associated with text within the web page.

11. The method of claim 1 , wherein receiving the annotation comprises receiving a selection of an image included on the web page.

12. analyzing content of the web page to identify suggested annotation content; presenting the proposed annotation content as a selectable element; receiving a selection of the suggested annotation content; The method of claim 1 , wherein receiving the annotation occurs in response to the selection of the suggested annotation content.

13. A computer program comprising instructions, said instructions being configured, when executed by at least one processor, to cause a computing system to perform a method according to any of claims 1 to 12.

14. 1. A computing system comprising: At least one processor; and a non-transitory computer-readable storage medium comprising instructions, the instructions being stored on the non-transitory computer-readable storage medium and configured, when executed by the at least one processor, to cause the computing system to perform a method according to any of claims 1 to 12.

Images