Storage of content associated with resource locator

Abstract

To provide a browser-integrated annotation system, a method thereof, and a non-temporary computer readable medium which integrate an annotation system with a browser to improve usability, facilitate integrating a web page, a tab, and a retrieval session, and store annotation in association with a user account.SOLUTION: The present invention is directed to a method of execution through a browser. The method includes presenting at least a part of a resource address (e.g., URL), and presenting a rendered webpage to a browser content window 120. The method includes presenting an annotation region 130 in the rendered webpage which is associated with the resource address. The method also includes receiving, in association with a user account, an annotation in the annotation region which is located outside of the browser content window, and storing the annotation in association with the user account and the resource address.SELECTED DRAWING: Figure 1A

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Application No. 63 / 266,291, filed December 31, 2021, and entitled "BROWSER-BASED ANNOTATION INTERFACE FOR INTERNET DOCUMENTS," the disclosure of which is incorporated herein by reference. [Background technology]

[0002] Websites provide useful information or functionality to users, and many users use the Internet to research products, places, companies, services, etc. The web pages associated with a website are typically stored on a remote server. Summary of the Invention

[0003] Websites provide useful information or functionality to users, and many users use the Internet to research products, places, companies, services, and the like. Web pages associated with websites are typically stored on remote servers. Research across various sites and various web pages within those sites can be complex, often requiring integration, comparison, and decision-making. Users can view web pages within a browser running on their local computing device. In embodiments, browser-integrated tools are provided to assist with these and other tasks. These tools include novel user interfaces and novel browser features, such as storing information associated with web pages and / or websites. Stored information associated with web pages and / or websites is also associated with a user account. Stored information associated with web pages and / or websites is generated by the user who owns and / or manages the user account. Stored information is not accessible from the server that maintains the website with which the user-generated information is associated. A novel user interface, called an annotation user interface, helps users organize, understand, and interact with Internet-based documents without coordination and / or communication with the servers and / or entities that maintain the websites. The annotation user interface provides a highly integrated information generation (e.g., note-taking) experience that solves the decision-making funnel. In particular, the annotation user interface can include intelligence that enables users to extract the most important details from a website or collection of websites, compare information across websites, and return to the original content (web page) without having to bookmark the content and without having to provide information to the servers and / or entities that maintain the websites. The annotation user interface can extract rich content, provide note and / or image suggestions, and / or organize annotations around the annotation target.Annotation targets can represent web pages (source documents), topics, tab groups, etc. The annotation user interface allows users to integrate user-created annotations with suggested content, track annotations across websites (for example, by associating annotations with topics or tab groups), and compare notes across sites and web pages. The user interface can be viewed in multiple ways relative to a user account and can be presented at various levels of detail (e.g., in a side panel, across multiple tables, or as a standalone document). Integrating the annotation interface into a browser enables the browser to separate the annotation data from the rendering process. Such separation allows the interface to appear integrated, but the annotations themselves are not exposed to the rendering process, and information about the annotations is not available to the server and / or administrators maintaining the website. become.

[0004] A method may be performed in a browser, the method including presenting at least a portion of a resource address and presenting a rendered web page in a browser content window, the rendered web page associated with the resource address, the method further including presenting an annotation area, the annotation area outside the browser content window, the method further including receiving an annotation in the annotation area in association with a user account, and storing the annotation in association with the user account and the resource address.

[0005] A non-transitory computer-readable storage medium including instructions stored on the non-transitory computer-readable storage medium and configured, when executed by at least one processor, to cause a computing system to present at least a portion of a resource address and present a rendered web page in a browser content window, the rendered web page being associated with the resource address, the instructions, when executed by the at least one processor, to cause the computing system to present an annotation area, the annotation area being outside the browser content window, and the instructions, when executed by the at least one processor, to further cause the computing system to receive an annotation in the annotation area in association with a user account and store the annotation in association with the user account and the resource address.

[0006] The computing system may include at least one processor and a non-transitory computer-readable storage medium that may include instructions stored on the non-transitory computer-readable storage medium and configured, when executed by the at least one processor, to cause the computing system to present at least a portion of the resource address and present a rendered web page in a browser content window, the rendered web page being associated with the resource address, the instructions, when executed by the at least one processor, to cause the computing system to present an annotation region, the annotation region being outside the browser content window, and the instructions, when executed by the at least one processor, to receive an annotation in the annotation region in association with a user account and store the annotation in association with the user account and the resource address.

[0007] The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings. [Brief explanation of the drawings]

[0008] [Figure 1A] 1 illustrates an exemplary technique for displaying a browser annotation interface, according to some aspects. [Figure 1B] 1 illustrates an exemplary technique for displaying a browser annotation interface, according to some aspects. [Figure 1C] 1 illustrates an exemplary technique for displaying a browser annotation interface, according to some aspects. [Figure 1D] 1 illustrates an exemplary technique for displaying a browser annotation interface, according to some aspects. [Figure 2A] FIG. 10 illustrates another exemplary technique for displaying a browser annotation interface, according to one aspect. [Figure 2B] FIG. 10 illustrates another exemplary technique for displaying a browser annotation interface, according to one aspect. [Figure 3] 3A and 3B illustrate an exemplary browser annotation interface for a computing device with limited display area, according to one aspect. [Figure 4A] FIG. 1 illustrates an exemplary browser annotation interface with suggested annotation content, according to some aspects. [Figure 4B] FIG. 1 illustrates an exemplary browser annotation interface with suggested annotation content, according to some aspects. [Figure 4C] FIG. 1 illustrates an exemplary browser annotation interface with suggested annotation content, according to some aspects. [Figure 4D] FIG. 1 illustrates an exemplary browser annotation interface with suggested annotation content, according to some aspects. [Figure 4E]FIG. 1 illustrates an exemplary browser annotation interface with suggested annotation content, according to some aspects. [Figure 4F] FIG. 1 illustrates an exemplary browser annotation interface with suggested annotation content, according to some aspects. [Figure 5A] FIG. 1 illustrates an exemplary browser annotation interface with grouped annotation content, according to some aspects. [Figure 5B] FIG. 1 illustrates an exemplary browser annotation interface with grouped annotation content, according to some aspects. [Figure 5C] FIG. 1 illustrates an exemplary browser annotation interface with grouped annotation content, according to some aspects. [Figure 5D] FIG. 1 illustrates an exemplary browser annotation interface with grouped annotation content, according to some aspects. [Figure 6] FIG. 1 illustrates an exemplary annotation rendering, according to some aspects. [Figure 7] FIG. 1 illustrates a system for providing a browser annotation interface, according to one aspect. [Figure 8] 1 is a block diagram of components for a browser application that implements a system for providing a browser annotation interface, according to one aspect. [Figure 9] 4 is a timing diagram illustrating exemplary operations for loading an annotated Internet document, according to one aspect. [Figure 10] 4 is a timing diagram illustrating exemplary operations for navigating using annotations, according to one aspect. [Figure 11] FIG. 1 illustrates an example of an exemplary computing device, according to one aspect. DETAILED DESCRIPTION OF THE INVENTION

[0009] Like reference numbers refer to like elements. Embodiments relate to providing a browser-integrated annotation system for annotating and storing annotations in Internet-based documents, such as websites and web pages. Integrating the annotation system into the browser improves usability, facilitates integration across web pages, tabs, and search sessions, and allows annotations to be associated with and stored in a user account. Browser-based annotation systems can also facilitate organization and navigation and provide automated assistance, such as automatically generated annotation suggestions. In some embodiments, such suggestions may be based on machine learning models, document structure, and / or historical information (e.g., analysis of anonymized search records). Because the annotation interface is integrated into the browser, embodiments can provide a variety of techniques for displaying the annotation interface and may serve as a navigation aid.

[0010] FIG. 1A illustrates an exemplary browser user interface according to embodiments described herein. 1 illustrates a user interface (UI) 100. Generally, the UI 100 is generated and rendered by a browser application (e.g., browser application 718 of FIG. 7 ) executing on an operating system (e.g., operating system 710) of a computing device. The UI 100 may include a tab strip 108 associated with the browser application. In this example, the UI 100 includes five browser tabs 105, although any number of browser tabs or tab groups can be opened by the browser application. Each browser tab, e.g., browser tab 105, can be associated with web content (a web page) that is displayed in a corresponding browser content window 120 of the browser application. As used herein, a web page refers to any content that is at least partially generated / served by / from a remote server hosting a website. A web page can be associated with a resource address, such as a uniform resource locator (URL) or a uniform resource indicator (URI). A web page can include hypertext markup language (HTML) content. The browser application can request a web page from a remote server hosting a website based on the resource address (e.g., a URL). A browser application can request a web page, for example, by sending a Hypertext Transfer Protocol (HTTP) GET request to a remote server hosting a website. Web page can refer to progressive web applications as well as traditional web pages. Additionally, as used herein, source document refers to the web page currently rendered in browser content window 120 and any other data (metadata) used to render the document (e.g., markup and scripts that are not actually displayed).

[0011] The UI 100 may also include an address bar area 110. The address of a web page displayed in the browser content window 120 may be displayed in the address bar area 110 (e.g., in the address input area 115). The address of the web page may be represented as a resource address. The resource address included in the address bar area 110 may be associated with the web page displayed in the browser content window 120. In some examples, the address bar area 110 may display at least a portion of the resource address. Other controls, icons, etc. may be included in the address bar area 110. For example, the address bar area 110 may include a user icon 125 (shown in FIG. 1B ). The user icon 125 may provide an indication of a user account associated with the browser session. The user icon 125 may be an image, text, or other representation representing the user account. In some examples, the user account is associated with a user, and the user may be required to enter a unique username as well as a passcode associated with the username to join a browser session associated with the user account. In some examples, no login is required to join a browser session, and the user account may be considered to be associated with the local computing device on which the browser application is running. Other controls / icons included in the address bar area 110 may include a forward control, a back control, a refresh control, a home control, an advanced features control, and / or a bookmarks control. The address bar area 110 may be controlled by and / or associated with a browser (e.g., browser application 718). While the content of the address bar area 110 may be controlled by the browser application, the content of the browser content window 120 is controlled by the web page being displayed and / or the provider of the web page.Thus, the appearance and user interface elements displayed in the address bar area 110 are controlled by the browser application.

[0012] The UI 100 includes an annotation user interface shown as annotation area 130. In the example of FIG. 1A , annotation (or note) area 130 is located within or implemented as a browser sidebar. The browser sidebar may be a contiguous area, for example, a combined area representing address bar area 110 and annotation area 130. Thus, the content displayed (rendered) in the sidebar is controlled (determined) locally by the browser, rather than by a server hosting the website displayed (rendered) in the browser content window 120. While the sidebar is shown as a contiguous area with address bar area 110, it could also be a split window, a pop-up window, or some other portion of the browser UI 100. Annotations entered in annotation area 130 can be stored locally on the computing device on which the browser is running or remotely in association with a user account. Annotations entered in annotation area 130 may not be accessible by the server hosting the website. Notification of the annotation is not sent to the server hosting the website. In other words, the annotations are associated locally with the website without notifying or interacting with a server associated with the website. Storing annotations locally on a computing device and / or remotely in association with a user account allows a user to securely generate and store annotations without the knowledge of the server hosting the website and / or the website administrators.

[0013] Annotations and / or notes may include content entered by a user into annotation area 130. In some examples, annotations may include text entered by a user into annotation area 130. In some examples, annotations may include changes to the appearance and / or formatting of a portion of text included in a rendered web page. Changes to the appearance and / or formatting of a portion of text may include highlighting or typeface changes. In some examples, annotations may include resource addresses (e.g., URLs) of other web pages. In some examples, annotations may include files uploaded by a user.

[0014] Because annotation area 130 can be integrated as part of browser user interface 100 (e.g., address bar area 110), annotation area 130 cannot be spoofed (imitated), for example, by a third party or the owner of the content displayed in browser content window 120. Because annotation area 130 is part of the application of the browser application, the integration of annotation area 130 is difficult to imitate. In some implementations, annotation area 130 may be displayed (invoked, rendered) in response to a menu option (e.g., menu option 122). Menu option 122 may be an option provided in a menu displayed in response to a command to display the tab menu (e.g., right-clicking tab 105).

[0015] In the example of FIG. 1A , tab 105 currently has no notes. Therefore, annotation area 130 includes new note UI element 132. New note UI element 132 can be any UI element that allows a user to enter text (e.g., keyboard, speech-to-text, stylus, etc.). Annotation area 130 may include a display of the tab / website with which the note is associated. Annotation area 130 may include controls for adding an annotation (note) to be associated with the webpage currently displayed in browser content window 120. The controls are configured to perform a predetermined task / operation when activated by the user. The controls may include a text control 133 for adding a text-based note. The text control 133 may open a new note UI element, such as UI element 132. The controls may include a to-do control 134. The controls may include an image control 135. The image control 135 allows a user to upload, take, and / or search for an image to add as a note. The controls may include a link control 136. The link control 136 allows a user to add a link (address, such as a URL) to another Internet-based document. The controls may include an auto-suggest control 137. The auto-suggest control 137 causes the browser application to analyze the annotation target (e.g., both the visible and invisible content of the browser content window 120) and provide one or more suggestions for information items to include in the annotation target's note. Examples of auto-suggestions are shown in Figures 4A-4F.

[0016] In some implementations, annotation region 130 may be displayed as shown in FIG. 1B . In FIG. 1B , annotation region 130 is displayed in response to menu option 123. Menu option 123 may be provided within a menu displayed in response to a command to display a selected text menu. Thus, for example, a user may select content displayed in browser content window 120 and right-click within the selection to display a menu including menu option 123. In FIGS. 1B , 2A , 2B , 3A , and 3B , text with a shaded background is considered highlighted and is an example of a text change. The highlighted text may be considered content selected by the user when the user right-clicked a mouse or otherwise entered a human interface device (HID) to display a menu including menu option 123. Selecting menu option 123 may cause annotation region 130 of FIG. 1B to be rendered by a browser application, e.g., by the browser application's rendering process. In the example of Figure 1B, the annotation target already has one existing note (which may be considered a first annotation associated with the web page), so annotation area 130 includes note 138 (which may be considered a second annotation associated with the web page). Additionally, in response to selection of menu option 123, an additional new note UI element 132 is included in annotation area 130. The technique for displaying annotation area 130 shown in Figure 1B can be combined with the technique for displaying annotation area 130 shown in Figure 1A. In other words, a browser can be configured to use either or both techniques.

[0017] In some implementations, such as the example shown in FIG. 1C , the annotation area 130 can include a control 139 configured to expand the tab strip 108. The expanded tab strip 108′ can display a preview area 140 for each tab. Each tab can be associated with a different annotation target, i.e., a resource (e.g., a first resource, a second resource, and a third resource), such as a web page and / or a website. The preview area 140 can include any existing notes for the annotation target associated with the tab, but may lack controls (e.g., controls 133-137). Thus, the preview area 140 is a scaled-down version of the annotation area 130 for the tab. The preview area 140 can display portions of multiple rendered web pages and their associated annotations, such as a portion of a first rendered web page and a first annotation associated with the first rendered web page, a portion of a second rendered web page and a second annotation associated with the second rendered web page, and a portion of a third rendered web page and a third annotation associated with the third rendered web page. Thus, the expanded tab strip 108' provides a novel way to quickly and conveniently compare notes associated with various annotation targets, for example, between URLs, web pages and / or websites, tabs, tab groups, etc. The clip 108' reduces the area of ​​the browser content window 120, causing some previously viewable content to be off-screen and / or hidden or invisible in the browser. The technique for displaying the preview area 140 shown in Figure 1C can be combined with the technique for displaying the annotation area 130 shown in Figures 1A and 1B. In other words, a browser can be configured to use one, two, or all three techniques.

[0018] In some examples (not shown), the browser can display a tab's preview area 140 in response to a user performing a tab pre-selection action, such as hovering a cursor over the tab or sliding a finger over the tab. The browser can display a web page associated with the tab in response to a user performing a second type of selection on the tab, such as clicking or tapping on the tab. In some implementations, the preview area 140 can include a selectable control that, when selected, causes the browser to display a web page associated with a resource address (e.g., a URL) and simultaneously display the annotation area 130.

[0019] In some implementations, such as the example shown in FIG. 1D , the annotation region (e.g., annotation region 130) may be displayed in response to menu option 124. Menu option 124 may be provided in a menu that is displayed in response to selection of a control that displays a browser menu, as shown in FIG. 1D , for example. Although not shown in FIG. 1D , selecting menu option 124 may cause annotation region 130 of FIG. 1A or 1B to be rendered (depending on whether the source document of the annotation target (e.g., the current tab) has existing notes (e.g., 1B) or not (e.g., 1A)). The technique for displaying annotation region 130 shown in FIG. 1D can be combined with the techniques described in FIGS. 1A-1C. In other words, a browser can be configured to use one, two, three, or all four techniques.

[0020] In some embodiments, annotation area 130 may be displayed as shown in FIG. 2A . In the example of FIG. 2A , control 126 is provided by a browser application. In some embodiments, control 126 may be provided in address bar area 110. In some embodiments (not shown), control 126 may be located before address entry area 115. In some embodiments, control 126 may be located in address entry area 115. In some embodiments (not shown), control 126 may be located after address entry area 115. Control 126 may be provided when an annotation target (e.g., a document currently displayed in browser content window 120) has notes (e.g., annotations) associated with it. In some embodiments, control 126 may be provided regardless of whether there are notes and / or annotations associated with the annotation target. In some embodiments, the appearance of control 126 may change depending on whether there are notes and / or annotations associated with the annotation target. For example, the text, color, shading of control 126, or an icon included in control 126 may differ depending on whether there are notes and / or annotations associated with the annotation target. In some implementations, the text may indicate the number (quantity) of existing notes and / or annotations for the annotation target. In the example shown in Figure 2A, the shaded background represents text highlighting of the text portion associated with the notes and / or annotations for the annotation target.

[0021] When a user activates (selects) a control 126, such as by tapping and / or clicking on the control 126, the browser can render an annotation region 130 containing notes about the annotation target, as shown in Figure 2B. Existing notes and / or annotations (e.g., 238a, 238b, etc.) can be added to the software associated with the annotation target. The interface of FIG. 2B may be configured to navigate within a source document. For example, when a user activates (selects, clicks on) an existing note 238b and / or annotation, the browser may scroll the source document to the location associated with the existing note 238b and / or annotation. The interface of FIG. 2B also illustrates that the content of the note and / or annotation may have full highlighting and / or partial highlighting. For example, the content corresponding to the selected note 238a and / or annotation may be fully highlighted (displayed with a shaded background), while the text associated with other notes and / or annotations may be partially highlighted (no shading in FIG. 2B). If the user selects note 238b, the browser may fully highlight the content associated with note 238b (displayed with a shaded background) and change the content associated with note 238a (if still displayed in the browser content window 120) to a partial highlight.

[0022] 3A and 3B illustrate an exemplary preview area 340 for a device with limited display area, such as a mobile device, tablet, or wearable. Preview area 340 is similar to preview area 140 of FIG. 1C. Thus, preview area 340 displays first notes and / or annotations associated with a source document, which may expand into annotation area 330, as shown in FIG. 3B. Annotation area 330 is similar to and / or includes similar features and / or functionality as annotation area 130 of FIGS. 1A-2B.

[0023] 4A-4F illustrate exemplary browser annotation interfaces with suggested annotation content, according to some aspects. In the example of FIG. 4A, a user navigates to a web page that is now at least partially displayed in browser content window 120 and opens annotation region 130. In some implementations, annotation region 130 can include a description 431 of the annotation target, which in this case is the title of the web page displayed in browser content window 120. In some implementations, the title is stored as a data element with the annotation record (e.g., the annotation record stored in local notes storage 726 of FIG. 7). In the example of FIG. 4A, a user has activated auto-suggestion control 137. When auto-suggestion control 137 is activated, annotation region 130 is updated with a suggested text-based content element 432. The suggested text-based content element 432 is a content element selected from the content of the annotation target. Auto-suggestion can be based on the schema markup of the annotation target. Auto-suggestion can be based on historical search records. Auto-suggestion can be based on the document structure of the annotation target. The user can select one or more of the suggested text-based content elements 432. In the example of FIG. 4B, the user has selected four of the suggested text-based content elements 432, as indicated by the dotted background of the four suggested text-based content elements 432 selected by the user. The dotted background may represent a change in color or highlighting of the suggested text-based content elements 432. FIG. 4B shows four notes 433 added in response to the selection of the four suggested text-based content elements 432.

[0024] In the example of FIG. 4C, the user scrolled to another portion of the annotation target and changed the appearance of text 421. In this example, changing the appearance of text 421 includes highlighting text 421. Selecting auto-suggest control 137 may display auto-suggested element 434, which includes the selected highlighted text 421 and image 422. In response to the user selecting the suggested text of auto-suggested element 434, FIG. 4D shows that the browser application added a new note 435 and / or annotation to annotations region 130. This new note 435 includes the same text as note 433 and / or annotation. Although not shown, selecting one of the notes 433 and / or annotations causes the annotation target to scroll so that the content corresponding to the selected one of the notes 433 and / or annotations is displayed in browser content window 120, as shown, for example, in FIG.

[0025] In the example of FIG. 4E , selecting auto-suggest control 137 may add suggested image elements 436 to the annotation of the annotation target (e.g., displayed in browser content window 120). The user may select one or more of the images, which may be added to the annotation of the annotation target as image notes 437 and / or image annotations, as shown in FIG. 4F. In some implementations, suggested text-based content elements 432 and auto-suggested elements 434 are selectable elements. These selectable elements may be configured, upon selection, to add suggested annotation content as annotations, as described above.

[0026] 5A-5D illustrate exemplary browser annotation interfaces with grouped annotation content, according to some embodiments. In the example of FIG. 5A, a user has two tabs 105 (e.g., 105a and 105b) open. The user has annotation region 130 open, which displays two notes about the annotation target of the website corresponding to tab 105b, as indicated by annotation target description 531 included in annotation region 130. The user can select tab 105a, which causes the browser application to display the interface of FIG. 5B. In response to the selection of tab 105a, the browser displays in annotation region 130 annotations for the annotation target corresponding to the resource address (e.g., URL) associated with the selected tab 105a. The annotation target now corresponds to the website associated with tab 105a, and annotation region 130 indicates this change by changing description 531. In the example of FIG. 5B , the current annotation target does not have any existing annotations and / or notes. However, as indicated by the dashed arrow in FIG. 5B , the user can create an annotation and / or note that is an image included in the web page by dragging image 521 onto new note UI element 132. In addition to being a text-based UI element (e.g., configured to receive text), new note UI element 132 can be configured as a drop location. Thus, when content, such as an image, copied text, or some other content, is dragged (in whole or in part) onto new note UI element 132 included in annotation region 130 and released, the browser application can interpret the dragging and / or dropping of the content as a request to store the content as a new annotation and / or note in the annotation target (e.g., the annotation target corresponding to description 531). FIG. 5C illustrates the result of this request, in which, for example, the browser application adds the new image as an annotation to the annotation target. If the annotation is an image, the image is downloaded to the user's device, allowing the user to access the note even without a network connection.

[0027] In some implementations, as shown in FIG. 5D, a user may create a tab group by grouping two tabs 105a and 105b into tab group 505. The user may have given tab group 505 a description of "Mountain Climbing." Tab groups can also be annotation targets. In some implementations, creating tab group 505 may also cause the browser application to group existing annotations for annotation targets contained in the tab group and associate the tab group as an annotation target as well. This may be done directly (e.g., by adding an index entry that associates the annotation data with the tab group's identifier) ​​or indirectly (e.g., by adding an index entry that associates a web page with the tab group). Annotation region 130 may be updated to reflect the new association; for example, description 531 of annotation region 130 may be changed to the description of tab group 505. Annotation regions representing groups of annotation targets may also be added to the annotation targets. An annotation target can include additional information, such as the number of annotation targets (two tabs in this example) and / or the number of notes and / or annotations in the group of annotation targets. In some implementations, annotation region 130 can add a group control 535, which allows a user to select between annotation targets in the group upon actuation of control 535. The annotation region 130 of an annotation target that represents multiple annotation targets (e.g., tab group 505) can display annotations associated with one or more of the annotation targets. In the example of FIG. 5D , annotation data for both tab 105a and tab 105b is displayed (rendered) in annotation region 130.

[0028] FIG. 6 illustrates exemplary variations in annotation rendering, according to some aspects. These exemplary variations may be applied to any note (annotation) displayed in the annotation area 130 of any of the aforementioned figures. The shaded background represents a highlighted region and / or text highlighting. Exemplary annotation 602 includes text but not a title. Exemplary annotation 604 includes a title ("Note Title") and text. Exemplary annotation 606 includes text and a date and time (timestamp) when annotation 606 was entered by a user. Exemplary annotation 608 includes text with a modified appearance, such as highlighted text ("Highlight Hint Text") and additional text. Exemplary annotation 610 includes text with a modified appearance, such as highlighted text ("Highlight Hint Text"), a title, and additional text. Exemplary annotation 612 includes text with a modified appearance, such as highlighted text ("Highlight Hint Text"), a title, a date and time (timestamp), and additional text. Exemplary annotation 614 includes a title. Exemplary annotation 616 includes a title and text. Exemplary annotation 618 includes a date and time (timestamp) and text. Exemplary annotation 620 includes a title and includes text with a visual change, such as highlighted text. Exemplary annotation 622 includes a title, unaltered text, and text with a visual change, such as highlighted text. Exemplary annotation 624 includes a date and time (timestamp), text, and text with a visual change, such as highlighted text.

[0029] 7 illustrates a system 700 for providing a browser annotation interface, according to one embodiment. The browser annotation interface may include a browser user interface 100 and / or an annotation area 130.

[0030] The system 700 may include a computing system 702. The computing system 702 may include a local computing device and / or a client device, such as, but not limited to, a smartphone, a tablet, a mobile computing system, a laptop or notebook computer, or a desktop computer.

[0031] Computing system 702 may include memory 704. Memory 704 may include a non-transitory computer-readable storage medium that includes instructions that, when stored on the non-transitory computer-readable storage medium and executed by at least one processor (e.g., CPU / GPU 732 included in computing system 702), are configured to cause computing system 702 to perform any combination of the methods, functions, and / or techniques described herein.

[0032] The memory 704 may include instructions for executing an operating system (O / S) 710. The operating system 710 controls the various components of the computing system 702, such as the memory 704, the output device(s) 724, the input device(s) 730, the CPU / GPU 732, and / or the camera(s) 734. You can control which applications have access to the components.

[0033] The operating system 710 enables multiple application(s) 712, such as a browser application 718, to run on the computing system 702. The browser application 718 may perform any combination of the methods, functions, and / or techniques described herein. For each application 712, the operating system 710 may maintain and / or store application information 715 and / or application layers, which enable the application 712 to communicate with other applications running on the computing system 702. For each application 712, the operating system 710 may maintain and / or store session data 717, which enables sessions between application processes to be initiated, terminated, and managed.

[0034] The memory 704 may include a local note storage 726. The local note storage 726 may store notes and / or annotations, such as those described above and shown in the previous figures. The local note storage 726 may store notes and / or annotations in annotation records. In some implementations, the annotation records may include a user identifier. In some implementations, the location (directory) in which the annotation records are stored reflects their association with a particular user. Each annotation record may include a metadata portion, a body portion, and a target portion. The metadata portion of the annotation record may include an identifier, a creation date, a modification date, etc. The identifier allows a browser to uniquely identify each annotation record. The ID may be unique per user. The ID may be unique per client device. In some implementations, the body of the annotation record may include a note type indicating the type of annotation. In some implementations, the metadata portion of the annotation record may include the note type. The note type may identify the type of annotation, metadata such as text, image, rich text, handwritten, audio, video, location, geolocation, and / or file. The target portion includes an annotation target identifier. The annotation target identifier identifies the resource to which the annotation applies (e.g., a web page, a website, a document, a file, a tab group, etc.). The annotation target identifier can be a resource address, such as a URL. The annotation target identifier can be a group identifier, such as a tab group identifier. The body portion of the annotation record can include text. In some examples, the body portion of the annotation record can include an address, such as a URL, that links to and / or identifies an external information source and / or data source. The text represents the text selected / provided by the user for the annotation. The body portion of the annotation record can include an image identifier. For annotations that are tied to a specific element of the annotation target (e.g., specific text, a specific image, etc.), the annotation target element can be included in the metadata portion of the annotation record.In some examples, an annotation target element can be included in the annotation record. The annotation target element can be a selector (e.g., a CSS selector) that identifies specific content within the annotation target. In some examples, the annotation target element can include a portion of text contained in a web page. In some examples, the annotation target can identify a file (e.g., an image), and / or the annotation target identifier can include a filename and / or a hash value associated with the file. The hash value can be determined based on any hash algorithm, such as Message Digest 5 (MD5) or Secure Hash Algorithm (SHA).

[0035] The local notes storage 726 can store notes and / or annotations in association with addresses and / or groups, as well as in association with a user account, such as the user account associated with the user who created the note and / or annotation. In some examples, notes and / or annotations may be considered to be stored in association with a user account if they are stored in a directory associated with the user, such as a local directory accessible to the user. The notes and / or annotations may be stored locally in memory 704 of the computing system 702 on which the browser application is running. The notes and / or annotations may be stored locally in long-term storage, such as non-volatile memory, that remains accessible after the computing system is powered off and back on. In some examples, rendered web pages are not stored in long-term memory, and the notes and / or annotations may be retrieved and displayed when the rendered web pages are retrieved again from the service(s) / website(s) 770 and rendered in the browser application 718. Storing notes and / or annotations locally may prevent the server hosting the website associated with the notes and / or annotations from recognizing the notes and / or annotations.

[0036] The memory 704 can store preferences 728. The preferences 728 can be received from a user and can include, but are not limited to, web browser appearance, favorite sites or bookmarks, and / or the first application to launch.

[0037] The computing system 702 may include one or more camera(s) 734. The camera(s) 734 may capture images, such as images of users for videoconferencing.

[0038] Computing system 702 may include at least one processor, such as a central processing unit (CPU) and / or a graphics processing unit (GPU) 732. The CPU / GPU may execute instructions, such as instructions stored in memory 704, to perform any combination of the methods, functions, and / or techniques described herein.

[0039] The computing system 702 may include one or more input device(s) 730. The input device(s) may include, but are not limited to, a keyboard, a mouse, a microphone, or a human interface device such as a touchscreen.

[0040] The computing system 702 may include one or more output device(s) 724. The output device(s) may include, but are not limited to, output devices such as a display, speakers, or a printer.

[0041] The input device(s) 730 and / or output device(s) 724 may also include interfaces for communicating with other computing devices. The interface(s) for communicating with other computing devices may receive and / or transmit data to a server and / or may receive input from and provide output to a user. The input and output functions of the interface(s) may be combined into a single node or may be split into separate input and output nodes. An input / output node may include one or more wired or wireless interfaces for communicating with other computing devices.

[0042] The system 700 may include a server computing system 750. The server computing system 750 may store and / or maintain user accounts 760. The user accounts 760 stored and / or maintained by the server computing system 750 may be associated with and / or owned by users. The server computing system 750 may store the notes 756 and / or annotations in association with each user account. The notes 756 and / or annotations may have the functionality described above and may be stored in association with the user account 760 and an annotation target, such as a URL. The server computing system 750 may prevent servers that host and / or maintain resources associated with the annotation target, such as server(s) / website(s) 770, from accessing the notes 756 and / or annotations.

[0043] The server computing system 750 can store session data 762 in association with the user account 760, allowing the user to initiate, terminate, and manage sessions while logged in using the user's user account 760. The server computing system 750 can store a profile 766 in association with the user account 760, including browsing patterns, bookmarks, and / or appearance preferences.

[0044] Server computing system 750 may include at least one processor(s) 752. The at least one processor(s) 752 execute instructions, such as instructions stored on at least one memory device(s) 754, to cause server computing system 750 to perform any combination of the methods, functions, and / or techniques described herein.

[0045] Server computing system 750 can include at least one memory device(s) 754. The at least one memory device(s) 754 can include a non-transitory computer-readable storage medium. The at least one memory device(s) 754 can store data and instructions that, when executed by at least one processor, such as processor(s) 752, are configured to cause server computing system 750 to perform any combination of the methods, functions, and / or techniques described herein. Thus, in any embodiment described herein (even if not explicitly mentioned in connection with a particular embodiment), software (e.g., processing modules, stored instructions) and / or hardware (e.g., processors, memory devices, etc.) associated with or included in server computing system 750 can be configured to perform any combination of the methods, functions, and / or techniques described herein, alone or in combination with computing system 702.

[0046] The system 700 can include service(s) / website(s) 770. The service(s) / website(s) 770 can be maintained and / or hosted by one or more remote servers. The service(s) / website(s) 770 can be addressed by a resource address (e.g., a URL) and / or store files addressed by the resource addresses (e.g., URLs). The computing system 702, the server computing system 750, and / or the service(s) / website(s) 770 can communicate with each other over a network 740, such as the Internet.

[0047] 8 is a block diagram of components for a browser application, such as browser application 718 of FIG. 7, that implements a system for providing a browser annotation interface, according to one aspect. Browser application 718 includes an annotation UI coordinator 805, which is a platform-specific component of browser process 850. In other words, annotation UI coordinator 805 is configured for a particular operating system and device type. Annotation UI coordinator 805: The annotation UI 805 may have two main components: a note creation UI 810 and a note rendering UI 815. The note creation UI 810 includes components configured to add new notes and / or annotations 837 and associate the new notes and / or annotations 837 with annotation targets 836 (e.g., source documents or tab groups). Exemplary note creation UI 810 components include 132-137 in FIG. 1A and / or suggested elements 432, 434, 436 in FIGS. 4A-4F. The note rendering UI 815 is configured to render annotation areas (e.g., annotation area 130) and / or preview areas (e.g., preview area 140) in which the notes and / or annotations 837 are displayed. In some implementations, the annotation UI coordinator 805 may also include a note sharing coordinator 820. Annotation UI coordinator 805 may communicate with annotation renderer coordinator 835, for example, sending an identifier for annotation 837 to annotation renderer coordinator 835. Upon receiving the identifier for annotation 837 from annotation UI coordinator 805, annotation renderer coordinator 835 may identify the content of the annotation source associated with the annotation identifier and display (render) that portion of annotation target 836 in browser content window 120. In some implementations, messages may be passed through annotation browser coordinator 825.

[0048] The browser application 718 can include an annotation browser coordinator 825. The annotation browser coordinator 825 is a cross-platform (i.e., operating system-independent) component of the browser process 850 that contains the application's business logic. The annotation browser coordinator 825 also maintains annotation data in private storage (e.g., local to the computing device on which the browser application 718 is running). In some examples, the annotation data itself may be excluded from the renderer process 855. In other words, the annotation data is rendered in a rendering process different from the renderer process 855, and the annotation data is not provided to the annotation renderer coordinator 835 or another renderer process. This separation of the rendering processes ensures security and privacy. For example, the renderer process 855 (e.g., the annotation renderer coordinator 835) can receive information about the annotation's ID and annotation target 836 (e.g., what to highlight fully, what to highlight partially, etc.). This mitigates the risk of a malicious renderer leaking sensitive user data within the annotation itself. In some implementations, upon a user's request, annotations 837 may be synchronized with the user's account, for example so that if the user uses multiple computing devices, the user's annotation data can be accessed from any computing device when the user is logged into the user account. The user can stop synchronization at any time.

[0049] In some embodiments, the locally stored annotation data (which may represent the annotation 837) may have a format based on the Web Annotations W3C Recommended Web Annotation Data Model. The annotation data may consist of three main parts: a metadata part, a body part, and a target part. The metadata part may include information such as an ID, a creation date, and a modification date. The ID can be used to communicate about a particular annotation between the annotation UI coordinator 805, the annotation renderer coordinator 835, and the annotation browser coordinator 825. The body part of the annotation 837 includes the text, images, rich text, links, etc. of the annotation 837. The body part of the annotation 837 is not shared with the rendering process. The annotation target 836 (target part) defines what the annotation 837 is attached to. The annotation target 836 can be a web page element, a web page, a tab group, a topic (a user-defined group), etc. In some embodiments, in addition to a web page as the annotation target 836, an annotation target element is associated with the annotation 837. The annotation target element can be identified via a selector. The selector is The annotation 837 may be a text fragment selector and / or fragment finder 838. The selector and / or fragment finder 838 may be a CSS selector. Because the annotation 837 may be linked to elements of the web page that may change, it is possible that the annotation renderer coordinator 835 may not be able to find a particular annotation target element. In some implementations, these “orphaned” notes and / or annotations may continue to be displayed in the annotation region 130 of the web page. In some implementations, such orphaned annotations may be displayed with an additional message indicating that the target element cannot be found. If the web page associated with a uniform resource locator (URL) is inaccessible, the annotation 837 for that annotation target 836 may be displayed as an orphaned note and / or annotation in the annotation region 130. For example, a notes menu option, such as menu option 124, allows a user to select notes and / or annotations for annotation targets 836 that are not associated with an open tab.

[0050] The annotation browser coordinator 825 can retrieve the associated annotations 837 for the current page from local storage, as described in more detail in Figure 9. The annotation browser coordinator 825 can interact with the annotation UI coordinator 805 and the annotation renderer coordinator 835 to appropriately display the notes and / or annotations. The annotation browser coordinator 825 can determine which tasks to send to the annotation UI coordinator 805 and which tasks to send to the annotation renderer coordinator 835.

[0051] The annotation browser coordinator 825 can include a private note coordinator 830. The private note coordinator 830 can be configured to save and retrieve annotation data (notes and / or annotations) from local storage 831. The local storage 831 can be included in memory 704 of the computing system 702 on which the browser is executing. The private note coordinator 830 can associate annotations with the annotation target (e.g., source document or tab group) to which the note is associated. The note coordinator 830 can index the annotation data so that it can be retrieved, for example, by annotation target, tab group, topic, etc. The note coordinator 830 can also be configured to synchronize 832 the annotation data so that it can be accessed on other devices if the user is signed in and synced.

[0052] The annotation browser coordinator 825 can include an inline note coordinator 833. The inline note coordinator 833 can receive, store, retrieve, and / or present notes and / or annotations within annotation targets 836 and / or web pages, rather than within annotation regions 130.

[0053] Annotation browser coordinator 825 can include a remote note coordinator 834. Remote note coordinator 834 can receive, store, retrieve, and / or present notes and / or annotations that are stored remotely in association with a user account, such as notes 745 on server computing system 750.

[0054] The annotation renderer coordinator 835 can be configured to find and highlight selectors to which various annotations are attached. To do this, the annotation renderer coordinator 835 can maintain a subset of annotation data, such as ID and selector information. The annotation browser coordinator 825 coordinates the annotation renderer coordinator 835 and the annotation UI coordinator 805 to smoothly link annotations 837 to their target selectors using annotation IDs (GUIDs) and synchronize operations. Specifically, the annotation UI coordinator 805 is responsible for the annotation body (which holds all the data), and the annotation renderer coordinator 835 is responsible for the annotation target 836 (the browser content window). The annotation browser coordinator 825 is responsible for rendering the annotation 837 in the browser content window 120. In some implementations, the annotation 837 may not be rendered next to the highlighted text. In some implementations, the annotation 837 may be rendered next to the highlighted text. In some implementations, when a user clicks on an annotation 837 in the side panel (e.g., annotation region 130), the annotation browser coordinator 825 can cause the annotation renderer coordinator 835 to enhance the highlight on the annotation target 836 and scroll the highlight on the annotation target 836 into view. However, when a user clicks on a lightly highlighted (partially highlighted) annotation target 836 (e.g., in the browser content window 120), the annotation browser coordinator 825 can cause the annotation UI coordinator 805 to focus on the associated annotation in the side panel (annotation region 130). This is accomplished by including the ID of the selected annotation in inter-process messages so that other processes can activate the appropriate annotation.

[0055] 9 is a timing diagram illustrating exemplary operations for loading an annotated Internet document, according to one aspect. A user 900 selects a web page by entering a resource address, such as a URL, into address entry field 115 and / or selecting or opening a hyperlink displayed in browser content window 120 (902). Browser application 718 responds to the selection of the web page by performing a URL load operation (904), (906). The URL load operation (904), (906) may include generating and sending a request, such as an HTTP GET request, for the web page identified and / or addressed by the URL to a server, such as a server maintaining service(s) / website(s) 770. Browser application 718, in conjunction with renderer 835, may render and display the web page received from the server.

[0056] After performing a URL load operation (904), (906), such as receiving and rendering a web page, the browser application 718 may retrieve (908) an annotation 837 related to and / or associated with the URL. The browser application 718 may retrieve the annotation 837, for example, from local note storage 726, 831 or from notes 756 stored on a remote server computing system in association with a user account associated with the user 900. The browser application 718 may send (910) a current page annotation target 836 to the renderer 835, which may include a portion of the rendered web page.

[0057] The browser application 718 may display the annotation region 130 (e.g., a side panel or an overlay) with the annotation 837 obtained at 908. In instances where the annotation 837 includes a text change within the web page, such as lightly highlighting a portion of text contained within the web page (the annotation target 836 may include a web page associated with a URL), the renderer 835 may lightly highlight the annotation target (912).

[0058] 10 is a timing diagram illustrating example operations for navigating using annotations, according to one aspect. In some examples, the operations shown and described with respect to FIG. 10 may be performed after the operations shown and described with respect to FIG.

[0059] The user 900 may select 1002, such as by clicking on, a first note (e.g., note A) on the annotation region 130 (e.g., in a side panel or overlay). The browser application 718 may respond to the selection of note A by focusing 1004 on annotation A in the annotation region 130, such as by making annotation A visible in the annotation region and / or side panel.

[0060] The browser application 718 may also respond to the selection of annotation A by sending a request (1006) to the renderer 835 to highlight and scroll to annotation A. The renderer 835 may respond to the request (1006) by scrolling to and highlighting (1008) the annotation target. The annotation target may include a portion of the web page (e.g., a first portion of text) highlighted at (912), and the annotation may include modifying the first portion of text by highlighting the portion of the text. The highlighting of the annotation target (1008) may be an example of performing an action on the web page based on the stored annotation, including modifying the web page.

[0061] The user 900 can scroll and click 1010 on a second annotation target, which can be considered annotation B. The annotation of the annotation target can be a first modification to a second portion of text contained in the web page associated with the URL. The first modification can include lightly highlighting the second portion of text. In one example, the annotation of annotation target B can include a second modification to the second portion of text, and the renderer 835 can respond to the user's click 1010 on the annotation B target by performing 1012 the second modification of the annotation B target. The second modification can include, but is not limited to, fully highlighting the second portion of text and / or a change that is visually noticeable to the user, such as making it darker, less transparent, bolder, or more saturated. Renderer 835 can send a request (1014) to browser application 718 to focus annotation B, and / or browser application 718 can receive a request (1014) to focus annotation B from renderer 835. Browser application 718 can then focus (1016) annotation B in annotation region 130 and / or the side panel, such as by displaying annotation B in the side panel and / or annotation region 130.

[0062] FIG. 11 illustrates an example of an exemplary computing device, according to one aspect. In some implementations, computing device 1500 is an example of computing system 702. In some implementations, computing device 1500 is an example of server computing system 750. In some implementations, mobile computing device 1550 is an example of computing system 702. Computing device 1500 is intended to represent various types of digital computers, such as laptops, desktops, tablets, workstations, personal digital assistants, televisions, servers, blade servers, mainframes, and other suitable computing devices. Computing device 1550 is intended to represent various forms of mobile devices, such as personal digital assistants, mobile phones, smartphones, and other similar computing devices. The components, their connections and relationships, and their functions illustrated herein are intended for illustration only and are not intended to limit the implementation of the embodiments of the invention described and / or claimed herein.

[0063] The computing device 1500 includes a processor 1502, a memory 1504, a storage device 1506, a high-speed interface 1508 connecting to the memory 1504 and a high-speed expansion port 1510, and a low-speed interface 1512 connecting to a low-speed bus 1514 and the storage device 1506. The processor 1502 may be a semiconductor-based processor. The memory 1504 may be a semiconductor-based memory. The components 1502, 1504, 1506, 1508, 1510, and 1512 are interconnected using various buses and may be mounted on a common motherboard or mounted in other manners as needed. The processor 1502 may be connected to the memory 1504 via a high-speed expansion port 1510. The computing device 1500 may process instructions for execution within the computing device 1500, including instructions stored on the BIOS or storage device 1506, and display graphical information for a GUI on an external input / output device, such as a display 1516 connected to a high-speed interface 1508. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and types of memory, where appropriate. Also, multiple computing devices 1500 may be connected, each performing some of the required operations (e.g., as a bank of servers, a group of blade servers, or a multiprocessor system).

[0064] The memory 1504 stores information within the computing device 1500. In one implementation, the memory 1504 is a volatile memory unit(s). In another implementation, the memory 1504 is a non-volatile memory unit(s). The memory 1504 may be another form of computer-readable medium, such as a magnetic disk or an optical disk.

[0065] The storage device 1506 can provide mass storage for the computing device 1500. In one embodiment, the storage device 1506 can be or include a computer-readable medium such as a floppy disk device, a hard disk device, an optical disk device, or an array of devices including a tape device, a flash memory or other similar solid-state memory device, or a device in a storage area network or other configuration. The computer program product can be tangibly embodied in an information medium. The computer program product can also include instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 1504, the storage device 1506, or memory on the processor 1502.

[0066] The high-speed controller 1508 manages bandwidth-intensive operations of the computing device 1500, while the low-speed controller 1512 manages less bandwidth-intensive operations. This allocation of functionality is merely exemplary. In some implementations, the high-speed controller 1508 is coupled to the memory 1504, the display 1516 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 1510 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 1512 is coupled to the storage device 1506 and the low-speed expansion port 1514. Various communication ports (e.g., USB, BLUETOOTH, The low-speed expansion ports, which may include Ethernet, wireless Ethernet, etc., can be coupled to one or more input / output devices, such as a keyboard, pointing device, scanner, or a network device such as a switch or router, for example, via a network adapter.

[0067] Computing device 1500, as shown in the figure, can be implemented in many different forms. For example, it can be implemented as a standard server 1520, or multiple times within a group of such servers. It can also be implemented as part of a rack server system 1524. Additionally, it can be implemented in a personal computer, such as a laptop computer 1522. Alternatively, components from computing device 1500 can be combined with other components in a mobile device (not shown), such as device 1550. Each such device can include one or more of computing devices 1500, 1550, and the entire system can be composed of multiple computing devices 1500, 1550 communicating with each other.

[0068] Computing device 1550 includes components such as a processor 1552, memory 1564, input / output devices such as a display 1554, a communications interface 1566, and a transceiver 1568, among other components. Device 1550 may also be provided with a storage device such as a microdrive or other device to provide additional storage. Each of components 1550, 1552, 1564, 1554, 1566, and 1568 are interconnected using various buses, and some of the components may be mounted on a common motherboard or otherwise mounted as needed.

[0069] The processor 1552 can execute instructions within the computing device 1550, including instructions stored in the memory 1564. The processor can be implemented as a chipset of chips including separate analog and digital processors. The processor can provide coordination of other components of the device 1550, such as control of a user interface, applications run by the device 1550, and wireless communications by the device 1550.

[0070] The processor 1552 can communicate with a user via a control interface 1558 and a display interface 1556 connected to a display 1554. The display 1554 can be, for example, a TFT LCD (thin film transistor liquid crystal display) or an OLED (organic light emitting diode) display, or other suitable display technology. The display interface 1556 can include appropriate circuitry for driving the display 1554 to present graphical and other information to the user. The control interface 1558 can receive commands from the user and convert them for transmission to the processor 1552. Additionally, an external interface 1562 can be provided in communication with the processor 1552, enabling short-range communication between the device 1550 and other devices. The external interface 1562 can, for example, provide wired communication in some implementations and wireless communication in other implementations; multiple interfaces can also be used.

[0071] Memory 1564 stores information within computing device 1550. Memory 1564 may be implemented as one or more of a computer-readable medium(s), a volatile memory unit(s), or a non-volatile memory unit(s). Expansion memory 1574 may also be provided and connected to device 1550 via expansion interface 1572, which may include, for example, a SIMM (single in-line memory module) card interface. Such expansion memory 1574 may provide additional storage space for device 1550 or may store applications or other information for device 1550. Specifically, expansion memory 1574 may include instructions for performing or supplementing the processes described above and may also include secure information. Thus, for example, expansion memory 1574 may be provided as a security module for device 1550 and may be programmed with instructions that enable secure use of device 1550. Additionally, secure applications can be provided via SIMM cards along with additional information, such as by placing identifying information on the SIMM card in an unhackable manner.

[0072] The memory may include, for example, flash memory and / or NVRAM memory, as described below. In one embodiment, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier may include computer-readable or external memory, such as memory 1564, expansion memory 1574, or memory on processor 1552, which may be received, for example, via transceiver 1568 or external interface 1562. is a machine-readable medium.

[0073] Device 1550 can communicate wirelessly via communication interface 1566, which may optionally include digital signal processing circuitry. Communication interface 1566 can provide for communication in various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication can occur, for example, via radio frequency transceiver 1568. In addition, BLUETOOTH (registered trademark), Wi-Fi (registered trademark), or other such transceiver (not shown) In addition, a GPS (Global Positioning System) receiver module 1570 may provide navigation to the device 1550. Additional location- and position-related wireless data may be provided and used as appropriate by applications running on device 1550.

[0074] Device 1550 may also perform voice communications using audio codec 1560, which receives voice information from a user and converts it into usable digital information. Audio codec 1560 may similarly generate sounds that are audible to the user, such as through a speaker on a handset or the like of device 1550. Such sounds may include voice telephone calls, recorded sounds (e.g., voice messages, music files, etc.), and sounds generated by applications running on device 1550.

[0075] The computing device 1550, as shown in the figure, can be implemented in many different forms. For example, it can be implemented as a mobile phone 1580. It can also be implemented as part of a smartphone 1582, a personal digital assistant, a tablet, a wearable 1590, or another similar mobile device.

[0076] In some implementations, the illustrated computing device may include sensors that interface with a wearable (e.g., AR headset / HMD) device 1590 to generate an augmented environment for displaying inserted content within a physical space. For example, one or more sensors included in the illustrated computing device 1550 or other computing devices may provide input to the AR headset 1590 or to the AR space generally. The sensors may include, but are not limited to, a touchscreen, an accelerometer, a gyroscope, a pressure sensor, a biometric sensor, a temperature sensor, a humidity sensor, and an ambient light sensor. The computing device 1550 may use the sensors to determine the absolute position and / or detected rotation of the computing device within the AR space and use it as input to the AR space. For example, the computing device 1550 may be incorporated into the AR space as a virtual object such as a controller, a laser pointer, a keyboard, a weapon, etc. User positioning of the computing device / virtual object when incorporating it into the AR space allows the user to position the computing device to display the virtual object in a particular manner in the AR space. For example, if the virtual object represents a laser pointer, the user can manipulate the computing device as if it were a real laser pointer. The user can move the computing device left and right, up and down, in circles, etc., in a manner similar to using a laser pointer. In some implementations, the user can aim the virtual laser pointer at a target location.

[0077] In some implementations, the computing device 1550 may include or One or more input devices connected to the computing device 1550 can be used as input to the AR space. The input devices can include, but are not limited to, a touchscreen, a keyboard, one or more buttons, a trackpad, a touchpad, a pointing device, a mouse, a trackball, a joystick, a camera, a microphone, earphones or earbuds with input capabilities, a game controller, or other connectable input devices. When the computing device is integrated into the AR space, a user manipulating an input device included in the computing device 1550 can cause specific actions in the AR space.

[0078] In some implementations, the touchscreen of the computing device 1550 can be rendered as a touchpad in the AR space. A user can manipulate the touchscreen of the computing device 1550. The manipulations are rendered, for example, in the AR headset 1590, as movements on the rendered touchpad in the AR space. The rendered movements can control virtual objects in the AR space.

[0079] In some implementations, one or more output devices included in computing device 1550 can provide output and / or feedback to a user of AR headset 1590 within the AR space. The output and feedback can be visual, tactical, or audio. The output and / or feedback can include, but is not limited to, vibration, turning one or more lights or strobes on and off or blinking and / or flashing, sounding an alarm, playing a chime, playing a song, playing an audio file, etc. Output devices include, but are not limited to, vibration motors, vibration coils, piezoelectric devices, electrostatic devices, light-emitting diodes (LEDs), strobes, speakers, etc.

[0080] In addition to the above, users may be provided with controls that allow them to make choices about both whether and when the systems, programs, or features described herein may enable the collection of user information (e.g., information about the user's social network, social behavior, or activities, occupation, user preferences, or the user's current location), as well as whether content or communications are sent from the server to the user. Additionally, certain data may be processed in one or more ways so that personally identifiable information is removed before it is stored or used. For example, the user's identity may be processed so that personally identifiable information about the user cannot be determined, or if location information is obtained (e.g., to the city, zip code, or state level), the user's geographic location may be generalized so that the user's specific location cannot be identified. Thus, users may control what information is collected about them, how that information is used, and what information is provided to them.

[0081] Various implementations of the systems and techniques described herein may be realized in digital electronic circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, which may be specialized or general-purpose, coupled to receive data and instructions from the storage system, and to transmit data and instructions to the storage system.

[0082] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be written in high-level procedural and / or object-oriented programming languages, and and / or assembly / machine language. As used herein, "machine-readable medium" or "computer-readable medium" refers to any computer program product, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic circuit (PLD)) used to provide machine instructions and / or data to a programmable processor that includes a machine-readable medium that receives the machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0083] To provide for user interaction, the systems and techniques described herein may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices may also be used to provide for user interaction. For example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user may be received in any form, such as acoustic, verbal, or tactile input.

[0084] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or that includes middleware components (e.g., an application server), or that includes front-end components (e.g., a client computer having a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a combination of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0085] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0086] As used herein and in the appended claims, the singular forms "a," "an," and "the" do not exclude plural referents unless the context clearly dictates otherwise. Furthermore, conjunctions such as "and," "or," and "and / or" are inclusive unless the context clearly dictates otherwise. For example, "A and / or B" includes A only, B only, and A and B. Furthermore, connecting lines or connectors shown in the various illustrated figures are intended to illustrate example functional relationships and / or physical or logical couplings between the various elements. In an actual device, many alternative or additional functional relationships, physical connections, or logical connections may be present. Furthermore, unless an element is specifically described as "essential" or "critical," no item or component is essential to the practice of the embodiments disclosed herein.

[0087] Without limitation, terms such as "approximately," "substantially," and "generally" are used herein to indicate that an exact value or range thereof is not required and need not be specified. As used herein, the above terms have meanings that are readily apparent to one of ordinary skill in the art.

[0088] Additionally, terms such as top, bottom, upper, lower, side, end, front, back, etc., as used herein are used with reference to the orientation currently considered or illustrated, and it should be understood that when these are considered from another orientation, such terms should be changed accordingly.

[0089] Although certain exemplary computer-implemented methods, apparatus, and articles of manufacture have been described herein, the scope of this patent is not limited thereto. It is to be understood that the terminology used herein is for the purpose of describing particular aspects and is not intended to be limiting. To the contrary, this patent covers all methods, apparatus, and articles of manufacture fairly falling within the scope of the claims of this patent.

[0090] In one aspect, a non-transitory computer-readable medium stores instructions that, when executed by a processor on a receiving computing device, cause the receiving computing device to perform any of the methods disclosed herein.

[0091] In one aspect, a computing device may be configured with at least one processor and a memory that stores instructions that, when executed by the at least one processor, perform any of the methods disclosed herein.

Claims

1. A method implemented by a browser, comprising: providing at least a portion of a resource address; presenting the rendered web page in a browser content window, the rendered web page being associated with the resource address, the method further comprising: presenting an annotation area, the annotation area being outside the browser content window, the method further comprising: receiving an annotation in the annotation area in association with a user account; and storing the annotation in association with the user account and the resource address.

2. storing the annotation includes storing the annotation locally in long-term storage in association with the user account and the resource address; The method of claim 1 , wherein the rendered web page is not stored locally in the long-term storage.

3. The method of claim 1 or claim 2, wherein the annotation causes a change in appearance of a portion of text contained in the rendered web page.

4. The method of claim 3 , wherein the modification is performed using a rendering process different from the rendering process that rendered the web page.

5. the annotations include a first annotation; storing the first annotation includes storing the first annotation in association with the user account and the resource address; The method further includes receiving a second annotation in the annotation area associated with the user account, the second annotation being for a tab group; 10. The method of any preceding claim, wherein the method further comprises storing the second annotation in association with an identifier of the user account and the tab group.

6. 10. The method of any preceding claim, wherein storing the annotation occurs without notifying a server associated with the resource address.

7. 10. The method of any preceding claim, wherein the annotations are stored on a local computing device on which the browser is running.

8. 10. The method according to any preceding claim, wherein the annotations are stored in annotation records comprising a metadata part, a body part and a target part.

9. the resource address is a first resource address; the rendered web pages include a first rendered web page in a first tab; the annotations include a first annotation; The method comprises: Detecting a preselection action on a second resource address of a second tab; 6. The method of claim 1, further comprising: in response to detecting the pre-selection action, presenting a preview area of ​​the second resource address, the preview area including a second annotation of the second resource address.

10. receiving a request to display a web page associated with the resource address; retrieving the annotations in response to receiving the request to display the web page; In response to retrieving the annotations, adding a notice to the address bar area indicating that the web page has at least one annotation; receiving a selection of the notification; 10. The method of any preceding claim, wherein presenting the annotation area is in response to receiving the selection of the notification.

11. receiving a selection of the annotation; 10. The method of any preceding claim, further comprising, in response to receiving the selection of the annotation, scrolling to a portion of the rendered web page associated with the annotation.

12. 10. The method of any preceding claim, wherein receiving the annotation comprises receiving a selection of an image to be included in the rendered web page.

13. analyzing content of the web page to identify suggested annotation content; presenting the proposed annotation content as a selectable element; receiving a selection of the suggested annotation content; 10. The method of any preceding claim, wherein receiving the annotation occurs in response to the selection of the suggested annotation content.

14. 14. A non-transitory computer-readable storage medium comprising instructions, the instructions being stored on the non-transitory computer-readable storage medium and configured to cause a computing system, when executed by at least one processor, to perform a method according to any of claims 1 to 13.

15. 1. A computing system comprising: at least one processor; and a non-transitory computer-readable storage medium containing instructions, the instructions being stored on the non-transitory computer-readable storage medium and configured, when executed by the at least one processor, to cause the computing system to perform a method according to any one of claims 1 to 13.

Images