Methods for creating, organizing, displaying, and connecting document annotations within a web browser, computer-readable media, and systems.

Abstract

This invention provides a method, medium, and system for generating text for identifying similar information and for automatic annotation of web documents. [Solution] An exemplary method is for a teacher to visit the Wikipedia page for Abraham Lincoln on the first day and create question annotations, visit the Wikipedia page for Woodrow Wilson and create question annotations, visit Zillow's page on the second day and create dagger annotations for houses for sale near the workplace, set up two collections, the American Presidents Study Guide and Real Estate, add the student to the Guides collection and the roommate to the Real Estate collection, move the question annotations for Abraham Lincoln and Woodrow Wilson to the Guides collection and move the dagger annotations for houses for sale near the workplace to the Real Estate collection and share the Guides collection with the student.

Description

Background Art

[0001] The World Wide Web (WWW), web browsers, and browser extensions are well known. The WWW is a global information source in which assets such as web pages, PDF documents, images, and videos are identified by a unique Uniform Resource Locator (URL). A web browser is a software application designed for accessing information on the WWW. Furthermore, browser extensions are software created to extend the functionality of browsers. It is known that the functionality of a web browser can be modified by modifying the software codebase of the web browser, or by providing a web browser extension for the web browser that supports adding functionality through such means.

[0002] As the popularity of the World Wide Web has increased, on average people now spend more time each day using the World Wide Web than in the past. Due to the increase in the average time spent using the World Wide Web, people have growing needs for: searching for information they have previously viewed on the WWW; sending and discussing snippets of text within web documents with friends and colleagues; referencing specific text fragments or contexts within web documents; asking specific questions about parts of web documents; connecting information between web documents; finding and organizing content and snippets for later use; evaluating the value of crowdsourced information; and accessing other people's public comments on web documents and placing these comments in context, among other needs.

[0003] To address these use cases, people typically repeatedly search to find previously accessed content, bookmark web documents, rescan articles for snippets to recall what they've seen, copy and paste sections of text into email messages or other documents to share with friends and colleagues, refer to sentences or paragraphs by number or webpage location while discussing on the phone, ask questions in the comment section of an article, cut and paste within comments to reference desired text, download web pages, or copy snippets into personal word processing or spreadsheet documents. This results in inefficiency and wasted time for many users. Furthermore, the lack of tools to identify inaccurate information and disseminate this knowledge to other users allows inaccurate information to persist longer than necessary. Providing methods, computer-readable media, and systems to handle these tasks more efficiently would be beneficial. [Overview of the project]

[0004] The object of the present invention is to provide users with an annotation-enabled web browser that allows users in other locations on the Internet to annotate web pages and documents, enabling them to identify questions, points of interest, statements they agree with, and statements they disagree with, without having to read user annotation comments or annotated highlighted text. Such a mechanism improves the efficiency with which users sift through the vast amount of information on the WWW.

[0005] A further object of the present invention is to provide users with an annotation-enabled web browser for annotating images, videos, and audio recordings uploaded or embedded on web pages, so that users in other locations on the Internet can also view these image, video, and audio annotations using such an annotation-enabled web browser.

[0006] Another objective of the present invention is to ensure that, after annotations are added, the document viewing context is modified in the least amount of way compared to a document viewing context without annotations. By using a document viewing context that is modified in the least amount of way, users can consume content without being distracted by annotations, and at the same time find and use the annotations at the appropriate time.

[0007] A further objective of the present invention is to present advertisements to users in a non-intrusive manner. One objective is to show advertising content to users after they have initiated an action triggered by curiosity, which is triggered by the presentation of the advertisement in a specific location or context within a web page. This location or context suggests that the advertisement may be relevant to the user's current needs. Furthermore, an advertisement presented without content may contain several aspects of a complete advertisement, but the full details are not displayed until the user initiates an action triggered by curiosity, which has been planted by a contextual method that selectively and transparently targets the user based on their current needs.

[0008] Another objective of the present invention is to provide an annotation browsing context area separate from the World Wide Web document browsing context area. This annotation browsing context area has a mechanism to present annotations to the user so that, in addition to search and filtering, the user can understand the approximate location of familiar sites and annotations. Furthermore, other elements of the context of annotations and highlighted text may become clear without reading the highlighted text or user annotation comments. This mechanism makes it easier to find a specific annotation among a large number of annotations.

[0009] A further objective of the present invention is to provide an annotation organization context area that enables users to scan annotations and organize them into meaningful groups. The advantage of this object is that it allows users to focus on creating annotations initially and then on organizing them after they have been created. Furthermore, the default organization of annotations does not need to be undone.

[0010] Another objective of the present invention is to provide the user with an annotation display context area that allows the user to view a page of annotations, keep some annotations available for later review, and continue searching for other annotations. Furthermore, it is a way to facilitate mental tracking by keeping annotations in the visual view while moving annotations of no interest out of the visual view.

[0011] An additional objective of the present invention is to always communicate privacy settings transparently to the user. By communicating privacy settings, it is possible to avoid accidentally sharing information that the user does not want to share.

[0012] A further object of the present invention is a method for creating a guided tour through a series of websites on the global internet, while simultaneously drawing the user's attention to notable points of interest at each site. This object further includes a method for modifying the tour and presenting it to the user.

[0013] Another object of the present invention is to provide a method by which a group manages access control permissions and communicates through messaging. [Brief explanation of the drawing]

[0014] For a more complete understanding of the present invention, refer to the following description and accompanying drawings.

[0015] Figure 1 shows a flowchart illustrating how annotations are created, moved to two different collections, and shared with others through those collections.

[0016] Figure 2 shows a flowchart of annotations viewed on two separate web pages on two separate days.

[0017] Figure 3 represents an annotable web browser in an exemplary embodiment where a web browser that lacks the ability to create annotations before the annotation process begins is extended with a web browser extension that provides annotation capabilities.

[0018] Figure 4 shows an annotable web browser after text has been highlighted.

[0019] Figure 5 shows an annotable web browser before annotation symbols are selected.

[0020] Figure 6 shows the annotable web browsers after an annotation symbol has been selected.

[0021] Figure 7 shows an annotable web browser after annotations have been created and the displayed annotation symbols have been opened.

[0022] Figure 8 shows the annotable web browsers after annotation has been performed. This creates and displays the annotation symbol, indicating that it is in a closed state.

[0023] Figure 9 shows an annotation item box that provides context for annotations when it is presented in a separate context area rather than as an overlay on an existing web document adjacent to the highlighted annotation text.

[0024] FIG. 10 shows an annotation organization context region that presents a writable collection displayed with an annotation item box within a context region separate from the original web document containing the annotation emphasized text.

[0025] FIG. 11 shows an annotation organization context area showing a process of moving and copying annotations into a collection, which is a grouping of annotations and other items.

[0026] FIG. 12 shows an annotatable web browser displaying a web document having annotation symbols displayed in a closed state.

[0027] FIG. 13 shows an annotatable web browser that displays a web document with displayed annotation symbols in a closed state by corresponding displayed annotation symbol beacons.

[0028] FIG. 14 is a sequence diagram showing communication among an annotatable web server, a web server providing data of a visited URL, an annotation server, an electronic database, and a CDN.

[0029] FIG. 15 is a sequence diagram showing communication among a web browser for organizing annotations, an annotation server, an electronic database, and a CDN.

[0030] FIG. 16 is a sequence diagram showing communication among an annotatable web browser, a web server providing data of a visited URL, an annotation server, and an electronic database.

[0031] FIG. 17 is a flowchart showing receiving and storing a bundle of data elements.

[0032] FIG. 18 is a flowchart showing compiling and transmitting a bundle display set.

[0033] Figure 19 shows a flowchart illustrating the interactions between a user, a web browser, a web browser extension content script, and a web browser extension background script, which enable the user to view a web page and create annotations on the page.

[0034] Figure 20 shows a flowchart illustrating how the user, web browser, web browser extension background script, and web browser extension content script interact to enable the user to view a webpage with annotations on the page.

[0035] Figure 21 shows a flowchart illustrating how a nurse using this exemplary embodiment can create a notepad on glycemic index to share with a patient.

[0036] Figure 22 shows a flowchart illustrating how patients use the notepad provided by the nurse.

[0037] Figure 23 shows the user interface of the notepad within the extension.

[0038] Figure 24 shows a sequence diagram illustrating how a notepad is created and how one annotation is created and added to the notepad.

[0039] Figure 25 shows a sequence diagram illustrating how Notepad is viewed, the first annotation is clicked, the browser navigates to its URL, scrolls to the appropriate location, and then displays the annotation.

[0040] Figure 26 shows a flowchart illustrating how a notepad is created and how previously created annotations are added to it.

[0041] Figure 27 shows a flowchart illustrating how a notepad is viewed and how the notepad annotations are opened.

[0042] Figure 28 shows an exemplary embodiment of the system diagram.

[0043] Figure 29 shows another exemplary embodiment of a component diagram for an annotation server and an electronic database.

[0044] Figure 30 shows an exemplary embodiment illustrating a client electronic device for an annotation server.

[0045] Figure 31 presents an exemplary embodiment of a collection item page for an American history collection.

[0046] Figure 32 shows an exemplary embodiment of a collection page.

[0047] Figure 33 shows an exemplary embodiment of an inbox page displaying system messages.

[0048] Figure 34 shows an exemplary embodiment of the inbox page after a user has responded to a system message and other users have also sent messages.

[0049] Figure 35 shows the image annotation display and image annotation group creation interfaces.

[0050] Figure 36 shows the image annotation group creation interface where one annotation group list item has been added to the annotation group candidate list.

[0051] Figure 37 shows the image annotation group creation interface with one annotation group list item where the second non-collision image annotation has not been added to the candidate annotation group list.

[0052] Figure 38 shows the image annotation group creation interface where two annotation group list items have been added to the annotation group candidate list.

[0053] Figure 39 shows the image annotation group creation interface where one annotation group list item has been added to the candidate annotation group list, and a second collision image annotation has not been added to the candidate annotation group list.

[0054] Figure 40 shows the video annotation viewing and video annotation group creation interfaces.

[0055] Figure 41 shows the video annotation display and video annotation group creation interface with one video annotation added to the annotation group candidate list box that has a motion path for the video annotation element.

[0056] Figure 42 shows the video annotation group creation interface with two annotation group list items added to the candidate annotation group list, which has a motion path for a second video annotation element.

[0057] Figure 43 shows the video annotation group creation interface when one annotation group list item has been added to the candidate annotation group list, and the second collision video annotation has not yet been added to the candidate annotation group list.

[0058] Figure 44 shows the timeline sound annotation and derived sound annotation creation interface, as well as the sound annotation viewing and sound annotation group creation interface.

[0059] Figure 45 shows the video annotation display and video annotation group creation interfaces, which have timeline video annotation and derived video annotation creation interfaces.

[0060] Figure 46 shows an annotable web browser displaying a video with an embedded video annotation beacon.

[0061] Figure 47 shows an annotation-enabled web browser after an image annotation has been created and the embedded, displayed image annotation is open.

[0062] Figure 48 shows an annotable web browser after video annotation has been created, indicating that the embedded and displayed video annotation remains open throughout the video annotation playback range.

[0063] Figure 49 shows an annotable web browser after an audio annotation has been created and the embedded audio annotation is open during the playback range of the audio annotation. [Modes for carrying out the invention]

[0064] A method, non-temporary computer-readable media (CRM), and system are provided for annotating web pages, documents, images, videos, and recordings within a web browser, organizing these annotations, viewing these annotations, and linking these annotations to a useful digital notepad containing a set of annotations. Web documents include web pages, PDF documents, and other document types that have URLs and are displayed in a web browser. Furthermore, these methods, non-temporary CRM, and system provide facilities for content curation, content sharing with other users, and access control management. This specification includes flowcharts illustrating the user rationale for using this system and non-temporary CRM. Furthermore, graphical user interface (GUI) diagrams show the final result of computer-readable media instructions and what the system displays. Furthermore, sequence diagrams show the interactions and data flows between various components, and flowcharts explain the methods. Furthermore, swimlane diagrams show the operation of various components.

[0065] Annotation refers to explanatory notes, comments, or drawings added to text, diagrams, images, documents, videos, or recordings. When describing methods, non-temporary CRM, and systems, the term is introduced to describe annotations in separate contexts, along with data elements and structures to represent annotations and objects that are added to the context of annotations in separate context areas, as well as within web pages, web documents, documents, images, videos, and recordings.

[0066] Figure 1 shows a flowchart of a teacher using an exemplary embodiment of an annotable web browser. This teacher has two distinct goals. The first goal is to create a list of learning questions about presidents for the students in the teacher's class. The teacher's class is small, and in this example within the exemplary embodiment depicted, the students can see each other's answers. The second goal is to find a house for sale near the school where the teacher teaches. As shown in Figure 1, on day one, the teacher visits the Wikipedia page for Abraham Lincoln and creates a question annotation [4], then visits the Wikipedia page for Woodrow Wilson and creates a question annotation [5]. As shown in Figure 1, on day two [2], the teacher visits the Zillow page and creates a dagger annotation [6] for a house for sale near the workplace. As shown in Figure 1, on day 10, the teacher[3] sets up two collections, the U.S. Presidents Study Guide and Real Estate[7], adds a student to the U.S. Presidents Study Guide collection and a roommate to the Real Estate collection[8], moves the Abraham Lincoln and Woodrow Wilson question annotations to the U.S. Presidents Study Guide[9], moves the dagger annotations about houses for sale near the workplace to the Real Estate collection

[10] , shares the U.S. Presidents Study Guide collection with the student

[11] , and further adds a roommate to both the U.S. Presidents Study Guide and the Real Estate collections.

[0067] Figure 2 shows a flowchart of a student using an exemplary embodiment of an annotable web browser. The student's goal is to answer a learning question provided by the teacher. As shown in Figure 2, on day 11, the student

[12] visits the Wikipedia page for Abraham Lincoln

[14] , reads the Wikipedia page for Abraham Lincoln

[15] , sees the displayed annotation symbol for the teacher's closed question while reading the Wikipedia page for Abraham Lincoln

[16] , clicks the displayed annotation symbol for the teacher's closed question to open the displayed annotation symbol for the question and display the teacher's annotation comment

[17] , and responds to the teacher's question annotation comment with an answer.

[18] As shown in Figure 2, on the 12th day,

[13] a student visits the Wikipedia page for Woodrow Wilson,

[19] reads the Wikipedia page for Woodrow Wilson,

[20] sees the annotation symbol for a closed question from the teacher while reading the Wikipedia page for Woodrow Wilson,

[21] clicks on the annotation symbol for a closed question from the teacher, which opens the annotation symbol for the question,

[22] and the student responds to the teacher's question annotation comment with an answer.

[23]

[0068] Figure 3 shows an exemplary embodiment of an annotation-enabled web browser

[24] . The horizontally elongated, short rectangles displayed in the web browser's display area represent text. The first paragraph has 11 rectangles representing 11 lines of text, and the second paragraph has 5 rectangles representing 5 lines of text. In this exemplary embodiment, the web browser that supports the addition of web extensions is modified with an exemplary example of a web browser extension

[25] that adds annotation functionality. A web browser extension

[25] is a software package that typically exists in a non-temporary, computer-readable medium that can be loaded into memory by a web browser, supports the addition of web extensions, and the web browser application itself is a software application that is loaded into memory and executed on the processor of a computer. Once the web browser application loads a web browser extension

[25] into memory, the instruction set of the web browser extension

[25] can be triggered by the instruction set of the web browser application at a specific point in the web browser's lifecycle. Some of the effects of the web browser extension

[25] instruction set on the user interface presented to the user are shown in Figures 5, 6, 7, 8, 12, 13, 23, 46, 47, 48, and 49.

[0069] In Figure 4, an exemplary embodiment of an annotable web browser

[24] shows the period after a user has selected text, referred to as regular highlighted text

[26] , before a set of instructions for a web browser extension

[25] is triggered.

[0070] Figure 5 shows an exemplary embodiment of an annotable web browser

[24] after the user has selected text, triggered a web browser extension

[25] instruction set, and displayed an annotation symbol selection button bar

[28] . Furthermore, the selected text will be referred to as annotated highlighted text

[27] to distinguish it from ordinary highlighted text

[26] , as shown in the exemplary embodiment depicted in Figure 4. Ordinary highlighted text

[26] is the selected text before the web browser extension

[25] instruction set is triggered, or the selected text in a web browser that does not have annotation capabilities. The web browser extension

[25] instruction set processes the input after the user clicks one of the buttons shown on the annotation symbol selection button bar

[28] in Figure 5.

[0071] In an exemplary embodiment, Figure 6 illustrates the effect of a user clicking a question mark annotation symbol button in the annotation symbol selection button bar

[28] shown in Figure 5. In an alternative exemplary embodiment, the annotation symbols shown in the annotation symbol selection button bar

[28] of Figure 5 may be arranged in vertical columns, as opposed to horizontal rows. In the exemplary embodiment depicted in Figure 6, an open question annotation symbol

[29] and an annotation comment input box

[30] are shown. The open question annotation symbol

[29] may indicate that the user intends to enter a question about the annotation highlighted text

[27] into the annotation comment input box

[30] . The annotation comment input box

[30] is an input area for a user of an annotationable web browser

[24] to enter text stored on one or more computer systems. When the user finishes typing text and signals the annotable web browser

[24] that the text typing action is complete, a set of instructions for the web browser extension

[25] causes the annotable web browser

[24] to transition to the state shown in Figure 7.

[0072] In the exemplary embodiment depicted in Figure 7, an annotation is created and the displayed annotation symbol is visually shown in an open state, followed by the state of an annotable web browser

[24] . The displayed annotation symbol in the open state displays screen elements such as annotation highlighted text

[27] , annotation comments

[32] , a reputation selector

[33] , an open question displayed annotation symbol

[29] , and an annotation comment box

[31] . The annotation comments

[32] are derived from text entered by the user, as previously mentioned. The annotation comment box

[31] also includes a context variant annotation control bar

[34] . The context variant annotation control bar

[34] allows the user to change privacy settings, share via social media or email, flag annotations, and delete annotations according to access control rules. The reputation selector

[33] allows the user to select a rating to assign to the annotation comment

[32] . For example, a user can assign a thoughtful rating to an annotation comment

[32] if they find the comment thoughtful. By clicking outside the annotation highlighting text

[27] , the displayed annotation symbol for an open question

[29] , or the annotation comment box

[31] , the user can close the annotation by transitioning the displayed annotation symbol to a closed state. In another embodiment, the reputation selector

[33] may appear inside the context variant annotation control bar

[34] , as opposed to being located adjacent to the annotation comment

[32] .

[0073] In an exemplary embodiment depicted in Figure 8, the state of an annotable web browser

[24] after annotation has been created and the displayed annotation symbol is shown in a closed state. Figure 8 shows the displayed annotation symbol

[35] of a closed question. The displayed annotation symbol

[35] of a closed question is visible, but is designed so as not to distract the user from the rest of the text.

[0074] After a user creates an annotation using an annotation-enabled web browser

[24] , a bundle of data elements representing the annotation is sent to the server. This bundle of data elements can be used to construct an annotation item box

[36] in a separate context area, as shown in the exemplary embodiment depicted in Figure 9. In the exemplary embodiment depicted in Figure 9, the annotation item box

[36] includes a screenshot

[37] with marked text, a separate contextual annotation highlighting text

[38] , a separate contextual question displayed annotation symbol

[39] , a separate contextual annotation comment

[40] , a separate contextual variant annotation control bar

[41] , and a separate contextual reputation selector

[42] . In this exemplary embodiment, the screenshot

[37] with marked text is a thumbnail-sized, relevant marked screenshot intended to allow the user to determine the web page on which the annotation resides and the location of the annotation within the web page. Figures 7 and 9 can be viewed as two different contexts of the same exemplary embodiment, or as two different exemplary embodiments. For the purpose of understanding the separate contextual annotation highlighting text

[38] , the separate contextual question displayed annotation symbol

[39] , and the separate contextual annotation comment

[40] , it is useful to view these two figures as two different contexts of the same exemplary embodiment. The separate contextual annotation highlighting text

[38] presents the annotation highlighting text

[27] in a separate context area of ​​the annotation item box

[36] . The description of the separate context area in this situation indicates that the context is different from the original web page or web document in which the annotation highlighting text

[27] originates. The separate contextual question displayed annotation symbol

[39] presents the question annotation symbol in a separate context area of ​​the annotation item box

[36] .A separate contextual annotation comment

[40] presents the annotation comment

[32] in a separate context area of ​​the annotation item box

[36] . A separate contextual variant annotation control bar

[41] allows the user to change privacy settings, share on social media or email, associate classification ratings with annotations, or delete annotations.

[0075] When an item box represents an annotation, it is called an annotation item box

[36] , and is depicted in the exemplary embodiment of Figure 9. In the exemplary embodiment, an item box may represent a web page, document, image, video, or recording, as well as annotations for a web page, document, image, video, or recording. Item boxes may be web page item boxes, document item boxes, image item boxes, video item boxes, audio item boxes, annotation item boxes, image and annotation item boxes, video and annotation item boxes, audio and annotation item boxes, embedded image and annotation item boxes, embedded video and annotation item boxes, embedded audio and audio item boxes, annotation group item boxes, derived annotation group item boxes, and derived annotation-attached annotation group item boxes. A collection is a representative of a group of items containing annotations, and a writable collection is a collection that a user can modify in some way. One form of modification is adding items to a writable collection. Collections are used to organize items containing annotations into meaningful groups. As an example, in Figure 1, the teacher created two collections, one titled "American Presidents Study Guide" and the other titled "Real Estate."

[0076] In the exemplary embodiment depicted in Figure 9, the annotation item box

[36] includes a separate contextual reputation selector

[42] in a separate contextual variant annotation control bar

[41] , allowing users sharing annotations in the form of the annotation item box

[36] to vote on their reputation by selecting from a dropdown. This reputation is assigned to the annotation. This reputation has value within the social community. Annotations with reputations such as useful, interesting, or thoughtful, or annotations expressing dissent against the annotation-emphasized text, can be selectively displayed based on various criteria selected by the user and deemed useful by the annotation server

[61] , as shown in several other figures. Users can filter annotations based on these criteria, and a user's aggregated reputation can also be linked to the reputation of annotations created by the user. In contrast to star-based voting, detailed annotation-specific semantic information about why an annotation is valuable to other users in the online community can be collected from the user.

[0077] Furthermore, annotation systems function better when users can reply to annotations, in addition to giving them a reputation. Other users can also vote on the reputation of replies. This system helps the community identify potentially inaccurate annotations, as users provide feedback through replies and reputations.

[0078] Both the exemplary embodiments depicted in Figure 7 and Figure 9 represent annotations. Figure 7 represents an annotation embedded within a web document that it is annotating, while Figure 9 represents an annotation outside of a web document. Since Figure 9 lacks the context surrounding the web document, a screenshot with a thumbnail mark is provided to give users viewing the annotation item box

[36] more context. Figures 7 and 9 could potentially depict two contexts of the same exemplary embodiment, or they could potentially depict two different exemplary embodiments. Both Figures 7 and 9 enable reputation voting. In Figure 7, a reputation selector

[33] allows users to assign reputation to annotation comments

[32] . In Figure 9, another context reputation selector

[42] allows users to assign reputation to annotation comments

[32] . This allows users sharing annotations to evaluate the reputation of the annotation from different contexts.

[0079] In the exemplary embodiment depicted in Figure 9, the annotation item box

[36] includes a separate contextual variant annotation control bar

[41] that allows the user to change privacy settings. In the exemplary embodiment, dark gray indicates the most private setting, where no one other than the annotation creator can see the annotation; light gray indicates that the annotation can be shared with a specific group of users; and white indicates that the annotation can be seen by all users on the platform.

[0080] One of the separate context areas that can render the annotation item box

[36] depicted in Figure 9 is the annotation organization context area

[43] of the exemplary embodiment depicted in Figure 10. The annotation organization context area

[43] in Figure 10 also includes an item bin

[44] for displaying item boxes and a writable collection bin

[45] for displaying writable collections. In Figure 9, the annotation item box

[36] is depicted in the item bin

[44] and the writable collection

[50] displayed in the writable collection bin

[45] is depicted. Furthermore, a search filter

[46] can be used to change the item boxes displayed in the item bin

[44] . Furthermore, a collection search filter

[47] can be used to change the display of writable collections displayed in the writable collection bin

[45] . Furthermore, a paging mechanism

[48] can be used to change the page of item boxes displayed in the item bin

[44] , and a collection paging mechanism

[49] can be used to change the collection page of displayed writable collections displayed in the writable collection bin

[45] .

[0081] In the exemplary embodiment depicted in Figure 10, an item bin

[44] has a paging mechanism

[48] , and a writable collection bin

[45] has a collection paging mechanism

[49] . These paging mechanisms operate independently. The paging mechanism

[48] in the item bin

[44] moves forward and backward through pages of six items in the exemplary embodiment shown. The items shown in Figure 10 represent annotations. All of these items are arranged sequentially. Moving the paging mechanism forward displays the next six items of the sequentially arranged items. Moving the paging mechanism backward displays the previous six items in the sequentially ordered set of items. The collection paging mechanism

[49] in the writable collection bin

[45] moves forward and backward through pages of three collections in the exemplary embodiment shown. The collections shown in Figure 10 represent groups of items containing annotations. Each collection represents a group of one or more items, including web page items, document items, image items, video items, audio items, annotation items, image annotation items, video annotation items, audio annotation items, embedded image annotation items, embedded video annotation items, embedded audio items, annotation group items, derived annotation group items, and annotation groups with derived annotation items. All of these collections are arranged sequentially. Moving the paging mechanism forward displays the next three items in the sequentially ordered set of collections. Moving the paging mechanism backward displays the previous three items in the sequentially arranged collections. After a resize operation, the visible area of ​​the browser changes, which changes the number of displayed items in the item bin

[44] and the number of displayed writable collections in the writable collections bin

[45] . In alternative exemplary embodiments, explicit writable permissions may not exist.In this case, the writable collection bin

[45] will contain the displayed collection. The paging mechanism

[48] and collection paging mechanism

[49] take this shift into account and operate with the new page size.

[0082] In Figure 10, in addition to the paging mechanism

[48] , the pins of all annotation item boxes

[36] located in the item bin

[44] can also be seen. In the exemplary embodiment shown in Figure 10, six items representing the annotation are shown. Each of these six items can be unpinned or pinned. As shown, the page size is 6, and if one of these items is pinned, the page size is reduced to 5. As a result of being pinned, the pinned item remains in the viewing area even when the paging mechanism

[48] is moved back and forth. When the paging mechanism

[48] is moved forward, the pinned item remains in the viewing area, and the remaining 5 items move out of the viewing area. Then, in sequential order, the next 5 items replace the 5 items that were removed. When the paging mechanism is moved backward, the 5 unpinned items are removed from the viewing area, and the 5 items that preceded them in sequential order replace the 5 items that were removed from the viewing area. If two items are pinned, the page size is reduced to 4. Pinning works similarly for collections in a writable collection bin

[45] .

[0083] Each time an item is pinned, the pinned item is excluded from the set of items sequentially ordered by the annotation server

[61] , as shown in several other diagrams. This prevents the pinned item from appearing twice in the display area. If two items are pinned, the two pinned items are excluded from the set of items sequentially ordered by the annotation server

[61] . Exclusion from sequential ordering occurs in addition to changes in page size. The exclusion from sequential ordering by the annotation server

[61] works similarly for collections in a writable collection bin

[45] .

[0084] Filters, such as the search filter

[46] shown in the exemplary embodiment of Figure 10, are used to receive input from the user. In the case of the search filter

[46] for the item bin

[44] , it is a search phrase. These inputs are converted into parameter values ​​that are sent to an annotation server

[61] shown in several other figures. These parameter values ​​are used by the annotation server

[61] to input parameters into a request query used by an electronic database

[62] shown in several other figures. The request query includes parameter values ​​and a page number. The request query, including the parameter values, is used to specify a set of items and the order of those items. The page number specifies a particular page. In addition, the request query may include a specified number of items to be sent to a web browser running on a client electronic device. This request query includes parameters of filter values, such as the search phrase, and an exclusion list of items to exclude from the specified list of items, and the request query, along with the parameters and the exclusion list of items, introduces sequential ordering to that list of items.

[0085] Filters such as the collection search filter

[47] shown in the exemplary embodiment of Figure 10 are used to receive input from the user. In the case of the collection search filter

[47] of a writable collection bin

[45] , it is a collection search phrase. These inputs are converted into parameter values ​​that are sent to an annotation server

[61] shown in several other figures. These parameter values ​​are used by the annotation server

[61] to input parameters into a collection request query used by an electronic database

[62] shown in several other figures. The collection request query includes parameter values ​​and a page number. The collection request query is used to specify a set of collections and the ordering of those collections, along with the collection parameter values. The page number specifies a particular page. In addition, the collection request query may include a specified number of collections to be sent to a web browser running on a client electronic device. This collection request query includes parameters of collection filter values, such as the collection search phrase, and an exclusion list of collections to exclude from the specified list of collections, and this collection request query, along with the parameters and the exclusion list of collections, introduces ordering to the list of collections.

[0086] After a user creates an annotation, in the exemplary embodiment depicted in Figure 10, the annotation appears in an item bin

[44] . Using a drag-and-drop process, the user can move or duplicate the annotation item box

[36] to the collection, as in the exemplary embodiment depicted in Figure 11. In the exemplary embodiment depicted in Figure 11, the annotation item box

[51] is shown being moved. As the moving annotation item box

[51] approaches the displayed writable collection

[50] , as in the state of the exemplary embodiment depicted in Figure 10, the visual form of the displayed writable collection

[50] changes as depicted in Figure 11, showing screen elements including the move target

[52] and the duplicate target

[53] . In this exemplary embodiment, the move action is triggered when the mouse pointer depicting the drag of the moving annotation item box

[51] is within the area indicated by the move target

[52] and then released. In this exemplary embodiment, the duplicate operation begins when the mouse pointer indicating the drag of the moving annotation item box

[51] is released in the area specified by the duplicate target.

[0087] In the exemplary embodiment depicted in Figure 12, the user is presented with a displayed annotation symbol for a closed question

[35] , a displayed annotation symbol for a closed exclamation

[54] , a displayed annotation symbol for a closed check

[55] , and a displayed annotation symbol for a closed incorrect answer

[56] . These four annotation symbols are examples of displayed annotation symbols. In the exemplary embodiment, a displayed annotation symbol has at least two possible states. These states include an open state and a closed state. In the closed state, the displayed annotation symbol is visible, and is designed to have low visibility so as not to distract the user while reading the web document. In the open state, additional information is displayed in the web browser's display area. In the exemplary embodiment of Figure 7, a displayed annotation symbol in the open state is depicted. This displayed annotation symbol is a displayed annotation symbol for a question. In the exemplary embodiment of Figure 7, the displayed annotation symbol for an open question

[29] is shown together with annotation highlighting text

[27] and an annotation comment box

[31] .

[0088] Annotation symbols are associated with elements of a group of annotation semantic relationships. Each annotation semantic relationship represents a typically user-selection-based relationship between annotated highlighted text

[27] and an annotation comment

[32] , as depicted in the exemplary embodiment of Figure 7. Different annotation symbols are associated with each relationship within a group of annotation semantic relationships. For example, as depicted in the exemplary embodiment of Figure 8, a closed question displayed annotation symbol

[35] indicates that the annotation creator has marked the annotation comment as a question about the annotation highlighted text. This closed question displayed annotation symbol

[35] allows a user familiar with how the exemplary embodiment works to know, before actually performing the click action, that they can click the closed question displayed annotation symbol

[35] , as depicted in Figure 7, to see the annotation comment, which is a question about the annotation highlighted text

[27] .

[0089] Once clicked, the annotation symbol

[35] displaying a closed question, as in the exemplary embodiment shown in Figure 8, becomes the annotation symbol

[29] displaying an open question, showing annotation highlighting text

[27] and an annotation comment box

[31] , as in the exemplary embodiment shown in Figure 7. Furthermore, in the exemplary embodiment shown in Figure 8, hovering the mouse over the annotation symbol

[35] displaying a closed question makes the annotation highlighting text

[27] , the annotation symbol

[29] displaying an open question, and the annotation comment box

[31] visible, and they can be fixed by clicking. To enable users to perform this action reflexively, the annotation highlighting text

[27] in Figure 7 can be assigned a color based on its annotation symbol. That is, the color of the annotation highlighting text

[27] is determined based on the semantic relationship between the annotation highlighting text

[27] and the annotation comment.

[0090] To further illustrate exemplary embodiments, observe the following: A closed exclamation mark annotation symbol

[54] indicates that the annotation author has marked an annotation comment as an exclamation statement indicating a note or interesting fact about the annotation-highlighted text. A closed checkmark annotation symbol

[55] indicates that the annotation author has marked an annotation comment as a statement agreeing with the annotation-highlighted text. Also, a closed incorrect annotation symbol

[56] indicates that the annotation author has marked an annotation comment as a statement disagreeing with the annotation-highlighted text.

[0091] In addition to the annotation symbols discussed so far and the displayed annotation symbols shown in Figure 12, it is possible to use advertising annotation symbols to show another relationship within the group of semantic relationships of annotations. This relationship indicates that an annotation comment

[32] , as in the exemplary embodiment shown in Figure 7, is an advertisement related to the annotation highlighted text

[27] . In this situation, the displayed annotation symbol

[35] for a closed question would be replaced with the displayed annotation symbol for an advertisement, as in the exemplary embodiment shown in Figure 8.

[0092] In the exemplary embodiment depicted in Figure 12, several displayed annotation symbols can be seen. In this exemplary embodiment, the five relationships in the group of semantic relationships of annotations correspond to the question annotation symbol, disagreement annotation symbol, agreement annotation symbol, exclamation annotation symbol, and advertisement annotation symbol. These semantic relationships of annotations are question relationship, disagreement relationship, agreement relationship, exclamation relationship, and advertisement relationship.

[0093] In exemplary embodiments, machine learning and deep learning models can be used to automatically determine the semantic relationships between annotations, such as highlighted text and annotation comments. Furthermore, user data collected by users determining semantic relationships and writing annotation comments can be used to train deep learning models, and the by-products of these trained models can be used for other natural language processing purposes. Users can be given the option to opt in or opt out of using their created data to train deep learning-based natural language processing models. A Bidirectional Long-Term Short-Term Memory (BiLSTM) model with a final max pooling layer can be used to compute sentence embeddings. Other machine learning and deep learning techniques can also be used to create sentence embeddings. Sentence embeddings can be used to determine the similarity between two sentences. Once sentence embeddings have been computed for all sentences in a collection through analysis of highlighted text, annotation comments, and other sentences occurring in items and replies, vectors of these sentence embeddings can be used to determine whether an item is similar to other items in the collection, or to determine the similarity between two collections. Based on this similarity analysis, items can be suggested to the user for reading, discussing with other users, or adding to a collection. Similar collections can also be suggested based on collections the user has searched, viewed, or posted. Furthermore, annotations on web pages can be linked to similar annotations and content in other web pages or documents through similarity analysis based on sentence embedding vectors. In combination with a sentence similarity classifier, a convolutional neural network (CNN) model can be used to verify the semantic relationships of annotations specified by the user. The semantic relationship data of annotations provided by the user can be combined with user reputation to verify the semantic relationships of annotations selected by the user and potentially identify inappropriate content generated and posted by the user.The Bidirectional Encoder Representation (BERT) from Transformers can be used to identify named entities within web pages and web documents and automatically annotate items that users may be potentially interested in. These named entities could be entities present on Wikipedia or products sold on online marketplaces. For moderating user-generated annotations, a pre-trained BERT fine-tuned for sentiment analysis can be used to flag inappropriate user-generated content.

[0094] In the exemplary embodiment depicted in Figure 13, the user can trigger a symbol beacon by pressing a symbol beacon trigger button

[57] to make a closed, displayed annotation symbol more prominent. The symbol beacon may be in an untriggered or triggered state. When the symbol beacon trigger button

[57] is pressed, the symbol beacon transitions from an untriggered state to a triggered state. The trigger mechanism for the symbol beacon may also be a sequence of keyboard key presses or mouse button clicks. The trigger mechanism may also be located in a menu option within an annotation-enabled web browser

[24] . Whether it be a button available in the display area of ​​the annotation-enabled web browser

[24] , a menu option in the annotation-enabled web browser

[24] , a combination of keyboard key presses and mouse button clicks, an input gesture on a touchscreen device, or an input gesture via an input device, it is a set of instructions in the web browser extension

[25] that makes these triggers available and triggers one displayed annotation symbol beacon

[58] for each displayed annotation symbol.

[0095] Next, sequence diagrams are presented showing how messages are transmitted between different entities. These messages include user input to an annotable web browser

[24] or a regular web browser, and electronic messages between an electronic device and a computer system. An annotable web browser

[24] is shown in an exemplary embodiment in Figure 3. These sequence diagrams apply to a class of exemplary embodiments. After the sequence diagrams are presented, several message variations are presented. Specific exemplary embodiments can be discussed by more precisely identifying the messages.

[0096] Figure 14 is a sequence diagram applicable to a class of exemplary embodiments. In Figure 14, the sequence diagram shows the interaction between a user

[59] , an annotable web browser

[24] , a URL server

[60] , an annotation server

[61] , an electronic database

[62] , and a content delivery network

[63] . When the user

[59] visits a URL

[64] , the annotable web browser

[24] requests a web document

[65] , and when the URL server

[60] returns the web document

[66] , the annotable web browser

[24] displays the document to the user

[59]

[67] . Next, the user

[59] enters input to create an annotation

[68] , the annotable web browser

[24] sends a bundle of data elements representing the annotation to the annotation server

[61]

[69] , the server sends an image for storage

[70] to the content delivery network

[63] and returns the URL of the sent image

[71] . Next, the annotation server

[61] instructs the electronic database

[62] to store the data elements and non-image bundle of URLs from the CDN as an annotation object

[72] , and the electronic database confirms the storage of the annotation object

[73] . The annotation server

[61] then confirms the received bundle of data elements and sends the annotation display bundle to an annotation-enabled web browser

[24]

[74] , and displays the displayed annotation bundle to the user

[59] indicating that the annotation has been successfully created

[75] .

[69] refers to the bundle of data elements that represents the annotation. The bundle of data elements may vary depending on the implementation details of the exemplary embodiment and the type of annotation.

[0097] In Figure 14, a sequence diagram applicable to a class of exemplary embodiments is presented. Depending on the characteristics of the bundle of data elements representing annotations or highlighted text, the annotation server

[61] may record different data, and the annotable web browser

[24] may display different views due to the fact that different data is received and selectable when generating a display representation of the annotations or highlighted text. Next, screenshot bundles and annotation symbol bundles, which may serve as data element bundles in Figure 14, are discussed in exemplary embodiments. Furthermore, Figure 14 may be applied to exemplary embodiments where the annotation display bundle is a screenshot display bundle or an annotation symbol display bundle. The corresponding screenshot bundle is the screenshot display bundle, and the corresponding annotation symbol bundle is the annotation symbol display bundle. These corresponding pairs are used when Figure 14 uses data element bundles and annotation display bundles.

[0098] An exemplary embodiment of a screenshot bundle includes a uniform resource locator (URL) of the document, data of selected text pieces from the document containing selected text pieces representing annotations, and marked screenshots of the portion of the document displaying the selected text pieces with the selected text pieces visually marked. The screenshot bundle may also include information for finding selected text pieces within the document. This information for finding selected portions of text within the document typically includes Document Object Model (DOM) information. An annotation screenshot bundle is a screenshot bundle that also includes annotation comments

[32] , as shown in the exemplary embodiment depicted in Figure 7. The screenshot bundle may also include data for representing annotation symbols. Alternatively, a screenshot bundle can be used that does not include annotation comments

[32] . In such a case, data representing the highlighted text without annotation comments

[32] is included in the screenshot display bundle along with the relevant marked screenshots representing the highlighted text.

[0099] In an exemplary embodiment, the annotation symbol bundle includes the document's uniform resource locator (URL), data for representing selected text fragments from the document, where the selected text fragments represent annotations, annotation comments

[32] , and annotation symbols. An exemplary embodiment in Figure 7 may be used for reference. The screenshot bundle also includes information for finding selected text fragments within the document. This information for finding selected text fragments within the document typically includes document object model (DOM) information. The annotation symbol bundle may also include marked screenshots of the document portion showing the selected text fragments, where the selected text fragments are visually marked. Depending on the desired behavior, marked screenshots may or may not be necessary.

[0100] In an exemplary embodiment, DOM information for finding a selected portion of text in a document may include information for finding the element in the DOM containing the first character of the selected text, the element in the DOM containing the last character of the highlighted text, and information for finding the highlighted text itself. These two elements are called the highlighting start element and the highlighting end element. The information recorded for the highlighting start and end elements is their tags, IDs, classes, indexes, content, XPath, and parent. In addition, the ancestor elements of the highlighted text start element are also recorded. The ancestor elements of the highlighting start element have their tags, IDs, classes, indexes, and content recorded. To find the specified highlighted text at a later point in time, the DOM is searched using tags and IDs, XPath, tags and classes and indexes, tags and classes, tags and indexes, and tags alone. A simple XPath search is not sufficient because web pages may be edited or the DOM structure of a web page may change as a result of the execution of JavaScript code that inserts elements into the DOM. In this exemplary embodiment, the goal is to find the selected text even if the web page has been edited. Multiple search methods are used to facilitate finding the selected text in different scenarios. Finding the start and end elements of the highlighted text makes it easier to locate the selected highlighted text.

[0101] Another concern is that the JavaScript code of the visited webpage inserts elements into the webpage's DOM. These inserted elements can be annotated. To facilitate this process, the browser's mutation observer can be used. When a change is detected, the DOM structure detection code can be rerun. If the start and end elements of the highlighted text are found, the highlighted text can be inserted. In the case of annotated highlighted text, the displayed annotation symbols can also be inserted.

[0102] If the submitted screenshot bundle or annotation symbol bundle contains marked screenshots, a storage set of images is created from these marked screenshots through various operations, including resizing, cropping, and image modification. These operations are performed to produce images that look good in different viewing areas on different sized screens. Once this storage set of marked screenshots is created, it makes sense to decide whether to store these marked screenshots in a Content Delivery Network (CDN). For thumbnail-marked screenshots, there may be merit in storing the data together with all other data in the screenshot bundle or annotation symbol bundle. For larger marked screenshots, it is usually better to store these images using a deliverable CDN. As shown in some figures, the annotation server

[61] typically decides which images in the storage set to store in an electronic database

[62] and which images to send to the CDN, as shown in some figures. The CDN typically returns URLs once the images are stored. These URLs are stored along with the rest of the screenshot bundle or annotation symbol bundle.

[0103] A deliverable screenshot is either an actual image or a URL stored in an electronic database

[62] , as shown in some figures. When sending data to a client device, the image can be loaded from the CDN using the URL. Alternatively, if the image is sent directly, it can simply be displayed. Finally, what is stored in the electronic database is an annotation object

[62] . An annotation object typically contains all the received data in a screenshot bundle or annotation symbol bundle, and a deliverable screenshot of each marked screenshot in the storage set.

[0104] Ad screenshot bundles and ad annotation symbol bundles can potentially be deployed with their respective ad screenshot display bundles and ad annotation symbol display bundles to help advertisers reach customers with relevant ads related to the content the user is currently engaging with. A displayed ad screenshot bundle is a visual representation of the data within an ad screenshot display bundle, and a displayed ad annotation symbol bundle is a visual representation of the data within an ad annotation symbol display bundle. Figure 7 can be used for illustrative purposes, but the displayed annotation symbols are replaced with the displayed annotation symbols of the ad. In an exemplary embodiment, an ad screenshot bundle includes annotation comments

[32] created by the advertiser related to annotation highlighting text

[27] . An ad annotation symbol bundle includes annotation symbols that indicate an ad. An annotation symbol bundle includes annotation symbols, but a screenshot bundle may also include annotation symbols. A screenshot bundle may include marked screenshots, while an annotation symbol bundle may include marked screenshots. Figure 9 can be used for illustrative purposes, but the displayed annotation symbols are replaced with the displayed annotation symbols of the ad. Furthermore, while annotation comments are created by the advertiser along with other elements, the overall form is preferably similar to other annotation item boxes. If marked screenshots are included, designing a functional system may require the creation of storage sets and the use of a CDN for distributing these marked screenshots. In exemplary embodiments, these ad screenshot bundles and ad annotation symbol bundles are often stored and managed via an ad server that communicates with an annotation server. The ad server is not explicitly shown in the sequence diagram but can be inserted.The page size can be reduced by one to accommodate one advertisement. As shown in some diagrams, after the annotation server

[61] has prepared a page for the item representing the annotation, the annotation server

[61] can communicate with the ad server to retrieve one advertisement.

[0105] The semantic web provides semantic descriptions of several web pages and documents available on the WWW through ontologities, which are implementations of a kind of descriptive logic. These ontologities need to be created for different domains and have the ability to represent concepts within their domain in logical form, as done in descriptive logic. Collections of existing ontologities are published by projects such as KBPedia. Descriptive logic is a type of logic defined in mathematical logic and has some features of first-order logic. The computational properties differ depending on the properties of the logical family. Using existing ontologities, logical statements can usually be created. In exemplary embodiments, these logical statements can be translated into natural language, and some sentences in natural language can be translated into statements created within descriptive logic using neural machine translation techniques. Other deep learning techniques can also be used. Machine learning and deep learning models can translate some sentences in natural language into statements in a particular ontology. In this exemplary embodiment, this methodology may be used for advertising purposes. Advertisers can express concepts about their products or services and write logical statements within an ontology related to those products or services. These logical statements can then be entered into the advertising platform. The ad server can identify web pages and documents that contain natural language text that can be translated into statements within the ontology selected by the advertiser to express concepts about the advertised products or services. Furthermore, the ad server can determine which of these web pages and documents contain natural language text that expresses statements similar to or logically related to those expressed by the advertiser. Once these web pages and documents are identified along with their corresponding locations within the documents, advertisers can be given the opportunity to place ads in specific locations somewhere that link to those web pages or documents or those specific locations. A logic analyzer can be used to determine the various consequences of statements occurring on web pages and documents.Based on this information, web pages and documents and their locations within them can be narrowed down to find the optimal location for the advertiser. Images, videos, and audio can be converted into text through image captioning, speech recognition, and transcription. In some cases, this text can also be translated into the same statements as the ontology used by the advertiser. After this, a logic analyzer is used to determine the best candidates to show the advertiser, which the advertiser can then review. Based on the advertiser's selection, the advertising platform can generate ads to be placed on the ad server. Furthermore, in this exemplary embodiment, ontologs can be created for the purpose of representing facts about products and services sold in different domains such as insurance, financial services, legal services, and healthcare. These ontologs can then be used by various groups advertising and selling services in these domains to create logically accurate statements about those products and services for searching media such as web pages, web documents, images, videos, and recordings. This media can then reach the audience that consumes it, and the advertiser can know that this audience is likely to be familiar with the statements made by the advertiser in the ontology. Furthermore, in this exemplary embodiment, the media source itself has a logical relationship with the advertiser's statement, and the advertiser can use this logical relationship to provide more tailored products and more tailored targeting that meet customer needs.

[0106] To specify the details of an exemplary embodiment shown in Figure 14, bundles of data elements should be specified along with details regarding storage sets and deliverable screenshots. A bundle of data elements can be a bundle of screenshots having their associated data elements. A bundle of data elements can also be a bundle of annotation symbols having their associated data elements. The storage sets, size, resolution, cropping, and other characteristics of different types of marked screenshots should be determined. This is usually based on considerations of the type of display on which these marked screenshots will be displayed, along with the size of the displayed view. In some cases, marked screenshots with thumbnails may be included in the storage set. If the size of these thumbnails is small, storing these images using a CDN may not be the most efficient approach. It may be more efficient to store images in an electronic database

[62] , a local file system, or other local location than in a content delivery network

[63] . Specifying whether the deliverable screenshot is an actual image or a URL stored in an electronic database is another decision to consider in order to specify implementation details. It will be obvious to those skilled in the art that various fine-grained variations can be introduced for storing and retrieving images. It will also be obvious to those skilled in the art that these variations are compatible and the differences between them are not substantial.

[0107] The electronic database

[62] in Figure 14 and several other figures can take various forms, ranging from relational to non-relational databases. Furthermore, the electronic database

[62] may be a distributed electronic database system implemented on one or more computer systems. The most advantageous characteristics of the database may vary depending on the number of database clients and the usage patterns of these database clients. In the exemplary embodiment, the annotation server

[61] is a database client. The characteristics of the electronic database or distributed electronic database system may vary to accommodate a large number of annotation servers with different latency requirements, as well as the needs for consistency, availability, and persistence. Furthermore, a distributed electronic database system can be implemented in a variety of ways.

[0108] Furthermore, it should be noted that storing images in an electronic database

[62] can sometimes impact performance. In such cases, it is beneficial to store references in the database and use a different storage mechanism for specific images. In the case of a content delivery network

[63] , the references are URLs. However, it is possible to use alternative mechanisms to store references that are not URLs and then store images in a searchable manner using the references stored in the electronic database. In exemplary embodiments, images may be stored on a local file system, and references in the database may correspond to file paths. In such cases, the storage location with the electronic database should be considered a distributed electronic database that stores images in a more performance-oriented manner. It will be obvious to those skilled in the art that various minor variations of this technique can be introduced for the storage and retrieval of images. It will also be obvious to those skilled in the art that these variations are interchangeable and the differences between them are not substantial.

[0109] Figure 15 is a sequence diagram applicable to a class of exemplary embodiments. For Figure 15 to specify the details of an exemplary embodiment, an annotation display bundle and a displayed annotation bundle must be specified. Furthermore, Figure 15 may apply to exemplary embodiments in which the annotation display bundle is replaced by a screenshot display bundle, and the displayed annotation bundle is replaced by a displayed screenshot bundle. In this case, a screenshot display bundle with data elements to be included must be specified, and a mechanism for obtaining deliverable screenshots must also be specified. User interface diagrams, data elements described for screenshot bundles, and previously discussed considerations can be used to specify the annotation display bundle and the displayed annotation bundle. A displayed screenshot bundle is generally used when there are marked screenshots with annotation highlighting and annotation comments. A displayed annotation screenshot bundle is used when there are marked screenshots, annotation highlighting, and annotation comments. Figure 15 is a sequence diagram illustrating the interaction between a user

[59] , an annotation-enabled web browser

[24] , an annotation server

[61] , an electronic database

[62] , and a content delivery network

[63] . When the user

[59] visits the annotation server's URL

[76] , the annotation-enabled web browser

[24] requests the annotation server

[61] for the annotation server's organized page

[77] , requests the electronic database

[62] for the annotation objects needed to construct the page

[78] , and the annotation objects are returned to the annotation server

[61]

[79] . Next, the annotation server

[61] returns the annotation server's organized page along with the annotation display bundle

[80] to the annotation-enabled web browser

[24] , and requests an image of the annotation display bundle URL returned when the annotation server's organized page was returned

[81] .Finally, the content delivery network

[63] returns the requested image to the annotation-enabled web browser

[24]

[82] and displays the organized page with the annotation bundle displayed to the user

[59]

[83] . Next, the user

[59] provides input to the annotation-enabled web browser

[24] to move the annotations

[84] , and the annotation server

[61] requests the annotations to be moved by issuing a move instruction to the electronic database

[62]

[86]

[85] . In response, the electronic database

[62] authorizes the annotation server

[61] to move the annotation object

[87] , the annotation server acknowledges that the annotations have been moved

[88] , and the annotation-enabled web browser

[24] displays the content with the annotation bundle displayed in the new location to the user

[59] .

[0110] The displayed annotation screenshot bundles and annotation item boxes are created using data from annotation screenshot display bundles sent from the annotation server

[61] , as shown in part of the figure. The annotation screenshot display bundles are created using data from annotation objects retrieved from an electronic database, as shown in part of the figure

[62] . When viewing a page of annotations in a screenshot bundle display area, such as the item bin

[44] in the annotation organization context area

[43] of an organized web page, as shown in an exemplary embodiment in Figure 10, the data sent from the annotation server

[61] is the page on which the thumbnail display bundles are displayed. These thumbnail display bundles include, at a minimum, deliverable screenshots of thumbnail-marked screenshots in storage groups of annotation objects stored in an electronic database

[62] . Figure 9 can be used for illustrative purposes. In an exemplary embodiment, the thumbnail display bundle also includes data representing the displayed annotation symbols for separate contextual annotation highlighting text

[38] , separate contextual annotation comments

[40] , and separate contextual questions

[39] in a web browser-readable format. Furthermore, the displayed annotation symbols for separate contextual annotation highlighting text

[38] and separate contextual questions

[39] are color-coded. All of this data works together to allow the user to get more information at a glance. Additionally, screenshots of the displayed annotations can be displayed on the annotation details page, which can provide more detailed information about a single annotation and many user comments and replies about that annotation.

[0111] Figure 16 is a sequence diagram applicable to a class of exemplary embodiments. For Figure 16 to explicitly specify an exemplary embodiment, an annotation symbol display bundle and a corresponding displayed annotation symbol bundle must be specified, along with the data elements to be included along with the visual appearance. This specification can be made using a user interface diagram, data elements describing the annotation symbol bundle, and the considerations described above. Figure 16 is a sequence diagram showing the interaction between a user

[59] , an annotable web browser

[24] , a URL server

[60] , an annotation server

[61] , and an electronic database

[62] . When the user

[59] visits a URL

[90] , the annotable web browser

[24] requests a web document from the URL server

[60]

[91] , and the server returns the web document to the annotable web browser

[24]

[92] . Next, the annotable web browser

[24] requests an annotation symbol display bundle representing the annotation from the annotation server

[61]

[93] , the annotation server requests the annotation object from the electronic database

[62]

[94] . The electronic database

[62] then returns the annotation object to the annotation server

[61]

[95] , the annotation server returns an annotation symbol display bundle representing the annotation to the annotable web browser

[24]

[96] , and displays the web document with closed, displayed annotation symbols and annotations that can be opened by hovering over or clicking them

[97] .

[0112] Figure 17 shows a flowchart applicable to a class of exemplary embodiments. A particular exemplary embodiment can be specified if the bundle of data elements to be received and stored is precisely specified. For example, a screenshot bundle can be explicitly specified with all of its data elements, or an annotation symbol bundle can be explicitly specified with all of its elements. In this case, the screenshot bundle or annotation symbol bundle becomes the bundle of data elements to be received. Furthermore, if the bundle of data elements includes marked screenshots, the storage set and storage mechanism for the screenshots to be delivered must be specified. These details are helpful in illustrating the implementation. In an exemplary embodiment, the storage set may include marked screenshots for display on large monitors, laptops, tablets, and phones, as well as thumbnail marked screenshots for item boxes. The exact size can be adjusted to the most common size currently used on these displays. Furthermore, in a client-server architecture, the server receives a bundle of data elements representing annotations

[98] and stores the received and newly created data elements using one or more computer systems to satisfy requests made by an annotable web browser on a user's client device to create annotations or simple highlights

[99] . Figure 17 shows a flowchart of a method implemented by one or more computer systems representing a server in a client-server architecture. This method is implemented by the server to enable the sharing of data that may be requested by client user interfaces of the same or different clients. The steps of this method include receiving a bundle of data elements representing annotations

[98] and using one or more computer systems to store the received data elements and newly created elements

[99] .

[0113] Figure 18 shows a flowchart applicable to a class of exemplary embodiments. A particular exemplary embodiment may be specified if the data elements included in the bundle display set are precisely specified. This means specifying the data elements included in the display bundle. In an exemplary embodiment, the display bundle may be a screenshot display bundle or an annotation symbol display bundle. If the display bundle is a screenshot display bundle, the bundle display set will be the screenshot display bundle display set in Figure 18. If the display bundle is an annotation symbol display bundle, the bundle display set will be the annotation symbol bundle display set in Figure 18. Furthermore, in a client-server architecture, the server will use one or more computer systems to compile a bundle display set containing the pages of the display bundle

[0100] and send the bundle display set in a format that the display bundle can be displayed by a web browser running on an electronic device to satisfy a request made by an annotable web browser running on a client device of a user viewing a web document or a page of annotations in another context area

[0101] . The flowchart shown in Figure 18 is implemented by one or more computer systems.

[0114] Figure 19 is a flowchart applicable to the class of exemplary embodiments. Certain exemplary embodiments may be specified when a bundle of data elements is explicitly specified. For example, a bundle of screenshots having data elements to be included may be explicitly specified, or a bundle of annotation symbols having data elements to be included may be explicitly specified. Figure 19 shows an annotation creation flowchart illustrating the steps performed by a set of instructions for a web browser extension that converts a web browser to an annotation-enabled web browser that supports the addition of browser extensions in an exemplary embodiment

[24] . Annotation creation is performed by the browser extension, as shown in some figures. Figure 19 shows the steps performed by the user

[59] , the web browser

[0102] , the web browser extension content script

[0103] , and the web browser extension background script

[0104] when creating an annotation. The user

[59] opens the browser

[0105] , and the web browser

[0102] loads the browser extension

[0106] . Next, the user

[59] visits a URL

[0107] , and the web browser

[0102] reads the data sent from the accessed URL

[0108] and injects a content script

[0109] . Here, the user

[59] views the web page

[0110] and highlights text

[0111] . At this time, the web browser extension content script

[0103] executes the instructions of the web browser extension and makes the annotation symbol selection button bar available

[0112] . Next, the user

[59] presses the button corresponding to one of the annotation symbols on the annotation symbol selection button bar

[0113] , and the web browser extension content script

[0103] makes the annotation input box available accordingly

[0114] . Next, when the user

[59] enters text and presses the Enter key

[0115] The web browser extension content script

[0103] sends a bundle of data elements to a background script.

[0116] At this time, the background script

[0104] of the web browser extension receives a bundle of data elements from the content script.

[0117] Send the bundle of data elements to the annotation server.

[0118] .

[0115] In Figure 19, after the user

[59] enters text and presses Enter

[0115] , it is desirable that the web browser extension content script

[0103] generates either an annotation symbol bundle or a screenshot bundle. An exemplary embodiment may require that the annotation symbol bundle include a marked screenshot. If the marked screenshot is generated on the server side, the web page may have to be re-rendered or the screenshot may have to be marked. Doing so would be computationally intensive and prone to errors. Generating a marked screenshot using a browser extension avoids these problems. The browser extension must ensure that the user's privacy is protected. The screenshot should not include the desktop or other browser tabs. Furthermore, in the exemplary embodiment, it is desirable that the screenshot be annotated with highlighted text

[27] and an annotation comment box

[31] . The annotated highlighted text

[27] and annotation comment box

[31] are shown in Figure 7, which may be useful for illustration in this case. In the exemplary embodiment, a simple delay may be implemented before the screenshot is taken. This delay should be sufficient to ensure that the annotation-highlighted text

[27] and annotation-comment boxes

[31] are present on most devices. In an alternative exemplary embodiment, a more performance-oriented method could be implemented in which a web browser extension content script

[0103] searches the document's DOM structure for the presence of the annotation-highlighted text

[27] and annotation-comment boxes

[31] and takes a screenshot if these elements are present. The search for the presence of the annotation-highlighted text

[27] and annotation-comment boxes

[31] in the document's DOM structure would occur at periodic intervals on a shorter timescale than the simple delay time described above.In an exemplary embodiment, the periodic interval can be one-third of the length of the simple delay that would have been selected.

[0116] Figure 20 is a flowchart applicable to a class of exemplary embodiments. A particular exemplary embodiment can be specified, provided that the displayed annotation symbol bundle is explicitly specified in all of its data elements. Figure 20 presents a flowchart of annotation display showing the steps performed by a set of instructions for a web browser extension that, in an exemplary embodiment, transforms a web browser that supports the addition of browser extensions into an annotable web browser. Figure 20 shows the steps performed by a user

[59] , a web browser

[0102] , a web browser extension content script

[0103] , and a web browser extension background script

[0104] when viewing annotations. The user

[59] opens the browser

[0119] The web browser

[0102] loads browser extensions.

[0120] Next, the user

[59] accesses the URL

[0121] The web browser

[0102] reads the data sent from the accessed URL.

[0122] Inject content scripts

[0123] Here, the web browser extension background script

[0104] loads an annotation symbol display bundle that represents annotations for URLs.

[0124] Send an annotation symbol display bundle to the content script.

[0125] Now, the web browser extension content script

[0103] receives the annotation symbol display bundle from the background script.

[0126] , display annotation symbols in a closed state on the web page

[0127] Now, the user

[59] is browsing a webpage with annotation symbols displayed in a closed state.

[0128] Click the annotation symbol that is displayed in the closed state.

[0129] In response, the web browser extension content script

[0103] displays the annotation symbol in an open state.

[0130] Next, the user

[59] views a webpage that displays the selected annotation symbols in open format.

[0131] .

[0117] In an exemplary embodiment depicted in Figure 31, a collection item web page is shown. The collection item page context area

[0258] is the display area for the displayed items, the collection item paging mechanism

[0259] , the collection item search filter

[0260] , and the collection action menu trigger

[0261] . The collection action menu

[0262] displays the options to send a message to the collection

[0263] , invite a user

[0264] , and change user permissions

[0265] . The collection action menu trigger

[0261] is used to trigger the collection action menu

[0262] . The collection item paging mechanism

[0259] may be used to change the page of the displayed collection displayed in the collection item page context area

[0258] . The option to send a message to the collection

[0263] creates a message chain with the first text message and sends it to a member of the collection who has read-only permissions or higher permissions for the collection. The message chain appears in the user's inbox. An exemplary embodiment of the user's inbox is depicted in Figure 33. In an exemplary embodiment shown in Figure 33, the American History Collection Message Chain

[0281] is illustrated. Messages sent to members of the collection using the Send Message to Collection option

[0263] appear in the inboxes of recipients categorized under the collection from which the message was sent. All members belonging to the collection with Read or higher permissions can read the message, and all members with Messageable or higher permissions can reply. The Invite User option

[0264] is used to invite new users to the collection. The invitation is sent to the invited user's inbox. The Change User Permissions option

[0265] can be used to change the permissions of members of the collection.A notification of the change may be sent to the inbox of the user whose permissions were changed, in the form of a message within the message chain.

[0118] A collection item filter, such as the collection item search filter

[0260] shown in the exemplary embodiment of Figure 31, is used to receive input from a user. In the case of the collection item search filter

[0260] , the input is a collection item search phrase. These inputs are converted into collection item parameter values ​​that are sent to the annotation server

[61] , as shown in some figures. These collection item parameter values ​​are used by the annotation server

[61] to input the collection item parameters into a collection item request query used by the electronic database

[62] , as shown in some figures. The collection item request query includes the collection item parameter values ​​and the collection item page number. The collection item request query, including the collection item parameter values, is used to specify a set of collection items and the sequential order of those collection items. The collection item page number specifies a particular collection item page. Furthermore, the collection item request query may include a specified number of collection items to be sent to a web browser running on a client electronic device. This collection item request query includes collection item parameters for collection item filter values, such as a collection item search phrase, an exclusion list of collection items to exclude from the specified list of collection items, and a sequential ordering for that list of collection items.

[0119] In an exemplary embodiment depicted in Figure 32, a collection web page is shown. The collection page context area

[0266] is the display area for the displayed collection, the collection page paging mechanism

[0267] , the collection page search filter

[0268] , and the collection page action menu trigger

[0269] . The collection page action menu

[0270] displays the Owner filter option

[0271] , the Administrative filter option

[0272] , the Writable filter option

[0273] , the Messaging filter option

[0274] , the Viewable filter option

[0275] , and the Subscribe filter option

[0276] . The collection page action menu trigger

[0269] is used to trigger the action menu

[0270] of the collection page. The collection page paging mechanism

[0267] can be used to change the web page of the displayed collection displayed in the collection page context area

[0266] . The Owner filter option

[0271] , Administrator filter option

[0272] , Writable filter option

[0273] , Messengerable filter option

[0274] , Viewable filter option

[0275] , and Subscribe filter option

[0276] filter collections based on user permissions. Owner permission is the highest level of permission and is granted to the creator and owner of a collection. Owner permission grants other users administrator permission for that collection, allowing them to do anything that lower permissions allow. Administrator permission is a lower level of permission than owner permission and can raise non-administrator permissions, handle other administrative tasks, and do anything that lower permissions allow.Writable permissions are lower than administrator permissions, allowing users to add items to a collection and do anything permitted by lower-level permissions. Messaging permissions are lower than writable permissions, allowing users to send messages to other members of the collection and do anything permitted by viewable permissions. Readable permissions are lower than messageable permissions, allowing users to view all items in a collection and read all collection messages for that collection. Subscribe permissions grant users readable permissions to public collections. Subscribe permissions are usually granted to users requesting permission without administrator intervention because the collection and associated messages are public. Background colors are assigned based on color to help users visualize their privacy settings. Colors associated with shorter wavelengths signify greater permissions, and colors associated with longer wavelengths signify smaller permissions. In other words, the colors of the rainbow are associated with permissions. Purple corresponds to owned collections. Blue corresponds to admirable collections. Green corresponds to writable collections. Yellow corresponds to messageable collections. Orange corresponds to viewable collections. Red then corresponds to the subscribed collection. For design purposes, pastel color variations of these colors may be used. Alternatively, a different set of backgrounds representing a spectrum of values ​​can be used. These backgrounds are applied to the displayed collection in the exemplary embodiment of Figure 32. For the exemplary embodiment of Figure 31, the background of the web page is changed to correspond to the color associated with the permission of that collection.

[0120] In exemplary embodiments, to facilitate browsing through a large number of items containing annotations in a single collection, annotations can be grouped into stack sets of displayed annotation items. Such stack sets of displayed annotation items appear as a single item, having the visual appearance of stacked items. It is beneficial to place up to nine annotations or another fixed configurable value, displayed annotation items from the same URL into a stack set of displayed annotation items. To facilitate such display, the paging mechanism must support the return of items in this form. As an example, imagine that eight out of ten items are annotation items belonging to the same URL. When these eight items are grouped into a stack set of displayed annotation items, there will be three items on the collection item page. To ensure that there are still ten items even when stack sets of displayed annotation items are used, the annotation server needs to return more items

[61] . In exemplary embodiments, a relational database

[0246] may be used, and complex Structured Query Language (SQL) queries using windowing may be required. Generally, object-relational mapping (ORM) systems are used by most modern web frameworks to access relational databases

[0246] . In an exemplary embodiment, the ORM system did not support window functions. To overcome this limitation, the underlying relational algebra library used by the ORM system was employed. Using this relational algebra, complex SQL queries were generated with appropriate window functions. This made it possible to display pages of collection items stacked with the displayed sets of annotation items.

[0121] Collection filters, such as the collection page search filter

[0268] shown in the exemplary embodiment of Figure 32, are used to receive input from the user. In the case of the collection page search filter

[0268] , this is a collection search phrase. These inputs are converted into collection parameter values ​​that are sent to the annotation server

[61] , as shown in some figures. These collection parameter values ​​are used by the annotation server

[61] to input collection parameters into a collection request query used by the electronic database

[62] , as shown in some figures. The collection request query includes collection parameter values ​​and collection page numbers. The collection request query, including collection parameter values, is used to specify a set of collections and the ordering of those collections. The collection page number specifies a particular collection page. In addition, the collection request query may include a specified number of collections to be sent to a web browser running on a client electronic device. This collection request query includes collection parameters of collection filter values ​​such as the collection search phrase, an exclusion list of collections to exclude from the list of specified collections, and the sequential ordering of that list of collections.

[0122] In the exemplary embodiment depicted in Figure 33, the inbox page is shown. The Collection Message Chains section and title

[0277] includes the American History Collection Message Chains section and title

[0279] and the Car Collection Message Chains section and title

[0280] . Collection message chains are message chains associated with a collection. These collection message chains appear in the “Collection Messages” section under the name of the collection to which the collection message chain belongs. In this exemplary embodiment, the American History Collection Message Chains section and title

[0279] lists the American History Collection Message Chain

[0281] , and the Car Collection Message Chains section and title

[0280] lists the Car Collection Message Chains 1

[0282] and the Car Collection Message Chains 2

[0283] . Untagged message chains are message chains that are not associated with any collection. Untagged message chains appear in the “Untagged Messages” section. In an exemplary embodiment, the untagged message chain section and title

[0278] includes untagged message chain 1

[0284] , untagged message chain 2

[0285] , and untagged message chain 3

[0286] . When a user clicks on a message chain, the message chain is displayed in the message chain context area

[0287] . The message chain context area

[0287] displays the first system message

[0288] of the American History message chain, along with a system message acceptance button

[0289] and a system message rejection button

[0290] . The user can accept the invitation to the collection by pressing the system message acceptance button

[0289] or reject the invitation to the collection by pressing the system message rejection button

[0290] .

[0123] In the exemplary embodiment depicted in Figure 34, the inbox page is displayed after the user accepts an invitation to join the collection. A system message notification

[0291] indicates that the user has accepted an invitation to join the collection. A message chain splitter

[0292] separates one user's messages from other users' messages. It should be obvious to those with common skill in the art that the message chain splitter

[0292] can also be implemented using various visual design techniques to separate one user's messages from another user's messages in this context. Next, the first text message

[0293] of the American History message chain, the second text message

[0294] of the American History message chain, and the third text message

[0295] of the American History message chain are shown. Then, the message composition section and title

[0296] are displayed along with the message composition box

[0297] and the send message button

[0298] . This inbox has connections to the authorization management system and the collection. Granting permissions to a message chain allows users who didn't originally have access to the message chain when it was created to see all other messages within it. This facilitates discussion and controls access at the message chain level. Furthermore, system message chains typically ensure that collection administrators are informed about the activities and permissions of group members. Additionally, users with messageable permissions can send a message chain in a single message to users within a collection or to selected groups within a collection. The ability to create collection chains clarifies the subject of discussion and allows users to focus their attention on the collections and tasks they want to concentrate on.

[0124] The inbox is an effective way to invite users to collections when they are already on the site. The inbox is also effective for sharing items with others when users are already on the site. To facilitate sharing items and collections when users are not on the site, links with cryptographic hashes are used. These links are unpredictable, and users with the link do not need to create an account to access them. Sharing is possible simply by possessing the link. Additionally, public items such as annotations and collections can be shared via social media.

[0125] Figure 21 shows a flowchart of a nurse using an exemplary embodiment of an annotable web browser

[24] , as shown in several figures. The nurse wants to inform the patient that certain foods are not good for diabetic patients and then show that their favorite foods can be modified to be more suitable for diabetic patients. Once this notepad is created, it can be shared with the patient. Day 1

[0132] The nurse visited the Wikipedia page for glycemic index, created a check annotation with the annotation comment "Supported by the American Diabetes Association" and "Most doctors strongly believe that high glycemic index foods are detrimental to the health of people with diabetes"

[0134] , and added the annotation to Notepad

[0135] . Next, the nurse visited allrecipes.com and found a recipe for "Flavorful Spanish Rice and Beans," highlighted "1 cup of raw white rice," created an exclamation annotation with the annotation comment "Rice has a very high glycemic index, but the same recipe can be made with pearl barley"

[0136] , and added the annotation to Notepad

[0137] . Next, the nurse visited the Harvard Health website, found the page listing the glycemic index of over 100 foods, highlighted "cooked white rice*", and added an exclamation mark annotation stating, "Most people don't realize white rice has such a high glycemic index. Note that it's 73, which is higher than most foods."

[0138] and added the annotation to her notepad.

[0139] Next, the nurse visited the Harvard Health website, found the page listing the glycemic index of over 100 foods, highlighted "pearl barley" with a dagger mark annotation stating, "Most people don't realize pearl barley has such a low glycemic index. Note that it's 28, which is lower than most foods."

[0140] and added the annotation to her notepad.

[0141] The nurse then visited the target website, found the pearl barley product page, selected "Pearl Barley - 1lb - Market Pantry" with dagger annotations, and wrote, "Pearl barley can be cooked in a rice cooker just like rice. Furthermore, cooked pearl barley has the appearance and texture of grain and is an excellent substitute for rice. It can be used as a substitute for rice in recipes that require rice."

[0142] Then she added the annotation to her notepad.

[0143] Day 2 after creating the notepad.

[0133] A nurse shares a notepad with a diabetic patient who likes to make Spanish rice.

[0144]

[0126] In Figure 22, a flowchart is shown of a patient using an exemplary embodiment of an annotable web browser with notepad functionality

[24] to view a notepad created by a nurse. On day 2, patient

[0145] opens the notepad link, clicks on the first displayed annotation screenshot bundle embedded in the notepad

[0147] , reads the Wikipedia title of glycemic index, observes the checkmark annotation symbol, then reads the highlighted text “Endorsed by the American Diabetes Association,” and reads the annotation comment “Most physicians strongly believe that high glycemic index foods are detrimental to the health of people with diabetes”

[0148] . Next, the patient clicks on the screenshot bundle of the second annotation embedded in Notepad

[0149] , reads the title "Flavorful Spanish Rice and Beans" from allrecipes.com, looks at the displayed exclamation mark annotation symbol, reads the highlighted text "1 cup of raw white rice", and then reads the annotation comment "Rice has a very high glycemic index, but the same recipe can be made with pearl barley"

[0150] . Subsequently, the patient clicks on the screenshot bundle of the third annotation embedded in Notepad

[0151] , reads the title "Glycemic Index of 100+ Foods" from the Harvard Health website, and after looking at the exclamation mark annotation symbol, reads the highlighted text "Boiled white rice*", and then reads the annotation comment "Most people don't realize that white rice has such a high glycemic index. Note that it's 73, which is higher than most foods."

[0152] . Next, the patient clicks on the screenshot bundle of the fourth annotation embedded in Notepad

[0153] , reads the title of the Harvard Health website, “Glycemic Index of More Than 100 Foods,” and after seeing the displayed annotation symbol of a dagger, reads the highlighted text, reading, “Pearl Barley,” and “Many people are unaware that pearl barley has such a low glycemic index.”The patient then reads an annotation comment

[0154] that reads, “Note that it is 28, lower than most foods.” The patient then clicks on the fifth displayed annotation screenshot bundle embedded in the notepad

[0155] , reads the title of the target website and realizes it is the product page for pearl barley, reads the highlighted text “Pearl Barley - 1lb - Market Pantry” after seeing the displayed annotation symbol of a dagger, and reads an annotation comment

[0156] that reads, “Pearl barley can be cooked in a rice cooker just like rice. In addition, cooked pearl barley has the look and feel of grain and can be used as a substitute for rice. It can be used as a substitute for rice in recipes that require rice.” After understanding the contents of the notepad, the patient goes to Target on grocery shopping day

[0146] , buys pearl barley and makes “Flavorful Spanish Barley and Beans”

[0157] .

[0127] In an exemplary embodiment depicted in Figure 23, an annotable web browser with notepad functionality

[24] is presented. The annotable web browser with notepad functionality tracks whether an active notepad exists and allows the user to select an active notepad from a group of accessible notepads. Figure 23 shows an active notepad

[0159] and annotations

[0160] embedded in the notepad. If an active notepad

[0159] exists, the annotable web browser with notepad functionality is in a state where an active notepad exists; if an active notepad

[0159] does not exist, the annotable web browser with notepad functionality is in a state where an active notepad does not exist. The displayed notepad contains many notepad embedded annotations

[0160] . A notepad embedded annotation

[0160] is a container containing a displayed annotation bundle that shows the data contained in the annotation display bundle. In an exemplary embodiment, a notepad embedded annotation

[0160] contains a displayed annotation screenshot bundle that also includes annotation symbols. In an alternative embodiment, the Notepad embedded annotation

[0160] may include a displayed screenshot bundle that does not contain annotation symbols or annotation comments. The user can add the annotation to Notepad by clicking the Add to Notepad icon button

[0158] as depicted in Figure 23. The Notepad contains a list of references to annotation objects. As shown in some figures, the electronic database

[62] stores the Notepad object along with the list of references to annotation objects. When the annotation server

[61] receives a request from an annotable web browser

[24] , as shown in some figures, it retrieves the Notepad object and annotation objects from the electronic database

[62] and sends the Notepad to the annotable web browser. It is displayed in a web browser

[24] . When the active notepad

[0159] is displayed, the user can navigate to the annotation by simply clicking the notepad embedded annotation

[0160] , as depicted in the notepad embedded annotation

[0160] in Figure 23. In the class of exemplary embodiments, each notepad is associated with a bundle display set. In exemplary embodiments, each notepad may be associated with a screenshot bundle display set. In another exemplary embodiment, each notepad may be associated with an annotation screenshot bundle display set.

[0128] The notepad contains rich information through links to other sources, along with annotation comments. This information may be supplemented by explanations written by the notepad's creator. These notepads can be used to create presentations in various formats, including PDF, Microsoft Word, and various other open-source and proprietary document formats. These presentations are created by sequentially including, for each annotation, a relevant screenshot with the marked text

[37] , separate contextual annotation highlighting text

[38] , a separate contextual question displayed annotation symbol

[39] , and a separate contextual annotation comment

[40] , as shown in Figure 9. Furthermore, this information can be supplemented with text summaries of linked articles generated using natural language processing techniques along with image captioning techniques. In addition, users can create explanatory text to add to specified locations, and these explanatory texts will be included in the presentation output.

[0129] In an exemplary embodiment depicted in Figure 24, a sequence diagram illustrates the interaction between a user

[59] for creating a notepad, an annotable web browser with notepad functionality

[0161] , a URL server

[60] , an annotation server

[61] , an electronic database

[62] , and a content delivery network

[63] . First, the user

[59] visits the URL of the annotable web browser with notepad functionality

[0161]

[0162] , requests a web document from the URL server

[60]

[0163] , the URL server returns the web document to the annotable web browser with notepad functionality

[0161]

[0164] , and displays the web document to the user

[59]

[0165] . Next, the user

[59] creates a notepad for "Flavoured Spanish Barley and Beans" in an annotationable web browser with notepad functionality

[0161]

[0166] , sends a request to the notepad server

[61] to create a notepad for "Flavoured Spanish Barley and Beans"

[0167] , and sends an instruction to the electronic database

[62] to create a notepad object

[0168] . Next, the electronic database

[62] approves the creation of the notepad object to the annotation server

[61]

[0169] , the annotation server approves the creation of the notepad in the annotationable web browser with notepad functionality

[0161]

[0170] , and displays the displayed notepad to the user

[59]

[0171] . Next, the user

[59] provides input to create a blood glucose annotation

[0172] , sends the annotation screen bundle to the annotation server

[61]

[0173] , and the server sends the image for storage to the content delivery network

[63]

[0174] , the content delivery network

[63] returns the URL of the stored image to the annotation server

[61]

[0175] , and sends a glycemic index annotation object creation instruction with the CDN URL to the electronic database

[62]

[0176] .Next, the electronic database

[62] authorizes the annotation server

[61] to create an annotation object

[0177] , the annotation server sends the received annotation screenshot bundle to an annotation-enabled web browser with notepad functionality

[0161]

[0178] , and the displayed annotation screenshot bundle is presented to the user

[59]

[0179] . Next, the user

[59] adds a blood glucose annotation to the notepad of the annotation-enabled web browser with notepad functionality

[0161]

[0180] , the annotation server

[61] requests that a blood glucose annotation object reference be added to the notepad

[0181] , and instructions to add a glycemic index annotation object reference to the notepad are sent to the electronic database

[62]

[0182] . The electronic database

[62] then allows the annotation server

[61] to add an annotation object reference

[0183] , the annotation server allows an annotation-enabled web browser with notepad functionality

[0161] to add a blood glucose annotation object reference to the notepad

[0184] , and the user

[59] is shown a web document with notepad that includes the displayed annotation screenshot bundle

[0185] .

[0130] In an exemplary embodiment depicted in Figure 25, the sequence diagram shows the interaction between a user

[59] , an annotable web browser with notepad functionality

[0161] , a URL server

[60] , an annotation server

[61] , an electronic database

[62] , and a content delivery network

[63] for the purpose of viewing a notepad. First, the user

[59] accesses an arbitrary URL with the annotable web browser with notepad functionality

[0161]

[0186] , requests a notepad for "Tasty Spanish Barley and Beans" from the annotation server

[61]

[0187] , and requests a notepad object from the electronic database

[62]

[0188] . Next, the electronic database

[62] returns the notepad object to the annotation server

[61]

[0189] , and the annotation server returns the notepad to the annotable web browser with notepad functionality

[0161]

[0190] . Next, the annotation-enabled web browser with notepad functionality

[0161] requests a blood glucose level annotation screen display bundle with annotation symbol data from the annotation server

[61]

[0191] , requests a blood glucose level annotation object from the electronic database

[62]

[0192] , returns the annotation object with CDN URL to the annotation server

[61]

[0193] , and returns the CDN URL and the annotation screen display bundle with annotation symbol data to the annotation-enabled web browser with notepad functionality

[0161]

[0194] . Next, the annotation-enabled web browser with notepad functionality

[0161] requests an image from the CDN using the CDN URL from the content delivery network

[63]

[0195] , returns the image from the CDN at the specified URL to the annotation-enabled web browser with notepad functionality

[0161]

[0196] , and displays the displayed notepad to the user

[59]

[0197] .Next, the user

[59] clicks the first displayed annotation screenshot bundle in the annotation-enabled web browser with notepad functionality

[0161]

[0198] , requests the web document from the server for the URL

[60]

[0199] , the web document is returned to the annotation-enabled web browser with notepad functionality

[0161]

[0200] , and the annotation symbols displayed in the open state are displayed on the web document

[0201] . Then, the annotation-enabled web browser with notepad functionality

[0161] displays the web document to the user

[59] with the displayed annotation symbols open, notepad placed to the side, and the browser scrolled to the appropriate position

[0202] .

[0131] In an exemplary embodiment depicted in Figure 26, a flowchart of notepad creation is presented showing the steps performed by a set of instructions for a web browser extension that transforms a web browser that supports the addition of browser extensions into an annotable web browser with notepad functionality. User

[59] opens the browser

[0203] , then the web browser

[0102] loads the browser extension

[0204] . Next, User

[59] visits a URL

[0205] . Subsequently, the web browser

[0102] reads the data sent from the visited URL

[0206] and injects a content script

[0207] . Then, User

[59] browses a web page

[0208] and creates a notepad

[0209] . As a result, the web browser extension content script

[0103] creates the notepad

[0210] , and the web browser extension background script

[0104] sends a notepad creation request to the annotation server

[0211] . Next, the web browser extension content script

[0103] displays the Notepad

[0212] , the user

[59] views the displayed Notepad and adds the already created annotations to the Notepad

[0213] . After that, the web browser extension content script

[0103] adds the already created annotations to the Notepad

[0214] , and the web browser extension background script

[0104] sends a request to the annotation server to add an annotation object reference to the Notepad

[0215] .

[0132] In an exemplary embodiment depicted in Figure 27, a flowchart of a notepad display is presented showing the steps performed by a set of instructions for a web browser extension that transforms a web browser that supports the addition of browser extensions into an annotable web browser with notepad functionality. User

[59] opens the browser

[0216] , and the web browser

[0102] loads the browser extension

[0217] . User

[59] then visits an arbitrary URL

[0218] . Next, the web browser

[0102] reads the data sent from the visited URL

[0219] and injects a content script

[0220] . User

[59] then opens notepad

[0221] , and the web browser extension content script

[0103] requests notepad

[0222] . Next, the web browser extension background script

[0104] requests the annotation server to provide a notepad containing annotation screenshot display bundles and annotation symbol data for all corresponding annotation objects in the reference list to annotation objects

[0223] , and the web browser extension content script

[0103] displays the displayed notepad

[0224] . The user

[59] then browses the displayed notepad and clicks the first displayed annotation screenshot bundle

[0225] , and the web browser extension content script

[0103] sends a message to the browser to visit the URL

[0226] . The web browser

[0102] then accesses the URL

[0227] and reads the data sent from the URL

[0228] . Next, the web browser extension background script

[0104] requests additional annotation object information, such as replies to annotation comments, from the annotation server

[0229] , and retrieves the additional annotation object information from the annotation server

[0230] . Next, the web browser extension content script

[0103] displays the annotation symbol in an open state

[0231] and scrolls to the position of the displayed annotation symbol

[0232] .The user

[59] then scrolls to the location of the annotation symbol that is displayed in the open state and views the page

[0233] .

[0133] Figure 28 presents a system diagram illustrating how a first user

[0234] , a second user

[0235] , a third user

[0236] , the first user's browser

[0237] , the second user's browser

[0238] , the third user's browser

[0239] , an annotation server

[61] , a content delivery network

[63] , and an electronic database

[62] interact. The user communicates with the annotation server

[61] , and the annotation server pushes images to the content delivery network

[63] . The user also communicates with the content delivery network

[63] to retrieve images and other files. The user does not communicate directly with the electronic database

[62] . In this exemplary embodiment, the annotation server

[61] is located on one computer system, and the electronic database

[62] is located on a second computer system. The number of machines used can be reduced by placing the electronic database

[62] on the same machine as the annotation server

[61] . The issue here is the number of concurrent users of the system. If this issue is resolved, it may be possible to determine the size of the system. In this exemplary embodiment, the content delivery network

[63] is operated by a third party. The content delivery network (CDN) may be provided by a third party. Many third parties provide CDNs. These CDNs can be configured using a web interface, and files can be uploaded and accessed via an application programming interface (API). This API may be a REST (representational state transfer) API. Setting up and using such a third-party CDN according to its third-party documentation should be obvious to those skilled in the art.

[0134] In an exemplary embodiment depicted in Figure 29, a component diagram is presented illustrating how to build a more robust system that allows for the isolation of different components. Isolating concerns can be advantageous in improving security and facilitating reproducible software deployment of machines to a network. The added security, isolation of concerns, and reproducible deployment can facilitate scaling the system to handle a large number of users. Figure 29 shows a simplified version of an annotation server

[61] and electronic database

[62] communicating next to an annotation server

[61] which includes a load balancer

[0241] , a front-end web server

[0242] , an annotation API server

[0243] , a deep learning API

[0244] , and a message broker

[0245] . Also shown is an electronic database

[62] which includes a relational database

[0246] and a key-value store database

[0247] . In this exemplary embodiment, the load balancer

[0241] , front-end web server

[0242] , annotation API server

[0243] , deep learning API

[0244] , message broker

[0245] , relational database

[0246] , and key-value store database

[0247] each run on a separate machine having one or more central processing units (CPUs), memory, and non-temporary computer-readable storage media. Furthermore, the deep learning API

[0244] comprises at least one of a graphical processing unit (GPU), a tensor processing unit (TPU), or a field-programmable gate array (FPGA). In the exemplary embodiment shown in Figure 29, the load balancer

[0241] machine comprises one central processing unit (CPU) having four cores, 8 GB of random access memory (RAM), and a 250 GB solid-state drive (SSD), while the front-end web server

[0242] machine comprises one CPU having four cores, 8 GB of RAM, and a 250 GB SSD.The Annotation API Server

[0243] machine has two CPUs, each with 32 cores, for a total of 64 cores, 64GB of RAM, and a 2TB SSD. The Deep Learning API

[0244] machine has one 32-core CPU, 128GB of RAM, two GPUs, each with 11GB of RAM, and a 2TB SSD. The Message Broker

[0245] machine has one 4-core CPU, 32GB of RAM, and a 1TB SSD. The Relational Database

[0246] machine has one 32-core CPU, 128GB of RAM, and a 2TB SSD, while the Key-Value Store Database

[0247] machine has one 4-core CPU, 128GB of RAM, and a 1TB SSD. Machines meeting these hardware specifications enable this system to support many users with multiple concurrent users.

[0135] The hardware specifications were described in the exemplary embodiment shown in Figure 29. Next, the rationale for these specifications is explained. The load balancer

[0241] simply routes requests to the appropriate locations. As a result, the machine's requirements for non-temporary computer-readable storage should be met by a 250 GB SSD. Compared to the computations performed by the rest of the system, the load balancer

[0241] has a low computational and memory load. As a result, 2 GB per core is sufficient, and 4 cores are sufficient considering the rest of the system. As the system scales up, the number of instructions executed per second increases, and the number of CPU cores increases, which may prevent the load balancer

[0241] from becoming a bottleneck. However, given the specifications given for the exemplary embodiment shown in Figure 29, it is unlikely that the load balancer

[0241] will become a bottleneck. The front-end web server

[0242] also does not require a lot of non-temporary computer-readable storage, a lot of CPU cores, or a lot of memory. This machine does not store user data, and any front-end code it stores can be supported by allocated storage space. Furthermore, the CPU cores of the front-end web server

[0242] machine only send the requested resources to the user, and in typical use, the user does not often request front-end code so that the user's browser makes API calls to the annotation API server

[0243] . As a result, the CPU cores of the front-end web server

[0242] machine have lower computational load and memory requirements than the annotation API server

[0243] machine. The annotation API server

[0243] has a higher computational load because API requests are made from the user's web browser when creating, organizing, and viewing annotations. The total computing power of the annotation API server

[0243] machine can become a bottleneck if there are many concurrent users. In an exemplary embodiment, this machine has 64 CPU cores and 64 GB of RAM.If the Annotation API Server

[0243] code is written in a thread-safe manner, 1GB of RAM per CPU core is sufficient. If the code is not thread-safe, or if the web framework consumes a lot of memory, it may be desirable to use 2GB of RAM per CPU core for the Annotation API Server

[0243] . To scale the Annotation API Server

[0243] to accommodate many users, it is more cost-effective to add machines than to provide many CPU cores and a lot of RAM on each machine. The hard disk size of the Annotation API Server

[0243] is mainly to allow for the storage of log files. The Deep Learning API

[0244] would benefit from two GPUs with sufficient GPU RAM to run deep learning models. The CPU cores and RAM are used to handle CPU-intensive data processing steps and to bring data consumed by the GPUs into memory. A 2TB SSD can store large deep learning models and other data for these models. The message broker

[0245] should operate whenever the system is running, but it will not face a shortage of computing power. 32 GB of RAM is present to ensure sufficient RAM for storing large messages that may contain images. Since these images will not be stored for long periods, 32 GB of RAM is sufficient. Furthermore, four cores are sufficient for the message broker

[0245] . In this exemplary embodiment, the message broker

[0245] has lower computing requirements in terms of instructions per second than the annotation API server

[0243] . The key-value store database

[0247] uses one core in this exemplary embodiment. However, the key-value store database

[0247] uses RAM to store in memory the reputation of various item objects, such as annotation objects, and user reputation votes for these item objects.Furthermore, disk space is used for logging and temporarily storing backups of the key-value store database in memory. The relational database

[0246] has 128GB of RAM so that all the data in the database can be stored in RAM during operation, 2TB of SSDs to ensure sufficient storage space for logging, and 32 cores to provide the computing power to process queries sent from the annotation API server

[0243] .

[0136] To scale up the system depicted in the exemplary embodiment of Figure 29, the following steps can be taken. The new exemplary embodiment will have one or more load balancers

[0241] , one or more front-end web servers

[0242] , one or more annotation API servers

[0243] , and one or more deep learning APIs

[0244] . Load balancers, front-end web servers, annotation API servers, and deep learning API servers can be added to improve the overall system performance. Often, message brokers can be organized into message broker clusters. Message broker clusters can be used to scale up the new exemplary embodiment. Most key-value store database

[0247] software also has the ability to create key-value store database clusters.

[0137] In a new exemplary embodiment, it is ideal to use a relational database

[0246] that uses threads as opposed to processes, so that the relational database

[0246] can handle the most users. In some cases, threads take up about 256kb of memory per thread, while processes take up 10mb of memory per process. Therefore, thread-dependent relational databases can have more than 20 times the number of open connections compared to process-dependent relational databases. Even with the connection pooling tools used by process-dependent relational databases, thread-based connections are suitable for the style of connections made by the annotation API server

[0243] . The annotation API server

[0243] is transaction-dependent and requires performance. As a result, it is ideal to use SQL transaction pooling rather than SQL session pooling or SQL statement pooling. SQL session pooling is less performant, and SQL statement pooling makes transaction implementation more difficult than using SQL transaction pooling. When using SQL transaction pooling, thread-based connections offer the best performance when computer instructions need to be executed on processors not being used by the relational database

[0246] because the connection remains open while work is being done on the external processor. As a result, the relational database

[0246] may be given fewer SQL statements to execute than it can handle. This results in a decrease in overall throughput.

[0138] To further enhance the capabilities of the relational database

[0246] , it is advantageous to read-scale the relational database

[0246] by creating read replicas. The system can be further scaled up by reducing the write load of the relational database

[0246] . The write load can be reduced by implementing reputation voting through a key-value store database

[0247] in a new exemplary embodiment. To further expand the capacity of the relational database, in a new exemplary embodiment, the number of CPU cores and RAM of the relational database

[0246] machine can be increased. Another step that can be taken is to use an in-memory relational database

[0246] that implements SQL without the full endurance guarantee of most relational databases. Such a system can use an in-memory database with checkpoint saving and write-ahead logging to ensure that in the event of a power outage, database data other than database data that arrived in the in-memory database within the last few seconds is not lost. Through these steps, the relational database can be further expanded. In an HPC cluster, remote direct memory access (RDMA) via InfiniBand networking allows relational databases to use more memory than is locally available on a single machine. Because this HPC cluster can scale to hundreds or thousands of machines, the corresponding relational database

[0246] can support a large number of concurrent users.

[0139] In the exemplary embodiment depicted in Figure 30, different types of client electronic devices

[0248] capable of communicating with an annotation server

[61] and a content delivery network

[63] are shown. The devices shown are a laptop

[0249] , a desktop

[0250] , an iPad

[0251] , an Android tablet

[0252] , an iPhone

[0253] , an Android mobile phone

[0254] , an Apple TV via Air-Web

[0255] , an Android TV

[0256] , and a Raspberry Pi

[0257] . The laptop

[0249] , desktop

[0250] , Android tablet

[0252] , Android mobile phone

[0254] , and Raspberry Pi

[0257] support an annotation-enabled web browser with notepad functionality through the introduction of a web browser and a web browser extension. Additionally, iPads

[0251] and iPhones

[0253] support annotable web browsers by installing the Safari web browser and app extensions. Apple TVs

[0255] and Android TVs

[0256] via Air Web may not currently be suitable for installing an annotable web browser, including the web browser and web browser extensions. Furthermore, their form factors may not be ideal for creating annotations. However, it is possible to create a progressive web application for these devices that communicates with an annotation server. This would allow the annotated material to be displayed, much like a digital photo frame displays images. This would enable these devices to display the rich annotations created using other devices, as shown in Figure 30.

[0140] Next, the annotation of images, videos, and audio recordings will be described. A GUI diagram in an exemplary embodiment will be presented as showing what the system displays and the final result of non-temporary computer-readable media instructions performed by a web browser running on the processor. In the exemplary embodiment, the non-temporary computer-readable media instructions may be for a web browser running on the processor, or they may be incorporated into an annotable web browser. As previously described, the annotable web browser in the exemplary embodiment is a web browser with web browser extensions. When using an annotable web browser, images, videos, and audio recordings embedded within any web page can be annotated. The website being annotated does not need to include computer-readable media instructions for annotating embedded images, videos, and recordings. When using a regular web browser, images, videos, and recordings can be annotated if non-temporary computer-readable media instructions are delivered via a specific website and the content annotation is performed on that specific website.

[0141] In addition to annotable web browsers that annotate documents and media on the World Wide Web, software applications for mobile devices and internet televisions may also use non-temporary computer-readable media instructions that implement the methods, techniques, and interfaces described herein to annotate images, videos, and recordings available to groups of users on the Internet or a local intranet.

[0142] After describing the GUI, the section will discuss data bundles created for transmitting data between components. As the relevant data bundles are discussed, the methods and system architecture will be discussed within the context of the system architecture and components already discussed.

[0143] In the exemplary embodiment depicted in Figure 35, an image

[0299] is shown with an image annotation subject

[0300] , a bounding box for the image annotation subject

[0301] , an image annotation arrow

[0302] , and an image annotation comment

[0303] within the box. The image annotation subject

[0300] is an entity of interest contained in the image. The image annotation subject

[0300] may be a person, a car, a soccer ball, a tennis ball, or any other object identifiable in the image by a person or computer system. The bounding box for the image annotation subject

[0301] is a bounding box drawn around the image annotation subject

[0300] by a user or computer system. The image annotation comment

[0303] within the box is created by a user, and the annotation comment must be related to the image annotation subject

[0300] . The image annotation arrow

[0302] is a line that may have arrowheads on both sides, connecting the bounding box

[0301] of the image annotation subject and the image annotation comment

[0303] within the box. The bounding box

[0301] of the image annotation subject, the image annotation arrow

[0302] , and the image annotation comment

[0303] within the box are user-created objects included in the image annotation. Users can select the rectangle tool from a panel on the web page or from a floating toolbar to draw the subject bounding box

[0301] of the image annotation. Furthermore, users can input text and the position of the image annotation comment

[0303] within the box. The subject bounding box

[0301] of the image annotation and the image annotation comment

[0303] within the box are connected by the image annotation arrow

[0302] . Image and annotation objects may not all be displayed simultaneously.For example, it is possible to display only the bounding box

[0301] of an image annotation subject, while other objects of the image annotation become visible when the mouse is moved over the image annotation subject

[0300] or the bounding box

[0301] of the image annotation subject. Visible elements of an image annotation are elements that are made visible without hovering over other elements. Mouseover elements of an image annotation are elements that become visible when the user mouses over a visible element of the image annotation.

[0144] Below the image

[0299] , a control panel is displayed that includes the previous image annotation arrow button

[0304] , the next image annotation arrow button

[0305] , the "Include Image Collision" button

[0306] , the "Add Image Candidate Annotation Group" button

[0307] , the "Image Candidate Annotation Group List Box"

[0308] , and the "Create Image Annotation Group" button

[0309] . The label of the "Add Image to Annotation Group Candidate" button

[0307] is the short name "Add to Annotation Group". However, an image annotation is first added to an image candidate annotation group, and then becomes an image annotation group when the user clicks the "Create Image Annotation Group" button

[0309] . An image annotation contains objects such as the bounding box for the image annotation subject

[0301] , the image annotation arrow

[0302] , and the image annotation comment

[0303] within the box. An image annotation may contain one or more annotation subjects

[0300] . In the electronic database

[62] , image annotations are stored as image annotation objects. These image annotation objects are given a sequential order by the electronic database

[62] . This ordering of image annotations gives meaning to the concepts of previous and next image annotations. The previous image annotation arrow button

[0304] removes the objects contained in the current image annotation from the displayed view and adds the objects of the previous image annotation to the displayed view, assuming that the current image annotation is not an image annotation candidate. The next image annotation arrow button

[0305] removes the objects contained in the current image annotation from the displayed view and adds the objects of the next image annotation to the displayed view, assuming that the current image annotation is not an image annotation candidate. An image annotation group contains multiple image annotations.Before creating an image annotation group, you must select an image annotation. The selected image annotation becomes a candidate image annotation. The candidate image annotation is listed in the image candidate annotation group list box

[0308] . Figure 35 illustrates a scenario where no image annotation has yet been added to the image candidate annotation group list box

[0308] . Therefore, the message in the image candidate annotation group list box

[0308] is the image candidate annotation group list empty message

[0310] .

[0145] The button

[0306] containing image collisions determines the display properties of image annotations that collide with the image annotation candidates listed in the image annotation candidate group list box

[0308] . The possible states are to exclude collisions and to include collisions. When the button

[0306] containing image collisions is clicked, image annotations that collide with the candidate image annotations listed in the image annotation candidate group list box

[0308] are displayed. In the exemplary embodiment shown in Figure 35, the button's label is changed to "Exclude Collisions". When this button is clicked again, image annotations that collide with the image annotation candidates listed in the image annotation candidate group list box

[0308] are no longer displayed. Clicking the Image Annotation Group Create button

[0309] creates an item that displays an Image Annotation Group containing all the Images Annotation Group Candidate List Box

[0308] at the time the Image Annotation Group Create button

[0309] is clicked.

[0146] In alternative embodiments, the bounding box

[0301] and image annotation arrow

[0302] of the image annotation subject may not be displayed. In alternative embodiments, only the bounding box

[0301] of the image annotation subject and the image annotation comment

[0303] within the box exist. Alternatively, in another alternative embodiment, only the image annotation comment

[0303] within the box exists near the image annotation subject

[0300] .

[0147] In the exemplary embodiment shown in Figure 36, an image

[0299] is displayed in the image candidate annotation group list box

[0308] with the first image candidate annotation group list item

[0311] added. This state is generated by clicking the add button

[0307] to the image annotation candidate group in the exemplary embodiment shown in Figure 35. After the image annotation is added to the image annotation candidate group list, the state shown in Figure 37 is created by clicking the next image annotation arrow button

[0305] in the exemplary embodiment shown in Figure 36.

[0148] In the exemplary embodiment shown in Figure 37, the second image annotation subject

[0312] is displayed along with the user-created second image annotation subject bounding box

[0313] , second image annotation arrow

[0314] , and second image annotation comment

[0315] within the box. When the add button

[0307] to the image annotation candidate group is clicked in the state of the exemplary embodiment shown in Figure 37, the state of the exemplary embodiment shown in Figure 38 is generated.

[0149] In the exemplary embodiment shown in Figure 38, a second image candidate annotation group list item

[0316] is shown. When the image creation annotation group button

[0309] is pressed, an annotation group containing the two annotations shown in the image candidate annotation group list box

[0308] in Figure 38 is created.

[0150] In the exemplary embodiment shown in Figure 39, the user previously clicked the Image Collision button

[0306] in Figure 36, which allows the collision image annotation to be displayed. After pressing the Image Next Annotation arrow button

[0305] from the state shown in Figure 36, the state shown in Figure 39 may be obtained. In Figure 39, the collision image annotation arrow

[0317] and the collision image annotation comment in the box

[0318] are displayed.

[0151] In the exemplary embodiment depicted in Figure 40, a video

[0319] is shown along with video annotation playback

[0320] . The video

[0319] shows a video annotation subject

[0321] , a bounding box

[0322] for the video annotation subject, a video annotation arrow

[0323] , and a video annotation comment

[0324] within the box. The annotation subject

[0321] is a notable entity in the video. The video annotation subject

[0321] could be a person, a car, a soccer ball, a tennis ball, or any other object identifiable in the video by a person or a computer system. The bounding box

[0322] for the video annotation subject is a bounding box drawn around the video annotation subject

[0321] by a user or a computer system. The video annotation comment box

[0324] is created by the user, and the annotation comment must be related to the video annotation subject

[0321] . The video annotation arrow

[0323] is a line that can have arrowheads on both sides, connecting the video annotation subject bounding box

[0322] and the video annotation comment in box

[0324] . The video annotation subject bounding box

[0322] , video annotation arrow

[0323] , and video annotation comment input box

[0324] are objects created by the user and included in the video annotation. The user can select the rectangle tool from a panel on the web page or from the floating toolbar and draw the first video annotation subject bounding box

[0322] . Furthermore, they can enter text, place the video annotation comment in box

[0324] , and connect it with the video annotation arrow

[0323] . When video

[0319] is played, video annotations

[0320] are also played.During video annotation playback

[0320] , the user creating the video annotation can move the bounding box

[0322] of the video annotation subject using the mouse to verify that the bounding box

[0322] of the video annotation subject is properly positioned throughout the video. The user can also move the video annotation comment within box

[0324] with the mouse to verify that the video annotation comment within box

[0324] is properly positioned throughout the video. If the motion paths of the video annotation comment in box

[0322] and box

[0324] of the video annotation subject are known, the motion path of the video annotation arrow

[0323] can be calculated along with temporal rotation, temporal contraction, temporal expansion, and temporal position that correspond to the motion paths of the video annotation comment in box

[0322] and box

[0324] of the video annotation subject. Additionally, users can pause the video and resize the video annotation comments within the bounding boxes

[0322] and

[0324] of the video annotation subject. Not all video annotation objects are necessarily displayed simultaneously. For example, after a user mouses over the video annotation subject

[0321] or the bounding box

[0322] of the video annotation subject, other video annotation objects may appear, leaving only the bounding box

[0322] of the video annotation subject visible. Visible elements of a video annotation are elements that are visible without mouseover over other elements. Mouseover elements of a video annotation are elements that become visible when a user mouses over a visible element of the video annotation.

[0152] Below the video

[0319] , a control panel is displayed that includes the video previous annotation arrow button

[0325] , the video next annotation arrow button

[0326] , the video collision button

[0327] , the "Add to Video Candidate Annotation Group" button

[0328] , the video candidate annotation group list box

[0329] , and the "Create Video Annotation Group" button

[0330] . The label of the "Add to Video Annotation Group Candidate" button

[0328] is the short name "Add Annotation Group". However, video annotations are first added to the video candidate annotation group, and only become a video annotation group after the user clicks the "Create Video Annotation Group" button

[0330] . Video annotations include objects such as the video annotation subject bounding box

[0322] , video annotation arrows

[0323] , and video annotation comment in box

[0324] . A video annotation can contain multiple video annotation subjects

[0321] . In the electronic database

[62] , video annotations are stored as video annotation objects. These video annotation objects are then given a sequential order by the electronic database

[62] . This ordering of video annotations gives meaning to the concepts of previous and next video annotations. The previous video annotation arrow button

[0325] removes the objects contained in the current video annotation from the displayed view and adds the objects of the previous video annotation to the displayed view, assuming that the current video annotation is not a candidate video annotation. The next video annotation arrow button

[0326] removes the objects contained in the current video annotation from the displayed view and adds the objects of the next video annotation to the displayed view, assuming that the current video annotation is not a candidate video annotation.A video annotation group contains multiple video annotations. Before a video annotation group is created, video annotations must be selected. During the selection process, video annotations become candidate video annotations. Candidate video annotations are listed in the video candidate annotation group list box

[0329] . Figure 40 illustrates a scenario where no video annotations have yet been added to the video annotation candidate group list box

[0329] . Therefore, the message in the video annotation candidate group list box

[0329] is the video annotation candidate group list empty message

[0331] .

[0153] The "Include Video Collision" button

[0327] determines the display properties of video annotations that collide with candidate video annotations listed in the "Video Candidate Annotation Group List Box"

[0329] . For video annotations, a collision means that two video annotations collide at some point. Collisions may be displayed in red on the video annotation's playback line

[0320] . When the "Include Video Collision" button

[0327] is clicked, video annotations that collide with candidate video annotations listed in the "Video Candidate Annotation Group List Box"

[0329] are displayed. In the exemplary embodiment shown in Figure 40, the button label is changed to "Exclude Collisions". When this button is clicked again, video annotations that collide with candidate video annotations listed in the "Video Candidate Annotation Group List Box"

[0329] are no longer displayed. Clicking the "Create Video Annotation Group" button

[0330] creates the following items. When the video creation annotation group button

[0330] is clicked, a video annotation group is generated that includes all video annotations listed in the video candidate annotation group list box

[0329] . When the video add candidate annotation group button

[0328] is clicked in the state depicted in the exemplary embodiment of Figure 40, the state depicted in the exemplary embodiment of Figure 41 is created by adding the first video candidate annotation group list item

[0335] to the video candidate annotation group list box

[0329] .

[0154] In an alternative embodiment, the bounding box

[0322] and video annotation arrow

[0323] of the video annotation subject may not be displayed. In an alternative embodiment, only the bounding box

[0322] and the video annotation comment within box

[0324] of the video annotation subject are present. Or, in another alternative embodiment, only the video annotation comment within box

[0324] is present near the video annotation subject

[0321] .

[0155] In the exemplary embodiment shown in Figure 40, the video

[0319] player control is separated from the video annotation playback

[0320] . In an alternative embodiment, both controls can be integrated into a single playback control. This may result in a more complex input interface, but it can reduce the number of elements displayed on the screen. This comment regarding the integration of the two playback controls also applies to the exemplary embodiments shown in Figures 41, 42, 43, 44, and 45.

[0156] In an exemplary embodiment depicted in Figure 41, the bounding box motion path

[0332] of a video annotation subject, the video annotation arrow motion path

[0333] , and the video annotation comment input box motion path

[0334] are shown. The bounding box motion path

[0332] of the video annotation subject shows the motion path of the bounding box

[0322] as the video annotation subject

[0321] moves within the video, and is intended to function as the bounding box of the video annotation subject

[0321] . The video annotation arrow motion path

[0333] shows the motion path of the video annotation arrow

[0323] as it moves as the video

[0319] progresses over time. The video annotation arrow

[0323] can be seen to be temporally translated, rotated, expanded, and contracted relative to the video annotation subject's bounding box

[0322] and the video annotation comment

[0319] . Box

[0324] moves in time. The video annotation comment input box motion path

[0334] shows the motion path as the video annotation comment within box

[0324] moves in accordance with the temporal progression of video

[0319] .

[0157] In the exemplary embodiment shown in the figure, 42 shows a second video annotation subject

[0336] , a bounding box of the second video annotation subject

[0337] , a second video annotation arrow

[0338] , a second video annotation comment within the box

[0339] , a bounding box motion path of the second video annotation subject

[0340] , a second video annotation arrow motion path 1

[0341] , a second video annotation arrow motion path 2

[0342] , a final bounding box position of the second video annotation subject

[0343] , and a final second video annotation arrow position

[0344] . The bounding box

[0337] of the second video annotation subject follows the second video annotation subject

[0336] to the final bounding box position

[0343] of the second video annotation subject along the bounding box motion path

[0340] of the second video annotation subject. The second video annotation arrow

[0338] is transformed over time to the final second video annotation arrow position

[0344] . Transformations including translation, rotation, and contraction are depicted by motion path 1

[0341] and motion path 2

[0342] of the second video annotation arrow. The second video annotation comment within the box

[0339] remains fixed in time. The second video annotation candidate group list item

[0345] appears in the video annotation candidate group list box

[0329] , which is created when the user clicks the Add to Video Annotation Candidate Group button

[0328] .

[0158] In the exemplary embodiment shown in Figure 43, the user has previously clicked the collision button

[0327] including video in the state shown in the exemplary embodiment of Figure 41. As a result, a video annotation that has a collision with a video annotation in the candidate annotation group list is displayed. If, in the state shown in the exemplary embodiment of Figure 41, the video next annotation arrow button

[0326] is clicked, and then the collision button

[0327] including video is clicked, a collision video annotation arrow

[0346] and a collision video annotation comment in the box

[0347] may be displayed. In the exemplary embodiment of Figure 43, the collision is depicted. However, the collision may occur at a point not depicted in a particular video frame. In such a case, the collision may be depicted in red on the timeline of the video annotation playback

[0320] in the exemplary embodiment.

[0159] In the exemplary embodiment depicted in Figure 44, recording playback

[0348] and recording annotation playback

[0349] are shown. Below the recording annotation playback

[0349] are the recording annotation arrow

[0350] , the recording annotation comment box

[0351] , the recording annotation connection arrow

[0352] , and the recording derived annotation comment box

[0353] . Below the recording annotation playback

[0349] is a control panel including the pre-recording annotation arrow button

[0354] , the next-recording annotation arrow button

[0355] , the recording candidate annotation group add button

[0356] , the recording derived annotation create button

[0357] , the recording candidate annotation group list box

[0358] , the recording derived annotation group create button

[0359] , and the recording derived annotation comment input box

[0360] . The label for the "Add to Recording Candidate Annotation Group" button

[0356] is the short name "Add Annotation Group". However, recording annotations are first added to the recording candidate annotation group, and only become a recording annotation group after the user clicks the "Create Recording Annotation Group" button

[0359] .

[0160] In the exemplary embodiment depicted in Figure 44, to create recording annotations, the user uses the mouse to select a short segment of the recording annotation playback

[0349] . After the selection is made, a recording annotation arrow

[0350] and a recording annotation comment box

[0351] appear, originating from the center of the selection, and the user can create recording annotation comments in the recording annotation comment box

[0351] . Recording annotations include objects such as recording annotation arrows

[0350] and recording annotation comment boxes

[0351] . The short segment of the recording annotation playback

[0349] can be marked as shown in the embedded recording annotation playback range

[0393] in Figure 49. Figure 49 shows an embedded recording where the marked short segment is on the playback bar for recording playback

[0348] , but the same style of marked segment can be used in recording annotation playback

[0349] . Furthermore, it is possible to leave segments unmarked, or to display marked segments when the mouse hovers over a selected segment, recording annotation arrow

[0350] , or recording annotation comment box

[0351] . In addition, a question sound annotation symbol that operates similarly to the question embedded sound annotation symbol

[0394] in Figure 49 can be introduced.

[0161] A recording annotation can include one or more selected sound segments. In the electronic database

[62] , recording annotations are stored as recording annotation objects. These recording annotation objects are then given a sequential order by the electronic database

[62] . This ordering of recording annotations gives meaning to the concepts of previous and next recording annotations. The previous recording annotation arrow button

[0354] removes the objects contained in the current recording annotation from the displayed view and adds the objects contained in the previous recording annotation to the displayed view, assuming that the current recording annotation is not a candidate for recording annotation. The next recording annotation arrow button

[0355] removes the objects contained in the current recording annotation from the displayed view and adds the objects of the next recording annotation to the displayed view, assuming that the current recording annotation is not a candidate for recording annotation. A recording annotation group is a group of recording annotations that the user wants to group together. These groupings are displayed as a single item, and you can use one or more recording annotations from a recording annotation group to create a derived recording annotation that associates these one or more recording annotations. Furthermore, the derived recording annotation and the recording annotations it depends on can be displayed as a single item. Recording annotations can be added to a recording annotation group by clicking the Add to Recording Annotation Candidate Group button

[0356] . As a result, the new recording annotation will appear in the Recording Annotation Candidate Group list box

[358] . Once a recording annotation is added to a recording annotation group, the user can create recording-derived annotations. A recording-derived annotation is an annotation that links other recording annotations and adds comments to a recording-derived annotation comment box

[0353] that is connected to multiple recording annotation comment boxes.The recording-derived annotation comment box

[0353] is connected to each sound source annotation comment box

[0351] using the recording annotation connection arrow

[0352] . Recording-derived annotations are created by selecting the desired recording annotation and clicking the recording-derived annotation create button

[0357] . Next, enter the recording-derived annotation comment

[0361] into the recording-derived annotation comment input box

[0360] and press Enter. The recording candidate annotation group list box

[0358] contains the recording candidate annotation group list item

[0362] . This list contains both recording annotations and recording-derived annotations. The first item in the recording candidate annotation group list item

[0362] is the first recording candidate annotation group list item

[0363] .

[0162] In the exemplary embodiment depicted in Figure 45, a video

[0319] and video annotation playback

[0320] are shown. Below video annotation playback

[0320] are a video annotation arrow

[0364] , a video annotation comment box

[0365] , a video annotation connection arrow

[0366] , and a video derived annotation comment box

[0367] . Below video annotation playback

[0320] is a control panel including a video previous annotation arrow button

[0325] , a video next annotation arrow button

[0326] , a video collision button

[0327] , an add to video candidate annotation group button

[0328] , a video derived annotation create button

[0368] , a video candidate annotation group list box

[0329] , a video creation annotation group button

[0330] , and a video derived annotation comment input box

[0369] . The label for the Add Video button

[0328] is the short name "Add Annotation Group". However, video annotations are first added to a candidate video annotation group and become a video annotation group after the user clicks the Create Video Annotation Group button

[0330] . To create a video annotation in an exemplary embodiment, the user uses the mouse to select a short segment of video annotation playback

[0320] . After the selection is made, a video annotation arrow

[0364] and a video annotation comment box

[0365] appear, originating from the center of the selection, and the user can create a video annotation comment in the video annotation comment box

[0365] . The short segment of video annotation playback

[0320] can be marked as shown in the embedded video annotation playback range

[0375] in Figure 46. Figure 46 shows an embedded video, where the marked short segment is on the playback bar of the video

[0319] , but marked segments of the same style can be used in video annotation playback

[0320] .It is possible to prevent segments from being marked, or to display marked segments when the mouse hovers over a selected segment, a video annotation arrow

[0364] , or a video annotation comment box

[0365] . Furthermore, a question video annotation symbol that behaves similarly to the question embedded video annotation symbol

[0376] in Figure 46 can be introduced.

[0163] As previously described in another exemplary embodiment, a video annotation includes objects such as a bounding box for the video annotation subject

[0322] , a video annotation arrow

[0323] , and a video annotation comment within the box

[0324] . A video annotation can include multiple video annotation subjects

[0321] . In the electronic database

[62] , video annotations are stored as video annotation objects. These video annotation objects are then given a sequential order by the electronic database

[62] . This ordering of video annotations gives meaning to the concepts of previous and next video annotations. The previous video annotation arrow button

[0325] removes the objects contained in the current video annotation from the displayed view and adds the objects of the previous video annotation to the displayed view, because the current video annotation is not a candidate for video annotation. The next video annotation arrow button

[0326] removes the objects contained in the current video annotation from the displayed view, assuming that the current video annotation is not a video candidate annotation, and adds the objects of the next video annotation to the displayed view. A video annotation group is a collection of video annotations that the user wants to group together. These groupings are displayed as a single item, and one or more video annotations from a video annotation group can be used to create derived video annotations that associate these one or more video annotations. Furthermore, derived video annotations can be displayed as a single item along with the video annotations they depend on. To add a video annotation to a video candidate annotation group, click the Add to Video Candidate Annotation Group button

[0328] . As a result, the new video annotation will appear in the video annotation candidate group list box

[329] .When a video annotation is added to a video candidate annotation group, the user can create a video-derived annotation. A video-derived annotation is an annotation that links other video annotations and adds comments to a video-derived annotation comment box

[0367] that is connected to multiple video annotation comment boxes. The video-derived annotation comment box

[0367] is connected to each video annotation comment box

[0365] using the video annotation connection arrow

[0366] . A video-derived annotation is created by selecting the desired video annotation and clicking the video-derived annotation create button

[0368] . Next, enter the video-derived annotation comment

[0370] in the video-derived annotation comment input box

[0369] and press Enter. The video candidate annotation group list box

[0329] displays the video candidate annotation group list item

[0371] . This list includes both video annotations and video-derived annotations. The first item in the video candidate annotation group list item

[0371] is the first video candidate annotation group list item

[0372] . Clicking the video creation annotation group button

[0330] creates an annotation group containing the video candidate annotations in the video candidate annotation group list box

[0329] . The video contain collision button

[0327] functions the same as the video contain collision button

[0327] in the exemplary embodiments shown in Figures 40, 41, 42, and 43.

[0164] In the exemplary embodiment depicted in Figure 46, an embedded video

[0373] and embedded video playback

[0374] are shown. The embedded video

[0373] and embedded video playback

[0374] reside within a web page being viewed in an annotation-enabled web browser. In the exemplary embodiment depicted in Figure 46, the embedded video annotation playback range

[0375] , the question embedded video annotation symbol

[0376] , the disagreement embedded video annotation symbol

[0377] , the embedded video annotation beacon

[0378] , the embedded video annotation beacon question symbol

[0379] , the embedded video annotation beacon disagreement symbol

[0380] , and the embedded video annotation beacon playback time

[0381] are shown. The symbol beacon trigger button

[57] triggers the display of one fully displayed video annotation beacon for each video annotation in the embedded video

[0373] unless the embedded video

[0373] contains three or more video annotations. If there are three or more video annotations, an alternative display methodology allows the user to better visualize the video annotations that can be seen. The fully displayed video annotation beacon includes the embedded video annotation playback range

[0375] , the embedded video annotation symbol, the embedded video annotation beacon

[0378] , the embedded video annotation beacon symbol, and the embedded video annotation beacon playback time

[0381] . The embedded video annotation playback range

[0375] highlights the time range that the user considers relevant to the video annotation. Embedded video annotation symbols, such as the question embedding video annotation symbol

[0376] and the disagreement embedding video annotation symbol

[0377] , convey the semantic relationship of the annotation between the selected video segment and the video annotation comment. Embedded video annotation beacons

[0378] serve to draw the user's attention to the presence and some of the characteristics of the video annotation.The embedded video annotation beacon symbol places the semantic relationship of annotations between the selected video segment and the video annotation comment near the embedded video annotation beacon

[0378] . Two examples of embedded video annotation beacon symbols are the embedded video annotation beacon question symbol

[0379] and the embedded video annotation beacon disagreement symbol

[0380] . The embedded video annotation beacon playback time

[0381] , placed near the embedded video annotation beacon

[0378] , conveys the length of the video segment related to the video annotation comment.

[0165] In the exemplary embodiment shown in Figure 46, the embedded video annotation playback range

[0375] and the embedded video annotation symbol

[0376] are shown in the context of the embedded video

[0373] and embedded video playback

[0374] . However, the video annotation playback range and video annotation symbol can also be used on the video annotation playback

[0320] shown in Figure 45.

[0166] In the exemplary embodiment depicted in Figure 47, an embedded image

[0382] is shown. The embedded image

[0382] resides within a web page, and the embedded image annotation subject

[0383] occurs within the embedded image

[0382] . In the exemplary embodiment depicted in Figure 47, the bounding box

[0384] of the embedded image annotation subject, the embedded image annotation arrow

[0385] , and the embedded image annotation comment within the box

[0386] are shown. The embedded image annotation subject

[0383] is the object of interest within the image. The embedded image annotation subject

[0383] may be a person, a car, a soccer ball, a tennis ball, or any other object identifiable within the image by a person or computer system. The bounding box

[0384] of the embedded image annotation subject is the bounding box drawn around the embedded image annotation subject

[0383] by a user or computer system. The user can select the rectangle tool from a panel or floating toolbar anywhere on the webpage and draw a bounding box

[0384] for the embedded image annotation subject. Furthermore, the user can enter text and place an embedded image annotation comment within the box

[0386] . The embedded image annotation comment within the box

[0386] must be created by the user and must be related to the embedded image annotation subject

[0383] . The embedded image annotation arrow

[0385] is a line that can have arrowheads on both sides, connecting the bounding box

[0384] of the embedded image annotation subject and the embedded image annotation comment in the box

[0386] . The subject bounding box

[0384] , the embedded image annotation arrow

[0385] , and the image annotation comment within the embedded box

[0386] are embedded objects created by the user and included in the image annotation.

[0167] Embedded image and annotation objects may not all be displayed simultaneously. For example, only the bounding box

[0384] of the embedded image / annotation subject may appear. The image annotation is visible along with other embedded objects of the image annotation, which become visible after the user mouses over the embedded image annotation subject

[0383] or the bounding box of the embedded image annotation subject

[0384] . Visible elements of the image annotation are embedded elements that are visible without mouseover to other elements. Mouseover embedded elements of the image annotation are embedded elements that become visible when the user mouses over a visible embedded element of the image annotation. When the mouse hovers over an embedded object of the image annotation, the context variant annotation control bar box

[0387] is displayed. Users can use the contextual variant annotation control bar

[34] embedded in the contextual variant annotation control bar box

[0387] to change privacy settings, share via email, share via social media, flag an image annotation as inappropriate, delete an image annotation object from an electronic database

[62] with sufficient permission, or edit an embedded object consisting of an image annotation. In addition, users can use the reputation selector

[33] to vote on the reputation of an image annotation.

[0168] In an exemplary embodiment depicted in Figure 48, an embedded video

[0373] is shown. The embedded video

[0373] and embedded video playback

[0374] reside within a web page, and the embedded video annotation subject

[0388] occurs within the embedded video

[0373] . In the exemplary embodiment depicted in Figure 48, the bounding box

[0389] , embedded video annotation arrow

[0390] , and embedded video annotation comment within box

[0391] of the embedded video annotation subject are shown. The embedded video annotation subject

[0388] is an entity of interest within the video. The embedded video annotation subject

[0388] could be a person, a car, a soccer ball, a tennis ball, or any other object identifiable within the video by a person or a computer system. The bounding box

[0389] of the embedded video annotation subject is a bounding box drawn around the embedded video annotation subject

[0388] by the user or computer system. As the embedded video annotation subject

[0388] moves within the video, it is desirable that the bounding box

[0389] of the embedded video annotation subject continue to function as the bounding box of the embedded video annotation subject

[0388] . The embedded video annotation comment in box

[0391] is created by the user, and the annotation comment must be related to the embedded video annotation subject

[0388] . The embedded video annotation arrow

[0390] is a line that can have arrowheads on both sides, connecting the bounding box

[0389] of the embedded video annotation subject and the embedded video annotation comment in box

[0391] .The bounding box for embedded video annotation subjects

[0389] , the arrow for embedded video annotations

[0390] , and the comment inbox for embedded video annotations

[0391] are embedded objects created by the user and included in the video annotation.

[0169] The user can select the rectangle tool from a panel or floating toolbar anywhere on the webpage and draw the bounding box

[0389] of the initial embedded video annotation subject. Furthermore, the user can enter text, place embedded video annotation comments in the box

[0391] , and connect them with embedded video annotation arrows

[0390] . To ensure that objects are properly positioned when the video plays, either the user or the computer system needs to determine the motion path of the bounding box

[0389] of the embedded video annotation subject. The user creating the video annotation has the option of tracing the path of the bounding box

[0389] of the embedded video annotation subject as the video plays over time. While the embedded video playback

[0374] is playing, the user creating the video annotation can move the bounding box

[0389] of the embedded video annotation subject with the mouse to verify that the bounding box

[0389] of the embedded video annotation subject is properly positioned throughout the embedded video. The user can also move the embedded video annotation comment within box

[0391] with the mouse to verify that the embedded video annotation comment within box

[0391] is properly positioned throughout the video. Either an annotation-enabled web browser, a web browser extension, or the code of the web page records the motion path when the user drags the mouse. The next time the video is played, the paths of the bounding box

[0389] of the embedded video annotation subject and the embedded video annotation comment within box

[0391] evolve along their respective motion paths.If the bounding box

[0389] of the embedded video annotation subject and the motion path of the embedded video annotation comment within box

[0391] are known, the motion path of the embedded video annotation arrow

[0390] and its temporal rotation, temporal contraction, temporal expansion, and temporal position can be calculated corresponding to the motion path of the embedded video annotation comment within the bounding box

[0389] and box

[0391] of the embedded video annotation subject. Furthermore, if the desired size of the bounding box

[0388] of the embedded video annotation subject changes as the video plays, the user can pause the video and resize the bounding box

[0389] of the embedded video annotation subject and the embedded video annotation comment

[0391] within the box.

[0170] Not all embedded objects in a video annotation are visible at the same time. For example, only the bounding box of an embedded video annotation subject

[0389] may be visible, so that other embedded objects of the video annotation become visible after the user mouses over the embedded video annotation subject

[0388] or the bounding box of the embedded video annotation subject

[0389] . Visible elements of a video annotation are embedded elements that are visible without mouseing over other elements. Mouseover embedded elements of a video annotation are embedded elements that become visible when the user mouses over a visible embedded element of the video annotation. When you mouse over an embedded object of a video annotation, the contextual variant annotation control bar box

[0387] is displayed. Users can use the contextual variant annotation control bar

[34] embedded in the contextual variant annotation control bar box

[0387] to change privacy settings, share via email, share via social media, flag video annotations as inappropriate, delete video annotation objects from an electronic database

[62] with sufficient permission, or edit embedded objects consisting of video annotations. In addition, users can vote on the reputation of video annotations using the reputation selector

[33] . Each relationship within a group of annotation semantic relationships is associated with a different video annotation embedding symbol. For example, the question embedding video annotation symbol

[0376] indicates that the creator of the video annotation has designated the semantic relationship between a selected portion of the video and a video annotation comment as a question relationship.The question embedding video annotation symbol

[0376] allows a user familiar with how the exemplary embodiment works to know, before actually performing the click action, that they can click the question embedding video annotation symbol

[0376] to view the video annotation comment, which is a question about the embedded video annotation playback range

[0375] . The question annotation symbol, disagreement annotation symbol, agreement annotation symbol, exclamation annotation symbol, and advertising annotation symbol correspond to five relationships in a group called annotation semantic relationships. These annotation semantic relationships are question relationship, disagreement relationship, agreement relationship, exclamation relationship, and advertising relationship. These video annotation symbols and video annotation semantic relationships apply to the relationship between the embedded video annotation playback range

[0375] and the embedded video annotation comment input field

[0391] .

[0171] In the exemplary embodiment depicted in Figure 49, an embedded sound recording

[0392] is shown together with an embedded sound recording annotation playback range

[0393] and a question-embedded sound annotation symbol

[0394] . The embedded recording annotation playback range

[0393] specifies a selected portion of the recording corresponding to the sound annotation. The embedded sound annotation symbol is associated with an element of a group of annotation semantic relationships. Each annotation semantic relationship indicates the relationship between the embedded recording annotation playback range

[0393] and the sound annotation comment. A different embedded sound annotation symbol is associated with each relationship in the group of annotation semantic relationships. As an example, the question-embedded sound annotation symbol

[0394] indicates that the creator of the sound annotation has designated the annotation semantic relationship between a selected portion of the sound in the recording and the sound annotation comment as a question relationship. The embedded question sound annotation symbol

[0394] allows a user familiar with how the exemplary embodiment works to know, before actually performing the click action, that they can click the embedded question sound annotation symbol

[0394] to see the sound annotation comment, which is a question about the embedded recording annotation playback range

[0393] . The annotation comment box

[31] appears when the mouse hovers over the embedded recording annotation playback range

[0393] . The user can use the contextual variant annotation control bar

[34] embedded in the annotation comment box

[31] to change privacy settings, share via email, share via social media, flag the sound annotation as inappropriate, delete the sound annotation object from an electronic database

[62] with sufficient permission, or edit the embedded object consisting of the sound annotation. In addition, the user can use the reputation selector

[33] to vote on the reputation of the sound annotation.The five relationships in the group known as semantic relationships of annotations correspond to the question annotation symbols, disagreement annotation symbols, agreement annotation symbols, exclamation annotation symbols, and advertising annotation symbols. These semantic relationships of annotations are question relationships, disagreement relationships, agreement relationships, exclamation relationships, and advertising relationships. These semantic relationships of sound annotation symbols and sound annotations apply to the relationship between embedded recording annotation playback ranges

[0393] and sound annotation comments

[32] .

[0172] Aspects of the present invention are as illustrated in Figures 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, and 49, and it will be understood that each item in these GUI diagrams can be implemented by computer-readable program instructions. Next, we will describe methods and systems related to the features and capabilities depicted in the exemplary embodiments of Figures 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, and 49, relating them to previously presented material. Describing these methods and systems includes describing bundles of data elements used to receive and store data, and display bundles used to transmit and display visual elements to client devices. When describing bundles of display elements, various item boxes will be described. After describing these item boxes, their relationship to other presented elements, including the user interface, will be described. The data element bundles, display bundles, and item boxes described below are to be interpreted as being described within the context of exemplary embodiments.

[0173] In the context of exemplary embodiments, a bundle of data elements used to receive data from a client device and store this data using one or more computer systems is described below. The bundle of data elements is an image annotation bundle, a video annotation bundle, an audio annotation bundle, an embedded image annotation bundle, an embedded video annotation bundle, an embedded audio annotation bundle, an annotation group bundle, a derived annotation group bundle, and an annotation group bundle with derived annotations.

[0174] In the context of the exemplary embodiment, an image annotation bundle includes one or more image annotation atoms. An image annotation atom is a unit that can be used meaningfully on its own to annotate an image. For example, an image annotation atom may include a bounding box

[0301] of an image annotation subject, an image annotation comment

[0303] within the box, and an image annotation arrow

[0302] , as shown in the exemplary embodiment depicted in Figure 36. However, an image annotation comment

[0303] within a box can also be used alone near an image annotation subject

[0300] . Thus, an image annotation comment

[0303] within a box can itself be an image annotation atom. In addition to the image annotation atoms, the image annotation bundle includes frame information for the image to be annotated. The frame information includes the image size, specified by the number of pixels on the x-axis and the number of pixels on the y-axis. When an image size is specified, relative coordinates are assigned to the bounding box

[0301] of the image annotation subject, the image annotation comment

[0303] within the box, and the image annotation arrow

[0302] , which are elements that make up the image annotation atom, based on the frame information. If the bounding box

[0301] of the image annotation subject is specified by its center's X, Y coordinates, width, and length, the coordinates are specified relative to the lower left corner of the image. The image annotation comment

[0303] within the box can be specified in a similar manner, along with the markup information of the annotation comment contained within the image annotation comment

[0303] within the box. This markup information includes the font size and formatting information of the text. Information regarding the text creation mechanism is also included in the image annotation bundle. The creation mechanism may be user-written, image captions created using artificial intelligence (AI) or deep learning models, another annotation, or a combination of multiple methods.Furthermore, the username, machine learning model, and at least one of the other annotation data are recorded. If a combination of techniques is used, that information is also included in the image annotation bundle. Image annotation arrows

[0302] can be specified by the X and Y coordinates of their start and end points, and by information specifying whether the arrowhead is at the start point or the end point. In this case as well, the coordinates are specified relative to the lower left corner of the image. Each element included in the annotation atom specifies the generation mechanism in addition to the relative coordinate information. The generation mechanism specifies whether the element was generated by a user, a machine learning model, or a simple algorithm. At least one of the username, machine learning model name, or algorithm name is also recorded. Furthermore, ovals can also be used. Ellipses are specified by the X and Y coordinates of the center relative to the lower left corner of the image, and by the width, height, and rotation of the ellipse. Furthermore, the generation mechanism is recorded along with at least one of the username, machine learning model name, or algorithm name.

[0175] In the context of the exemplary embodiment, the embedded image annotation bundle contains not only all the information within the image annotation bundle, but also the URL of the web page or document in which the image is located. It is preferable that descriptions written about the image annotation bundle be interpreted as substituting embedded image annotation subject

[0383] for image annotation subject

[0300] , bounding box of embedded image annotation subject

[0384] for bounding box of image annotation subject

[0301] , embedded image annotation arrow

[0385] for image annotation arrow

[0302] , and embedded image annotation comment in box

[0386] for image annotation comment in box. Figures FIG. 36 and FIG. 47 should be examined in detail. Additionally, if the document is a PDF or a complex web page, information for locating the image, such as the DOM locators for the PDF document's pages and images, is also stored.

[0176] In the context of the exemplary embodiment, a video annotation bundle includes one or more video annotation atoms. A video annotation atom is a unit that can be meaningfully used on its own to annotate a video. For example, a video annotation atom may include a bounding box for a video annotation subject

[0322] , a bounding box motion path for a video annotation subject

[0332] , a video annotation comment input box

[0324] , a video annotation comment input box motion path

[0334] , a video annotation arrow

[0323] , and a video annotation arrow motion path

[0333] , as shown in the exemplary embodiment depicted in Figure 41. However, it is also possible to use the box

[0324] and the video annotation comment input box motion path

[0334] alone, by placing the video annotation comment in the box

[0324] near the video annotation subject

[0321] during video playback. Therefore, the video annotation comment in box

[0324] and the motion path

[0334] of the video annotation comment input box can themselves be video annotation atoms. The video annotation bundle includes the video annotation atom as well as frame information about the video to be annotated. The frame information includes the video size, specified by the number of pixels in the x-axis direction and the number of pixels in the y-axis direction. This must include the video region

[0319] but not the video annotation playback region

[0320] .When a video size is specified, the video annotation atom constituent elements, such as the video annotation subject bounding box

[0322] , video annotation subject bounding box motion path

[0332] , video annotation comment input box

[0324] , video annotation comment input box motion path

[0334] , video annotation arrow

[0323] , and video annotation arrow motion path

[0333] , are specified in relative coordinates based on the frame information. If the video annotation subject bounding box

[0322] is specified by its center's X, Y coordinates, width, and length, those coordinates are specified relative to the lower left corner of the video. The video annotation subject bounding box motion path

[0332] needs to specify the temporal evolution of the video annotation subject bounding box

[0322] . This specifies the temporal changes in its center's X, Y coordinates, width, and length. This specification is made by specifying the X and Y coordinates relative to the lower left corner of the video at discrete points in time within the video annotation playback range, along with the interpolation mechanism between these discrete time steps. Since the relative x-coordinate, relative y-coordinate, width, and length are real numbers, the method for storing the values ​​at discrete points in time and specifying the interpolation method is sufficient if all four values ​​are the same. Possible interpolation methods include linear interpolation, spline curves, and Bézier curves. The video annotation comment in box

[0324] and the motion path

[0334] of the video annotation comment input box can be specified in the same way, and the markup information of the annotation comment included in the video annotation comment in box

[0324] can also be specified. This markup information includes the font size and formatting information of the text. Furthermore, information about the text creation mechanism is included in the video annotation bundle. The creation mechanism can be user-written, video captions using AI or deep learning models, other annotations, or a combination of multiple methods. Furthermore, at least one of the following is recorded: the username, the machine learning model, and other annotation data.When a combination of technologies is used, that information is also included in the video annotation bundle. Video annotation arrows

[0323] and video annotation arrow motion paths

[0333] can be specified by the relative X and Y coordinates of their start and end points, and by information specifying whether the arrowhead is at the start point or the end point. The specification of video annotation arrow motion paths

[0333] is done by specifying the relative X and Y coordinates of the start and end points at discrete points in time within the video annotation playback range, and the interpolation mechanism between these discrete time steps. Here again, the coordinates are specified relative to the lower left corner of the video. Each element included in the annotation atom specifies a generation mechanism in addition to relative coordinate information. The generation mechanism specifies whether the element was created by a user, an AI or deep learning model, another annotation, or a combination of technologies. In addition, at least one of the following is recorded: the user name, the AI ​​or deep learning model name, or the algorithm name. Ovals can also be used. The ellipse is specified by the X and Y coordinates of its center relative to the lower left corner of the image, its width, height, and rotation. Furthermore, the generation mechanism is recorded along with at least one of the following: the username, the name of the AI ​​or deep learning model, and the name of the algorithm. Video annotation symbol information can also be included in the video annotation bundle.

[0177] In the context of the exemplary embodiment, the embedded video annotation bundle includes not only all the information within the video annotation bundle, but also the URL of the webpage or document on which the video is located. The description written for the video annotation bundle is embedded video annotation subject

[0388] to video annotation subject

[0321] , embedded video annotation subject bounding box

[0389] to video annotation subject bounding box

[0322] , embedded video annotation arrow

[0390] to video annotation arrow

[0323] , embedded video annotation comment in box

[0391] to video annotation comment in box

[0324] , embedded The bounding box motion path of the embedded video annotation subject must be interpreted as the bounding box motion path of the video annotation subject

[0332] , the embedded video annotation arrow motion path as the video annotation arrow motion path

[0333] , the embedded video annotation comment motion path within a box as the video annotation comment input box motion path

[0334] , and the embedded video annotation symbol as the video annotation symbol. The shape of the diagram and Figure 48 need to be examined in detail. If the document is a PDF or a complex web page, information for finding the video, such as the PDF document's page numbers and the video's DOM locator, will also be stored.

[0178] In the context of an exemplary embodiment, a recording annotation bundle includes one or more recording annotation atoms. A recording annotation atom is a unit that can be meaningfully used to annotate a recording on its own. For example, a recording annotation atom may include a recording annotation playback range, a recording annotation symbol, and a recording annotation comment. However, it is also possible to use only a recording annotation playback range and a recording annotation comment. Thus, a recording annotation playback range and a recording annotation comment can constitute a recording annotation atom on their own. A recording annotation comment may have associated markup information. This markup information includes font size as well as text formatting information. Furthermore, information regarding the text creation mechanism is contained within the recording annotation bundle. The creation mechanism may include user-written text, speech recognition by AI or deep learning models, copying of other annotations, or a combination of multiple methods. Furthermore, the creation mechanism is recorded along with the username, the AI ​​or deep learning model name, and at least one of other annotation references. If a combination of techniques is used, one or more source name types are recorded. Also, the playback range of the recording annotation can be determined based on sound search. Therefore, it is desirable that the generation mechanism of the recording playback range be recorded along with the username, the AI ​​or deep learning model name, and at least one of other annotation comments. Multiple generation mechanisms and sound source information are recorded when multiple methods are used in combination. In addition, recording frame information is recorded to facilitate the placement of the recording annotation playback range in the recording annotation playback control.

[0179] In the context of the exemplary embodiment, the embedded recording annotation bundle contains not only all the information within the recording annotation bundle, but also the URL of the webpage or document in which the recording is located. Descriptions written about the recording annotation bundle should be interpreted by replacing embedded recording annotation playback range

[0393] with recording playback range, embedded recording annotation symbols with recording annotation symbols, and embedded recording annotation comments with recording annotation comments. Furthermore, it should be understood that embedded recording frame information is recorded in the embedded recording annotation bundle to facilitate the placement of embedded recording annotation playback range

[0393] in the embedded recording playback control. Details should be considered by referring to the forms in Figures 44 and 49. If the document is a complex webpage, information for locating the recording's position, such as a DOM locator for the recording, is also stored.

[0180] In the context of an exemplary embodiment, an annotation group bundle contains references to all annotations included in that particular annotation group. Furthermore, information regarding whether annotations conflict is stored along with metadata indicating when the annotation group was created and metadata indicating whether the conflict detection was performed by a user or an AI or deep learning model. At least one of the users and / or AI / deep learning models that performed the conflict detection is also stored in the annotation group bundle.

[0181] In the context of the exemplary embodiment, a derived annotation group bundle contains references to all annotations included in that particular derived annotation group. Furthermore, information regarding whether annotations conflict is stored, along with metadata indicating when the annotation group was created and whether the conflict detection was performed by a user or an AI or deep learning model. At least one of the users and / or AI / deep learning models that performed the conflict detection is also stored in the derived annotation group bundle. The derived annotation group bundle contains one derived annotation, along with references to all annotations included in that particular annotation group. Annotation connection arrows can be determined from the annotation references present within the derived annotation group bundle. In this exemplary embodiment, only one derived annotation exists, which provides all the information necessary to draw the annotation connection arrows. Furthermore, derived annotation comments are stored in the derived annotation group bundle. Markup information for the derived annotation comments is also stored. This markup information includes the font size and formatting information of the text. Furthermore, the derived annotation group bundle stores information about the text creation mechanism. This mechanism could be user-written, generated by an AI or deep learning model, copied from another annotation, or a combination of multiple techniques. Additionally, the username, machine learning model, and at least one of the other annotation data are recorded. If a combination of techniques was used, that information is also included in the derived annotation group bundle.

[0182] In the context of an exemplary embodiment, a derived annotation group bundle contains all derived annotations and references to annotations present in that particular derived annotation group. Furthermore, information regarding whether annotations conflict is stored, along with metadata indicating when the annotation group with derived annotations was created, and whether the conflict detection was performed by a user or an AI or deep learning model. At least one of the users and / or AI / deep learning models that performed the conflict detection is also stored in the annotation group bundle.

[0183] Next, several types of display bundles and item boxes are described. These descriptions are to be interpreted as being within the context of exemplary embodiments. In the context of exemplary embodiments, the display bundles described are image annotation display bundles, video annotation display bundles, audio annotation display bundles, embedded image annotation display bundles, embedded video annotation display bundles, embedded audio annotation display bundles, annotation group display bundles, derived annotation group display bundles, and annotation group display bundles with derived annotations. In the context of exemplary embodiments, the item boxes described are image annotation item boxes, video annotation item boxes, audio annotation item boxes, embedded image annotation item boxes, embedded video annotation item boxes, embedded audio item boxes, annotation group item boxes, derived annotation group item boxes, and annotation group item boxes with derived annotations. One purpose of the display bundles in exemplary embodiments is to provide data for a web browser to display the corresponding item boxes associated with that type of annotation or media item. Therefore, the item box will be described in relation to the annotation item box

[36] presented in the exemplary embodiment shown in Figure 9. In addition to the numbered items within the annotation item box, the annotation item box also includes item box annotation highlighting text, item box annotation symbols, item box annotation comments, and item box URLs. Now that various item boxes are described in the context of the exemplary embodiment, data structures corresponding to various display bundles can be illustrated. Furthermore, referring to the display bundle may be helpful for the explanation. The display bundle is a visual output created using the display bundle. This visual output is contained within the item box.In a context like Notepad, the displayed bundle can exist even if the entire item box does not exist.

[0184] In an exemplary embodiment, the image annotation item box would include a separate contextual variant annotation control bar

[41] and a separate contextual reputation selector

[42] , as shown in the exemplary embodiment of Figure 9

[36] . Instead of a screenshot with marked text

[37] , the image annotation item box would include an image thumbnail with image annotations displayed on top of the thumbnail image. Instead of separate contextual annotation highlighted text

[38] , separate contextual question displayed annotation symbols

[39] , and separate contextual annotation comments

[40] , the image annotation item box would display image annotation comments. URLs are not displayed in the image annotation item box, but they are displayed in embedded image annotation item boxes. In order for web browsers to display these image annotation item boxes, the image annotation display bundle must appropriately resize and overlay objects such as the image annotation subject bounding box

[0301] , the image annotation comment within the box

[0303] , and the image annotation arrow

[0302] , along with the displayed image and marked thumbnail image. Furthermore, the image annotation display bundle must include the image annotation comment, username, and reputation information.

[0185] In exemplary embodiments, the embedded image annotation item box would include a separate contextual variant annotation control bar

[41] and a separate contextual reputation selector

[42] , as in the annotation item box

[36] . Instead of a screenshot with marked text, the embedded image annotation item box would include a thumbnail of the embedded image and an image annotation displayed on top of the thumbnail image. Instead of separate contextual annotation highlighting text

[38] , separate contextual question displayed annotation symbol

[39] , and separate contextual annotation comment

[40] , the embedded image annotation item box would have an embedded image annotation comment. Its URL would correspond to a URL containing the image-annotated embedded image. To enable web browsers to display these embedded image and annotation item boxes, the embedded image and annotation display bundle must include a marked thumbnail image of the embedded image with the image and annotation displayed, with objects such as the bounding box of the embedded image and annotation subject

[0384] , the embedded image and annotation comments within the box

[0386] , and the embedded image and annotation arrows

[0385] appropriately resized and overlaid. Furthermore, the embedded image and annotation display bundle must include the embedded image and annotation comments, username, and reputation information.

[0186] In exemplary embodiments, the video annotation item box would include a separate contextual variant annotation control bar

[41] and a separate contextual reputation selector

[42] , as shown in the exemplary embodiment of Figure 9

[36] . Instead of a screenshot with marked text

[37] , the video annotation item box would include a thumbnail of a video frame at a selected point in time, along with video annotations displayed on the thumbnail of the video frame at the same selected point in time. Furthermore, a play button could play a thumbnail video showing the thumbnail video and a portion of the video annotations. Instead of separate contextual annotation highlighting text

[38] , video sound summary text, which is machine-generated text summarizing the sounds in the video, would be displayed. Next, the video annotation item box would display video annotation comments, and the video annotation item box would also display video annotation symbols that determine the semantic relationship of the annotations between the video sound summary text and the video annotation comments. The URL does not exist in the video annotation item box, but it does exist in the embedded video annotation item box. In order for web browsers to display these video annotation item boxes, the video annotation display bundle must include a marked thumbnail image of the video frame and the video at the selected point in time, with the bounding box of the video annotation subject

[0322] , the video annotation comment

[0324] within the box, and the video annotation arrow

[0323] appropriately resized and overlaid. When the user presses the play button, the video annotation display bundle must include the thumbnail video with the video annotation overlaid.Furthermore, the video annotation display bundle must include video audio summary text, video annotation symbols, video annotation comments, username, and reputation information. Additionally, users should be able to modify the machine-generated video annotations.

[0187] In exemplary embodiments, the embedded video annotation item box would include a separate contextual variant annotation control bar

[41] and a separate contextual reputation selector

[42] , as shown in the exemplary embodiment of Figure 9

[36] . Instead of a screenshot

[37] with marked text, the embedded video annotation item box would include a thumbnail of the embedded video frame at a selected point in time, along with the embedded video annotation displayed on the thumbnail of the embedded video frame at the same selected point in time. Furthermore, a play button may enable playback of the thumbnail video and a portion of the video annotation. Instead of a separate contextual annotation highlighting text

[38] , there would be embedded video sound summary text, which is part of machine-generated text summarizing the sound in the video. Next, the embedded video annotation item box would have embedded video annotation comments, and the embedded video annotation item box would also have embedded video annotation symbols that determine the semantic relationship of the annotations between the embedded video sound summary text and the embedded video annotation comments. The URL of the webpage in which the video is embedded is displayed in the embedded video annotation item box. In order for web browsers to display these embedded video annotation item boxes, the embedded video annotation display bundle must have objects such as the bounding box of the embedded video annotation subject

[0389] , the embedded video annotation comment within the box

[0391] , and the embedded video annotation arrow

[0390] appropriately resized and overlaid, with the embedded video annotation displayed on top of it, and must include a marked thumbnail image of the embedded video frame at a selected point in time.When a user presses the play button, the embedded video annotation display bundle must include a thumbnail video with the embedded video annotation overlaid. Furthermore, the embedded video annotation display bundle must include embedded video audio summary text, embedded video annotation symbols, embedded video annotation comments, username, and reputation information. Additionally, the user should be able to modify aspects of the machine-generated video annotation.

[0188] In an exemplary embodiment, the recording annotation item box would include a separate contextual variant annotation control bar

[41] and a separate contextual reputation selector

[42] , as shown in the exemplary embodiment of Figure 9

[36] . Instead of a screenshot with marked text

[37] , the recording annotation item box would include a thumbnail representing the recording, called a thumbnail-related recording image. This is done by first creating a recording summary text, which is a text representation of the recording. This is done using speech understanding techniques such as speech recognition. Once the text representation is obtained, an AI or deep learning model is used to find an image associated with that recording summary text. This image will be called an associated recording image. A thumbnail representation of the associated recording image is the thumbnail-related recording image. Instead of a separate contextual annotation highlighting text

[38] , the recording summary text is used. Next, the recording annotation item box will have recording annotation comments, and it will also have recording annotation symbols that determine the semantic relationship of the annotations between the recording summary text and the recording annotation comments. URLs are not present in the recording annotation item box, but they are present in the embedded recording annotation item box. In order for web browsers to display these recording annotation item boxes, the recording annotation display bundle must include the recording image associated with the thumbnail, the recording summary text, the recording annotation comments, the recording annotation symbols, the username, and levitation information. Furthermore, the user will be able to modify the machine-generated portion of the recording annotation.

[0189] In an exemplary embodiment, the embedded recording annotation item box would include a separate contextual variant annotation control bar

[41] and a separate contextual reputation selector

[42] , as in the annotation item box

[36] . Instead of a screenshot

[37] containing marked text, the embedded recording annotation item box would include a thumbnail representing the embedded recording, called a thumbnail-associated embedded recording image. This is done first by creating an embedded recording summary text, which translates the embedded recording into a text representation. This is done using speech understanding techniques such as speech recognition. Once the text representation is available, an AI or deep learning model can be used to find an image associated with that embedded recording summary text. This image will be called the associated embedded recording image. The thumbnail representation of the associated embedded recording image will be called the thumbnail-associated embedded recording image. Instead of a separate contextual annotation highlighting text

[38] , it will be the embedded recording summary text. Next, the embedded recording annotation item box will contain the embedded recording annotation comment, and the embedded recording annotation item box will also contain the embedded recording annotation symbol that determines the semantic relationship of the annotation between the embedded recording summary text and the embedded recording annotation comment. The embedded recording annotation item box will display the URL of the website where the embedded recording was found. In order for web browsers to display these embedded recording annotation item boxes, the embedded recording annotation display bundle must include the embedded recording image associated with the thumbnail, the embedded recording summary text, the embedded recording annotation comment, the embedded recording annotation symbol, the username, and reputation information. Furthermore, the user will be able to modify the machine-generated portion of the recording annotation.

[0190] In exemplary embodiments, the annotation group item box would include a separate contextual variant annotation control bar

[41] and a separate contextual reputation selector

[42] , as shown in the exemplary embodiment of Figure 9

[36] . Instead of a screenshot with marked text

[37] , the annotation group item box would include a thumbnail representing the annotation group. For image annotations, this means a thumbnail image of an image containing all the image annotations included in the annotation group. Image annotations and annotation groups include annotations for embedded images. For video annotations, this means a thumbnail image of a video at a certain point in time and all the video annotations included in the annotation group at that same point in time. Furthermore, a thumbnail video would include a thumbnail video in which all the video annotations of the annotation group are displayed across the entire thumbnail video. Video annotations and annotation groups include video annotations for embedded videos. For recording annotations, for all recording annotations within an annotation group, a thumbnail image representing the sound for all recording playback ranges is used. Such images would be based on recording summary text for all recording playback ranges. Instead of the highlighted text of a separate contextual annotation

[38] , a list of annotations within that particular annotation group is displayed. Instead of a separate contextual annotation comment

[40] , a user-generated comment about that particular annotation group is used. Furthermore, if the machine-generated images and default text are inappropriate, these elements can be modified by the user. In another embodiment, the list of annotations in an annotation group can be replaced with more appropriate human-generated summary text.URLs are not present in annotation group item boxes that do not have embedded annotations, but they are present in annotation group item boxes that do have embedded annotations. Embedded annotation group annotation symbols can be used to indicate the semantic relationship between source material and annotations within the annotation group. The annotation group item box also displays the number of annotations in the annotation group. In order for web browsers to display these annotation group annotation item boxes, the annotation group display bundle must include the relevant thumbnail image or thumbnail video, a list of references to annotation objects, relevant summary text, relevant annotation comments, relevant annotation group annotation symbols, username, and reputation information. Furthermore, users will be able to modify the machine-generated aspects of the annotation group.

[0191] In exemplary embodiments, the derived annotation group item box would include a separate contextual variant annotation control bar

[41] and a separate contextual reputation selector

[42] , as shown in the exemplary embodiment of Figure 9, as an annotation item box

[36] . Instead of a screenshot with marked text

[37] , the derived annotation group item box would include a thumbnail representing the derived annotation group. For image annotations, this means a thumbnail image of an image containing all image annotations included in the derived annotation group. Image annotations and derived annotation groups include image annotations of embedded images. For video annotations, this means a thumbnail image of a video at a certain point in time and all video annotations included in the derived annotation group at that same point in time. Furthermore, a thumbnail video would include a thumbnail video in which all video annotations included in the derived annotation group are displayed across the entire thumbnail video. Video annotations and derived annotation groups include video annotations of embedded videos. In the case of recording annotations, this means a thumbnail image representing the sound for all recording playback ranges of all recording annotations in a derived annotation group. Such images would be based on the recording summary text for all recording playback ranges. Instead of the highlighted text of a separate contextual annotation

[38] , a list of annotations within that particular derived annotation group is displayed. Instead of a separate contextual annotation comment

[40] , a user-generated comment about that particular derived annotation group is displayed. Furthermore, if the machine-generated images and default text are inappropriate, these elements can be modified by the user. In an alternative embodiment, the list of annotations in a derived annotation group can be replaced with a more appropriate human-generated summary text.URLs are not present in derived annotation group item boxes that do not have embedded annotations, but they are present in derived annotation group item boxes that do have embedded annotations. Derived annotation group annotation symbols may be used to indicate the semantic relationship between source material and annotations within an annotation group. In order for web browsers to display these annotation group annotation item boxes, the annotation group display bundle must include the associated thumbnail image or thumbnail video, a reference list to annotation objects, associated summary text, associated derived annotation comments, associated derived annotation group annotation symbols, username, and reputation information. Furthermore, users will be able to modify the machine-generated portion of the derived annotation group.

[0192] At this point, annotation groups and derived annotation groups may appear similar in the context of the exemplary embodiments described so far. However, derived annotation groups, derived annotation group item boxes, and derived annotation group display bundles will be treated differently from annotation groups, annotation group item boxes, and annotation group display bundles when used. Annotation groups more closely indicate the links between annotations, and derived annotation groups can be incorporated into annotation groups. The impact of these differences can be seen by examining Figures 44 and 45.

[0193] In exemplary embodiments, a derived annotation group includes a separate contextual variant annotation control bar

[41] and a separate contextual reputation selector

[42] , such as an annotation item box

[36] , as shown in the exemplary embodiment of Figure 9. The derived annotation group item box and the corresponding derived annotation group display bundle are similar to the annotation group item box and annotation group display bundle, except that information about derived annotations, such as their constituent annotations and derived annotation comments, exists for each derived annotation group. This information enables the display of derived annotations as depicted in Figures 44 and 45. Furthermore, the derived annotation group item box displays the number of annotations within the derived annotation group and the number of derived annotation groups. All of this information is reflected in the derived annotation group display bundle. The derived annotation group item box displays a separate contextual variant annotation control bar

[41] and a separate contextual reputation selector

[42] , similar to the annotation item box

[36] . Instead of a screenshot containing the marked text, the derived annotation group item box displays a representative thumbnail of the derived annotation group

[37] . For image annotations, this means a thumbnail image of the image containing all the image annotations included in the derived annotation group. The important thing is the set of all annotations included in the derived annotation group, as the derived annotation group modifies the image with the annotations included in the derived annotation group. Image annotations and annotation groups contain annotations on embedded images.In the case of video annotation, it means a thumbnail image of the video at a given point in time and all video annotations included in the annotation group that have derived annotations at that same point in time. Furthermore, the thumbnail video will include a thumbnail video in which all video annotations included in the annotation group that have derived annotations are displayed across the entire thumbnail video. Again, the important thing is the set of all annotations included in the annotation group that have derived annotations. Video annotations and annotation groups include video annotations of embedded videos. In the case of audio annotation, it means a thumbnail image representing the sound for all recording playback ranges of all recording annotations included in the annotation group that have derived annotations. Such an image will be based on the recording summary text for all recording playback ranges in the set of all annotations in the annotation group that have derived annotations. Instead of the highlighted text of a separate contextual annotation

[38] , the list of annotations in that particular annotation group becomes the derived annotation. Instead of separate contextual annotations, comments

[40] are user-generated comments relating to that particular group of annotations. Furthermore, if machine-generated images and default text are inadequate, these elements can be modified by the user. In an alternative embodiment, the list of annotations in an annotation group with derived annotations can be replaced with more appropriate human-generated summary text. URLs will not be present in an annotation group item box with derived annotations that does not have embedded annotations, but will be present in an annotation group item box with derived annotations that does have embedded annotations. Embedded annotation groups containing derived annotation symbols can be used to show semantic relationships between source material and annotations within an annotation group containing derived annotations.The item box for an annotation group with derived annotations displays the number of annotations within the annotation group and the number of derived annotation groups. For web browsers to display these item boxes for annotation groups with derived annotations, the annotation group display bundle must include associated thumbnail images or videos, a list of references to annotation objects, associated summary text, associated annotation comments, associated derived annotation group symbols, usernames, and reputation information. Furthermore, users will be able to modify the machine-generated aspects of annotation groups with derived annotations.

[0194] While various types of data element bundles, display bundles, and item boxes have been described within the context of exemplary embodiments, changes that need to be made to Figures 14, 15, 16, 17, and 18 will now be described. These changes are intended to adapt these sequence diagrams and flowcharts to the described descriptions and implementation methods and systems related to the features and capabilities shown and described in the exemplary embodiments depicted in Figures 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, and 49. The sequence diagram in Figure 14 shows that when different types of annotations are implemented, the annotation creation process applies to one of the data element bundles described earlier, along with its corresponding display bundle and its corresponding displayed bundle, which differ for the different types of annotations described earlier. A display bundle occurs when Figure 14 uses an annotation display bundle, and a displayed bundle occurs when Figure 14 uses a displayed annotation bundle. The sequence diagram in Figure 15 also applies to the display bundle and the displayed bundle described earlier. These displays occur where the displayed bundle of annotations occurs in Figure 15, and the displayed bundle occurs where Figure 15 uses the displayed annotation bundle. A display bundle has data for displaying either its corresponding item box or its corresponding displayed bundle. Depending on the exemplary embodiment, either the item box or the displayed bundle may be displayed in the user interface. Typically, an item box is a displayed bundle with a separate contextual variant annotation control bar

[41] and a separate contextual reputation selector

[42] . The separate contextual variant annotation control bar

[41] and the separate contextual reputation selector

[42] are shown for the annotation item box

[36] in the exemplary embodiment shown in Figure 9.The sequence diagram in Figure 16 is applied by replacing the annotation symbol display bundle with the appropriate embedded annotation display bundle and replacing the closed displayed annotation symbols with the appropriate embedded annotation visible elements. The embedded annotation display bundle and the embedded annotation visible elements are as previously described. Figures 17 and 18 are applied as is when the data element bundles and display bundles are selected from the types previously defined herein.

[0195] In an exemplary embodiment, the item bin

[44] within the annotation organization context area

[43] in Figure 10 may include annotation item boxes, embedded image annotation item boxes, embedded video annotation item boxes, embedded audio item boxes, annotation group item boxes, derived annotation group item boxes, derived annotation group item boxes, and notepads. Notepads may also be displayed as notepad item boxes. All of these types of item boxes have a separate context variant annotation control bar

[41] , as depicted for the annotation item box

[36] . Other elements of the annotation item box

[36] change for other types of item boxes, but the separate context variant annotation control bar

[41] and the displayed form factor of the item box do not change. This allows these new types of item boxes to be integrated into the item bin

[44] . All of these item types are created on an external site and appear in the item bin as soon as a user creates these items

[44] . This is easily achieved by using WebSockets to push information to all connected web browsers from an annotation server

[61] that should receive display bundles based on access control and use cases. Item boxes can be moved from an item bin

[44] to a writable collection represented by a displayed writable collection

[50] , as depicted in an exemplary embodiment of Figure 11. Furthermore, annotation item boxes, embedded image annotation item boxes, embedded video annotation item boxes, and embedded sound recording item boxes can be added to the active notepad

[0159] using the add to notepad icon

[0158] button in the context variant annotation control bar

[34] , as depicted in an exemplary embodiment of Figure 23.The icon

[0158] button for adding to Notepad is not displayed in the contextual variant annotation control bar

[34] labeled in all figures. However, such a button can be added to the contextual variant annotation control bar

[34] in the exemplary embodiment. Other item types can also be added to the active Notepad

[0159] using triggers to perform the addition to Notepad. In all cases, clicking an item box in the active Notepad

[0159] requires opening the relevant item in a browser tab, scrolling to the location of the annotation, and viewing the annotation. All different types of item boxes can be displayed in a collection. The collection displayed in the exemplary embodiment depicted in Figure 31 shows only annotation item boxes, but all types of item boxes can be present in a collection. All comments on the stacked set of displayed annotation items apply to stacked sets of various types of items. In addition, comments on window functionality also apply. Stacked sets of items can be grouped based on URLs, or based on the annotated image, video, audio, or document items. Furthermore, the types of items included in a collection do not change any other properties of the collection. The collection represented by the displayed collection can be displayed on a page similar to that shown in the exemplary embodiment of Figure 32.

[0196] As previously mentioned, filters, such as the search filter shown in the exemplary embodiment of Figure 10, are used to receive input from the user. In the case of the search filter

[46] in the item bin

[44] , this is a search phrase. These inputs are converted into parameter values ​​that are sent to the annotation server

[61] . These parameter values ​​are used by the annotation server

[61] to input parameters into a request query used in the electronic database

[62] . The request query includes parameter values ​​and page numbers. The request query, including the parameter values, is used to specify a set of items and the order of those items. These items include web page annotations, embedded image annotations, embedded video annotations, embedded audio annotations, image annotations, video annotations, audio annotations, annotation groups, derived annotation groups, annotation groups with derived annotations, PDF document annotations, PDF documents, images, videos, links, and regular comments. Filters can also be restricted to specific item types. The page number specifies a particular page. Furthermore, the request query may include a specified number of items to be sent to a web browser running on a client electronic device. This request query includes parameters for filter values, such as a search phrase, an exclusion list of items to be excluded from the specified list of items, and a sequential ordering for that list of items. As previously described, this sequential ordering provides information for paging in the exemplary embodiments shown in Figures 10 and 31. The collection filter and collection paging operate as previously described in the description of the exemplary embodiments shown in Figures 10 and 32. The same applies to the pinning function.

[0197] The inbox described in the exemplary embodiment shown in Figure 33 would function similarly if the collection contained different item types. The change would be that users could now discuss more details about item types within the collection message. Furthermore, functionality could be built within message chains specific to each type of item. Functionality could be implemented to modify the objects that constitute the annotation elements, approve changes to annotations, and fork annotations added to the current collection. Also, when discussing image annotations, the discussion could be facilitated by using selectors for each object in the image annotation. In addition, selectors for other item types, such as video annotations and audio annotations, could be incorporated into the message chain functionality. These features allow you to modify the subject bounding box of a video annotation

[0322] , the video annotation comment within the box

[0324] , or the annotation arrow

[0323] , as well as the motion path of the bounding box of the video annotation subject

[0332] , the video annotation arrow motion path

[0333] , and the video annotation comment input box motion path

[0334] . After the changes are proposed in the message chain, the item owner can approve the changes, and the administrator can approve forking the item into a new item in the context of the current collection.

[0198] The exemplary embodiments depicted in Figures 28 and 29 also apply when multiple item types exist. In this scenario, the annotation server

[61] and the electronic database

[62] will handle these different types of items. In the exemplary embodiments, the annotation server

[61] may have a REST API for handling different item types, and the electronic database

[62] has a schema for handling different data elements in various types of bundles. Hardware requirements are affected as follows: If the video

[0319] is not stored locally and is accessed via a URL, the presence of the video

[0319] will not increase storage requirements as much as if the video

[0319] were stored locally. If it is stored locally or via a content delivery network

[63] , the video

[0319] will have a significant impact on storage requirements. The presence of audio playback

[0348] may necessitate the use of deep learning models to handle speech and speech recognition. The presence of video

[0319] will increase the system's computational requirements, as it generally has higher computational requirements than other media types.

[0199] In exemplary embodiments, to provide users with additional annotation capabilities, machine learning and deep learning models can be deployed and configured via a web browser extension for an annotation-enabled web browser. If the annotation-enabled web browser is not built through the use of a web browser extension, deployment and configuration can also be handled directly through the annotation-enabled web browser. This functionality can be made available to the user through a panel or floating toolbar located somewhere on the web page. This functionality can also be exposed through a menu option or a fixed browser toolbar. The floating toolbar can be attached to the page assignment or placed within the web browser's side panel. For images and embedded images, this panel or floating toolbar can provide buttons to handle tasks such as recognizing objects in the image, recognizing text in the image, setting recognition characteristics for object and text detectors, and setting characteristics for annotation elements. Annotation elements include bounding boxes for annotation subjects, annotation arrows, and annotation comment boxes. The object recognition button draws a dashed bounding box for the annotation subject around the candidate annotation subject. Users can select this bounding box and write annotation comments in the annotation comment input box. Depending on the settings, annotation arrows may be automatically created. Users can also be provided with the option to select bounding boxes for multiple annotation subjects, resulting in one annotation arrow being created for each bounding box of an annotation subject. One end of each annotation arrow points to the bounding box of the annotation subject, and the other end points to the annotation comment input box or annotation comment box, depending on whether an annotation comment has been written. The placement of the arrows is a configurable property.Other configurable properties include the visibility properties of the annotation subject's bounding box, annotation arrows, and annotation comment boxes. Visibility properties include visible, mouseover, and invisible. Visible is for annotation elements that are always displayed. Mouseover corresponds to annotation elements that become visible when the mouse hovers over a target area. Invisible corresponds to annotation elements that are always invisible. The mouseover target area can be at least one of the visible annotation subject or annotation element. In an alternative embodiment, an annotation subject outline is used, enclosing a closed area containing the pixels of the annotation subject. These annotation subject outlines can be used in contrast to the annotation subject's bounding box. Furthermore, object outlines can be mouseover elements, as opposed to visible elements, and this visibility property can be controlled through configuration settings. It is also possible to have configuration settings that allow the user to choose whether to use the annotation subject's bounding box or the annotation subject outline. Determining the bounding box or outline of an annotation subject relies on a machine learning or deep learning model that can recognize the object and determine the bounding box or outline of the annotation subject. The deep learning model can run in a web browser on the client electronic device, using the CPU or GPU of the client electronic device. The GPU can be used through the use of the WebGL interface exposed by modern web browsers, thereby enabling the use of the client electronic device's GPU.The bounding box and outline of the annotation subject can then be determined through the use of an annotation object detector using a region-based convolutional neural network (R-CNN), or through object segmentation using a region-based convolutional neural network (R-CNN). In addition to R-CNN-based models, alternative embodiments can employ other machine learning or deep learning methods. Since different deep learning models have different recognition characteristics, the user can input the desired recognition characteristics for the object detector and text detector. Based on these characteristics, different machine learning or deep learning models can be selected. It is also possible to recognize characters in an image using a CNN (Convolutional Neural Network) that recognizes characters and numbers. Furthermore, in the settings options, it is possible to select whether to group individual characters into words, or whether to group individual words into n-grams. In addition, by inputting words or tags in the recognition characteristics settings options, it is also possible to limit recognition to specific types of objects entered by the user. A button has also been added to set the motion path of the annotation element in order to handle videos and embedded videos. Furthermore, the techniques described in the images can also be implemented on individual frames of a video. In addition, a deep learning model can track the annotation subject over time, ensuring that the bounding box and outline of the annotation subject move along with the annotation subject. This can be achieved by using discrete-point individual frame analysis and interpolation methods between the discrete points. This yields the motion paths of the bounding box and outline of the annotation subject. The contour movement of the annotation subject outline is also calculated.The settings for the evolution of these motion paths and annotation outlines are available to the user through an add button in a panel, menu option, browser toolbar, or floating toolbar that handles the motion path settings for annotation elements. Machine learning and deep learning models for speech and sound recognition in videos intended for annotation can be handled through separate settings options in a panel, menu option, browser toolbar, or floating toolbar.

[0200] As used herein, the word “exemplary” means “serving as an example, instance, or illustration.” Any implementation described herein as “exemplary” is not necessarily construed as preferable or advantageous to other implementations, nor is it intended to be construed as a model that must be literally replicated. Accordingly, this specification is not limiting and is considered illustrative.

[0201] Conditional language, particularly “can,” “could,” “might,” or “may,” is generally intended to convey that certain features, elements, and / or steps are included in certain embodiments and not included in other embodiments, unless otherwise specified or understood within the context in which they are used. Therefore, such conditional language is not generally intended to imply that features, elements, and / or steps are required in any way in one or more embodiments, or that one or more embodiments necessarily include logic, together with user input or prompts, to determine whether these features, elements, and / or steps are included in or performed in any particular embodiment. Headings used herein are for the convenience of the reader and are not intended to limit the scope of the invention or the claims.

[0202] As used herein, any phrase referring to “at least one” in a list of items refers to any combination of those items, including a single member. For example, “at least one of A, B, or C” is intended to cover A, B, C, A and B, A and C, B and C, as well as A, B, and C. Connecting language such as the phrase “at least one of X, Y, and Z” is understood to be used in context to convey that an item, term, etc., may be at least one of X, Y, or Z, unless otherwise specified. Thus, such connecting language is not intended to require that a particular embodiment must contain at least one of X, at least one of Y, and at least one of Z, respectively.

[0203] The flowcharts, GUIs, sequences, systems, components, and swimlane activity diagrams in the figures illustrate the use, functionality, architecture, and operation of possible implementations of methods, computer program products, and systems according to various embodiments of the present invention. In this regard, each item in the flowcharts, GUIs, sequences, systems, components, and swimlane activity diagrams may represent a module, segment, or portion of instructions consisting of one or more executable instructions for implementing a specified logical function(s). In some alternative implementations, the functions described in an item may occur out of the order shown in the figure. For example, two items shown consecutively may actually be executed substantially simultaneously, and items may sometimes be executed in reverse order depending on the functions involved. It will also be noted that each item and combination of items in the flowcharts, GUIs, sequences, systems, components, and swimlane activity diagrams that do not involve one or more users may be implemented by a special-purpose hardware-based system that does not involve one or more users and performs a specific function or action, or executes a combination of special-purpose hardware and computer instructions.

[0204] Aspects of the present invention are described herein with reference to flowcharts, GUIs, sequences, systems, components, and swimlane activity diagrams of methods, computer program products, and systems according to embodiments of the present invention. It will be understood that each item of a flowchart, GUI, sequence, system, component, and swimlane activity diagram that does not involve one or more users, as well as combinations of items of a flowchart, GUI, sequence, system, component, and swimlane activity diagram that does not involve one or more users, can be implemented by computer-readable program instructions.

[0205] These computer-readable program instructions are provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to generate instructions for the machine to be executed via the computer's processor or other programmable data processing device, creating means for implementing functions / actions that do not involve one or more users, specified in one or more items of a flowchart, GUI, sequence, system, component, and swimlane activity diagram, wherein one or more items do not involve one or more users. These computer-readable program instructions are also stored in computer-readable storage media that can instruct computers, programmable data processing devices, and other devices to function in a particular way, and the computer-readable storage media having the instructions stored therein consists of a product containing instructions for implementing aspects of functions / actions specified in or of items of a flowchart, GUI, sequence, system, component, and swimlane activity diagram, wherein the items and items may not involve one or more users.

[0206] Such code may be stored or adapted for storage in one or more tangible machine-readable media, such as memory chips, local or remote hard disks, optical discs, or other media, which may be accessed and executed by a processor-based system. It should be noted that the tangible media may consist of paper on which the instructions are printed or other suitable media. For example, instructions may be electronically captured via an optical scan of paper or other media, then compiled, interpreted, or otherwise processed in an appropriate manner as needed, and then stored in computer memory. Furthermore, it should be noted that the tangible media may constitute a version control repository utilizing memory chips, local or remote hard disks, optical discs, or other media for storage. For example, instructions may be retrieved from a version control repository, then compiled, interpreted, or otherwise processed in an appropriate manner, and then stored in computer memory.

[0207] Computer-readable program instructions can be loaded into a computer, other programmable data processing device, or other device to generate a computer implementation process. These instructions can then be executed on the computer, other programmable device, or other device to perform a series of action steps, such as carrying out a function / action specified by an item or item in a flowchart, GUI, sequence, system, component, swimlane activity diagram.

[0208] Furthermore, the methods, non-temporary computer-readable media, systems, and apparatus described herein may be subject to various modifications and alternative forms, specific examples of which are shown in the drawings and described herein in detail. However, it should be understood that the present invention is not limited to any particular form or method disclosed, but rather covers all modifications, equivalents, and alternatives that fall within the spirit and scope of the various embodiments and appended claims described herein. Furthermore, any particular features, aspects, methods, characteristics, features, qualities, attributes, elements, etc., disclosed herein relating to an implementation or embodiment may be used in all other implementations or embodiments described herein. Any method disclosed herein does not need to be performed in the order mentioned herein. A method disclosed herein may include specific actions performed by a practitioner, but a method may also include, explicitly or implicitly, instructions by any third party for those actions. The scope disclosed herein also includes any and all overlaps, sub-scopes, and combinations thereof.

[0209] Words such as "at least" and "at a minimum" include the number mentioned. Numbers preceded by terms such as "about" or "approximately" should include the cited number and be interpreted on a contextual basis (e.g., as accurately as reasonably possible under the circumstances, e.g., -5%, 5%, 10%, 15%, etc.). For example, "about 3.5 mm" includes "3.5 mm". Phrases preceded by terms such as "substantially" should include the cited phrase and be interpreted on a contextual basis (e.g., as accurately as reasonably possible under the circumstances). For example, "substantially constant" includes "constant".

[0210] Detailed descriptions are provided to enable those skilled in the art to manufacture and use the invention and are provided in relation to the requirements for patentability. Various modifications to the exemplary embodiments will be readily apparent to those skilled in the art, the general principles of the invention may be applied to other embodiments, and some features of the invention may be used without corresponding use of other features. Accordingly, the invention is not intended to be limited to the embodiments shown, but rather to give the broadest possible scope consistent with the principles and features described herein.

Claims

1. A computer implementation method for similar information determination and text generation for automatic annotation, which is of potential user interest in web documents, Computing a source set of sentence embedding vectors associated with sentences obtained from multiple source contents, wherein the types of elements of the multiple source contents are selected from a group consisting of web pages, documents, and annotations for web pages and documents, and the calculation is performed using a deep learning or machine learning model for sentence embedding. When a user navigates their web browser to a URL corresponding to a web document, the web browser loads the web document, and The type of elements in multiple request content is selected from a group consisting of web documents, annotations for web documents, replies to annotations for web documents, and items associated with web documents belonging to a collection containing web documents. This is computed using a deep learning or machine learning model for sentence embedding vectors, and the request set of sentence embedding vectors associated with sentences obtained from multiple request content. Using a request set of sentence embedding vectors and a source set of sentence embedding vectors, retrieve a subset of multiple source contents via the API, Multiple request content and subsets of auto-annotation are of potential user interest, and using deep learning models for natural language processing, it is possible to auto-annotate subsets of multiple request content to generate auto-annotation using obtained subsets of multiple source content and multiple request content, and Displaying the automatic annotations to the user on the web document displayed in a web browser or an annotation-enabled web browser, A method for providing this.

2. The method according to claim 1, wherein the web document is a web page or web PDF document available at the corresponding URL.

3. The method according to claim 1, wherein the plurality of source contents include web PDF documents accessible by URL in an open access repository of PDF documents.

4. The method according to claim 1, wherein the automatic annotation is an annotation item that includes a link to similar content on another web document.

5. The method according to claim 1, wherein a subset of multiple request contents that represent the user's potential interests correspond to content within a web document.

6. The method according to claim 1, wherein the automated annotations displayed to the user on a web document are augmented with generated text summaries of linked articles, generated using natural language processing techniques.

7. The method according to claim 6, wherein the natural language processing technique utilizes a deep learning model that uses a transformer architecture.

8. One or more non-temporary computer-readable media, which store data representing software executable by an electronic device having at least one processor, for similar information determination and automatic annotation text generation of potential user interest within web documents, and the non-temporary computer-readable media comprises the following instructions: Computing a source set of sentence embedding vectors associated with sentences obtained from multiple source contents, wherein the types of elements of the multiple source contents are selected from a group consisting of web pages, documents, and annotations for web pages and documents, and the calculation is performed using a deep learning or machine learning model for sentence embedding. When a user navigates their web browser to a URL corresponding to a web document, the web browser loads the web document, and The type of elements in multiple request content is selected from a group consisting of web documents, annotations for web documents, replies to annotations for web documents, and items associated with web documents belonging to a collection containing web documents. This is computed using a deep learning or machine learning model for sentence embedding vectors, and the request set of sentence embedding vectors associated with sentences obtained from multiple request content. Using a request set of sentence embedding vectors and a source set of sentence embedding vectors, retrieve a subset of multiple source contents via the API, Multiple request content and subsets of auto-annotation are of potential user interest, and using deep learning models for natural language processing, it is possible to auto-annotate subsets of multiple request content to generate auto-annotation using obtained subsets of multiple source content and multiple request content, and Displaying the automatic annotations to the user on the web document displayed in a web browser or an annotation-enabled web browser, It includes instructions for One or more non-temporary computer-readable media.

9. The web document is one or more non-temporary computer-readable media according to claim 8, wherein the web document is a web page or web PDF document available at the corresponding URL.

10. The plurality of source contents include one or more non-temporary computer-readable media according to claim 8, which include web PDF documents accessible by URL in an open access repository of PDF documents.

11. The one or more non-temporary computer-readable media according to claim 8, wherein the automatic annotation is an annotation item that includes a link to similar content on another web document.

12. A subset of multiple request contents of potential user interest corresponds to one or more non-temporary computer-readable media according to claim 8, wherein the content in the web document is one or more of the above.

13. One or more non-temporary computer-readable media according to claim 8, wherein the automatically generated annotations displayed to the user on the web document are augmented with generated text summaries of linked articles, generated using natural language processing techniques.

14. The natural language processing technique utilizes a deep learning model using a transformer architecture, one or more non-temporary computer-readable media according to claim 13.

15. A system for determining similar information and generating text for automatic annotations that are of potential user interest within web documents, The types of elements of multiple source content are selected from a group consisting of web pages, documents, and annotations for web pages and documents, and a deep learning or machine learning model for sentence embeddings is used to compute a source set of sentence embedding vectors associated with sentences originating from multiple source content. An electronic database for storing source sets of sentence embedding vectors and the sentences associated with them, An annotable web browser that can load a web document into the web browser when the web browser navigates to the corresponding URL of the web document, and can send multiple web document request contents coming from the web document to an annotation server, The type of elements in multiple request content is selected from a group consisting of multiple web document request content, web document annotations, responses to web document annotations, and items associated with web documents belonging to a collection containing web documents. An annotation server capable of running a deep learning or machine learning model for sentence embeddings to compute a request set of sentence embedding vectors associated with sentences derived from multiple request contents, An annotation server capable of retrieving a subset of multiple source contents from an electronic database using a text embedding vector request set and a text embedding vector source set, A subset of multiple request contents and a group of auto-annotations represent the user's potential interests. An annotation server that uses a deep learning model for natural language processing to generate a group of automatically annotated subsets of multiple request contents using retrieved subsets of multiple source contents and multiple request contents, And, An annotation-enabled web browser displays a group of automatic annotations on a loaded web document, which is displayed in a web browser or an annotation-enabled web browser. A system equipped with these features.

16. The system according to claim 15, wherein the web document is a web page or web PDF document available at a URL on the World Wide Web.

17. The system according to claim 15, wherein the multiple source contents include web PDF documents accessible by URLs in an open access repository of PDF documents.

18. The system according to claim 15, wherein the automatic annotation is an annotation item that includes a link to similar content on other web documents.

19. The system according to claim 15, wherein a subset of multiple request contents that represent the user's potential interest is a subset of multiple web document request contents.

20. The system according to claim 15, wherein automatic annotation for users viewing web documents is augmented with generated text summaries of linked articles, generated using natural language processing techniques.

21. The natural language processing technology is the system according to claim 20, which utilizes a deep learning model that uses a transformer architecture.

Images