Method, computer readable medium, and system for creating, organizing, viewing and connecting annotations of documents within web browsers

Abstract

To provide a method, a computer readable medium, and a system for creating, organizing, viewing, and connecting annotations of web documents within annotation-enabled web browsers.SOLUTION: A method transmits an annotation highlighted text, an annotation comment, a uniform resource locator (URL), a document object model (DOM) context, a screenshot including a marked annotation highlighted text, and an annotation symbol to an annotation server.SELECTED DRAWING: Figure 14

Description

Background Art

[0001] The World Wide Web (WWW), web browsers, and browser extensions are well known. The WWW is a global information source where assets such as web pages, PDF documents, images, videos, etc. are identified by unique Uniform Resource Locators (URLs), and a web browser is a software application designed to access information on the WWW. Also, browser extensions are software created to extend the functionality of browsers. It is known that the functionality of a web browser can be changed by changing the software codebase of the web browser or by providing a web browser extension for the web browser that supports adding functionality by such means.

[0002] As the popularity of the World Wide Web has grown, people, on average, are spending more time using the World Wide Web each day than in the past. Due to the increased average time spent using the World Wide Web, people have needs such as searching for information they previously saw on the WWW, sending snippets of text within web documents to friends and colleagues for discussion, referring to specific pieces of text or context within web documents, asking specific questions about parts of web documents, connecting information between web documents, finding and organizing content and snippets for later use, evaluating the value of crowd-sourced information, accessing other people's public comments on web documents and placing these comments in context, etc.

[0003] To address these use cases, people typically repeat searches to find previously accessed content, bookmark web documents, rescan articles for snippets they remember seeing, copy & paste sections of text into email messages or other documents to share with friends and colleagues, reference sentences or paragraphs by number or web page location while discussing on the phone, ask questions in the comment section of articles, cut & paste in comments to reference desired text, download web pages, or copy snippets into personal word processing or spreadsheet documents. This is inefficient and time-consuming for many users. Additionally, there is no tool for identifying inaccurate information and spreading this knowledge to other users, allowing inaccurate information to persist longer than necessary. It would be advantageous to provide a method, computer-readable medium, and system for more efficiently handling these tasks.

Summary of the Invention

[0004] An object of the present invention is to provide a user with an annotatable web browser that can annotate web pages and documents so that a user located at another place on the Internet can identify questions, interesting points, statements of agreement, and statements of disagreement without reading user annotations, comments, or annotation-emphasized text using the same annotatable web browser. Such an arrangement improves the efficiency of users when sifting through the vast amount of information on the WWW.

[0005] A further object of the present invention is to provide a user with an annotatable web browser for annotating images, videos, and recordings uploaded or embedded in web pages so that users located at other places on the Internet can also view these image, video, and recording annotations using such an annotatable web browser.

[0006] Another object of the present invention is to ensure that after the annotation is added, the document display context is changed in a minimal way compared to the document display context without the annotation. By using a document viewing context that is changed in a minimal way, the user can consume the content without being distracted by the annotation, and at the same time, can find and utilize the annotation at an appropriate timing.

[0007] A further object of the present invention is to non-intrusively present advertisements to the user. One object is to show the user advertisement content after these users have initiated an action caused by curiosity, which is caused by the presentation of an advertisement at a specific location or context within a web page. This location and context suggest the possibility that the advertisement is relevant to the user's current needs. Further, an advertisement presented without content can include some aspects of a complete advertisement, but the full details are not displayed until the user initiates an action caused by curiosity planted by a contextual method that selectively and transparently targets the user based on the user's current needs.

[0008] Another object of the present invention is to provide an annotation viewing context area separate from the World Wide Web document viewing context area. This annotation viewing context area has a mechanism for presenting annotations to the user so that the user can understand familiar sites and the approximate location of the annotations, in addition to search and filtering. Further, other elements of the context of the annotation or emphasized text may become apparent without the user having to read the emphasized text or the user's annotation comments. Such a mechanism makes it easy to find a specific annotation among a large number of annotations.

[0009] A further object of the present invention is to provide an annotation compilation context area that enables a user to scan annotations and organize them into meaningful groups. The advantage of this object is that it allows the user to first concentrate on creating annotations and then concentrate on organizing the annotations after they have been created. Further, the default compilation of the annotations need not be restored.

[0010] Another object of the present invention is to provide the user with an annotation display context area that enables the user to display the annotation page, keep some annotations available for later review, and continue to search for other annotations. Further, it is a way to facilitate mental tracking by maintaining the annotations in a visual view while moving uninteresting annotations outside the visual view.

[0011] An additional object of the present invention is to always transparently convey privacy settings to the user. By conveying the privacy settings, it is possible to avoid accidentally sharing information that the user does not want to share.

[0012] A further object of the present invention is a method of creating a guided tour through a series of websites on the global Internet and at the same time drawing the user's attention to prominent points of interest on each site. This object further includes a method of modifying the tour and presenting the tour to the user.

[0013] Another object of the present invention is to provide a method for a group to manage access control permissions and communicate through messaging.

Brief Description of the Drawings

[0014] For a more complete understanding of the present invention, reference may be made to the following description and the accompanying drawings.

[0015] Figure 1 shows a flowchart in which an annotation is created, moved to two different collections, and shared with others via the collections.

[0016] Figure 2 shows a flowchart of an annotation viewed on two separate web pages on two separate days.

[0017] Figure 3 represents an annotatable web browser in an exemplary embodiment in which a web browser without the ability to create an annotation is extended with a web browser extension that provides annotation capabilities before the annotation process is initiated.

[0018] Figure 4 shows an annotatable web browser after text has been highlighted.

[0019] Figure 5 shows an annotatable web browser before an annotation symbol is selected.

[0020] Figure 6 shows an annotatable web browser after an annotation symbol has been selected.

[0021] Figure 7 shows an annotatable web browser after an annotation has been created and the displayed annotation symbol is in an open state.

[0022] Figure 8 shows an annotatable web browser after an annotation has been made. Indicates that the created and displayed annotation symbol is in a closed state.

[0023] Figure 9 shows an annotation item box that provides the context of an annotation when it is presented not as an overlay on an existing web document adjacent to the highlighted annotation text but in a separate context area.

[0024] Figure 10 shows an annotation composition context area presenting an annotation item box and a writable collection displayed within a context area separate from the original web document containing the annotation-emphasized text.

[0025] Figure 11 shows an annotation composition context area demonstrating the process of moving and duplicating annotations to a collection that is a grouping of annotations and other items.

[0026] Figure 12 shows an annotatable web browser displaying a web document with an annotation symbol shown in a closed state.

[0027] Figure 13 shows an annotatable web browser that displays a web document with a displayed annotation symbol in a closed state using the corresponding displayed annotation symbol beacon.

[0028] Figure 14 shows a sequence diagram depicting the communication among an annotatable web server, a web server providing data of the visited URL, an annotation server, an electronic database, and a CDN.

[0029] Figure 15 shows a sequence diagram depicting the communication among a web browser for organizing annotations, an annotation server, an electronic database, and a CDN.

[0030] Figure 16 shows a sequence diagram depicting the communication among an annotatable web browser, a web server providing data of the visited URL, an annotation server, and an electronic database.

[0031] Figure 17 shows a flowchart depicting the reception and saving of a bundle of data elements.

[0032] Figure 18 shows a flowchart depicting the compilation and transmission of a bundle display set.

[0033] Figure 19 shows a flowchart illustrating the interaction among a user, a web browser, a web browser extension content script, and a web browser extension background script that enables the user to view a web page and create an annotation on the page.

[0034] Figure 20 shows a flowchart illustrating the way of interaction among a user, a web browser, a web browser extension background script, and a web browser extension content script that enables the user to view a web page with an annotation on the page.

[0035] Figure 21 shows a flowchart illustrating the method for a nurse to create a memo regarding the glycemic index for sharing with a patient using this exemplary embodiment.

[0036] Figure 22 shows a flowchart illustrating how a patient uses the memo provided by the nurse.

[0037] Figure 23 shows the user interface of the memo within the extension.

[0038] Figure 24 shows a sequence diagram illustrating the state where a memo is created and one annotation is created and added to the memo.

[0039] Figure 25 shows a sequence diagram illustrating the state where the memo is viewed, the first annotation is clicked, the browser navigates to its URL, scrolls to the appropriate location, and then displays the annotation.

[0040] Figure 26 shows a flowchart illustrating the state where a memo is created and an already created annotation is added to the memo.

[0041] Figure 27 shows a flowchart illustrating the state where the memo is viewed and the annotation of the memo is opened.

[0042] Figure 28 shows an exemplary embodiment of a system diagram.

[0043] Figure 29 shows another exemplary embodiment of a component diagram for an annotation server and an electronic database.

[0044] Figure 30 shows an exemplary embodiment showing a client electronic device for an annotation server.

[0045] Figure 31 presents an exemplary embodiment of a collection item page for a collection of American history.

[0046] Figure 32 shows an exemplary embodiment of a collection page.

[0047] Figure 33 shows an exemplary embodiment of an inbox page showing system messages.

[0048] Figure 34 shows an exemplary embodiment of an inbox page after a user has responded to a system message and other users have also sent messages.

[0049] Figure 35 shows an image annotation display and an image annotation group creation interface.

[0050] Figure 36 shows an image annotation group creation interface in which one annotation group list item has been added to the annotation group candidate list.

[0051] Figure 37 shows an image annotation group creation interface having one annotation group list item to which a second non-colliding image annotation has not been added to the candidate annotation group list.

[0052] Figure 38 shows an image annotation group creation interface where two annotation group list items are added to the annotation group candidate list.

[0053] Figure 39 shows an image annotation group creation interface where one annotation group list item is added to the candidate annotation group list and the second collision image annotation is not added to the candidate annotation group list.

[0054] Figure 40 shows a video annotation viewing and video annotation group creation interface.

[0055] Figure 41 shows a video annotation display and video annotation group creation interface where one video annotation is added to the annotation group candidate list box having a motion path for the video annotation element.

[0056] Figure 42 shows a video annotation group creation interface where two annotation group list items are added to the candidate annotation group list having a motion path for the second video annotation element.

[0057] Figure 43 shows a video annotation group creation interface where one annotation group list item is added to the candidate annotation group list and the second collision video annotation is not added to the candidate annotation group list.

[0058] Figure 44 shows a sound annotation viewing and sound annotation group creation interface for timeline sound annotation and derived sound annotation creation interface.

[0059] Figure 45 shows a video annotation display and a video annotation group creation interface having a timeline video annotation and a derived video annotation creation interface.

[0060] Figure 46 shows an annotatable web browser that displays a video in which a video annotation beacon is embedded.

[0061] Figure 47 shows an annotatable web browser after an image annotation is created and the embedded displayed image annotation is in an open state.

[0062] Figure 48 shows an annotatable web browser after a video annotation is created, indicating that the embedded displayed video annotation is open during the video annotation playback range.

[0063] Figure 49 shows an annotatable web browser after a voice annotation is created and the embedded displayed voice annotation is in an open state during the playback range of the voice annotation.

Embodiments for Carrying Out the Invention

[0064] A method, non-transitory computer-readable medium (CRM), and system are provided for adding annotations to web pages, documents, images, videos, and recordings within a web browser, organizing these annotations, viewing these annotations, and connecting these annotations to a useful digital notebook containing a series of annotations. The web documents have URLs and include web pages, pdf documents, and other document types that are displayed in a web browser. Further, these methods, non-transitory CRM, and systems provide facilities for content curation, sharing of content with other users, and management of access control. This specification includes flowcharts showing the basis for users to use the present system and non-transitory CRM. Further, graphical user interface (GUI) diagrams show the final results of the instructions of the computer-readable medium and what the system displays. Further, sequence diagrams show the interactions and data flows between various components, and flowcharts explain the methods. Further, swimlane diagrams show the operations of various components.

[0065] An annotation is a note, comment, or drawing of an explanation added to text, a figure, an image, a document, a video, or a recording. When describing the method, non-transitory CRM, and system, the terms are introduced to describe annotations within a separate context, as well as data elements and structures representing annotations and objects added to the context of annotations within web pages, web documents, documents, images, videos, and recordings, as well as within another context area.

[0066] Figure 1 shows a flowchart of a teacher using an exemplary embodiment of an annotatable web browser. The teacher has two separate goals. The first goal is to create a list of learning questions about the president for the students in the teacher's class. The teacher's class is small, and in this example within the depicted exemplary embodiment, the students can see each other's answers. The second goal is to find a house for sale near the school where the teacher teaches. As shown in Figure 1, on the first day, the teacher [1] visits the Wikipedia page of Abraham Lincoln, creates a question annotation [4], then visits the Wikipedia page of Woodrow Wilson and creates a question annotation [5]. As shown in Figure 1, on the second day [2], the teacher visits the Zillow page and creates a dagger annotation [6] of a house for sale near the workplace. As shown in Figure 1, on the tenth day, the teacher [3] sets up two collections, the US President Learning Guide and Real Estate [7], adds students to the US President Learning Guide collection, adds roommates to the Real Estate collection [8], moves the question annotations of Abraham Lincoln and Woodrow Wilson to the US President Learning Guide [9], moves the dagger annotation regarding the house for sale near the workplace to the Real Estate collection

[10] , shares the US President Learning Guide collection with the students

[11] , and further adds roommates to the US President Learning Guide and Real Estate collections.

[0067] In FIG. 2, a flowchart of a student using an exemplary embodiment of an annotatable web browser is shown. The student's goal is to answer learning questions provided by the teacher. As shown in FIG. 2, on the 11th day, student

[12] visits the Wikipedia page of Abraham Lincoln

[14] , reads the Wikipedia page of Abraham Lincoln

[15] , views the displayed annotation symbols of the teacher's closed questions while reading the Wikipedia page of Abraham Lincoln

[16] , clicks on the displayed annotation symbols of the teacher's closed questions, transitions to the state where the displayed annotation symbols of the questions are open, and displays the teacher's annotation comments

[17] , and responds to the teacher's question annotation comments with an answer.

[18] As shown in FIG. 2, on the 12th day, student

[13] visits the Wikipedia page of Woodrow Wilson

[19] , reads the Wikipedia page of Woodrow Wilson

[20] , views the displayed annotation symbols of the teacher's closed questions while reading the Wikipedia page of Woodrow Wilson

[21] , clicks on the displayed annotation symbols of the teacher's closed questions, the displayed annotation symbols of the questions move to the open state, and the teacher's annotation comments are displayed

[22] , and responds to the teacher's question annotation comments with an answer.

[23]

[0068] In FIG. 3, an exemplary embodiment of an annotatable web browser

[24] is shown. The horizontally long and short rectangles displayed in the display area of the depicted web browser represent text. There are 11 rectangles representing 11 lines of text in the first paragraph and 5 rectangles representing 5 lines of text in the second paragraph. In this exemplary embodiment, the web browser that supports the addition of web extensions is modified with an exemplary example of a web browser extension

[25] that adds an annotation function. The web browser extension

[25] is a software package typically present in a non-transitory computer-readable medium that can be loaded into memory by the web browser, supports the addition of web extensions, and the web browser application itself is a software application loaded into memory and executed on the processor of an electronic computing device. When the web browser application loads the web browser extension

[25] into memory, the instruction set of the web browser extension

[25] can be triggered by the instruction set of the web browser application at a specific point in the web browser's life cycle. Some of the effects of the web browser extension

[25] instruction set on the user interface presented to the user are shown in FIGS. 5, 6, 7, 8, 12, 13, 23, 46, 47, 48, and 49.

[0069] In FIG. 4, an exemplary embodiment of an annotatable web browser

[24] shows the state before the user selects text, which is called normal emphasized text

[26] , and before the instruction set of the web browser extension

[25] is triggered.

[0070] In FIG. 5, an exemplary embodiment of an annotatable web browser

[24] is shown after the user selects text, the web browser extension

[25] instruction set is triggered, and the annotation symbol selection button bar

[28] is displayed. Further, the selected text will be referred to as annotation emphasized text

[27] in order to distinguish it from normal emphasized text

[26] , as shown in the state of the exemplary embodiment depicted in FIG. 4. Normal emphasized text

[26] is the selected text before the web browser extension

[25] instruction set is triggered, or the selected text in a web browser that does not have an annotation function. The web browser extension

[25] instruction set processes the input after the user clicks any of the buttons shown on the annotation symbol selection button bar

[28] in FIG. 5.

[0071] In an exemplary embodiment, FIG. 6 depicts the effect of a user clicking on the question mark annotation symbol button in the annotation symbol selection button bar

[28] illustrated in FIG. 5. In an alternative exemplary embodiment, the annotation symbols shown in the annotation symbol selection button bar

[28] of FIG. 5 may be arranged one above the other in a vertical column as opposed to a horizontal row. In the exemplary embodiment depicted in FIG. 6, the displayed annotation symbol symbol

[29] of the open question and the annotation comment input box

[30] are shown. The displayed annotation symbol symbol

[29] of the open question can indicate that the user intends to enter a question regarding the annotation emphasis text

[27] into the annotation comment input box

[30] . The annotation comment input box

[30] is an input area for a user of the annotatable web browser

[24] to enter text that is stored in one or more computer systems. When the user finishes entering text and conveys the completion of the text entry operation to the annotatable web browser

[24] , a set of web browser extension function

[25] instructions causes the annotatable web browser

[24] to transition to the state depicted in FIG. 7.

[0072] In the exemplary embodiment depicted in FIG. 7, after an annotation is created and the displayed annotation symbol is visually displayed in an open state, the state of the annotatable web browser

[24] is displayed. The displayed annotation symbol in the open state displays screen elements such as annotation emphasis text

[27] , annotation comment

[32] , reputation selector

[33] , the displayed annotation symbol symbol

[29] of the opened question, and annotation comment box

[31] . The annotation comment

[32] is derived from the text input by the user as described above. The annotation comment box

[31] also has a context-variant annotation control bar

[34] . With the context-variant annotation control bar

[34] , the user can change privacy settings, share via social media or email, flag an annotation, and delete an annotation according to access control rules. With the reputation selector

[33] , the user can select an evaluation to assign to the annotation comment

[32] . For example, if the user thinks the comment is thoughtful, the user can assign a thoughtful evaluation to the annotation comment

[32] . By clicking outside the annotation emphasis text

[27] , the displayed annotation symbol

[29] of the opened question, and the annotation comment box

[31] , the user can transition the displayed annotation symbol to a closed state and close the annotation. In another embodiment, the reputation selector

[33] may be displayed inside the context-variant annotation control bar

[34] , as opposed to being arranged adjacent to the annotation comment

[32] .

[0073] In the exemplary embodiment depicted in FIG. 8, the state of the annotatable web browser

[24] after an annotation has been created and the displayed annotation symbol is shown in a closed state is illustrated. FIG. 8 shows the displayed annotation symbol

[35] of a closed question. The displayed annotation symbol

[35] of the closed question is visible but is designed so that the user is not distracted from the remainder of the text.

[0074] After the user creates an annotation using an annotatable web browser

[24] , a bundle of data elements representing the annotation is sent to the server. This bundle of data elements can be used to construct an annotation item box

[36] in a different context area, as shown in the exemplary embodiment depicted in FIG. 9. In the exemplary embodiment depicted in FIG. 9, the annotation item box

[36] includes a screenshot

[37] with marked text, a separate context annotation emphasis text

[38] , an annotation symbol

[39] with a displayed separate context question, a separate context annotation comment

[40] , a separate context variant annotation control bar

[41] , and a separate context reputation selector

[42] . In this exemplary embodiment, the screenshot

[37] with marked text is a thumbnail-sized related marked screenshot intended to enable the user to determine the web page where the annotation exists and the location of the annotation within the web page. FIGS. 7 and 9 can be viewed as two different contexts of the same exemplary embodiment or as two different exemplary embodiments. For the purpose of understanding the separate context annotation emphasis text

[38] , the displayed annotation symbol

[39] with a separate context question, and the separate context annotation comment

[40] , it is useful to view these two figures as two different contexts of the same exemplary embodiment. The separate context annotation emphasis text

[38] presents the annotation emphasis text

[27] in a separate context area of the annotation item box

[36] . The description of the separate context area in this situation indicates that the context is different from the original web page or web document where the annotation emphasis text

[27] occurred. The displayed annotation symbol

[39] with a separate context question presents a question annotation symbol in a separate context area of the annotation item box

[36] .Separate context annotation comments

[40] present annotation comments

[32] in a separate context area of the annotation item box

[36] . The separate context variant annotation control bar

[41] enables the user to change privacy settings, share on social media or by email, associate classification ratings with the annotation, or delete the annotation.

[0075] When the item box represents an annotation, it is called an annotation item box

[36] and is depicted in the exemplary embodiment of FIG. 9. In the exemplary embodiment, the item box can represent a web page, document, image, video, and recording, as well as the annotation of a web page, document, image, video, and recording. The item box may be a web page item box, document item box, image item box, video item box, recording item box, annotation item box, image-annotation item box, video-annotation item box, recording annotation item box, embedded image-annotation item box, embedded video-annotation item box, embedded recording item box, annotation group item box, derived annotation group item box, derived annotation-attached annotation group item box. A collection is a representative of a group of items including annotations, and a writable collection is a collection that the user can change in some way. One form of change is to add an item to a writable collection. Collections are used to organize items including annotations into meaningful groups. As an example, in FIG. 1, the teacher created two collections, one titled "US President Learning Guide" and the other titled "Real Estate".

[0076] In the exemplary embodiment depicted in FIG. 9, an annotation item box

[36] includes a separate context reputation selector

[42] in a separate context - variant annotation control bar

[41] , enabling users sharing annotations in the form of the annotation item box

[36] to vote on the reputation by selecting from a drop - down. This reputation is assigned to the annotation. This reputation has value within the social community. Annotations with reputations such as useful, interesting, or considerate, or annotations expressing opposition to the annotation - emphasized text, can be selectively displayed based on various criteria selected by the user and determined to be useful by the annotation server

[61] shown in some other figures. The user can filter the annotations based on this criterion, and the aggregated reputation of the user can also be connected to the reputation of the annotations created by the user. In contrast to star - based voting, detailed annotation - specific semantic information about why an annotation is valuable to other users within the online community can be collected from the user.

[0077] Furthermore, in addition to giving a reputation to an annotation, the annotation system functions better if users can reply to the annotation. Additionally, other users can also vote on the reputation of the reply. This system helps the community identify potentially inaccurate annotations as users provide feedback through replies and reputations.

[0078] Both the exemplary embodiment depicted in FIG. 7 and the exemplary embodiment depicted in FIG. 9 represent annotations. FIG. 7 represents an annotation embedded within the web document to which it is attached, while FIG. 9 represents an annotation outside of the web document. Since FIG. 9 has no context around the web document, a thumbnail-marked screenshot is provided to give the user viewing the annotation item box

[36] more context. FIGS. 7 and 9 can potentially depict two contexts of the same exemplary embodiment, or they can potentially depict two different exemplary embodiments. Both FIGS. 7 and 9 enable reputation voting. In FIG. 7, the reputation selector

[33] enables the user to assign a reputation to the annotation comment

[32] . In FIG. 9, another context reputation selector

[42] enables the user to assign a reputation to the annotation comment

[32] . This allows users sharing an annotation to evaluate the reputation of the annotation from different contexts.

[0079] In the exemplary embodiment depicted in FIG. 9, the annotation item box

[36] includes a separate context-variant annotation control bar

[41] that enables the user to change privacy settings. In the exemplary embodiment, dark gray indicates the most private setting where no one other than the creator of the annotation can view the annotation, light gray indicates that the annotation can be shared with a specific group of users, and white indicates that the annotation can be viewed by all users of the platform.

[0080] One of the separate context areas that can render the annotation item box

[36] depicted in FIG. 9 is the annotation composition context area

[43] of the exemplary embodiment depicted in FIG. 10. The annotation composition context area

[43] of FIG. 10 also includes an item bin

[44] for displaying an item box and a writable collection bin

[45] for displaying a writable collection. In FIG. 9, the annotation item box

[36] is depicted within the item bin

[44] , and the writable collection

[50] displayed within the writable collection bin

[45] is depicted. Further, the search filter

[46] can be used to change the item box displayed in the item bin

[44] . Further, the collection search filter

[47] can be used to change the display of the writable collection displayed in the writable collection bin

[45] . Further, the paging mechanism

[48] can be used to change the pages of the item boxes displayed in the item bin

[44] , and the collection paging mechanism

[49] can be used to change the collection pages of the displayed writable collections displayed in the writable collection bin

[45] .

[0081] In the exemplary embodiment depicted in FIG. 10, it can be seen that the item bin

[44] has a paging mechanism

[48] , and the writable collection bin

[45] has a collection paging mechanism

[49] . These paging mechanisms operate independently. The paging mechanism

[48] of the item bin

[44] moves forward and backward through the 6-item pages of the exemplary embodiment being displayed. The items shown in FIG. 10 represent annotations. All of these items are arranged sequentially. When the paging mechanism is moved forward, the next 6 items among the sequentially arranged items are displayed. When the paging mechanism is moved backward, the previous 6 items in the set of sequentially ordered items are shown. The collection paging mechanism

[49] within the writable collection bin

[45] moves forward and backward through the 3 collection pages in the exemplary embodiment being displayed. The collections shown in FIG. 10 represent groups of items that include annotations. Each collection represents a group of one or more items, and the items include web page items, document items, image items, video items, recording items, annotation items, image-annotation items, video-annotation items, recording annotation items, embedded image-annotation items, embedded video-annotation items, embedded recording items, annotation-group items, derived annotation-group items, annotation groups with derived annotation items. All of these collections are arranged sequentially. When the paging mechanism is moved forward, the next 3 items in the set of sequentially ordered collections are displayed. When the paging mechanism is moved backward, the previous 3 items in the sequentially arranged collections are shown. When the displayable area of the browser changes after a sizing operation, the number of items displayed in the item bin

[44] and the number of writable collections displayed in the writable collection bin

[45] change. In an alternative exemplary embodiment, there may be no explicit writable permission.In this case, the writable collection bin

[45] will include the displayed collection. The paging mechanism

[48] and the collection paging mechanism

[49] take this shift into account and operate with the new page size.

[0082] In FIG. 10, in addition to the paging mechanism

[48] , the pins of all the annotation item boxes

[36] located in the item bin

[44] can also be seen. In the exemplary embodiment shown in FIG. 10, six items representing annotations are shown. Each of these six items can be in an unpinned state or a pinned state. As shown, the page size is 6, and when one of these items is pinned, the page size decreases to 5. As a result of the fixation, the pinned item remains in the display area even when the paging mechanism

[48] is moved back and forth. When the paging mechanism

[48] is advanced, the pinned item remains in the viewing area, and the remaining five items move out of the viewing area. Then, the next five items in sequential order replace the five removed items. When the paging mechanism moves backward, the five unpinned items are removed from the display area, and the previous five items in the sequential order of the items replace the five items removed from the display area. When two items are pinned, the page size is reduced to 4. Pinning functions similarly for the collections within the writable collection bin

[45] .

[0083] Each time an item is pinned, the pinned item is excluded from the set of items that are sequentially ordered by the annotation server

[61] shown in some other figures. This can avoid the pinned item being displayed twice in the display area. When two items are pinned, the two pinned items are excluded from the set of items that are ordered sequentially by the annotation server

[61] . The exclusion from the sequential rearrangement is performed in addition to the page size change. The exclusion of the sequential order by the annotation server

[61] functions similarly for the collection in the writable collection bin

[45] .

[0084] Filters such as the search filter

[46] shown in the exemplary embodiment of FIG. 10 are used to receive input from the user. In the case of the search filter

[46] of the item bin

[44] , it is a search phrase. These inputs are converted into parameter values that are sent to the annotation server

[61] shown in some other figures. These parameter values are used by the annotation server

[61] to input parameters into the request query used by the electronic database

[62] shown in some other figures. The request query includes parameter values and page numbers. The request query containing parameter values is used to specify a set of items and their order. The page number specifies a particular page. Further, the request query can include the number of items to be sent to the web browser running on the client electronic device. This request query includes parameters of filter values such as search phrases and an exclusion list of items to be excluded from the list of specified items, and the request query introduces a sequential ordering for the list of those items along with the parameters and the item exclusion list.

[0085] Filters such as the collection search filter

[47] shown in the exemplary embodiment of FIG. 10 are used to receive input from a user. In the case of the collection search filter

[47] of the writable collection bin

[45] , it is a collection search phrase. These inputs are converted into parameter values that are sent to the annotation server

[61] shown in some of the other figures. These parameter values are used by the annotation server

[61] to input parameters into the collection request query used by the electronic database

[62] shown in some of the other figures. The collection request query includes parameter values and page numbers. The collection request query is used to specify a set of collections, along with the collection parameter values, and an ordering for those collections. The page number specifies a particular page. Further, the collection request query can include the number of collections to be specified that are to be sent to the web browser running on the client electronic device. This collection request query includes parameters for collection filter values such as the collection search phrase, and an exclusion list of collections to be excluded from the list of the specified collections, and this collection request query introduces an ordering for the list of those collections, along with the parameters and the exclusion list of collections.

[0086] After the user creates an annotation, in the exemplary embodiment depicted in FIG. 10, the annotation is displayed in the item bin

[44] . Using a drag-and-drop process, the user can move or duplicate the annotation item box

[36] to a collection, as in the exemplary embodiment depicted in FIG. 11. In the exemplary embodiment depicted in FIG. 11, a moving annotation item box

[51] is depicted. As the moving annotation item box

[51] approaches the displayed writable collection

[50] , as in the state of the exemplary embodiment depicted in FIG. 10, the visual form of the displayed writable collection

[50] changes to that depicted in FIG. 11, which shows screen elements including a move target

[52] and a duplicate target

[53] . In this exemplary embodiment, when the mouse pointer depicting the drag of the moving annotation item box

[51] is within the area indicated by the move target

[52] and is released, a move action is triggered. In this exemplary embodiment, when the mouse pointer indicating the drag of the moving annotation item box

[51] is released within the area specified by the duplicate target, a duplicate operation is initiated.

[0087] In the exemplary embodiment depicted in FIG. 12, the user is presented with the displayed annotation symbol

[35] for a closed question, the displayed annotation symbol

[54] for a closed exclamation, the displayed annotation symbol

[55] for a closed check, and the displayed annotation symbol

[56] for a closed incorrect answer. These four annotation symbols are examples of displayed annotation symbols. In the exemplary embodiment, the displayed annotation symbol has at least two possible states. These states include an open state and a closed state. In the closed state, the displayed annotation symbol is visible and is designed to have low visibility so as not to distract the user while the user is reading the web document. In the open state, additional information is displayed in the display area of the web browser. In the exemplary embodiment of FIG. 7, the displayed annotation symbol in the open state is depicted. This displayed annotation symbol is the displayed annotation symbol for a question. In the exemplary embodiment of FIG. 7, the displayed annotation symbol

[29] for an open question is shown together with the annotation emphasis text

[27] and the annotation comment box

[31] .

[0088] Annotation symbols are associated with elements of a group of semantic relationships of an annotation. Each semantic relationship of an annotation indicates a relationship, typically based on user selection, between an annotation emphasis text

[27] and an annotation comment

[32] , as depicted in the exemplary embodiment of FIG. 7. Associated with each relationship within the group of semantic relationships of an annotation is a different annotation symbol. As an example, as depicted in the exemplary embodiment of FIG. 8, the displayed annotation symbol

[35] for a closed question indicates that the creator of the annotation has marked the annotation comment as a question regarding the annotation emphasis text. This displayed annotation symbol

[35] for a closed question enables a user familiar with how the exemplary embodiment functions to know that, before the user actually performs the action of clicking, as depicted in FIG. 7, clicking on the displayed annotation symbol

[35] for a closed question allows viewing of the annotation comment that is a question regarding the annotation emphasis text

[27] .

[0089] Once clicked, the displayed annotation symbol

[35] for a closed question, like the state of the exemplary embodiment depicted in FIG. 8, becomes the displayed annotation symbol

[29] for an open question that shows the annotation emphasis text

[27] and the annotation comment box

[31] , like the state of the exemplary embodiment depicted in FIG. 7. Also, in the state of the exemplary embodiment depicted in FIG. 8, when the mouse hovers over the displayed annotation symbol

[35] for a closed question, like the state of the exemplary embodiment depicted in FIG. 8, the annotation emphasis text

[27] , the displayed annotation symbol

[29] for an open question, and the annotation comment box

[31] become visible and can be fixed by clicking. To make this process reflexive for the user, a color based on the annotation symbol can be assigned to the annotation emphasis text

[27] in FIG. 7. That is, the color of the annotation emphasis text

[27] is determined based on the semantic relationship of the annotation between the annotation emphasis text

[27] and the annotation comment.

[0090] To further illustrate the exemplary embodiment, observe the following. The displayed annotation symbol

[54] for a closed exclamation mark indicates that the creator of the annotation marked the annotation comment as an exclamation statement indicating a point of attention or an interesting fact regarding the annotation emphasis text. The displayed annotation symbol

[55] for a closed checkmark indicates that the creator of the annotation marked the annotation comment as a statement agreeing with the annotation emphasis text. Also, the closed incorrect annotation symbol

[56] indicates that the creator of the annotation marked the annotation comment as a statement not agreeing with the emphasized text of the annotation.

[0091] In addition to the annotation symbols discussed so far and the displayed annotation symbols depicted in FIG. 12, advertising annotation symbols can be used to indicate another relationship within a group of semantic relationships of the annotations. This relationship indicates that an annotation comment

[32] such as in the state of the exemplary embodiment depicted in FIG. 7 is an advertisement related to the annotation emphasis text

[27] . In this situation, the displayed annotation symbol

[35] of the closed question will be replaced by the displayed annotation symbol of the advertisement, as in the state of the exemplary embodiment depicted in FIG. 8.

[0092] In the exemplary embodiment depicted in FIG. 12, several displayed annotation symbols can be seen. In this exemplary embodiment, corresponding to the question annotation symbol, the disagreement annotation symbol, the agreement annotation symbol, the exclamation annotation symbol, and the advertising annotation symbol are five relationships in a group of semantic relationships of the annotations. These semantic relationships of the annotations are the question relationship, the disagreement relationship, the agreement relationship, the exclamation relationship, and the advertising relationship.

[0093] In an exemplary embodiment, machine learning and deep learning models can be used to automatically determine the semantic relationships of annotations between annotation-emphasized text and annotation comments. Further, user data collected by the user determining the semantic relationships of annotations and writing annotation comments can be used to train deep learning models, and the by-products of these trained models can be used for other natural language processing purposes. An opt-in or opt-out option can be provided to the user for using the data created by the user for the purpose of training a deep learning-based natural language processing model. To calculate sentence embeddings, a bidirectional long short-term memory (BiLSTM) model with a final max pooling layer can be used. Also, other machine learning and deep learning techniques can be used to create sentence embeddings. Sentence embeddings can be used to determine the similarity between two sentences. After calculating sentence embeddings for all sentences in the collection through the analysis of annotation-emphasized text, annotation comments, and other sentences that occur in items and replies, these sentence embedding vectors can be used to determine whether an item is similar to other items in the collection or to determine the similarity between two collections. Based on this similarity analysis, items can be proposed to the user for reading, discussion with other users, or addition to the collection. Also, based on the collections searched, browsed, or posted by the user, similar collections can be proposed to the user. Also, annotations on a web page can be linked to similar annotations and content in other web pages or documents through similarity analysis based on sentence embedding vectors. Also, in combination with a sentence similarity classifier, a convolutional neural network (CNN) model can be used to verify the semantic relationships of annotations specified by the user. The semantic relationship data of the annotations provided by the user, in combination with the user reputation, can be used to verify the semantic relationships of the annotations selected by the user and potentially identify inappropriate content generated and posted by the user.Bidirectional Encoder Representations from Transformers (BERT) can be used to identify named entities within web pages and web documents and automatically annotate items that a user may potentially be interested in. These named entities can be entities that exist in Wikipedia or products that are sold on online marketplaces. For moderation of user-generated annotations, pre-trained BERT fine-tuned for sentiment analysis can be used to flag inappropriate content generated by the user.

[0094] In the exemplary embodiment depicted in FIG. 13, a user can trigger a symbol beacon by pressing a symbol beacon trigger button

[57] to make a closed displayed annotation symbol more prominent. The symbol beacon can be in a non-triggered state or a triggered state. When the symbol beacon trigger button

[57] is pressed, the symbol beacon transitions from a non-triggered state to a triggered state. The triggering mechanism of the symbol beacon can also be a sequence of keyboard key presses or mouse button clicks. The triggering mechanism can also be placed in menu options within an annotatable web browser

[24] . It is the web browser extension

[25] instruction set that makes these triggers available, whether they are buttons available in the display area of the annotatable web browser

[24] , menu options of the annotatable web browser

[24] , combinations of keyboard key presses and mouse button clicks, input gestures on a touch screen device, or input gestures via an input device, and triggers one displayed annotation symbol beacon

[58] for each displayed annotation symbol.

[0095] Next, a sequence diagram is shown that indicates how messages are sent between different entities. The messages include user input to an annotatable web browser

[24] or a normal web browser, and electronic messages between an electronic device and a computer system. The annotatable web browser

[24] is shown in the exemplary embodiment of FIG. 3. These sequence diagrams apply to the classes of the exemplary embodiments. After the sequence diagrams are presented, variations of some messages are presented. By more precisely identifying the messages, specific exemplary embodiments can be discussed.

[0096] Figure 14 is a sequence diagram applied to the classes of exemplary embodiments. In Figure 14, the sequence diagram shows the interaction among user

[59] , an annotatable web browser

[24] , a URL server

[60] , an annotation server

[61] , an electronic database

[62] , and a content delivery network

[63] . When user

[59] visits a URL

[64] , the annotatable web browser

[24] requests a web document

[65] , and when the URL server

[60] returns the web document

[66] , the annotatable web browser

[24] displays the document to user

[59]

[67] . Next, user

[59] inputs an input for creating an annotation

[68] , and the annotatable web browser

[24] sends a bundle of data elements representing the annotation to the annotation server

[61]

[69] . The server sends an image for storage

[70] to the content delivery network

[63] and returns the URL of the sent image

[71] . Next, the annotation server

[61] issues an instruction to store in the electronic database

[62] the non-image bundle of data elements and the URL from the CDN as an annotation object

[72] , and the electronic database confirms the storage of the annotation object

[73] . Thereafter, the annotation server

[61] checks the received bundle of data elements, sends an annotation display bundle to the annotatable web browser

[24]

[74] , and displays to user

[59] the displayed annotation bundle indicating that the annotation has been successfully created

[75] . Reference is made in

[69] to the bundle of data elements representing the annotation. The bundle of data elements may vary depending on the details of the implementation of the exemplary embodiments and the type of annotation.

[0097] In FIG. 14, a sequence diagram applied to the classes of the exemplary embodiments is presented. Depending on the characteristics of the bundle of data elements representing the annotation or the highlighted text, the annotation server

[61] records different data, and the annotatable web browser

[24] may display different views due to the fact that different data is received and is selectable when generating the display representation of the annotation or the highlighted text. Next, the screenshot bundle and the annotation symbol bundle that can serve the role of the bundle of data elements in FIG. 14 are discussed in the exemplary embodiments. Further, FIG. 14 may be applied to the exemplary embodiments where the annotation display bundle is the screenshot display bundle or the annotation symbol display bundle. Corresponding to the screenshot bundle is the screenshot display bundle, and corresponding to the annotation symbol bundle is the annotation symbol display bundle. These corresponding pairs are used when FIG. 14 uses the bundle of data elements and the annotation display bundle.

[0098] A screenshot bundle in an exemplary embodiment includes a Uniform Resource Locator (URL) of a document, data of a text snippet selected from the document that includes a selected text snippet representing an annotation, and a marked screenshot of a portion of the document that visually marks and displays the selected text snippet. The screenshot bundle can also include information for finding the selected text snippet within the document. This information for finding the selected portion of text within the document typically includes Document Object Model (DOM) information. An annotation screenshot bundle is a screenshot bundle that also includes an annotation comment

[32] , as shown in the exemplary embodiment depicted in FIG. 7. The screenshot bundle can also include data for representing an annotation symbol. Also, a screenshot bundle that does not include the annotation comment

[32] can be used. In such a case, data representing emphasized text without the annotation comment

[32] is included in the screenshot display bundle along with the associated marked screenshot representing the emphasized text.

[0099] An annotation symbol bundle in an exemplary embodiment is data for a document's Uniform Resource Locator (URL) and a text snippet selected from the document, where the selected text snippet represents an annotation, an annotation comment

[32] , and an annotation symbol. The exemplary embodiment of FIG. 7 may be helpful for reference. A screenshot bundle also includes information for finding the selected text snippet within the document. This information for finding the selected portion of text within the document typically includes Document Object Model (DOM) information. The annotation symbol bundle can also include a marked screenshot of the portion of the document showing the selected portion of text where the selected portion of text is visually marked. Depending on the desired operation, the marked screenshot may or may not be needed.

[0100] In an exemplary embodiment, the DOM information for finding a selected text portion within a document can include information for finding an element within the DOM that contains the first character of the selected text, an element within the DOM that contains the last character of the highlighted text, and information for finding the highlighted text itself. These two elements are referred to as the highlighting start element and the highlighting end element. The information recorded about the highlighting start element and the highlighting end element is tag, ID, class, index, content, XPath, and parent. Further, the ancestor elements of the highlighted text start element are also recorded. The ancestor elements of the highlighting start element have tag, ID, class, index, and content recorded. To find the specified highlighted text at a later time, the DOM is searched by tag and ID, XPath, tag and class and index, tag and class, tag and index, and tag only. Since the web page may be edited or the DOM structure of the web page may change as a result of the execution of JavaScript code that inserts elements into the DOM, a simple XPath search is not sufficient. In this exemplary embodiment, the aim is to find the selected text even when the web page is edited. Multiple search methods are used to make it easier to find the selected text in different scenarios. Once the start element and the end element of the highlighted text are found, it becomes easy to find the selected highlighted text.

[0101] Another concern is that elements are inserted into the DOM of the visited web page by the JavaScript code of the web page. These inserted elements can be annotated. To facilitate this process, the browser's mutation observer can be used. When a change is detected, the DOM structure detection code can be re-executed. Once the start element and the end element of the highlighted text are found, the highlighted text can be inserted. In the case of highlighted text with an annotation, the displayed annotation symbol can also be inserted.

[0102] If the sent screenshot bundle or the screenshot marked with annotation symbol bundle contains a screenshot, the image storage set is created from this marked screenshot by various operations including resizing, trimming, and image alteration. These operations are performed to generate images that look good in different display areas on screens of different sizes. Once the storage set of this marked screenshot is created, it makes sense to determine whether to save these marked screenshots to a content delivery network (CDN). In the case of thumbnail-marked screenshots, there may be an advantage in saving the data together with all other data of the screenshot bundle or annotation symbol bundle. In the case of larger marked screenshots, it is usually better to save these images using a deliverable CDN. As shown in some figures, the annotation server

[61] typically determines which images in the storage set to store in the electronic database

[62] and which images to send to the CDN, as shown in some figures. The CDN typically returns a URL when an image is stored. These URLs are saved together with the rest of the screenshot bundle or annotation symbol bundle.

[0103] The deliverable screenshot is either the actual image stored in the electronic database

[62] or a URL, as shown in some figures. When sending data to a client device, the URL can be used to load the image from the CDN. Also, when sending the image directly, simply display the image. Finally, what is stored in the electronic database is the annotation object

[62] . The annotation object usually contains all the received data in the screenshot bundle or annotation symbol bundle and the deliverable screenshot of each marked screenshot in the storage set.

[0104] In order for an advertiser to reach customers with relevant ads related to the content that the user is currently engaged in, the ad screenshot bundle and the ad annotation symbol bundle can potentially be deployed together with their respective ad screenshot display bundles and ad annotation symbol display bundles. The displayed ad screenshot bundle is the visually displayed form of the data within the ad screenshot display bundle, and the displayed ad annotation symbol bundle is the visually displayed form of the data within the ad annotation symbol display bundle. Although FIG. 7 can be used for illustration, the displayed annotation symbol is replaced with the displayed annotation symbol of the ad. In an exemplary embodiment, the ad screenshot bundle includes an annotation comment

[32] created by the advertiser related to the annotation emphasis text

[27] . The ad annotation symbol bundle includes an annotation symbol indicating the ad. The annotation symbol bundle includes annotation symbols, although the screenshot bundle may also include annotation symbols. While the screenshot bundle includes marked screenshots, the annotation symbol bundle may include marked screenshots. Although FIG. 9 can be used for illustration, the displayed annotation symbol is replaced with the displayed annotation symbol of the ad. Further, the annotation comment is created by the advertiser together with other elements, and preferably, the overall form is similar to other annotation item boxes. When marked screenshots are included, creating a storage set and using a CDN to distribute these marked screenshots may be required to design a viable system. In an exemplary embodiment, these ad screenshot bundles and ad annotation symbol bundles are often stored and managed via an ad server that communicates with an annotation server. Although not explicitly shown in the sequence diagram, the ad server can be inserted.To incorporate one advertisement, the page size can be reduced by one. As shown in some figures, after the annotation server

[61] prepares a page of items representing annotations, the annotation server

[61] can communicate with an advertisement server to obtain one advertisement.

[0105] The semantic web provides a semantic description of some of the web pages and documents available on the WWW through ontologies, which are implementations of a certain description logic. These ontologies need to be created for different domains and have the ability to represent concepts within those domains in logical form, as done in description logic. Collections of existing ontologies are published in projects such as KBPedia. Description logic is a type of logic defined in mathematical logic and has some characteristics of first-order logic. Depending on the characteristics of the logic family, the computational properties are different. Using the existing ontologies, logical statements can usually be created. In an exemplary embodiment, these logical statements can be translated into natural language, and some sentences in natural language can be translated into statements created within description logic using neural machine translation technology. Other deep learning techniques can also be used. Machine learning and deep learning models can translate some sentences in natural language into statements within a specific ontology. Within this exemplary embodiment, this methodology can be used for advertising purposes. Advertisers can express concepts related to a product or service within an ontology related to the product or service and write logical statements. These logical statements can then be input into an advertising platform. The advertising server can identify web pages and documents that contain natural language text that can be translated into statements within the ontology selected by the advertiser to represent concepts related to the product or service being advertised. Furthermore, the advertising server can determine which of these web pages and documents contain natural language text that represents statements similar to or logically related to those expressed by the advertiser. Once these web pages and documents are identified along with the corresponding locations within the documents, the advertiser can be given the opportunity to place an advertisement at a specific location that links to the web page, document, or that specific location within it. Using a logic analyzer, various results of the statements occurring in web pages and documents can be determined.Based on this information, web pages, documents, and their locations can be narrowed down to find the most optimal locations for advertisers. Images, videos, and audio can be converted into text by image captioning, speech recognition, and textification. In some cases, this text can also be translated into statements in the same ontology used by the advertiser. After that, a logic analyzer can be used to determine the most optimal candidates to show to the advertiser, which the advertiser can then consider. Based on the advertiser's selection, the advertising platform can generate the advertisements to be placed on the ad server. Additionally, in this exemplary embodiment, ontologies can be created for the purpose of representing facts about products and services sold in different domains such as insurance, financial services, legal services, healthcare, etc. These ontologies can then be used by various groups advertising and selling services in these domains to create logically correct statements about their products and services for the purpose of searching media such as web pages, web documents, images, videos, and recordings. This media can then reach the audience that consumes this media, and the advertiser can know that there is a high likelihood that this audience is familiar with the statements made by the advertiser in the ontology. Further, in this exemplary embodiment, the media source itself has a logical relationship with the advertiser's statements, and the advertiser can use this logical relationship to provide more tailored products and more tailored targeting to meet the customer's needs.

[0106] To specify the details of an exemplary embodiment, as shown in FIG. 14, a bundle of data elements should be specified, along with details regarding the storage set and deliverable screenshots. The bundle of data elements can be a bundle of screenshots that have their associated data elements. The bundle of data elements can also be a bundle of annotation symbols that have their associated data elements. The storage set, size, resolution, cropping, and other characteristics of different types of marked screenshots should be determined. This is typically based on consideration of the type of display on which these marked screenshots will be presented, along with the size of the view being presented. In some cases, thumbnail-marked screenshots may be included in the storage set. If the size of these thumbnails is small, storing these images using a CDN may not be the most efficient approach. It may be more efficient to store the images in an electronic database

[62] , a local file system, or some other local location rather than in a content delivery network

[63] . Specifying whether the deliverable screenshots are actual images or URLs stored in an electronic database is another decision to consider in specifying the implementation details. It will be apparent to those skilled in the art that various minor variations can be introduced for image storage and retrieval. It will also be apparent to those skilled in the art that these variations are compatible and the differences are not substantial.

[0107] The electronic database

[62] in FIG. 14 and several other figures can take various forms, from relational databases to non-relational databases. Further, the electronic database

[62] may be a distributed electronic database system implemented on one or more computer systems. The most advantageous characteristics of the database can vary depending on the number of database clients and their usage patterns. In an exemplary embodiment, the annotation server

[61] is a database client. The characteristics of the electronic database or distributed electronic database system may vary to handle the needs for consistency, availability, and durability, as well as a number of annotation servers with different latency requirements. Further, the distributed electronic database system can be implemented in various ways.

[0108] Further, it should be noted that storing images in the electronic database

[62] can sometimes affect performance. In such cases, it can be beneficial to store references in the database and use a different storage mechanism for specific images. In the case of a content delivery network

[63] , the reference is a URL. However, it is possible to use alternative mechanisms to store non-URL references and then use the references stored in the electronic database to store images in a searchable manner. In an exemplary embodiment, the images can be stored on the local file system, and the references in the database can correspond to the file paths. In such cases, the storage location with the electronic database should be considered a distributed electronic database that stores images in a more performant way. It will be apparent to those skilled in the art that various minor variations of this technique can be introduced for image storage and retrieval. It will also be apparent to those skilled in the art that these variations are interchangeable and the differences are not substantial.

[0109] Figure 15 is a sequence diagram applied to the classes of an exemplary embodiment. To specify the details of the exemplary embodiment for Figure 15, an annotation display bundle and a displayed annotation bundle need to be specified. Further, Figure 15 can be applied to an exemplary embodiment in which the annotation display bundle is replaced by a screenshot display bundle and the displayed annotation bundle is replaced by a displayed screenshot bundle. In this case, a screenshot display bundle having data elements to be included needs to be specified, and a mechanism for obtaining the deliverable screenshot also needs to be specified. The user interface diagram, the data elements described for the screenshot bundle, and the considerations discussed previously can be used to specify the annotation display bundle and the displayed annotation bundle. The displayed screenshot bundle is generally used when there are screenshots marked with annotation emphasis without annotation comments. The displayed annotation screenshot bundle is used when there are marked screenshots, annotation emphasis, and annotation comments. Figure 15 is a sequence diagram showing the interaction between user

[59] , annotatable web browser

[24] , annotation server

[61] , electronic database

[62] , and content delivery network

[63] . When user

[59] visits the URL of the annotation server

[76] , the annotatable web browser

[24] requests the organize page of the annotation server from the annotation server

[61] , requests the annotation objects necessary for page configuration from the electronic database

[62] , and the annotation objects are returned to the annotation server

[61]

[79] . Next, the annotation server

[61] returns the organize page of the annotation server to the annotatable web browser

[24] together with the annotation display bundle, and requests an image of the annotation display bundle URL when the organize page of the annotation server is returned

[81] .Finally, the content delivery network

[63] returns the requested image to the annotatable web browser

[24] and displays an organized page with the annotation bundle presented to the user

[59] [82, 83]. Next, the user

[59] provides an input to move the annotation to the annotatable web browser

[24]

[84] , and the annotation server

[61] requests the annotation to be moved by issuing a movement instruction for the annotation subject to the electronic database

[62] [85, 86]. In response, the electronic database

[62] authorizes the movement of the annotation object to the annotation server

[61]

[87] , the annotation server acknowledges that the annotation has moved

[88] , and the annotatable web browser

[24] presents the user

[59] with the content with the annotation bundle displayed at the new location.

[0110] The presented annotation screenshot bundle and annotation item box are created using the data of the annotation screenshot display bundle transmitted from the annotation server

[61] , as shown in part of the figure. The annotation screenshot display bundle is created using the data of the annotation object obtained from the electronic database, as shown in some of the figures

[62] . As shown in the exemplary embodiment of FIG. 10, when displaying the annotation page in the screenshot bundle display area such as the item bin

[44] of the annotation compilation context area

[43] of the organize web page, the data sent from the annotation server

[61] is the page where the thumbnail display bundle is displayed. These thumbnail display bundles include at least the deliverable screenshots for the thumbnail-marked screenshots within the storage group of the annotation objects stored in the electronic database

[62] . FIG. 9 can be used for illustration. In the exemplary embodiment, the thumbnail display bundle also includes data representing separate context-annotation emphasized text

[38] , separate context-annotation comments

[40] , and the displayed annotation symbols

[39] of separate context questions in a readable format that can be displayed by a web browser. Further, the separate context-annotation emphasized text

[38] and the displayed annotation symbols

[39] of separate context questions are color-coded. By all these data working together, the user can obtain more information at a glance. Also, the screenshot of the displayed annotation can be displayed on the annotation detail page, providing more detailed information about one annotation and many user comments and replies regarding that annotation.

[0111] FIG. 16 is a sequence diagram applied to the classes of the exemplary embodiments. In order for FIG. 16 to explicitly specify the exemplary embodiments, an annotation symbol display bundle having data elements to be included along with the visual appearance and a corresponding displayed annotation symbol bundle need to be specified. For this specification, the user interface diagram, the data elements described for the annotation symbol bundle, and the considerations described above can be used. In FIG. 16, it is a sequence diagram showing the interaction between user

[59] , annotatable web browser

[24] , URL server

[60] , annotation server

[61] , and electronic database

[62] . When user

[59] visits a URL

[90] , the annotatable web browser

[24] requests a web document from the URL server

[60]

[91] , and the server returns the web document to the annotatable web browser

[24]

[92] . Next, the annotatable web browser

[24] requests an annotation symbol display bundle representing an annotation from the annotation server

[61]

[93] , and the annotation server requests an annotation object from the electronic database

[62]

[94] . Thereafter, the electronic database

[62] returns the annotation object to the annotation server

[61]

[95] , and the annotation server returns an annotation symbol display bundle representing an annotation to the annotatable web browser

[24]

[96] , and displays the web document with the closed displayed annotation symbol and an annotation that can be opened when moused over or clicked

[97] .

[0112] FIG. 17 shows a flowchart applied to a class of exemplary embodiments. A particular exemplary embodiment can be specified if the bundle of data elements received and stored is accurately specified. For example, a screenshot bundle can be explicitly specified along with all of its data elements, or an annotation symbol bundle can be explicitly specified along with all of its elements. In this case, the screenshot bundle or the annotation symbol bundle becomes the bundle of data elements received. Further, if the bundle of data elements includes marked screenshots, it is necessary to specify the storage set and storage mechanism of the screenshots to be distributed. These details are useful for explaining the implementation. In an exemplary embodiment, the storage set can include marked screenshots for display on a large monitor, a notebook computer, a tablet, and a telephone, as well as thumbnail-marked screenshots for an item box. The exact size can be adjusted to the most common size currently in use for these displays. Further, in a client-server architecture, the server receives a bundle of data elements representing an annotation

[98] and stores the received data elements and newly created elements using one or more computer systems to satisfy the requests made by the annotation-capable web browser of the user's client device that creates the annotation or simple emphasis

[99] . In FIG. 17, a flowchart of a method implemented by one or more computer systems representing the server in a client-server architecture is shown. This method is implemented by the server to enable sharing of data that can be requested by the client user interface of the same client or different clients. The steps of this method are to receive a bundle of data elements representing an annotation

[98] and store the received data elements and newly created elements using one or more computer systems

[99] .

[0113] FIG. 18 shows a flowchart applied to a class of exemplary embodiments. When the data elements included in the bundle display set are specified exactly, a particular exemplary embodiment can be specified. This means specifying the data elements included in the display bundle. In an exemplary embodiment, the display bundle can be a screenshot display bundle or an annotation symbol display bundle. If the display bundle is a screenshot display bundle, the bundle display set will be the screenshot display bundle display set of FIG. 18. If the display bundle is an annotation symbol display bundle, the bundle display set will be the annotation symbol bundle display set of FIG. 18. Further, the server, in a client-server architecture, compiles a bundle display set including pages of the display bundle using one or more computer systems, and in order to satisfy a request made by an annotation-capable web browser running on a client device of a user who views pages of annotations within a web document or another context area, transmits the bundle display set in a format in which the display bundle can be displayed by the web browser running on the electronic device. The flowchart shown in FIG. 18 is implemented by one or more computer systems.

[0114] FIG. 19 is a flowchart applied to a class of exemplary embodiments. Certain exemplary embodiments can be specified such that a bundle of data elements is explicitly specified. For example, a bundle of screenshots having the data elements to be included can be explicitly specified, or a bundle of annotation symbols having the data elements to be included can be explicitly specified. In FIG. 19, in an exemplary embodiment, a flowchart of annotation creation showing steps implemented by an instruction set of a web browser extension that converts a web browser supporting the addition of a browser extension into an annotatable web browser is shown

[24] . As shown in some figures, annotation creation is implemented by a browser extension. FIG. 19 shows steps executed by a user

[59] , a web browser

[0102] , a web browser extension content script

[0103] , and a web browser extension background script

[0104] when creating an annotation. The user

[59] opens the browser

[0105] , and the web browser

[0102] loads the browser extension

[0106] . Next, the user

[59] visits a URL

[0107] , and the web browser

[0102] reads the data sent from the accessed URL

[0108] and injects the content script

[0109] . Here, the user

[59] browses the web page

[0110] and highlights the text

[0111] . At this time, the web browser extension content script

[0103] makes the annotation symbol selection button bar available as a result of executing the instructions of the web browser extension

[0112] . Next, the user

[59] presses a button corresponding to one of the annotation symbols on the annotation symbol selection button bar

[0113] , and the web browser extension content script

[0103] makes the annotation input box usable accordingly

[0114] . Next, when the user

[59] enters text and presses the enter key

[0115] , the web browser extension content script

[0103] sends the bundle of data elements to the background script

[0116] At this time, the background script

[0104] of the web browser extension receives a bundle of data elements from the content script

[0117] , and sends the bundle of data elements to the annotation server

[0118] .

[0115] In FIG. 19, after the user

[59] enters text and presses Enter

[0115] , it is desirable for the web browser extension content script

[0103] to generate either an annotation symbol bundle or a screenshot bundle. An exemplary embodiment may require that the annotation symbol bundle includes a marked screenshot. If the marked screenshot is generated on the server side, the web page may have to be re-rendered or the screenshot may have to be marked. Doing so increases the computational load and makes errors more likely to occur. Generating a marked screenshot using a browser extension does not have these problems. With a browser extension, the user's privacy must be securely protected. The screenshot should not include the desktop or other browser tabs. Further, in an exemplary embodiment, it is desirable for the screenshot to be attached with annotation emphasis text

[27] and an annotation comment box

[31] . The annotation emphasis text

[27] and the annotation comment box

[31] are shown in FIG. 7 and may be useful for explanation in this case. In an exemplary embodiment, a simple delay can be implemented before the screenshot is taken. This delay should be sufficient to ensure that the annotation emphasis text

[27] and the annotation comment box

[31] are present on most devices. In an alternative exemplary embodiment, a more performant method can be implemented, and the web browser extension content script

[0103] searches for the presence of the annotation emphasis text

[27] and the annotation comment box

[31] in the DOM structure of the document and takes a screenshot if these elements are present. The search for the presence of the annotation emphasis text

[27] and the annotation comment box

[31] in the document DOM structure will occur at periodic intervals on a time scale shorter than the simple delay time described above.In an exemplary embodiment, the periodic interval can be one third of the length of a simple delay that would be selected.

[0116] FIG. 20 is a flowchart applied to a class of exemplary embodiments. A particular exemplary embodiment can be specified if all of its data elements are explicitly specified in the displayed annotation symbol bundle. In FIG. 20, in an exemplary embodiment, a flowchart of the display of an annotation showing steps performed by an instruction set of a web browser extension that converts a web browser that supports the addition of a browser extension into an annotatable web browser is presented. FIG. 20 shows steps performed by user

[59] , web browser

[0102] , web browser extension content script

[0103] , and web browser extension background script

[0104] when viewing an annotation. User

[59] opens a browser

[0119] , and web browser

[0102] loads a browser extension

[0120] . Next, user

[59] accesses a URL

[0121] , and web browser

[0102] reads data sent from the accessed URL

[0122] and injects a content script

[0123] . Here, web browser extension background script

[0104] reads an annotation symbol display bundle representing an annotation for the URL

[0124] and sends the annotation symbol display bundle to the content script

[0125] . Now, web browser extension content script

[0103] receives the annotation symbol display bundle from the background script

[0126] and displays annotation symbols in a closed state on the web page

[0127] . Now, user

[59] views a web page with annotation symbols displayed in a closed state

[0128] and clicks on the annotation symbols displayed in a closed state

[0129] In response to this, the web browser extension content script

[0103] displays the annotation symbol in an open state.

[0130] Next, the user

[59] views the web page on which the selected annotation symbol is displayed in an open format.

[0131] .

[0117] In the exemplary embodiment depicted in FIG. 31, a collection item web page is shown. The collection item page context area

[0258] is the display area of the displayed item, the collection item paging mechanism

[0259] , the collection item search filter

[0260] , and the collection action menu trigger

[0261] . The collection action menu

[0262] displays the message sending option to the collection

[0263] , the user invitation option

[0264] , and the user permission change option

[0265] . The collection action menu trigger

[0261] is used to trigger the collection action menu

[0262] . The collection item paging mechanism

[0259] may be used to change the page of the displayed collection displayed in the collection item page context area

[0258] . The option to send a message to the collection

[0263] creates a message chain with the first text message and sends it to the members of the collection who have more than readable permissions for the collection. The message chain is displayed in the user's receiving tray. An exemplary embodiment of the user's receiving tray is depicted in FIG. 33. In the exemplary embodiment of FIG. 33, the American history collection message chain

[0281] is shown. The message sent to the members of the collection using the option to send a message to the collection

[0263] is displayed in the receiving tray of the recipient classified under the collection to which the message was sent. Among the members belonging to the collection, all members with more than readable permissions can read the message, and all members with more than messageable permissions can reply. The user invitation option

[0264] is used to invite new users to the collection. The invitation is sent to the receiving tray of the invited user. The user permission change option

[0265] can be used to change the permissions of the members of the collection.In the form of a message within a message chain, a notification of a change may be sent to the inbox of the user whose permission has been changed.

[0118] A collection item filter, such as the collection item search filter shown in the exemplary embodiment of FIG. 31

[0260] , is used to receive input from a user. In the case of the collection item page context area

[0258] and the collection item search filter

[0260] , it is a collection item search phrase. These inputs are converted into collection item parameter values that are sent to the annotation server

[61] , as shown in some of the figures. These collection item parameter values are used by the annotation server

[61] to input collection item parameters into a collection item request query used by the electronic database

[62] , as shown in some of the figures. The collection item request query includes collection item parameter values and a collection item page number. The collection item request query containing the collection item parameter values is used to specify a set of collection items and their sequential order. The collection item page number specifies a particular collection item page. Further, the collection item request query can include a specified number of collection items to be sent to a web browser running on a client electronic device. This collection item request query includes collection item parameters such as collection item filter values like a collection item search phrase, an exclusion list of collection items to be excluded from a specified list of collection items, and a sequential ordering for that list of collection items.

[0119] In the exemplary embodiment depicted in FIG. 32, a collection web page is shown. The collection page context area

[0266] is a display area for the displayed collection, the collection page paging mechanism

[0267] , the collection page search filter

[0268] , and the collection page action menu trigger

[0269] . The collection page action menu

[0270] displays ownership filter options

[0271] , administratable filter options

[0272] , writable filter options

[0273] , messageable filter options

[0274] , viewable filter options

[0275] , and subscribe filter options

[0276] . The collection page action menu trigger

[0269] is used to trigger the action menu

[0270] of the collection page. The paging mechanism

[0267] of the collection page can be used to change the web page of the displayed collection shown in the collection page context area

[0266] . The ownership filter options

[0271] , administratable filter options

[0272] , writable filter options

[0273] , messageable filter options

[0274] , viewable filter options

[0275] , and subscribe filter options

[0276] filter the collection based on user permissions. The ownership permission is the highest level of permission and is given to the creator and owner of the collection. The ownership permission allows other users to have admin permissions for that collection and do anything permitted by the lower-level permissions. Admin permissions are lower-level permissions than ownership permissions and can elevate non-admin permissions, handle other administrative tasks, and execute anything permitted by the lower-level permissions.The writable permission is lower than the admin permission, and the user can add items to the collection and do anything permitted by the lower-level permissions. The messageable permission is lower than the writable permission, and the user can send messages to other members of the collection and do anything permitted by the viewable permission. The readable permission is lower than the messageable permission, and the user can view all items in the collection and read all collection messages for that collection. The subscribe permission gives the user readable permission for public collections. The subscribe permission is usually granted to the user who requested the permission without admin intervention because the messages associated with the collection are public. To enable the user to visualize the privacy settings, the background color is assigned based on color. Colors associated with shorter wavelengths mean greater permissions, and colors associated with longer wavelengths mean lesser permissions. That is, the colors of the rainbow are associated with permissions. Purple corresponds to the collections the user owns. Blue corresponds to the admirable collections. Green corresponds to the writable collections. Yellow corresponds to the messageable collections. Orange corresponds to the viewable collections. And red corresponds to the subscribe collections. For design purposes, variations of the pastel colors of these colors may be used. Or, another set of backgrounds that mean a spectrum of values can be used. These backgrounds are applied to the collections shown in the exemplary embodiment of FIG. 32. For the exemplary embodiment of FIG. 31, the background of the web page is changed to correspond to the color associated with the permission of that collection.

[0120] In an exemplary embodiment, to facilitate browsing through a number of items that contain annotations in a single collection, the annotations can be grouped into a stack set of displayed annotation items. Such a stack set of displayed annotation items appears as a single item with the visual appearance of stacked items. It is beneficial to place annotations from the same URL, up to nine annotations or another configurable value, the displayed annotation items, into a stack set of displayed annotation items. To facilitate such a display, the paging mechanism needs to support the return of items in this format. As an example, imagine that 8 out of 10 items are annotation items belonging to the same URL. When these 8 items are grouped into a stack set of displayed annotation items, there will be 3 items present on the collection item page. To ensure that there are 10 items present even when a stack set of displayed annotation items is used, more items need to be returned by the annotation server

[61] . In an exemplary embodiment, a relational database is used and a complex structured query language (SQL) query using window functions may be required. Generally, an object-relational mapping (ORM) system is used by most modern web frameworks to access a relational database . In an exemplary embodiment, the ORM system did not support window functions. To overcome this limitation, the underlying relational algebra library used by the ORM system was used. Using this relational algebra, complex SQL queries were generated with appropriate window functions. This enabled the display of a page of collection items with a stack of the set of displayed annotation items.

[0121] The collection page search filters, such as the exemplary embodiment shown in FIG. 32 [

[0268] ], are used to receive input from the user. Collection page context area [

[0266] ] In the case of the collection page search filter [

[0268] ], it is a collection search phrase. These inputs are converted into collection parameter values that are sent to the annotation server

[61] , as shown in some of the figures. These collection parameter values are used by the annotation server

[61] to input collection parameters into the collection request query used by the electronic database

[62] , as shown in some of the figures. The collection request query includes the collection parameter value and the collection page number. The collection request query containing the collection parameter value is used to specify a set of collections and an ordering for those collections. The collection page number specifies a particular collection page. Further, the collection request query can include the number of collections to be sent to the web browser running on the client electronic device. This collection request query includes a collection parameter of collection filter values, such as a collection search phrase, an exclusion list of collections to be excluded from the list of specified collections, and a sequential ordering for the list of those collections.

[0122] In the exemplary embodiment depicted in FIG. 33, a received tray page is shown. The collection message chain section and title

[0277] includes an American history collection message chain section and title

[0279] and a car collection message chain section and title

[0280] . The collection message chain is a message chain related to the collection. These collection message chains appear in the "Collection Message" section under the name of the collection to which the collection message chain belongs. In this exemplary embodiment, the section and title

[0279] of the American history collection message chain includes the American history collection message chain

[0281] , and the section and title

[0280] of the car collection message chain lists the car collection message chain 1

[0282] and the car collection message chain 2

[0283] . The tagless message chain is a message chain not associated with any collection. The tagless message chain is displayed in the "Tagless Message" section. In the exemplary embodiment, the tagless message chain section and title

[0278] includes the tagless message chain 1

[0284] , the tagless message chain 2

[0285] , and the tagless message chain 3

[0286] . When the user clicks on a message chain, the message chain is displayed in the message chain context area

[0287] . In the message chain context area

[0287] , the first system message

[0288] of the American history message chain is displayed along with the system message acceptance button

[0289] and the system message rejection button

[0290] . The user can press the system message acceptance button

[0289] to accept an invitation to the collection or press the system message rejection button

[0290] to reject an invitation to the collection.

[0123] In the exemplary embodiment depicted in FIG. 34, after the user accepts an invitation to participate in a collection, a received tray page is displayed. A system message notification

[0291] indicates that the user has accepted an invitation to participate in a collection. A message chain message splitter

[0292] separates a certain user's message from other users' messages. Also, it should be apparent to those of ordinary skill in the art that the message chain message splitter

[0292] can be implemented using various visual design techniques to separate the message of the user in this context from other users' messages. Next, the first text message

[0293] of the American history message chain, the second text message

[0294] of the American history message chain, and the third text message

[0295] of the American history message chain are shown. And a message creation section and title

[0296] are displayed together with a message creation box

[0297] and a message send button

[0298] . This received tray has a connection to an authority management system and the collection. When permissions are granted to a message chain, all other messages within the message chain will be displayed even for users who originally did not have access rights when the message chain was created. This facilitates discussion, and access is controlled at the message chain level. Further, the system's message chain usually ensures that the collection administrator obtains information about the activities and permissions of group members. Furthermore, a user with messageable permissions can send a message chain to a user within the collection or a selected group within the collection in one message. By being able to create a collection chain, the subject of discussion can be clarified, and users can focus their attention on the collections and tasks they want to perform.

[0124] The inbox is an effective means of inviting the user to a collection when the user is already on the site. Also, the inbox is effective for sharing items with others when the user is already on the site. To facilitate sharing of items and collections when the user is not on the site, links with cryptographic hashes are used. This link is impossible to guess, and the user with the link does not need to create an account to access the link. One can share just by possessing the link. Also, public items such as annotations and collections can be shared through social media.

[0125] In FIG. 21, as shown in some of the figures, a flowchart of a nurse using an exemplary embodiment of an annotatable web browser

[24] is shown. This nurse wants to teach the patient that rice is not good for diabetic patients and then show that the favorite dishes can be modified to be more friendly for diabetic patients. Once this memo is created, it can be shared with the patient. First day

[0132] The nurse visits the Wikipedia page for the Glycemic Index, creates a check annotation with an annotation comment highlighting "supported by the American Diabetes Association" and stating that "most physicians strongly believe that high glycemic index foods have an adverse effect on the health of diabetics"

[0134] , and adds the annotation to a memo

[0135] . Next, the nurse visits allrecipes.com, finds a recipe for "Flavorful Spanish Rice and Beans", highlights "1 cup of raw white rice", creates an exclamation annotation with an annotation comment stating that "the glycemic index of rice is very high, but the same recipe can be made with pearl barley"

[0136] , and adds the annotation to a memo

[0137] . Then, the nurse visits the Harvard Health website, finds the page for the glycemic index of over 100 foods, highlights "cooked white rice*", attaches an exclamation annotation, and states that "most people are not aware that white rice has such a high glycemic index. Note that it is 73, higher than most foods."

[0138] , and adds the annotation to a memo

[0139] . Next, the nurse visits the Harvard Health website again, finds the page for the glycemic index of over 100 foods, highlights "pearl barley" with a dagger annotation, and states that "most people are not aware that pearl barley has such a low glycemic index. Note that it is 28, lower than most foods."

[0140] , and adds the annotation to a memo

[0141] . After that, the nurse visits the Target website, finds the product page for pearl barley, selects "Pearl Barley - 1lb - Market Pantry" with a dagger annotation, and writes that "pearl barley can be cooked in a rice cooker just like rice. Additionally, cooked pearl barley has a grain-like appearance and texture and is ideal as a substitute for rice. It can be used in recipes that call for rice instead of rice."

[0142] , and adds the annotation to a memo

[0143] . After creating the memo, on the second day

[0133] The nurse shares a memo book with a patient who likes to make Spanish rice with diabetes.

[0144]

[0126] In FIG. 22, a flowchart of a patient using an exemplary embodiment of an annotatable web browser

[24] with a memo function is shown to view a memo created by a nurse. The patient

[0145] on the second day opens the memo link, clicks on the first displayed annotation screenshot bundle embedded in the memo

[0147] , reads the Wikipedia title of the glycemic index, reads the emphasized text "supported by the American Diabetes Association" after observing the checkmark-displayed annotation symbol, and reads aloud the annotation comment "Most doctors strongly believe that high glycemic index foods have an adverse effect on the health of diabetic patients"

[0148] . Next, the patient clicks on the second displayed annotation screenshot bundle embedded in the memo

[0149] , reads the title "Flavorful Spanish Rice and Beans" on allrecipes.com, sees the exclamation-displayed annotation symbol, reads the emphasized text "1 cup of raw white rice", and then reads the annotation comment "The glycemic index of rice is very high, but the same recipe can be made with pearl barley"

[0150] . Thereafter, the patient clicks on the third displayed annotation screenshot bundle embedded in the memo

[0151] , reads the title "Glycemic Index of Over 100 Foods" on the Harvard Health website, and reads the emphasized text "Boiled white rice*" after seeing the exclamation annotation symbol, and reads the annotation comment "Most people are not aware that the glycemic index of white rice is this high. Note that it is 73, higher than most foods."

[0152] . Next, the patient clicks on the fourth displayed annotation screenshot bundle embedded in the memo

[0153] , reads the title "Glycemic Index of Over 100 Foods" on the Harvard Health website, and reads the emphasized text, reads "pearl barley" after seeing the dagger-displayed annotation symbol, "Many people are not aware that the glycemic index of pearl barley is this low.Note the annotation comment that "it is lower than most foods at 28."

[0154] . The patient then clicks on the fifth-annotated screenshot bundle embedded in the memo book

[0155] , reads the title of the target website, notices that it is the product page for pearl barley, reads the highlighted text "Pearl Barley - 1 lb - Market Pantry" after seeing the annotated symbol with the dagger, and reads the annotation comment stating that "Pearl barley can be cooked in a rice cooker just like rice. Additionally, cooked pearl barley has the appearance and texture of a grain and can be used in place of rice. This can be used in recipes that call for rice, in place of rice."

[0156] . After understanding the contents of the memo book, on the day of grocery shopping

[0146] , the patient goes to Target and buys pearl barley to make "Flavorful Spanish Barley and Beans"

[0157] .

[0127] In the exemplary embodiment depicted in FIG. 23, an annotatable web browser with a notepad function

[24] is presented. The annotatable web browser with a notepad function tracks whether an active notepad exists and enables the user to select an active notepad from a group of notepads accessible to the user. FIG. 23 shows an active notepad

[0159] and an annotation

[0160] embedded in the notepad. When an active notepad

[0159] exists, the annotatable web browser with a notepad function is in a state where an active notepad exists, and when an active notepad

[0159] does not exist, the annotatable web browser with a notepad function is in a state where an active notepad does not exist. The displayed notepad contains many notepad-embedded annotations

[0160] . The notepad-embedded annotation

[0160] is a container that includes a displayed annotation bundle indicating the data included in the annotation display bundle. In the exemplary embodiment, the notepad-embedded annotation

[0160] includes a displayed annotation screenshot bundle that also includes an annotation symbol. In an alternative embodiment, the notepad-embedded annotation

[0160] can include a displayed screenshot bundle that does not include an annotation symbol or an annotation comment. The user can add an annotation to the notepad by clicking on an additional icon button

[0158] in the notepad depicted in FIG. 23. The notepad includes a list of references to annotation objects. As shown in some of the figures, an electronic database

[62] stores notepad objects along with a list of references to annotation objects. As shown in some of the figures, when the annotation server

[61] receives a request from the annotatable web browser

[24] , it retrieves the notepad object and the annotation object from the electronic database

[62] and sends the notepad to the annotatable web browser. It is displayed on the web browser

[24] . When the active notepad

[0159] is displayed, the user can navigate to the annotation by simply clicking on the notepad-embedded annotation

[0160] as depicted in the notepad-embedded annotation

[0160] of FIG. 23. In the classes of the exemplary embodiments, each notepad is associated with a bundle display set. In an exemplary embodiment, each notepad may be associated with a screenshot bundle display set. In another exemplary embodiment, each notepad may be associated with an annotation screenshot bundle display set.

[0128] The notepad contains rich information through links to other information sources along with annotation comments. This information may be reinforced by the description written by the creator of the notepad. Using these notepads, presentations can be created in various formats including PDF, Microsoft Word, and various other open-source and proprietary document formats. These presentations are created by sequentially including, for each annotation, the relevant screenshot

[37] containing the marked text, the separate context-annotation highlighted text

[38] , the displayed annotation symbol

[39] for the separate context question, and the separate context-annotation comment

[40] as shown in FIG. 9. Further, this information can be reinforced by the text summary of the linked article generated using natural language processing technology together with the image caption technology. Further, the user can create explanatory texts to add to the specified locations, and these explanatory texts will be included in the output of the presentation.

[0129] In the exemplary embodiment depicted in FIG. 24, the interactions among a user

[59] for creating a memo, an annotatable web browser with a memo function

[0161] , a URL server

[60] , an annotation server

[61] , an electronic database

[62] , and a content delivery network

[63] are shown by a sequence diagram. First, the user

[59] visits the URL of the annotatable web browser with a memo function

[0161]

[0162] , requests a web document from the URL server

[60]

[0163] , the URL server returns the web document to the annotatable web browser with a memo function

[0161]

[0164] , and displays the web document to the user

[59]

[0165] . Next, the user

[59] makes an input for creating a memo of "Flavorful Spanish Barley and Beans" in the annotatable web browser with a memo function

[0161]

[0166] , sends a request to create a memo of "Flavorful Spanish Barley and Beans" to the memo server

[61]

[0167] , and sends an instruction to create a memo object to the electronic database

[62]

[0168] . Next, the electronic database

[62] approves the creation of the memo object to the annotation server

[61]

[0169] , the annotation server approves the creation of the memo to the annotatable web browser with a memo function

[0161]

[0170] , and displays the created memo to the user

[59]

[0171] . Next, the user

[59] makes an input for creating a blood glucose annotation

[0172] , sends an annotation screen bundle to the annotation server

[61]

[0173] , and the server sends an image for storage to the content delivery network

[63] as needed

[0174] . The content delivery network

[63] returns the URL of the stored image to the annotation server

[61]

[0175] , and sends an instruction to create a glycemic index annotation object with the CDN's URL to the electronic database

[62]

[0176] .Next, the electronic database

[62] permits the annotation server

[61] to create annotation objects. The annotation server sends the received annotation screenshot bundle to an annotatable web browser with a notepad function

[0161] and the displayed annotation screenshot bundle is presented to the user

[59] . Next, the user

[59] adds a blood glucose value annotation to the notepad of the annotatable web browser with a notepad function

[0161] . The annotation server

[61] requests to add a reference to the blood glucose value annotation object to the notepad, and an instruction to add a reference to the glycemic index annotation object to the notepad is sent to the electronic database

[62] . Then, the electronic database

[62] permits the annotation server

[61] to add a reference to the annotation object. The annotation server permits the annotatable web browser with a notepad function

[0161] to add a reference to the blood glucose value annotation object to the notepad, and a web document with the notepad to which the displayed annotation screenshot bundle is added is shown to the user

[59] .

[0130] In the exemplary embodiment depicted in FIG. 25, the sequence diagram shows the interactions among a user

[59] , an annotatable web browser with a memo function

[0161] , a URL server

[60] , an annotation server

[61] , an electronic database

[62] , and a content delivery network

[63] for the purpose of memo viewing. First, the user

[59] accesses an arbitrary URL with the annotatable web browser with a memo function

[0161]

[0186] , requests a memo of "tasty Spanish barley and beans" from the annotation server

[61]

[0187] , and requests a memo object from the electronic database

[62]

[0188] . Next, the electronic database

[62] returns the memo object to the annotation server

[61]

[0189] , and the annotation server returns the memo to the annotatable web browser having the memo function

[0161]

[0190] . Next, the annotatable web browser with a memo function

[0161] requests an annotation screen display bundle with blood glucose value annotation symbols from the annotation server

[61]

[0191] , requests a blood glucose value annotation object from the electronic database

[62]

[0192] , returns the annotation object with the CDN URL to the annotation server

[61]

[0193] , and returns the annotation screen display bundle with the CDN URL and annotation symbol data to the annotatable web browser with a memo function

[0161]

[0194] . Next, the annotatable web browser with a memo function

[0161] requests an image from the CDN using the CDN URL from the content delivery network

[63]

[0195] , returns the image from the CDN of the specified URL to the annotatable web browser with a memo function

[0161]

[0196] , and displays the displayed memo to the user

[59]

[0197] .Next, the user

[59] clicks on the initially displayed annotation screenshot bundle in the annotatable web browser with a notepad function

[0161] , requests the web document from the server for the URL

[60]

[0199] , the web document is returned to the annotatable web browser with a notepad function

[0161]

[0200] , and the annotation symbol displayed in the open state is displayed on the web document

[0201] . Then, the annotatable web browser with a notepad function

[0161] displays to the user

[59] the web document with the notepad placed horizontally with the displayed annotation symbol in the open state and the browser scrolled to an appropriate position

[0202] .

[0131] In the exemplary embodiment depicted in FIG. 26, a flowchart of memo creation showing the steps executed by an instruction set of a web browser extension that converts a web browser supporting the addition of a browser extension into an annotatable web browser with a memo function is presented. User

[59] opens the browser

[0203] , and then, the web browser

[0102] loads the browser extension

[0204] . Next, user

[59] visits the URL

[0205] . Thereafter, the web browser

[0102] reads the data sent from the visited URL

[0206] and injects a content script

[0207] . Then, user

[59] browses the web page

[0208] and creates a memo

[0209] . As a result, the web browser extension content script

[0103] creates a memo

[0210] , and the web browser extension background script

[0104] sends a memo creation request to the annotation server

[0211] . Next, the web browser extension content script

[0103] displays the displayed memo

[0212] , and user

[59] browses the displayed memo and adds the already created annotations to the memo

[0213] . Thereafter, the web browser extension content script

[0103] adds the already created annotations to the memo

[0214] , and the web browser extension background script

[0104] sends a request to add an annotation object reference to the memo to the annotation server

[0215] .

[0132] In the exemplary embodiment depicted in FIG. 27, a flowchart of a notepad display showing steps executed by an instruction set of a web browser extension that converts a web browser supporting the addition of a browser extension into an annotatable web browser with a notepad function is presented. User

[59] opens the browser

[0216] , and the web browser

[0102] loads the browser extension

[0217] . Thereafter, user

[59] visits an arbitrary URL

[0218] . Next, the web browser

[0102] reads the data sent from the visited URL

[0219] and injects a content script

[0220] . Thereafter, user

[59] opens the notepad

[0221] , and the web browser extension content script

[0103] requests the notepad

[0222] . Next, the web browser extension background script

[0104] requests from the annotation server a notepad of an annotation screen shot display bundle and annotation symbol data for all corresponding annotation objects in the reference list to the annotation object

[0223] , and the web browser extension content script

[0103] displays the displayed notepad

[0224] . Thereafter, user

[59] browses the displayed notepad, clicks on the first displayed annotation screen shot bundle

[0225] , and the web browser extension content script

[0103] sends a message to the browser to visit the URL

[0226] . Then, the web browser

[0102] accesses the URL

[0227] and reads the data sent from the URL

[0228] . Next, the web browser extension background script

[0104] requests additional annotation object information such as a reply to the annotation comment from the annotation server

[0229] and obtains the additional annotation object information from the annotation server

[0230] . Next, the web browser extension content script

[0103] displays the displayed annotation symbol in an open state

[0231] and scrolls to the position of the displayed annotation symbol

[0232] .Thereafter, user

[59] views the page scrolled to the position of the annotation symbol displayed in the open state

[0233] .

[0133] In the exemplary embodiment depicted in FIG. 28, a system diagram is presented showing how a first user

[0234] , a second user

[0235] , a third user

[0236] , a browser of the first user

[0237] , a browser of the second user

[0238] , a browser of the third user

[0239] , an annotation server

[61] , a content delivery network

[63] , and an electronic database

[62] interact. The user communicates with the annotation server

[61] , and the annotation server pushes an image to the content delivery network

[63] . Also, the user communicates with the content delivery network

[63] to obtain an image and other files. The user does not communicate directly with the electronic database

[62] . In this exemplary embodiment, the annotation server

[61] is placed in one computer system, and the electronic database

[62] is placed in a second computer system. By placing the electronic database

[62] on the same machine as the annotation server

[61] , the number of machines used can be reduced. The issue here is the number of concurrent users of the system. If this issue is resolved, it may be possible to determine the size of the system. The content delivery network

[63] in this exemplary embodiment is operated by a third party. The content delivery network (CDN) may be provided by a third party. Many third parties provide CDNs. These CDNs can be configured using a web interface, and files can be uploaded and accessed via an application programming interface (API). This API may be a REST (representational state transfer) API. It should be apparent to those skilled in the art that setting up and using such a third-party CDN based on the documentation of that third party.

[0134] In the exemplary embodiment depicted in FIG. 29, a component diagram is presented showing a way to build a more robust system that enables separating different components. Separating concerns can be advantageous for improving security and facilitating reproducible software deployment to a network of machines. The added security, separation of concerns, and reproducible deployment can ease the scaling of a system for handling a large number of users. FIG. 29 is a diagram showing a simplified version of an annotation server

[61] and an electronic database

[62] that communicate next to an alternative exemplary embodiment showing an annotation server

[61] including a load balancer

[0241] , a front-end web server

[0242] , an annotation API server

[0243] , a deep learning API

[0244] , and a message broker

[0245] . Also shown is an electronic database

[62] including a relational database

[0246] and a key-value store database

[0247] . In this exemplary embodiment, the load balancer

[0241] , the front-end web server

[0242] , the annotation API server

[0243] , the deep learning API

[0244] , the message broker

[0245] , the relational database

[0246] , and the key-value store database

[0247] are each being executed on a different machine having one or more central processing units (CPUs), memory, and a non-transitory computer-readable storage medium. Further, the deep learning API

[0244] comprises at least one of a graphical processing unit (GPU), a tensor processing unit (TPU), or a field-programmable gate array (FPGA). In the exemplary embodiment depicted in FIG. 29, the load balancer

[0241] machine comprises one central processing unit (CPU) having 4 cores, 8 gb of random access memory (RAM), and a 250 gb solid state drive (SSD), and the front-end web server

[0242] machine comprises one CPU having 4 cores, 8 gb of RAM, and a 250 gb SSD.The machine of the Annotation API server

[0243] has two CPUs with 32 cores each, for a total of 64 cores, 64 GB of RAM, and 2 TB of SSD. The Deep Learning API

[0244] machine has one 32-core CPU, 128 GB of RAM, two GPUs with 11 GB of RAM each, and 2 TB of SSD. The Message Broker

[0245] machine has one 4-core CPU, 32 GB of RAM, and 1 TB of SSD. The Relational Database

[0246] machine has one 32-core, 128 GB of RAM, and 2 TB of SSD, and the Key-Value Store Database

[0247] machine has one 4-core CPU with 128 GB of RAM and 1 TB of SSD. Machines that meet these hardware specifications enable this system to support many users with multiple concurrent users.

[0135] In the exemplary embodiment depicted in FIG. 29, the hardware specifications were described. Next, the theoretical basis for these specifications will be explained. The load balancer

[0241] simply routes requests to the appropriate location. As a result, the requirements for the non-transitory computer-readable storage area of this machine should be met by a 250 gb SSD. Compared to the calculations performed in other parts of the system, the load balancer

[0241] has a low computational load and a low memory load. As a result, 2 gb per core is sufficient, and considering the rest of the system, 4 cores are sufficient. As the system grows in size, the number of instructions executed per second increases, and by increasing the CPU cores, it may be possible to prevent the load balancer

[0241] from becoming a bottleneck. However, based on the specifications given for the exemplary embodiment depicted in FIG. 29, the likelihood of the load balancer

[0241] becoming a bottleneck is low. The front-end web server

[0242] also does not require a large amount of non-transitory computer-readable storage area, a large number of CPU cores, or a large amount of memory. This machine does not store user data, and the front-end code it stores can be supported by the allocated storage space. Furthermore, the CPU cores of the front-end web server

[0242] machine simply send the requested resources to the user, and it is not common in normal use for the user to request front-end code such that the user's browser makes an API call to the annotation API server

[0243] . As a result, the CPU cores of the front-end web server

[0242] machine have a lower computational load and memory requirements than the annotation API server

[0243] machine. The annotation API server

[0243] has a high computational load because API requests are made from the user's web browser during the creation, organization, and viewing of annotations. The total computational power of the annotation API server

[0243] machine may become a bottleneck when there are a large number of concurrent users. In the exemplary embodiment, this machine is equipped with 64 CPU cores and 64 gb of RAM.If the code of the Annotation API Server

[0243] is written in a thread-safe manner, 1 GB of RAM per CPU core is sufficient. If the code is not thread-safe or the web framework consumes a large amount of memory, it may be desirable to use 2 GB of RAM per CPU core for the Annotation API Server

[0243] . To scale the Annotation API Server

[0243] to support more users, it is more cost-effective to add machines rather than providing more CPU cores and more RAM per machine. The size of the hard disk of the Annotation API Server

[0243] is mainly for enabling the storage of log files. For the Deep Learning API

[0244] , it would be good to have two GPUs with sufficient GPU RAM to run the deep learning model. The CPU cores and RAM are used to process the data processing steps concentrated on the CPU and to load the data consumed by the GPU into memory. A 2tb SSD can store large-scale deep learning models and other data for these models. The Message Broker

[0245] should be operational at all times when the system is running and should not face a shortage of computing power. 32 GB of RAM exists to ensure sufficient RAM to store large messages that may include images. Since these images will not be stored for a long time, 32 GB of RAM is sufficient. Additionally, the Message Broker

[0245] requires 4 cores. The Message Broker

[0245] has lower computational processing requirements in terms of the number of instructions per second than the Annotation API Server

[0243] in this exemplary embodiment. The Key-Value Store Database

[0247] uses one core in this exemplary embodiment. However, the Key-Value Store Database

[0247] uses RAM to store the reputations of various item objects such as annotation objects and user reputation votes for these item objects.Furthermore, disk space is used for logging and for temporarily storing backups of the key-value store database in memory. The relational database [

[0246] ] has 128 gb of RAM to store all data in the database in RAM during operation, 2 tb of SSD to ensure sufficient storage space for logging, and 32 cores to provide the computing power to process queries sent from the annotation API server [

[0243] ].

[0136] To scale up the system depicted in the exemplary embodiment of FIG. 29, the following steps can be taken. The new exemplary embodiment will have one or more load balancers [

[0241] ], one or more front-end web servers [

[0242] ], one or more annotation API servers [

[0243] ], and one or more deep learning APIs [

[0244] ]. The load balancer, front-end web server, annotation API server, and deep learning API server can be added to improve the overall performance of the system. In many cases, the message broker can be organized into a message broker cluster. The message broker cluster can be used to scale up the new exemplary embodiment. Most key-value store database [

[0247] ] software also has the ability to create a key-value store database cluster.

[0137] In a new exemplary embodiment, to enable a relational database [

[0246] ] to handle the most users, it is desirable to use a relational database [

[0246] ] that uses threads as opposed to processes. In some cases, a thread takes about 256 kb of memory per thread, while a process takes 10 mb of memory per process. For this reason, a relational database that relies on threads can have over 20 times more open connections than a relational database that relies on processes. Even if a connection pooling tool used by a process-dependent relational database is used, the connection style performed by the annotation API server is suitable for thread-based connections [

[0243] ]. The annotation API server [

[0243] ] is transaction-dependent and requires performance. As a result, it is desirable to use SQL transaction pooling rather than SQL session pooling or SQL statement pooling. SQL session pooling has low performance, and SQL statement pooling makes transaction implementation more difficult than using SQL transaction pooling. When using SQL transaction pooling, thread-based connections provide the best performance when the computer's instructions need to be executed on a processor not used by the relational database [

[0246] ]. This is because the connection remains open while work is being done on an external processor. As a result, the relational database [

[0246] ] may be given fewer SQL statements to execute than the relational database [

[0246] ] can handle. This results in an overall throughput decrease.

[0138] To further enhance the capabilities of a relational database [

[0246] ], it is advantageous to perform read scaling of the relational database [

[0246] ] by creating read replicas. By reducing the write load on the relational database [

[0246] ], the system can be further scaled up. In a new exemplary embodiment, the write load can be reduced by performing reputation voting through a key-value store database [

[0247] ]. To further expand the capacity of the relational database, in a new exemplary embodiment, the number of CPU cores and RAM of the relational database [

[0246] ] machine can be increased. Another step that can be taken is to use an in-memory relational database [

[0246] ] that implements SQL without full durability guarantees for most relational databases. Such a system can use an in-memory database with checkpointing and write-ahead logging to ensure that, in the event of a power outage, the database data other than the data that arrived at the in-memory database within the last few seconds is not lost. Through these steps, the relational database can be further expanded. In an HPC cluster, remote direct memory access (RDMA) via InfiniBand networking enables a relational database to use more memory than the memory locally available on a single machine. Since this HPC cluster can be scaled up to hundreds or thousands of machines, the corresponding relational database [

[0246] ] can support a large number of concurrent users.

[0139] In the exemplary embodiment depicted in FIG. 30, different types of client electronic devices

[0248] that can communicate with an annotation server

[61] and a content delivery network

[63] are shown. The devices shown are a laptop computer

[0249] , a desktop

[0250] , an iPad

[0251] , an Android tablet

[0252] , an iPhone

[0253] , an Android mobile phone

[0254] , an Apple TV via Air-Web

[0255] , an Android TV

[0256] , and a Raspberry Pi

[0257] . The laptop computer

[0249] , the desktop

[0250] , the Android tablet

[0252] , the Android mobile phone

[0254] , and the Raspberry Pi

[0257] support an annotation-enabled web browser with a notepad function through the introduction of a web browser and a web browser extension. Also, the iPad

[0251] and the iPhone

[0253] support an annotatable web browser by installing an extension function of the web browser "Safari" and an app. Apple TV

[0255] and Android TV

[0256] via Air Web may not currently be suitable for installing an annotatable web browser that includes a web browser and a web browser extension. Furthermore, the form factor may not be ideal for creating annotations in some cases. However, it is possible to create a progressive web application that communicates with the annotation server for these devices. Then, just as a digital photo frame displays an image, annotated materials can be displayed. Then, these devices will be able to display rich annotations created using other devices shown in FIG. 30.

[0140] Next, annotations for images, videos, and recordings will be described. The GUI diagrams in the exemplary embodiments are presented as showing what the system displays and the end result of non-transitory computer-readable media instructions executed on a web browser running on a processor. In an exemplary embodiment, the non-transitory computer-readable media instructions may be for a web browser executed on a processor, or the non-transitory computer-readable media instructions may be incorporated into an annotatable web browser. As described above, the annotatable web browser in the exemplary embodiment is a web browser having a web browser extension. When using an annotatable web browser, images, videos, and audio recordings embedded within any web page can be annotated. The web site to be annotated does not need to include computer-readable media instructions for annotating the embedded images, videos, and recordings. When using a normal web browser, if the instructions of the non-transitory computer-readable media are delivered via a specific web site and content annotation is performed on that specific web site, images, videos, and recordings can be annotated.

[0141] In addition to the annotatable web browser for annotating documents and media on the World Wide Web, software applications for mobile devices and Internet TVs can also use non-transitory computer-readable media instructions implementing the methods, techniques, and interfaces described in this document to annotate images, videos, and recordings available to a group of users on the Internet or a local intranet.

[0142] After describing the GUI, a bundle of data created to transfer data between components will be described. When the relevant data bundles are discussed, the methods and system architectures are discussed in the context of the system architectures and components already discussed.

[0143] In the exemplary embodiment depicted in FIG. 35, an image

[0299] is shown along with an image annotation subject

[0300] , a bounding box of the image annotation subject

[0301] , an image annotation arrow

[0302] , and an image annotation comment within the box

[0303] . The image annotation subject

[0300] is an entity of interest included in the image. The image annotation subject

[0300] can be a person, a vehicle, a soccer ball, a tennis ball, or any other object distinguishable within the image by a human or a computer system. The bounding box of the image annotation subject

[0301] is a bounding box drawn around the image annotation subject

[0300] by a user or a computer system. The image annotation comment within the box

[0303] is created by the user, and the annotation comment needs to be related to the image annotation subject

[0300] . The image annotation arrow

[0302] is a line that may have arrowheads on both sides connecting the bounding box of the image annotation subject

[0301] and the image annotation comment within the box

[0303] . The bounding box of the image annotation subject

[0301] , the image annotation arrow

[0302] , and the image annotation comment within the box

[0303] are created by the user and are objects included in the image annotation. The user can select a rectangle tool from a panel or a floating toolbar on the web page and draw the bounding box of the image annotation subject

[0301] . Further, the user can input the position of the text and the image annotation comment within the box

[0303] . The bounding box of the image annotation subject

[0301] and the image annotation comment within the box

[0303] are connected by the image annotation arrow

[0302] . The objects of the image annotation are not always displayed simultaneously.For example, only the bounding box

[0301] of the image / annotation subject is displayed, and other objects of the image / annotation can be made to be displayed when the mouse is moved over the image / annotation subject

[0300] or the bounding box

[0301] of the image / annotation subject. The visible element of the image / annotation is an element that can be visualized without the user moving the mouse over other elements. The mouse-over element of the image / annotation is an element that becomes visible when the user moves the mouse over the visible element of the image / annotation.

[0144] Below the image [

[0299] ], there is a control panel including an image previous annotation arrow button [

[0304] ], an image next annotation arrow button [

[0305] ], a button including image collision [

[0306] ], an add button to the image candidate annotation group [

[0307] ], an image candidate annotation group list box [

[0308] ], and an image-annotation group creation button [

[0309] ]. The label of the add button to the image candidate annotation group [

[0307] ] is a short name "Add to Annotation Group". However, the image annotation is first added to the image candidate annotation group, and then, when the user clicks the image creation annotation group button [

[0309] ], it becomes an image-annotation group. The image annotation includes objects such as a bounding box [

[0301] ] of the image-annotation subject, an image-annotation arrow [

[0302] ], and an image-annotation comment [

[0303] ] inside the box. One image annotation may include one or more annotation subjects [

[0300] ]. In the electronic database

[62] , the image annotation is stored as an image-annotation object. This image-annotation object is sequentially ordered for the image annotation by the electronic database

[62] . This ordering of the image annotation gives meaning to the concept of the previous and next image annotations. The image previous annotation arrow button [

[0304] ] deletes the objects included in the current image annotation from the displayed view assuming that the current image annotation is not an image annotation candidate, and adds the objects of the previous image annotation to the displayed view. The image next annotation arrow button [

[0305] ] deletes the objects included in the current image annotation from the displayed view when the current image annotation is not an image annotation candidate, and adds the objects of the next image annotation to the displayed view. An image-annotation group includes a plurality of image annotations.Before creating an image-annotation group, it is necessary to select an image-annotation. The selected image-annotation becomes a candidate image-annotation. The candidate image-annotations are listed in the image candidate annotation group list box

[0308] . Figure 35 depicts a scenario where the image-annotations have not yet been added to the image candidate annotation group list box

[0308] . Therefore, the message in the image candidate annotation group list box

[0308] is the image candidate annotation group list empty message

[0310] .

[0145] The button

[0306] including image collision determines the display properties of the image-annotations that collide with the image-annotation candidates listed in the image candidate annotation group list box

[0308] . The possible states are excluding the collision and including the collision. When the button

[0306] including image collision is clicked, the image-annotations that collide with the candidate image-annotations listed in the image candidate annotation group list box

[0308] are displayed. In the exemplary embodiment depicted in Figure 35, the label of the button is changed to "Exclude collision". When this button is clicked again, the image-annotations that collide with the image-annotation candidates listed in the image candidate annotation group list box

[0308] are no longer displayed. When the image-annotation group creation button

[0309] is clicked, an item is created that displays an image-annotation group including all the image-annotations listed in the image annotation group candidate list box

[0308] at the time the image-annotation group creation button

[0309] is clicked.

[0146] In an alternative embodiment, the bounding box

[0301] of the image-annotation subject and the image-annotation arrow

[0302] may not be displayed. In an alternative embodiment, only the bounding box

[0301] of the image-annotation subject and the image-annotation comment

[0303] within the box exist. Alternatively, in another alternative embodiment, only the image-annotation comment

[0303] within the box exists near the image-annotation subject

[0300] .

[0147] In the exemplary embodiment depicted in FIG. 36, an image

[0299] with a first image candidate annotation group list item

[0311] added to the image candidate annotation group list box

[0308] is displayed. This state is generated by clicking the add button

[0307] to the image-annotation candidate group in the state of the exemplary embodiment depicted in FIG. 35. After the image-annotation is added to the image-annotation candidate group list, in the state of the exemplary embodiment depicted in FIG. 36, by clicking the next image-annotation arrow button

[0305] , the state depicted in FIG. 37 is created.

[0148] In the exemplary embodiment depicted in FIG. 37, a second image-annotation subject

[0312] is displayed together with the bounding box

[0313] of the second image-annotation subject created by the user, the second image-annotation arrow

[0314] , and the second image-annotation comment

[0315] within the box. When the add button

[0307] to the image-annotation candidate group is clicked in the state of the exemplary embodiment shown in FIG. 37, the state of the exemplary embodiment shown in FIG. 38 is generated.

[0149] In the exemplary embodiment depicted in FIG. 38, a second image candidate annotation group list item

[0316] is shown. When the image creation annotation group button

[0309] is pressed, an annotation group including two annotations depicted in the image candidate annotation group list box

[0308] of FIG. 38 will be created.

[0150] In the exemplary embodiment depicted in FIG. 39, the user previously clicked the image collision button

[0306] of FIG. 36 that enables the display of colliding image annotations. After pressing the image next annotation arrow button

[0305] from the state depicted in FIG. 36, the state depicted in FIG. 39 can be brought about. In FIG. 39, the collision image annotation arrow

[0317] and the collision image annotation comment within the box

[0318] are displayed.

[0151] In the exemplary embodiment depicted in FIG. 40, video

[0319] is shown along with video annotation playback

[0320] . The video

[0319] shows a video annotation subject

[0321] , a bounding box of the video annotation subject

[0322] , a video annotation arrow

[0323] , and a video annotation comment within the box

[0324] . The annotation subject

[0321] is an entity of interest in the video. The video annotation subject

[0321] can be a person, a car, a soccer ball, a tennis ball, or other objects distinguishable within the video by a human or a computer system. The bounding box of the video annotation subject

[0322] is a bounding box drawn around the video annotation subject

[0321] by a user or a computer system. The video annotation comment box

[0324] is created by the user, and the annotation comment needs to be related to the video annotation subject

[0321] . The video annotation arrow

[0323] is a line with arrowheads on both sides that connects the bounding box of the video annotation subject

[0322] and the video annotation comment in the box

[0324] . The bounding box of the video annotation subject

[0322] , the video annotation arrow

[0323] , and the video annotation comment input box

[0324] are objects created by the user and included in the video annotation. The user can select the rectangle tool from a panel or a floating toolbar on the web page and draw the first bounding box of the video annotation subject

[0322] . Further, the user can input text, place the video annotation comment in the box

[0324] , and connect them with the video annotation arrow

[0323] . When the video

[0319] is played, the video annotation playback

[0320] is also performed.During the playback of video annotation playback

[0320] , the user creating the video annotation can move the bounding box

[0322] of the subject of the video annotation using the mouse and confirm that the bounding box

[0322] of the video annotation subject is properly placed throughout the video. The user can move the video annotation comment within the box

[0324] using the mouse and confirm that the video annotation comment within the box

[0324] is properly placed throughout the video. If the motion paths of the bounding box

[0322] of the video annotation subject and the video annotation comment in the box

[0324] are separated, the motion path of the video annotation arrow

[0323] can be calculated together with time rotation, time contraction, time expansion, and time position to correspond to the motion paths of the bounding box

[0322] of the video annotation subject and the video annotation comment in the box

[0324] . Also, the user can pause the video and resize the video annotation comment within the bounding box

[0322] of the video annotation subject and the box

[0324] . Not all objects of the video annotation are always displayed at the same time. For example, after the user mouses over the video annotation subject

[0321] or the bounding box

[0322] of the video annotation subject, other objects of the video annotation can be displayed, and only the bounding box

[0322] of the video annotation subject can be displayed. The visible elements of the video annotation are elements that are visualized without mousing over other elements. The mouse-over elements of the video annotation are elements that become visible when the user mouses over the visible elements of the video annotation.

[0152] Below the video

[0319] , a control panel including a video pre-annotation arrow button

[0325] , a video next annotation arrow button

[0326] , a collision button including the video

[0327] , a button to add to the video candidate annotation group

[0328] , a video candidate annotation group list box

[0329] , and a video annotation group creation button

[0330] is displayed. The label of the button to add to the video annotation group candidate

[0328] has a short name "Add Annotation Group". However, the video annotation is first added to the video candidate annotation group and becomes a video annotation group after the user clicks the video creation annotation group button

[0330] . The video annotation includes objects such as a bounding box

[0322] of the video annotation subject, a video annotation arrow

[0323] , and a video annotation comment input box

[0324] . The video annotation can include a plurality of video annotation subjects

[0321] . In the electronic database

[62] , the video annotation is stored as a video annotation object. And these video annotation objects are sequentially ordered by the electronic database

[62] for the video annotation. This ordering of the video annotation gives meaning to the concept of the previous and next video annotations. The video pre-annotation arrow button

[0325] deletes the objects included in the current video annotation from the view where they are displayed, assuming that the current video annotation is not a video annotation candidate, and adds the objects of the previous video annotation to the view where they are displayed. The video next annotation arrow button

[0326] deletes the objects included in the current video annotation from the view where they are displayed, assuming that the current video annotation is not a candidate video annotation, and adds the objects of the next video annotation to the view where they are displayed.A video annotation group includes a plurality of video annotations. Before a video annotation group is created, it is necessary to select video annotations. In the selection process, the video annotations become candidate video annotations. The candidate video annotations are listed in the video candidate annotation group list box

[0329] . FIG. 40 depicts a scenario where the video annotation has not yet been added to the video annotation candidate group list box

[0329] . Therefore, the message in the video annotation candidate group list box

[0329] is the video annotation candidate group list empty message

[0331] .

[0153] The collision button

[0327] including the video determines the display properties of the video annotation that collides with the candidate video annotations listed in the video candidate annotation group list box

[0329] . In the case of video annotations, a collision means that two video annotations collide at a certain point. The collision may be displayed in red on the playback line of the video annotation

[0320] . When the collision button

[0327] including the video is clicked, the video annotation that collides with the candidate video annotations listed in the video candidate annotation group list box

[0329] is displayed. In the exemplary embodiment depicted in FIG. 40, the button label is changed to "Exclude collision". When this button is clicked again, the video annotation that collides with the video annotation candidates listed in the video candidate annotation group list box

[0329] is no longer displayed. When the video annotation group creation button

[0330] is clicked, the following items are created. When the video creation annotation group button

[0330] is clicked, a video annotation group is generated that includes all the video annotations listed in the video candidate annotation group list box

[0329] . When the video addition candidate annotation group button

[0328] is clicked in the state depicted in the exemplary embodiment of FIG. 40, the state depicted in the exemplary embodiment of FIG. 41 is created with the first video candidate annotation group list item

[0335] added to the video candidate annotation group list box

[0329] .

[0154] In an alternative embodiment, the bounding box

[0322] of the video annotation subject and the video annotation arrow

[0323] may not be displayed. In an alternative embodiment, only the bounding box

[0322] of the video annotation subject and the video annotation comment within the box

[0324] exist. Or, in another alternative embodiment, only the video annotation comment within the box

[0324] exists near the video annotation subject

[0321] .

[0155] In the exemplary embodiment depicted in FIG. 40, the video

[0319] player controls are separated from the video annotation playback

[0320] . In an alternative embodiment, it is possible to integrate both controls into a single playback control. This may result in a more complex input interface, but it can reduce the number of elements displayed on the screen. This comment regarding the integration of the two playback controls also applies to the exemplary embodiments depicted in FIGS. 41, 42, 43, 44, and 45.

[0156] In the exemplary embodiment depicted in FIG. 41, the bounding box motion path

[0332] of the video annotation subject, the video annotation arrow motion path

[0333] , and the video annotation comment input box motion path

[0334] are shown. The bounding box motion path

[0332] of the video annotation subject shows the motion path of the bounding box

[0322] when the video annotation subject

[0321] moves within the video, and is supposed to function as the bounding box of the video annotation subject

[0321] . The video annotation arrow motion path

[0333] shows the motion path of the video annotation arrow

[0323] when the video annotation arrow

[0323] moves as the video

[0319] progresses in time. It can be seen that the video annotation arrow

[0323] is moving in parallel, rotating, expanding, and contracting temporally from the bounding box

[0322] of the video annotation subject and the video annotation comment

[0319] . The box

[0324] moves in time. The video annotation comment input box motion path

[0334] shows the motion path when the video annotation comment within the box

[0324] moves as the video

[0319] progresses in time.

[0157] In the exemplary embodiment depicted in the figure, 42 represents the second video annotation subject

[0336] , the bounding box of the second video annotation subject

[0337] , the second video annotation arrow

[0338] , the second video annotation comment within the box

[0339] , the bounding box motion path of the second video annotation subject

[0340] , the motion path 1 of the second video annotation arrow

[0341] , the motion path 2 of the second video annotation arrow

[0342] , the bounding box position of the final second video annotation subject

[0343] , and the position of the final second video annotation arrow

[0344] . The bounding box

[0337] of the second video annotation subject follows the second video annotation subject

[0336] along the bounding box motion path

[0340] of the second video annotation subject to the bounding box position

[0343] of the final second video annotation subject. The second video annotation arrow

[0338] is transformed to the final second video annotation arrow position

[0344] over time. The transformation including movement, rotation, and shrinkage is depicted by the motion path 1

[0341] of the second video annotation arrow and the motion path 2

[0342] of the second video annotation arrow. The second video annotation comment within the box

[0339] remains temporally fixed. The second video annotation candidate group list item

[0345] is displayed in the video annotation candidate group list box

[0329] that occurs when the user clicks the add button

[0328] for the video annotation candidate group.

[0158] In the exemplary embodiment depicted in FIG. 43, the user may have previously clicked on the collision button

[0327] including video in the state depicted in the exemplary embodiment of FIG. 41. As a result, video annotations having a collision with the video annotation in the candidate annotation group list are displayed. After clicking on the video next annotation arrow button

[0326] in the state depicted in the exemplary embodiment of FIG. 41 and then clicking on the collision button

[0327] including video, the collision video annotation arrow

[0346] and the collision video annotation comment in the box

[0347] may be displayed. In the exemplary embodiment of FIG. 43, a collision is depicted. However, the collision may occur at a point not depicted in a specific video frame. In such a case, the collision may be depicted in red on the timeline of the video annotation playback

[0320] within the exemplary embodiment.

[0159] In the exemplary embodiment depicted in FIG. 44, recording playback

[0348] and recording annotation playback

[0349] are shown. Below the recording annotation playback

[0349] , a recording annotation arrow

[0350] , a recording annotation comment box

[0351] , a recording annotation connection arrow

[0352] , and a recording-derived annotation comment box

[0353] are shown. Below the recording annotation playback

[0349] , there is a control panel such as a pre-recording annotation arrow button

[0354] , a next-recording annotation arrow button

[0355] , a recording candidate annotation group addition button

[0356] , a recording-derived annotation creation button

[0357] , a recording candidate annotation group list box

[0358] , a recording-derived annotation group creation button

[0359] , and a recording-derived annotation comment input box

[0360] . The label of the addition button

[0356] to the recording candidate annotation group is a short name "Add Annotation Group". However, the recording annotation is first added to the recording candidate annotation group and becomes a recording annotation group after the user clicks the recording annotation group creation button

[0359] .

[0160] In the exemplary embodiment depicted in FIG. 44, to create a recording annotation, the user uses the mouse to select a short segment of the recording annotation playback

[0349] . After the selection is made, a recording annotation arrow

[0350] and a recording annotation comment box

[0351] appear centered on the selection, and the user can create a recording annotation comment in the recording annotation comment box

[0351] . The recording annotation includes objects such as the recording annotation arrow

[0350] and the recording annotation comment box

[0351] . The short segment of the recording annotation playback

[0349] can be marked as shown by the embedded recording annotation playback range

[0393] in FIG. 49. FIG. 49 shows an embedded recording, and the marked short segment is on the playback bar for the recording playback

[0348] , but the same style of the marked segment can be used for the recording annotation playback

[0349] . Also, it is possible to leave the segment unmarked, or to display the marked segment when mousing over the selected segment, the recording annotation arrow

[0350] , or the recording annotation comment box

[0351] . Further, a question sound annotation symbol that behaves similarly to the question embedded sound annotation symbol

[0394] in FIG. 49 can be introduced.

[0161] A recording annotation can include one or more selected segments of sound. In an electronic database

[62] , the recording annotation is stored as a recording annotation object. These recording annotation objects are then sequentially ordered by the electronic database

[62] for the recording annotations. This ordering of the recording annotations gives meaning to the concepts of the previous recording annotation and the next recording annotation. The pre-recording annotation arrow button

[0354] deletes the objects included in the current recording annotation from the displayed view on the assumption that the current recording annotation is not a recording annotation candidate, and adds the objects included in the previous recording annotation to the displayed view. The next recording annotation arrow button

[0355] deletes the objects included in the current recording annotation from the displayed view on the assumption that the current recording annotation is not a recording annotation candidate, and adds the objects of the next recording annotation to the view being displayed. A recording annotation group is a group of recording annotations that the user wants to group. These groupings are displayed as one item, and derivative recording annotations can be created that use one or more recording annotations from the recording annotation group and associate these one or more recording annotations. Further, the derivative recording annotation and the recording annotations on which it depends can be displayed as one item. A recording annotation can be added to a recording annotation group by clicking the add button

[0356] to the recording annotation candidate group. As a result, a new recording annotation is displayed in the recording annotation candidate group list box <358>. When a recording annotation is added to a recording annotation group, the user can create an annotation derived from the recording. A recording derivative annotation is an annotation that links other recording annotations and adds a comment to the recording derivative annotation comment box

[0353] connected to a plurality of recording annotation comment boxes.The recorded sound derived annotation comment box

[0353] is connected to each sound source annotation comment box

[0351] using the recorded sound annotation connection arrow

[0352] . The recorded sound derived annotation is created by selecting a desired recorded sound annotation and clicking the recorded sound derived annotation creation button

[0357] . Next, enter the recorded sound derived annotation comment

[0361] in the recorded sound derived annotation comment input box

[0360] and press Enter. The recorded sound candidate annotation group list box

[0358] contains the recorded sound candidate annotation group list item

[0362] . This list includes both the recorded sound annotation and the annotation derived from the recorded sound. The first item of the recorded sound candidate annotation group list item

[0362] is the first recorded sound candidate annotation group list item

[0363] .

[0162] In the exemplary embodiment depicted in FIG. 45, video [

[0319] ] and video annotation playback [

[0320] ] are shown. Below video annotation playback [

[0320] ], a video annotation arrow [

[0364] ], a video annotation comment box [

[0365] ], a video annotation connection arrow [

[0366] ], and a video-derived annotation comment box [

[0367] ] are shown. Below video annotation playback [

[0320] ], a control panel including a previous video annotation arrow button [

[0325] ], a next video annotation arrow button [

[0326] ], a collision button including video [

[0327] ], an add button to video candidate annotation group [

[0328] ], a video-derived annotation creation button [

[0368] ], a video candidate annotation group list box [

[0329] ], a created video annotation group button [

[0330] ], and a video-derived annotation comment input box [

[0369] ] is shown. The label of the add video button [

[0328] ] has a short name of "Add Annotation Group". However, the video annotation is first added to the video candidate annotation group and becomes a video annotation group after the user clicks the video annotation group creation button [

[0330] ]. To create a video annotation in the exemplary embodiment, the user uses the mouse to select a short segment of video annotation playback [

[0320] ]. After the selection is made, a video annotation arrow [

[0364] ] and a video annotation comment box [

[0365] ] appear starting from the center of the selection, and the user can create a video annotation comment in the video annotation comment box [

[0365] ]. The short segment of video annotation playback [

[0320] ] can be marked as shown by the embedded video annotation playback range [

[0375] ] in FIG. 46. FIG. 46 shows an embedded video, and the marked short segment is on the playback bar of the video [

[0319] ], but the marked segments of the same style can be used in video annotation playback [

[0320] ].It is possible not to mark segments, or to display marked segments when hovering over a selected segment, an arrow

[0364] of a video annotation, or a comment box

[0365] of a video annotation. Furthermore, a question video annotation symbol that behaves in the same way as the question-embedded video annotation symbol

[0376] in FIG. 46 can be introduced.

[0163] As previously described for another exemplary embodiment, video annotation includes objects such as a bounding box of a video annotation subject [

[0322] ], a video annotation arrow [

[0323] ], and an in-box video annotation comment [

[0324] ]. Video annotation can include a plurality of video annotation subjects [

[0321] ]. In an electronic database

[62] , video annotation is stored as a video annotation object. And these video annotation objects are sequentially ordered by the electronic database

[62] for video annotation. This ordering of video annotation gives meaning to the concept of previous and next video annotations. The previous video annotation arrow button [

[0325] ] deletes the objects included in the current video annotation from the displayed view on the assumption that the current video annotation is not a video annotation candidate, and adds the objects of the previous video annotation to the displayed view. The next video annotation arrow button [

[0326] ] deletes the objects included in the current video annotation from the displayed view on the assumption that the current video annotation is not a video annotation candidate, and adds the objects of the next video annotation to the displayed view. A video annotation group is a collection of video annotations that a user wants to group. These groupings are displayed as one item, and derivative video annotations can be created that use one or more video annotations from the video annotation group and associate these one or more video annotations. Further, the derived video annotation can be displayed as one item together with the video annotation on which it depends. To add a video annotation to a video candidate annotation group, click the add button [

[0328] ] to the video candidate annotation group. As a result, a new video annotation is displayed in the video annotation candidate group list box <329>.When a video annotation is added to the video candidate annotation group, the user can create an annotation derived from the video. A video-derived annotation is an annotation that links other video annotations and adds a comment to a video-derived annotation comment box

[0367] connected to multiple video annotation comment boxes. The video-derived annotation comment box

[0367] is connected to each video annotation comment box

[0365] using a video annotation connection arrow

[0366] . A video-derived annotation is created by selecting a desired video annotation and clicking a video-derived annotation creation button

[0368] . Next, enter a video-derived annotation comment

[0370] in a video-derived annotation comment input box

[0369] and press Enter. A video candidate annotation group list box

[0329] displays video candidate annotation group list items

[0371] . This list includes both video annotations and video-derived annotations. The first item of the video candidate annotation group list item

[0371] is the first video candidate annotation group list item

[0372] . Clicking a video creation annotation group button

[0330] enables creation of an annotation group that includes the video candidate annotations in the video candidate annotation group list box

[0329] . The video inclusion collision button

[0327] operates in the same manner as the video inclusion collision button

[0327] in the exemplary embodiments depicted in FIGS. 40, 41, 42, and 43.

[0164] In the exemplary embodiment depicted in FIG. 46, embedded video

[0373] and embedded video playback

[0374] are shown. Embedded video

[0373] and embedded video playback

[0374] exist within a web page being viewed in an annotatable web browser. In the exemplary embodiment depicted in FIG. 46, an embedded video annotation playback range

[0375] , a question embedded video annotation symbol

[0376] , a disagreement embedded video annotation symbol

[0377] , an embedded video annotation beacon

[0378] , an embedded video annotation beacon question symbol

[0379] , an embedded video annotation beacon disagreement symbol

[0380] , and an embedded video annotation beacon playback time

[0381] are shown. The symbol beacon trigger button

[57] triggers the display of one fully displayed video annotation beacon for each video annotation of the embedded video

[0373] unless the embedded video

[0373] contains three or more video annotations. If three or more video annotations are present, an alternative display methodology allows the user to better visualize the video annotations that can be viewed. The fully displayed video annotation beacon includes the embedded video annotation playback range

[0375] , the embedded video annotation symbol, the embedded video annotation beacon

[0378] , the embedded video annotation beacon symbol, and the embedded video annotation beacon playback time

[0381] . The embedded video annotation playback range

[0375] highlights the time range that the user considers relevant to the video annotation. Embedded video annotation symbols such as the question embedded video annotation symbol

[0376] and the disagreement embedded video annotation symbol

[0377] convey the semantic relationship of the annotation between the selected video segment and the video annotation comment. The embedded video annotation beacon

[0378] serves to draw the user's attention to the presence of the video annotation and some of its characteristics.The embedded video annotation beacon symbol is placed in the vicinity of the embedded video annotation beacon

[0378] for the semantic relationship of the annotation between the selected video segment and the video annotation comment. Two examples of the embedded video annotation beacon symbol are the embedded video annotation beacon question symbol

[0379] and the embedded video annotation beacon disagreement symbol

[0380] . The embedded video annotation beacon playback time

[0381] placed in the vicinity of the embedded video annotation beacon

[0378] conveys the length of the video segment related to the video annotation comment.

[0165] In the exemplary embodiment depicted in FIG. 46, the playback range

[0375] of the embedded video annotation and the question embedded video annotation symbol

[0376] are shown in the context of the embedded video

[0373] and the embedded video playback

[0374] . However, the video annotation playback range and the video annotation symbol can also be used on the video annotation playback

[0320] depicted in FIG. 45.

[0166] In the exemplary embodiment depicted in FIG. 47, an embedded image

[0382] is shown. The embedded image

[0382] exists within a web page, and an embedded image - annotation subject

[0383] occurs within the embedded image

[0382] . In the exemplary embodiment depicted in FIG. 47, a bounding box

[0384] of the embedded image - annotation subject, an embedded image - annotation arrow

[0385] , and an embedded image - annotation comment within a box

[0386] are shown. The embedded image - annotation subject

[0383] is an object of interest within the image. The embedded image - annotation subject

[0383] can be a person, a vehicle, a soccer ball, a tennis ball, or other object distinguishable within the image by a human or a computer system. The bounding box

[0384] of the embedded image - annotation subject is a bounding box drawn around the embedded image - annotation subject

[0383] by a user or a computer system. The user can select a rectangle tool from some panel or floating toolbar on the web page and draw the bounding box

[0384] of the embedded image - annotation subject. Further, the user can input text and place an embedded image - annotation comment within the box

[0386] . The embedded image - annotation comment within the box

[0386] is created by the user, and the annotation comment needs to be related to the embedded image - annotation subject

[0383] . The embedded image - annotation arrow

[0385] is a line that connects the bounding box

[0384] of the embedded image - annotation subject and the embedded image - annotation comment within the box

[0386] and can have arrowheads on both sides. The subject boundary box

[0384] of the embedded image - annotation, the embedded image - annotation arrow

[0385] , and the image - annotation comment within the embedded box

[0386] are embedded objects created by the user and included in the image - annotation.

[0167] Image and annotation embedding objects may not all be displayed simultaneously. As an example, only the bounding box

[0384] of the embedded image and annotation sub-object is visible together with other embedded objects of the image and annotation that are visualized after the user hovers over the embedded image and annotation sub-object

[0383] or the bounding box

[0384] of the embedded image and annotation sub-object. The visible elements of the image and annotation are the embedded elements that are visualized without the user hovering over other elements. The mouse-over embedded elements of the image and annotation are the embedded elements that become visible when the user hovers over the visible embedded elements of the image and annotation. When the user hovers over the embedded object of the image and annotation, the context-variant annotation control bar box

[0387] is displayed. The user can use the context-variant annotation control bar

[34] embedded in the context-variant annotation control bar box

[0387] to change privacy settings, share by email, share by social media, flag the image and annotation as inappropriate, delete the image and annotation object from the electronic database

[62] on the premise of sufficient permission, or edit the embedded object consisting of the image and annotation. Furthermore, the user can vote on the reputation of the image and annotation using the reputation selector

[33] .

[0168] In the exemplary embodiment depicted in FIG. 48, an embedded video

[0373] is shown. The embedded video

[0373] and the embedded video playback

[0374] exist within a web page, and the embedded video annotation subject

[0388] occurs within the embedded video

[0373] . In the exemplary embodiment depicted in FIG. 48, the bounding box

[0389] of the embedded video annotation subject, the embedded video annotation arrow

[0390] , and the embedded video annotation comment within the box

[0391] are shown. The embedded video annotation subject

[0388] is an entity of interest within the video. The embedded video annotation subject

[0388] can be a person, a car, a soccer ball, a tennis ball, or other object distinguishable within the video by a human or a computer system. The bounding box

[0389] of the embedded video annotation subject is a bounding box drawn around the embedded video annotation subject

[0388] by a user or a computer system. As the embedded video annotation subject

[0388] moves within the video, it is desirable for the bounding box

[0389] of the embedded video annotation subject to continue to function as the bounding box of the embedded video annotation subject

[0388] . The embedded video annotation comment within the box

[0391] is created by a user, and the annotation comment needs to be related to the embedded video annotation subject

[0388] . The embedded video annotation arrow

[0390] is a line that can have arrowheads on both sides connecting the bounding box

[0389] of the embedded video annotation subject and the embedded video annotation comment within the box

[0391] .The bounding box of the embedded video annotation subject

[0389] , the embedded video annotation arrow

[0390] , and the embedded video annotation comment inbox

[0391] are embedded objects created by the user and included in the video annotation.

[0169] The user can select the rectangular tool from somewhere on the web page or the floating toolbar and draw the bounding box [

[0389] ] of the first embedded video annotation subject. Further, the user can input text, place the embedded video annotation comment in the box [

[0391] ], and connect them with the embedded video annotation arrow [

[0390] ]. In order to ensure that the objects are properly placed during video playback, either the user or the computer system needs to determine the motion path of the bounding box [

[0389] ] of the embedded video annotation subject. The user creating the video annotation has the option to trace the path of the bounding box [

[0389] ] of the embedded video annotation subject when the video is played back in time. During the playback of the embedded video playback [

[0374] ], the user creating the video annotation can move the bounding box [

[0389] ] of the embedded video annotation subject with the mouse to confirm that the bounding box [

[0389] ] of the embedded video annotation subject is properly placed throughout the embedded video. The user can move the embedded video annotation comment in the box [

[0391] ] with the mouse to confirm that the embedded video annotation comment in the box [

[0391] ] is properly placed throughout the video. Either the annotation-capable web browser, the extension of the web browser, or the code of the web page records the motion path when the user drags the mouse. Next, when the video is played back, the paths of the bounding box [

[0389] ] of the embedded video annotation subject and the embedded video annotation comment in the box [

[0391] ] evolve along their respective motion paths.If the motion paths of the bounding box of the embedded video annotation subject

[0389] and the embedded video annotation comment in box

[0391] are known, the motion path, temporal rotation, temporal contraction, temporal expansion, and temporal position of the embedded video annotation arrow

[0390] can be calculated corresponding to the motion paths of the bounding box of the embedded video annotation subject

[0389] and the embedded video annotation comment in box

[0391] . Also, when the desired size of the bounding box of the embedded video annotation subject

[0388] changes as the video is played, the user can pause the video and resize the bounding box of the embedded video annotation subject

[0389] and the embedded video annotation comment in the box

[0391] .

[0170] Video annotation embedded objects are not always all visualized simultaneously. As an example, it is possible that only the bounding box

[0389] of the embedded video annotation subject

[0388] or the embedded video annotation subject is visible, such that other embedded objects of the video annotation become visible after the user mouses over the embedded video annotation subject

[0388] or the bounding box

[0389] of the embedded video annotation subject. The visible elements of the video annotation are the embedded elements that are visualized without mousing over other elements. The mouse-over embedded elements of the video annotation are the embedded elements that are visualized when the user mouses over the visible embedded elements of the video annotation. When mousing over the embedded objects of the video annotation, the context-variant annotation control bar box

[0387] is displayed. The user can use the context-variant annotation control bar

[34] embedded in the context-variant annotation control bar box

[0387] to change privacy settings, share by email, share by social media, flag the video annotation as inappropriate, delete the video annotation object from an electronic database

[62] assuming sufficient permission, or edit the embedded objects consisting of the video annotation. Further, the user can vote on the reputation of the video annotation using the reputation selector

[33] . Associated with each relationship within the semantic relationship group of the annotation are the embedded symbols of different video annotations. As an example, the question-embedded video annotation symbol

[0376] indicates that the creator of the video annotation has specified the semantic relationship of the annotation between the selected part of the video and the video annotation comment as a question relationship.The question-embedded video annotation symbol

[0376] enables a user who is familiar with how an exemplary embodiment functions to click on the question-embedded video annotation symbol

[0376] before actually performing the click operation that the user will perform, so as to know that video annotation comments, which are questions about the embedded video annotation playback range

[0375] , can be viewed. What the question annotation symbol, the disagreement annotation symbol, the agreement annotation symbol, the exclamation annotation symbol, and the advertisement annotation symbol correspond to are five relationships in the group of semantic relationships of annotations. These semantic relationships of annotations are the question relationship, the disagreement relationship, the agreement relationship, the exclamation relationship, and the advertisement relationship. These video annotation symbols and the semantic relationships of video annotations are applied to the relationship between the embedded video annotation playback range

[0375] and the comment input field

[0391] of the embedded video annotation.

[0171] In the exemplary embodiment depicted in FIG. 49, the embedded sound recording [

[0392] ] is shown together with the embedded sound recording annotation playback range [

[0393] ] and the question embedded sound annotation symbol [

[0394] ]. The embedded recording annotation playback range [

[0393] ] specifies a selected portion of the recording corresponding to the sound annotation. The embedded sound annotation symbol is associated with an element of a group of semantic relationships of the annotation. The semantic relationship of each annotation indicates the relationship between the embedded recording annotation playback range [

[0393] ] and the sound annotation comment. Different embedded sound annotation symbols are associated with each relationship of the group of semantic relationships of the annotation. As an example, the question embedded sound annotation symbol [

[0394] ] indicates that the creator of the sound annotation has specified the semantic relationship of the annotation between the selected portion of the sound of the recording and the sound annotation comment as a question relationship. The question embedded sound annotation symbol [

[0394] ] enables a user familiar with how the exemplary embodiment functions to know that before the user actually performs the click operation, by clicking on the question embedded sound annotation symbol [

[0394] ], the user can view the sound annotation comment that is a question regarding the embedded recording annotation playback range [

[0393] ]. The annotation comment box

[31] is displayed when the mouse is placed over the embedded recording annotation playback range [

[0393] ]. The user can use the context-variant annotation control bar

[34] embedded in the annotation comment box

[31] to change privacy settings, share by email, share by social media, flag the sound annotation as inappropriate, delete the sound annotation object from the electronic database

[62] subject to sufficient permission, or edit the embedded object consisting of the sound annotation. Further, the user can vote on the reputation of the sound annotation using the reputation selector

[33] .The question annotation symbol, the disagreement annotation symbol, the agreement annotation symbol, the exclamation annotation symbol, and the advertisement annotation symbol correspond to five relationships in the group of semantic relationships of annotations. These semantic relationships of annotations are the question relationship, the disagreement relationship, the agreement relationship, the exclamation relationship, and the advertisement relationship. These sound annotation symbols and the semantic relationships of the sound annotations are applied to the relationship between the embedded recording annotation playback range

[0393] and the sound annotation comment

[32] .

[0172] Aspects of the present invention are as described in FIGS. 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, and 49, and it will be understood that each item of these GUI figures can be implemented by computer-readable program instructions. Next, methods and systems related to the features and capabilities depicted in the exemplary embodiments of FIGS. 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, and 49 will be described and associated with the previously presented materials. Describing these methods and systems will include describing a bundle of data elements used to receive and store data, and a display bundle used to send and display visual elements to a client device. When the bundle of display elements is described, various item boxes are described. After describing these item boxes, the relationship with other presented elements including the user interface will be described. The bundle of data elements, the display bundle, and the item boxes to be described hereinafter are to be explained and interpreted within the context of the exemplary embodiments.

[0173] In the context of an exemplary embodiment, a bundle of data elements used to receive data from a client device and store this data using one or more computer systems will now be described. The bundle of data elements includes an image-annotation bundle, a video-annotation bundle, an audio-recording annotation bundle, an embedded-image annotation bundle, an embedded-video annotation bundle, an embedded-audio-recording annotation bundle, an annotation-group bundle, a derived-annotation-group bundle, and an annotation-group bundle with derived annotations.

[0174] In the context of an exemplary embodiment, an image-annotation bundle includes one or more image-annotation atoms. An image-annotation atom is a unit that can be used meaningfully by itself to annotate an image. For example, an image-annotation atom can include a bounding box

[0301] of an image-annotation subject, an image-annotation comment

[0303] within the box, and an image-annotation arrow

[0302] as shown in the exemplary embodiment depicted in FIG. 36. However, the image-annotation comment

[0303] within the box can also be used alone near the image-annotation subject

[0300] . Thus, the image-annotation comment

[0303] within the box can itself be an image-annotation atom. In addition to the image-annotation atoms, the image-annotation bundle includes frame information of the image to be annotated. The frame information includes the image size specified by the number of pixels on the x-axis and the number of pixels on the y-axis. When the image size is specified, based on the frame information, relative coordinates are assigned to the bounding box

[0301] of the image-annotation subject, the image-annotation comment

[0303] within the box, and the image-annotation arrow

[0302] , which are elements constituting the image-annotation atom. When the bounding box

[0301] of the image-annotation subject is specified by the X and Y coordinates of its center, width, and length, coordinates based on the lower left corner of the image are specified. The image-annotation comment

[0303] within the box can be specified in a similar manner together with the markup information of the annotation comment included in the image-annotation comment

[0303] within the box. This markup information includes the font size and formatting information of the text. Also, information regarding the mechanism for creating the text is included in the image-annotation bundle. The creation mechanism can be what the user wrote, an image caption using an artificial intelligence (AI) or deep learning model, another annotation, or a combination of multiple methods.Furthermore, at least one of the user name, machine learning model, and other annotation data is recorded. When a combination of technologies is used, that information is also included in the image-annotation bundle. The image-annotation arrow [

[0302] ] can be specified by the X and Y coordinates of its start and end points, and information specifying whether the arrowhead is at the start point or the end point. In this case as well, the coordinates are specified based on the lower left corner of the image. Each element included in the annotation atom specifies, in addition to the relative coordinate information, the generation mechanism. The generation mechanism specifies whether the element was generated by the user, a machine learning model, or a simple algorithm. Also, at least one of the user name, machine learning model name, and algorithm name is recorded. Furthermore, an oval can also be used. The ellipse is specified by the X and Y coordinates of the center relative to the lower left corner of the image, the width, height, and rotation of the ellipse. Furthermore, the generation mechanism is recorded together with at least one of the user name, machine learning model name, and algorithm name.

[0175] In the context of an exemplary embodiment, the embedded image-annotation bundle includes not only all the information within the image-annotation bundle, but also the URL of the web page or document where the image is located. It is desirable to interpret the description written about the image-annotation bundle by replacing the embedded image-annotation subject [

[0383] ] with the image-annotation subject [

[0300] ], the bounding box of the embedded image-annotation subject [

[0384] ] with the bounding box of the image-annotation subject [

[0301] ], the embedded image-annotation arrow [

[0385] ] with the image-annotation arrow [

[0302] ], and the embedded image-annotation comment [

[0386] ] within the box with the image-annotation comment within the box. FIGS. FIG. 36 and FIG. 47 need to be examined in detail. Also, in the case of a document being a pdf or a complex web page, information for finding the image, such as the page of the pdf document or the DOM locator of the image, is also stored.

[0176] In the context of an exemplary embodiment, a video annotation bundle includes one or more video annotation atoms. A video annotation atom is a unit that can be meaningfully used by itself to annotate a video. For example, a video annotation atom can include a bounding box of a video annotation subject

[0322] , a bounding box motion path of a video annotation subject

[0332] , a video annotation comment input box

[0324] , a video annotation comment input box motion path

[0334] , a video annotation arrow

[0323] , and a video annotation arrow motion path

[0333] as shown in the exemplary embodiment depicted in FIG. 41. However, during video playback, it is also possible to place the video annotation comment in box

[0324] near the video annotation subject

[0321] and use box

[0324] and the video annotation comment input box motion path

[0334] alone. Therefore, the video annotation comment in box

[0324] and the motion path of the video annotation comment input box

[0334] can be a video annotation atom by itself. The video annotation bundle includes, in addition to the video annotation atoms, frame information regarding the video to be annotated. The frame information includes a video size specified by the number of pixels in the x-axis direction and the number of pixels in the y-axis direction. This includes the area of the video

[0319] and should not include the playback area of the video annotation

[0320] .When the video size is specified, based on the frame information, video annotation atom composition elements such as the bounding box of the video annotation subject [

[0322] ], the bounding box motion path of the video annotation subject [

[0332] ], the video annotation comment input box [

[0324] ], the video annotation comment input box motion path [

[0334] ], the video annotation arrow [

[0323] ], and the video annotation arrow motion path [

[0333] ] are specified in relative coordinates. When the bounding box of the video annotation subject [

[0322] ] is specified by the X and Y coordinates of its center, width, and length, the coordinates are specified as relative coordinates from the lower left corner of the video. The bounding box motion path of the video annotation subject [

[0332] ] needs to specify the temporal evolution of the bounding box of the video annotation subject [

[0322] ]. This specifies the temporal changes in the X and Y coordinates of its center, its width, and its length. This specification is done by specifying the X and Y coordinates based on the lower left corner of the video at discrete time points within the video annotation playback range, together with an interpolation mechanism between these discrete time steps. Since the relative x coordinate, relative y coordinate, width, and length are real numbers, it is sufficient to remember the values at discrete time points and the method of specifying the interpolation method is the same for all four values. Interpolation methods such as linear interpolation, spline curves, and Bézier curves can be considered. The video annotation comment of the box [

[0324] ] and the motion path of the video annotation comment input box [

[0334] ] can also be specified in a similar manner, and the markup information of the annotation comment included in the video annotation comment of the box [

[0324] ] can also be specified. This markup information includes font size and formatting information of the text. Furthermore, information regarding the text creation mechanism is included in the video annotation bundle. The creation mechanism can be what the user wrote, video captioning using AI or deep learning models, another annotation, or a combination of multiple methods. Furthermore, at least one of the user name, machine learning model, and other annotation data is recorded.When a combination of technologies is used, that information is also included in the video annotation bundle. The video annotation arrow

[0323] and the video annotation arrow motion path

[0333] can be specified by the relative X and Y coordinates of their start and end points and information specifying whether the arrowhead is at the start point or the end point. The specification of the video annotation arrow motion path

[0333] is done by specifying the relative X and Y coordinates of the start and end points at discrete time points within the video annotation playback range and the interpolation mechanism between these discrete time steps. Here too, the coordinates are specified with reference to the lower left corner of the video. Each element included in the annotation atom specifies, in addition to the relative coordinate information, the generation mechanism. The generation mechanism specifies whether the element was created by the user, an AI or deep learning model, another annotation, or a combination of technologies. Furthermore, at least one of the user name, AI or deep learning model name, and algorithm name is recorded. Furthermore, ovals can also be used. The ellipse is specified by the X and Y coordinates of the center relative to the lower left corner of the video, the width, height, and rotation of the ellipse. Furthermore, the generation mechanism is recorded together with at least one of the user name, AI or deep learning model name, and algorithm name. The video annotation symbol information can also be included in the video annotation bundle.

[0177] In the context of an exemplary embodiment, an embedded video annotation bundle includes not only all the information within the video annotation bundle, but also the URL of the web page or document where the video is located. The description written for the video annotation bundle should be interpreted as substituting, respectively, the embedded video annotation subject

[0388] for the video annotation subject

[0321] , the bounding box

[0389] of the embedded video annotation subject for the bounding box

[0322] of the video annotation subject, the embedded video annotation arrow

[0390] for the video annotation arrow

[0323] , the video annotation comment

[0391] within the embedded box for the video annotation comment

[0324] within the box, the bounding box motion path of the embedded video annotation subject for the bounding box motion path

[0332] of the video annotation subject, the motion path of the embedded video annotation arrow for the motion path

[0333] of the video annotation arrow, the motion path of the video annotation comment within the embedded box for the motion path

[0334] of the video annotation comment input box, and the embedded video annotation symbol for the video annotation symbol. Details need to be examined with reference to the form of the figure and FIG. 48. When the document is a pdf or a complex web page, information for finding the video, such as the page of the pdf document or the DOM locator of the video, is also stored.

[0178] In the context of an exemplary embodiment, a recording annotation bundle includes one or more recording annotation atoms. A recording annotation atom is a unit that can be meaningfully used to annotate a recording by itself. For example, a recording annotation atom can include a recording annotation playback range, a recording annotation symbol, and a recording annotation comment. However, it is also possible to use only the recording annotation playback range and the recording annotation comment. Therefore, the recording annotation playback range and the recording annotation comment can also constitute a recording annotation atom by themselves. A recording annotation comment can be associated with markup information of the recording annotation comment. This markup information includes, in addition to text format information, font size. Furthermore, information regarding the creation mechanism of the text is in the recording annotation bundle. The creation mechanism can be what the user wrote, speech recognition by an AI or a deep learning model, copying of other annotations, or a combination of multiple methods. Furthermore, the creation mechanism is recorded together with at least one of the user name, the AI or deep learning model name, and other annotation references. When a combination of technologies is used, one or more types of source names are recorded. Also, the playback range of the recording annotation can be determined based on a sound search. Therefore, it is desirable that the generation mechanism of the recording playback range is also recorded together with at least one of the user name, the AI or deep learning model name, and other annotation comments. When multiple methods are combined and used, multiple generation mechanisms and sound source information are recorded. Furthermore, recording frame information is recorded to facilitate placing the recording annotation playback range in the recording annotation playback control.

[0179] In the context of an exemplary embodiment, an embedded recording annotation bundle includes not only all the information within the recording annotation bundle, but also the URL of the web page or document where the recording is located. The description written about the recording annotation bundle needs to be interpreted by replacing the embedded recording annotation playback range

[0393] with the recording playback range, the embedded recording annotation symbol with the recording annotation symbol, and the embedded recording annotation comment with the recording annotation comment. Further, in order to make it easier to place the embedded recording playback range

[0393] in the embedded recording playback control, it should be understood that the embedded recording frame information is recorded in the embedded recording annotation bundle. Details need to be examined with reference to the forms of FIGS. 44 and 49. When the document is a complex web page, information for identifying the position of the recorded sound, such as the DOM locator of the recorded sound, is also stored.

[0180] In the context of an exemplary embodiment, an annotation group bundle includes references to all the annotations included in that particular annotation group. Further, information regarding whether the annotations conflict is stored together with metadata indicating when the annotation group was created and metadata indicating whether the conflict detection was performed by the user or AI or a deep learning model. At least one of the user and AI or the deep learning model that performed the conflict detection is also stored in the annotation group bundle.

[0181] In the context of an exemplary embodiment, a derived annotation group bundle includes references to all the annotations included in that particular derived annotation group. Further, information regarding whether the annotations conflict is stored along with metadata indicating when the annotation group was created and whether the conflict detection was performed by a user or an AI or a deep learning model. At least one of the user and the AI or the deep learning model that performed the conflict detection is also stored in the derived annotation group bundle. The derived annotation group bundle includes one derived annotation along with references to all the annotations included in that particular annotation group. The annotation connection arrows can be determined from the annotation references present within the derived annotation group bundle. In this exemplary embodiment, only one derived annotation exists, which provides all the information for drawing the annotation connection arrows. Further, the derived annotation comment is stored in the derived annotation group bundle. Markup information for the derived annotation comment is also stored. This markup information includes the font size and formatting information of the text. Further, information regarding the mechanism for creating the text is stored in the derived annotation group bundle. The creation mechanism can be something written by the user, something generated by an AI or a deep learning model, another annotation that was copied, or a combination of multiple techniques. Further, at least one of the user name, the machine learning model, and other annotation data is recorded. If a combination of techniques was used, that information is also included in the derived annotation group bundle.

[0182] In the context of an exemplary embodiment, an annotation group bundle with derived annotations includes all derived annotations and references to annotations that exist in that particular annotation group with derived annotations. Further, information regarding whether an annotation conflicts is stored along with metadata indicating when the annotation group with derived annotations was created, and whether the conflict detection was performed by a user or an AI or a deep learning model. At least one of the user and the AI or the deep learning model that performed the conflict detection is also stored in the annotation group bundle.

[0183] Next, several types of display bundles and item boxes will be described. These descriptions are to be construed in the context of the exemplary embodiments. In the context of the exemplary embodiments, the display bundles described are an image-annotation display bundle, a video-annotation display bundle, a recorded audio annotation display bundle, an embedded image-annotation display bundle, an embedded video-annotation display bundle, an embedded recorded audio annotation display bundle, an annotation-group display bundle, a derived annotation-group display bundle, and an annotation-group display bundle with derived annotations. In the context of the exemplary embodiments, the item boxes described are an image-annotation item box, a video-annotation item box, a recorded audio annotation item box, an embedded image-annotation item box, an embedded video-annotation item box, an embedded recorded audio item box, an annotation-group item box, a derived annotation-group item box, and an annotation-group item box with derived annotations. One purpose of the display bundles in the exemplary embodiments is for the web browser to provide data for displaying the corresponding item box associated with that type of annotation or media item. Thus, the item boxes are described in relation to the annotation item box

[36] presented in the exemplary embodiment depicted in FIG. 9. In addition to the numbered items within the annotation item box, the annotation item box also includes item box annotation emphasis text, item box annotation symbols, item box annotation comments, and item box URLs. When various item boxes are described in the context of the exemplary embodiments, data structures corresponding to the various display bundles can be illustrated. Further, referring to the displayed bundles may be useful for purposes of explanation. The displayed bundles are the visual output created using the display bundles. This visual output is included within the item boxes.In a context like a notepad, the displayed bundle can exist even if the entire item box does not exist.

[0184] In an exemplary embodiment, the image-annotation item box, as shown in the exemplary embodiment of FIG. 9, will include a separate context-variant annotation control bar

[41] and a separate context reputation selector

[42] , like the annotation item box

[36] . Instead of a screenshot

[37] with marked text, the image-annotation item box includes a thumbnail of the image along with an image-annotation displayed on top of the thumbnail. Instead of separate context annotation emphasized text

[38] , a displayed annotation symbol

[39] for a separate context question, and separate context annotation comments

[40] , an image-annotation comment is displayed in the image-annotation item box. The URL is not displayed in the image-annotation item box but is displayed in the embedded image-annotation item box. To enable a web browser to display these image-annotation item boxes, the image-annotation display bundle needs to appropriately resize and overlay objects such as the subject boundary box of the image-annotation

[0301] , the image-annotation comment in the box

[0303] , the image-annotation arrow

[0302] , etc., and the marked thumbnail image of the image and the image displayed thereon. Further, the image-annotation display bundle needs to include an image-annotation comment, a user name, and reputation information.

[0185] In an exemplary embodiment, the embedded image - annotation item box will include a separate context - variant annotation control bar

[41] and a separate context reputation selector

[42] , such as the annotation item box

[36] . Instead of a screenshot containing marked text, the embedded image - annotation item box includes a thumbnail of the embedded image and image - annotations displayed over the thumbnail image. Instead of the separate context - annotation highlighted text

[38] , the separately - context question's displayed annotation symbol

[39] , and the separate context - annotation comment

[40] , the embedded image - annotation item box has an embedded image - annotation comment. Its URL corresponds to the URL containing the embedded image to which the image - annotation was applied. For the web browser to be able to display these embedded image - annotation item boxes, the embedded image - annotation display bundle should include a marked thumbnail image of the embedded image with the image - annotation, where objects such as the bounding box of the embedded image - annotation subject

[0384] , the embedded image - annotation comment within the box

[0386] , the embedded image - annotation arrow

[0385] , etc. are appropriately resized and overlaid. Further, the embedded image - annotation display bundle should include the embedded image - annotation comment, the user name, and reputation information.

[0186] In an exemplary embodiment, the video annotation item box will include a separate context variant annotation control bar

[41] and a separate context reputation selector

[42] , such as the annotation item box

[36] , as shown in the exemplary embodiment of FIG. 9. Instead of a screenshot

[37] with marked text, the video annotation item box includes a thumbnail of a video frame at a selected point in time, along with a video annotation displayed on the thumbnail of the video frame at the same selected point in time. Further, the play button can play the thumbnail video and a thumbnail video showing a part of the video annotation. Instead of the highlighted text

[38] of a separate context annotation, a video sound summarization text, which is machine-generated text that summarizes the sound in the video, will be displayed. Next, a video annotation comment is displayed in the video annotation item box, and a video annotation symbol that determines the semantic relationship of the annotation between the video sound summary text and the video annotation comment is also displayed in the video annotation item box. The URL does not exist in the video annotation item box but exists in the embedded video annotation item box. In order for a web browser to be able to display these video annotation item boxes, the video annotation display bundle needs to include a marked thumbnail image of the video frame and the selected-time video on it, with objects such as the bounding box of the video annotation subject

[0322] , the video annotation comment

[0324] in the box, and the video annotation arrow

[0323] appropriately resized and overlaid. When the user presses the play button, the video annotation display bundle needs to include a thumbnail video with the video annotation overlaid.Furthermore, the video annotation display bundle must include video audio summary text, video annotation symbols, video annotation comments, user names, and reputation information. Additionally, users will be able to modify mechanically generated video annotations.

[0187] In an exemplary embodiment, the embedded video annotation item box will include a separate context - variant annotation control bar

[41] and a separate context reputation selector

[42] , such as the annotation item box

[36] , as shown in the exemplary embodiment of FIG. 9. Instead of a screenshot

[37] with marked text, the embedded video annotation item box includes a thumbnail of an embedded video frame at a selected point in time, along with an embedded video annotation displayed on the thumbnail of the embedded video frame at the same selected point in time. Further, the play button can be enabled to play the thumbnail video and a thumbnail video showing a portion of the video annotation. Instead of a separate context annotation emphasis text

[38] , it becomes an embedded video sound summary text which is part of the machine - generated text that summarizes the sound within the video. Next, the embedded video annotation item box has an embedded video annotation comment, and the embedded video annotation item box also has an embedded video annotation symbol that determines the semantic relationship of the annotation between the embedded video sound summary text and the embedded video annotation comment. The URL of the web page in which the video is embedded is displayed in the item box of the embedded video annotation. In order for a web browser to be able to display these embedded video annotation item boxes, the embedded video annotation display bundle needs to include an object such as a bounding box of the embedded video annotation subject

[0389] , an embedded video annotation comment within the box

[0391] , an embedded video annotation arrow

[0390] , etc., which are appropriately resized and overlaid, with the embedded video annotation displayed thereon, and a marked thumbnail image of the frame of the embedded video at a selected point in time.When the user presses the play button, the embedded video annotation display bundle shall include a thumbnail video with the embedded video annotation overlaid. Further, the embedded video annotation display bundle shall include the embedded video audio summary text, the embedded video annotation symbol, the embedded video annotation comment, the user name, and the reputation information. Further, the user shall be able to modify aspects of the mechanically generated video annotation.

[0188] In an exemplary embodiment, the recorded annotation item box will include a separate context - variant annotation control bar

[41] and a separate context reputation selector

[42] , like the annotation item box

[36] shown in the exemplary embodiment of FIG. 9. Instead of a screenshot

[37] with marked text, the recorded annotation item box will include a thumbnail representing the recording, called a thumbnail related recording image. This is done first by creating a recorded summary text that converts the recording into a text representation. This is done using speech understanding techniques such as speech recognition. Once the text representation is available, an AI or deep - learning model is used to find an image related to the recorded summary text. This image will be called the related recording image. The thumbnail representation of the related recording image is the thumbnail related recording image. Instead of the separate context annotation highlighted text

[38] , it is the recorded summary text. Next, the recorded annotation item box will have recorded annotation comments, and the recorded annotation item box will also have a recorded annotation symbol that determines the semantic relationship of the annotation between the recorded summary text and the recorded annotation comments. URLs do not exist in the recorded annotation item box but exist in the embedded recorded annotation item box. The recorded annotation display bundle needs to include the recording image associated with the thumbnail, the recorded summary text, the recorded annotation comments, the recorded annotation symbol, the user name, and the reputation information so that a web browser can display these recorded annotation item boxes. Further, the user will be able to modify the mechanically - generated parts of the recorded annotation.

[0189] In an exemplary embodiment, the embedded recording annotation item box will include a separate context - variant annotation control bar

[41] and a separate context reputation selector

[42] , such as the annotation item box

[36] . Instead of a screenshot

[37] containing marked text, the embedded recording annotation item box will include a thumbnail representing the embedded recording, called a thumbnail - associated embedded recording image. This is done by first creating an embedded recording summary text that converts the embedded recording into a text representation. This is done using speech understanding techniques such as speech recognition. Once the text representation is available, models of AI or deep learning can be used to find an image related to the embedded recording summary text. This image will be called the associated embedded recording image. The thumbnail representation of the associated embedded recording image will be the thumbnail - associated embedded recording image. Instead of the separate context - annotation highlighted text

[38] , it will be the embedded recording summary text. Next, the embedded recording annotation item box will have an embedded recording annotation comment, and the embedded recording annotation item box will also have an embedded recording annotation symbol that determines the semantic relationship of the annotation between the embedded recording summary text and the embedded recording annotation comment. The URL of the website where the embedded recording was found will be displayed in the embedded recording annotation item box. The embedded recording annotation display bundle needs to include the embedded recording image associated with the thumbnail, the embedded recording summary text, the embedded recording annotation comment, the embedded recording annotation symbol, the username, and the reputation information so that a web browser can display these embedded recording annotation item boxes. Further, the user will be able to modify the mechanically - generated parts of the recording annotation.

[0190] In an exemplary embodiment, the annotation group item box will include a separate context variant annotation control bar

[41] and a separate context reputation selector

[42] , such as the annotation item box

[36] shown in the exemplary embodiment of FIG. 9. Instead of a screenshot

[37] with marked text, the annotation group item box will include a thumbnail representing the annotation group. In the case of image annotation, this means a thumbnail image of an image having all the image annotations included in the annotation group. The image annotation and the annotation group include annotations of the embedded image. In the case of video annotation, it means a thumbnail image of the video at a certain point in time and all the video annotations included in the annotation group at the same point in time. Further, in the thumbnail video, a thumbnail video in which all the video annotations of the annotation group are displayed for the entire thumbnail video will be included. The video annotation and the annotation group include video annotations of the embedded video. For recorded audio annotations, for all recorded audio annotations within the annotation group, it means a thumbnail image representing the sound for all recorded playback ranges. Such an image will be based on the recorded summary text for all recorded playback ranges. Instead of the highlighted text

[38] of the separate context annotation, a list of the annotations within that particular annotation group is displayed. Instead of the separate context annotation comment,

[40] will be the user-generated comment regarding that particular annotation group. Further, if the machine-generated image and the default text are not appropriate, these elements can be changed by the user. In another embodiment, the list of annotations of the annotation group can be replaced by a more appropriate summary text generated by a human.A URL does not exist in an annotation group item box without embedded annotations, but will exist in an annotation group item box that contains embedded annotations. Embedded annotation group annotation symbols can be used to indicate the semantic relationship between the source material and the annotations within the annotation group. The number of annotations within the annotation group is also displayed in the annotation group item box. The annotation group display bundle should include a related thumbnail image or thumbnail video, a list of references to annotation objects, related summary text, related annotation comments, related annotation group annotation symbols, user names, and reputation information so that a web browser can display these annotation group annotation item boxes. Additionally, the user will be able to modify the mechanically generated aspects of the annotation group.

[0191] In an exemplary embodiment, a derived annotation group item box will include a separate context variant annotation control bar

[41] and a separate context reputation selector

[42] , like the annotation item box

[36] , as shown in the exemplary embodiment of FIG. 9. Instead of a screenshot

[37] with marked text, the derived annotation group item box will include a thumbnail representing the derived annotation group. In the case of image annotation, this means a thumbnail image of an image having all the image annotations included in the derived annotation group. The image annotation and the derived annotation group include image annotations of embedded images. In the case of video annotation, it means a thumbnail image of the video at a certain point in time and all the video annotations included in the derived annotation group at the same point in time. Further, the thumbnail video will include a thumbnail video in which all the video annotations included in the derived annotation group are displayed throughout the thumbnail video. The video annotation and the derived annotation group include video annotations of embedded videos. In the case of audio recording annotation, it means a thumbnail image representing the sound for all the audio playback ranges of all the audio recording annotations of the derived annotation group. Such an image will be based on the audio recording summary text for all the audio playback ranges. Instead of the highlighted text

[38] of the separate context annotation, a list of the annotations within that particular derived annotation group is displayed. Instead of the separate context annotation comment,

[40] will be the user-generated comment regarding that particular derived annotation group. Further, if the machine-generated images and default text are not appropriate, these elements can be modified by the user. In an alternative embodiment, the list of annotations of the derived annotation group can be replaced with a more appropriate summary text generated by a human.A URL does not exist in a derived annotation group item box without embedded annotations, but exists in a derived annotation group item box that includes embedded annotations. A derived annotation group annotation symbol may be used to indicate the semantic relationship between the source material and the annotations within the annotation group. In order for a web browser to be able to display these annotation group annotation item boxes, an annotation group display bundle should include the associated thumbnail image or thumbnail video, a list of references to annotation objects, the associated summary text, the associated derived annotation comments, the associated derived annotation group annotation symbols, the user name, and reputation information. Further, the user will be able to modify the mechanically generated parts of the derived annotation group.

[0192] At this point, the annotation group and the derived annotation group may seem similar within the context of the exemplary embodiments described so far. However, the derived annotation group, the derived annotation group item box, and the derived annotation group display bundle, when used, will be handled differently from the annotation group, the annotation group item box, and the annotation group display bundle. The annotation group shows the links between annotations more closely, and the derived annotation group can be incorporated into the annotation group. The effects of these differences can be seen by examining FIGS. 44 and 45.

[0193] In an exemplary embodiment, an annotation group with derived annotations will include a separate context variant annotation control bar

[41] such as an annotation item box

[36] and a separate context reputation selector

[42] , as shown in the exemplary embodiment of FIG. 9. The annotation group item box with derived annotations and the corresponding annotation group display bundle with derived annotations are similar to the annotation group item box and the annotation group display bundle, except that information about derived annotations such as its constituent annotations and derived annotation comments exists for each derived annotation group. The existence of this information enables the display of derived annotations as depicted in FIGS. 44 and 45. Further, the item box of the annotation group with derived annotations displays the number of annotations within the annotation group with derived annotations and the number of derived annotation groups. All of this information is reflected in the annotation group display bundle with derived annotations. The item box of the annotation group with derived annotations displays a separate context variant annotation control bar

[41] and a separate context reputation selector

[42] , similar to the annotation item box

[36] . Instead of a screenshot containing marked text, the item box of the annotation group with derived annotations displays a representative thumbnail of the annotation group with derived annotations

[37] . In the case of image annotations, it means a thumbnail image of an image containing all the image annotations included in the annotation group of the derived annotation. Since the derived annotation group changes the image by the annotations included in the derived annotation group, what is important is the set of all the annotations included in the derived annotation group. Image annotations and annotation groups include annotations of embedded images.In the case of video annotation, it means all video annotations included in an annotation group having a thumbnail image of the video at a certain point in time and the derived annotation at the same point in time. Further, the thumbnail video will include a thumbnail video in which all video annotations included in the annotation group having the derived annotation are displayed throughout the thumbnail video. Here too, what is important is the set of all annotations included in the annotation group having the derived annotation. Video annotations and annotation groups include video annotations of the embedded video. In the case of recording annotation, it means a thumbnail image representing the sound for all recording playback ranges of all recording annotations included in the annotation group having the derived annotation. Such an image will be based on the recording summary text for all recording playback ranges in the set of all annotations of the annotation group having the derived annotation. Instead of the highlighted text

[38] of the separate context annotation, the list of annotations of that specific annotation group becomes the derived annotation. Instead of the separate context annotation, the comment

[40] becomes the user-generated comment regarding that specific annotation group. Further, if the machine-generated image and default text are not appropriate, these elements can be modified by the user. In an alternative embodiment, the list of annotations of the annotation group having the derived annotation can be replaced by a more appropriate summary text generated by a human. The URL does not exist in the annotation group item box with the derived annotation having no embedded annotation, but will exist in the annotation group item box with the derived annotation including the embedded annotation. An embedded annotation group including the derived annotation symbol can be used to show the semantic relationship between the source material and the annotations in the annotation group including the derived annotation.In the item box of an annotation group with derived annotations, the number of annotations in the annotation group with derived annotations and the number of derived annotation groups are displayed. In order for a web browser to display the item boxes of these annotation groups with derived annotations, the annotation group display bundle with derived annotations must include related thumbnail images or thumbnail videos, a reference list to annotation objects, related summary text, related annotation comments, related derived annotation group symbols, user names, and reputation information. Furthermore, the user will be able to modify the mechanically generated aspects of the annotation groups with derived annotations.

[0194] While various types of data element bundles, display bundles, and item boxes have been described within the context of the exemplary embodiments, the changes that need to be made to FIGS. 14, 15, 16, 17, and 18 will be described. These changes are in line with the features and capabilities shown and described in the described explanations and the exemplary embodiments depicted in FIGS. 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, and 49, and are to adapt these sequence diagrams and flowcharts to the implementation methods and systems related thereto. The sequence diagram of FIG. 14 is applied with one of the data element bundles described above, along with its corresponding display bundle and its corresponding displayed bundle, which are different for different types of annotations when different types of annotations are implemented. The display bundle occurs when FIG. 14 uses the annotation display bundle, and the displayed bundle occurs when FIG. 14 uses the displayed annotation bundle. The sequence diagram of FIG. 15 is also applied with the display bundle and the displayed bundle described above. These displayed ones occur where the displayed bundle of annotations occurs in FIG. 15, and the displayed bundle occurs where FIG. 15 uses the displayed annotation bundle. The display bundle has data for displaying either its corresponding item box or its corresponding displayed bundle. Depending on the exemplary embodiment, either the item box or the displayed bundle can be displayed on the user interface. Typically, the item box is a bundle displayed with a separate context-variant annotation control bar

[41] and a separate context reputation selector

[42] . The separate context-variant annotation control bar

[41] and the separate context reputation selector

[42] are shown for the annotation item box

[36] in the exemplary embodiment shown in FIG. 9.The sequence diagram of FIG. 16 is applied by replacing the annotation symbol display bundle with an appropriate embedded annotation display bundle and replacing the closed displayed annotation symbol with a visible element of an appropriate embedded annotation. The embedded annotation display bundle and the visible element of the embedded annotation are as described above. FIGS. 17 and 18 are applied as they are when the bundle of data elements and the display bundle are selected from the types previously defined in this specification.

[0195] In an exemplary embodiment, the item bin

[44] within the annotation composition context area

[43] of FIG. 10 can include an annotation item box, an embedded image - annotation item box, an embedded video - annotation item box, an embedded recording item box, an annotation - group item box, a derived - annotation - group item box, a derived - annotation - group item box, and a memo pad. The memo pad can also be displayed as a memo - pad item box. All of these types of item boxes are equipped with a separate context - variant annotation control bar

[41] , as depicted for the annotation item box

[36] . Other elements of the annotation item box

[36] change for other types of item boxes, but the separate context - variant annotation control bar

[41] and the displayed form factor of the item box do not change. This allows these new types of item boxes to be integrated into the item bin

[44] . All of these item types are created on an external site and are displayed in the item bin as soon as the user creates them

[44] . This is easily achieved by using a web socket that pushes information to all connected web browsers from an annotation server

[61] that should receive a display bundle based on access control and use cases. The item box can be moved from the item bin

[44] to a writable collection represented by the displayed writable collection

[50] , as depicted in the exemplary embodiment of FIG. 11. Further, the annotation item box, the embedded image - annotation item box, the embedded video - annotation item box, and the embedded sound recording item box can be added to the active notepad

[0159] using the add icon

[0158] button in the notepad in the context - variant annotation control bar

[34] , as depicted in the exemplary embodiment of FIG. 23.The icon to be added to the notepad

[0158] button is not displayed in the context-variant annotation control bar

[34] labeled in all the figures. However, such a button can be added to the context-variant annotation control bar

[34] within the exemplary embodiments. Other item types can also be added to the active notepad

[0159] using a trigger for adding to the notepad. In all cases, clicking on the item box within the active notepad

[0159] should open the relevant item in a browser tab, scroll to the position of the annotation, and display the annotation. All different types of item boxes can be displayed within the collection. The collection shown in the exemplary embodiment depicted in Figure 31 shows only the annotation item box, but all types of item boxes can be present within the collection. All comments regarding the stack set of the displayed annotation items apply to the stack sets of various types of items. Additionally, comments regarding the window functionality also apply. The set of stacked items can be grouped based on the URL or based on the annotated image item, video item, recording item, or document item. Further, the type of item included in the collection does not change the other properties of the collection. The collection represented by the displayed collection can be displayed on a page similar to that shown in the exemplary embodiment of Figure 32.

[0196] As described above, filters such as the search filter shown in the exemplary embodiment of FIG. 10 are used to receive input from a user. In the case of the search filter

[46] of the item bin

[44] , it is a search phrase. These inputs are converted into parameter values that are sent to the annotation server

[61] . These parameter values are used at the annotation server

[61] to enter parameters into the request query used in the electronic database

[62] . The request query includes parameter values and a page number. The request query including parameter values is used to specify a set of items and the order of those items. These items include web page annotations, embedded image - annotations, embedded video - annotations, embedded recording annotations, image - annotations, video - annotations, recording annotations, annotation - groups, derived annotation - groups, annotation - groups with derived annotations, PDF document annotations, PDF documents, images, videos, links, normal comments, etc. The filter can also be restricted to a specific item type. The page number specifies a specific page. Further, the request query can include the specified number of items to be sent to the web browser running on the client electronic device. This request query includes parameters of filter values such as search phrases, an exclusion list of items to be excluded from the list of specified items, and a sequential ordering for the list of those items. As explained earlier, this sequential ordering provides information for paging in the exemplary embodiments depicted in FIGS. 10 and 31. Collection - filters and collection paging operate as explained earlier for the description of the exemplary embodiments depicted in FIGS. 10 and 32. The same applies to the pinning function.

[0197] The inbox described in the exemplary embodiment depicted in FIG. 33 would function in a similar manner if the collection included different item types. The difference would be that the user would be able to discuss more item type details within the collection message. Additionally, functions can be built into the message chains specialized for each type of item. And functions can be implemented to change the objects that are the compositional elements of the annotation, approve annotation changes, or fork the annotations added to the current collection. Also, when discussing image annotations, the discussion can proceed smoothly by using selectors for each object of the image annotation. Furthermore, selectors for other item types such as video annotations and audio recordings can be incorporated into the message chain function. With such functions, the subject bounding box

[0322] of the video annotation, the video annotation comment

[0324] within the box, or the annotation arrow

[0323] can be changed, and also the motion path

[0332] of the bounding box of the video annotation subject, the video annotation arrow motion path

[0333] , and the motion path

[0334] of the video annotation comment input box can be changed. After a change is proposed in the message chain, the owner of the item can approve the change, and the admin can approve forking the item into a new item in the context of the current collection.

[0198] The exemplary embodiments depicted in FIGS. 28 and 29 are applicable even when there are multiple item types. In this situation, the annotation server

[61] and the electronic database

[62] will process these different types of items. In an exemplary embodiment, the annotation server

[61] may have a REST API for processing different item types, and the electronic database

[62] has a schema for processing different data elements within various types of bundles. The hardware requirements are affected as follows. If the video

[0319] is not stored locally and is referenced via a URL, the presence of the video

[0319] will not increase the storage requirements as much as if the video

[0319] were stored locally. If stored locally or via a content delivery network

[63] , the video

[0319] will have a significant impact on the storage requirements. The presence of the recorded playback

[0348] may create a need to use a deep learning model to process speech and speech recognition. The presence of the video

[0319] will increase the system's computational processing requirements because it generally has greater computational processing requirements than other media types.

[0199] In an exemplary embodiment, to provide additional annotation functionality to the user, machine learning and deep learning models can be deployed and configured via a web browser extension of an annotatable web browser. If the annotatable web browser is not constructed through the use of a web browser extension, it is also possible to directly process the deployment and configuration through the annotatable web browser. This functionality can be made available to the user through a panel or floating toolbar located somewhere on the web page. This functionality can also be exposed through menu options or a fixed browser toolbar. The floating toolbar can be attached to the page assignment or placed within the side panel of the web browser. In the case of images and embedded images, this panel or floating toolbar can be made available with buttons for tasks such as recognizing objects within the image, recognizing text within the image, setting the recognition characteristics of object and text detectors, and setting the characteristics of annotation elements. Annotation elements include a bounding box for the annotation subject, an annotation arrow, and an annotation comment box. When the object recognition button is pressed, a dashed bounding box of the annotation subject candidate is drawn around the annotation subject candidate. The user can select this bounding box and write an annotation comment in the annotation comment input box. Depending on the settings, an annotation arrow may be automatically created. The user can also be provided with the option to select the bounding boxes of multiple annotation subjects, resulting in one annotation arrow being created for each bounding box of the annotation subject, with one end of each annotation arrow pointing to the bounding box of the annotation subject and the other end pointing to the annotation comment input box or the annotation comment box depending on whether an annotation comment has been written. The placement of the arrow is a configurable property.Other configurable characteristics are the visibility characteristics of the annotation subject's bounding box, annotation arrow, and annotation comment box. The visibility characteristics are visible, mouse over, and invisible. Visible is for annotation elements that are always displayed. Mouse over corresponds to annotation elements that are visualized on mouse over of the target area. And invisible corresponds to annotation elements that are always invisible. The mouse over target area can be at least one of the visible annotation subject or annotation element. In an alternative embodiment, an annotation subject outline surrounding a closed area including the pixels of the annotation subject is used. In contrast to the bounding box of the annotation subject, these annotation subject outlines can be used. Further, the object outline can be a mouse over element as opposed to a visible element, and this visible characteristic can be controlled through configuration settings. It is also possible to have a configuration setting that allows the user to select which of the bounding box of the annotation subject and the annotation subject outline to use. Determining the bounding box of the annotation subject or the annotation subject outline depends on a machine learning model or deep learning model that can recognize the object and determine the bounding box of the annotation subject or the outline of the annotation subject. The deep learning model can be executed on a web browser operating on the client electronic device by using the CPU or GPU of the client electronic device. The GPU can be used through the use of the WebGL interface exposed by the latest web browsers, and as a result, the GPU of the client electronic device can be used.The bounding box of the annotation subject and the outline of the annotation subject can then be determined through the use of an annotation object detector using a region-based convolutional neural network (R-CNN), or through object segmentation using a region-based convolutional neural network (R-CNN). In addition to R-CNN-based models, alternative embodiments can employ other machine learning or deep learning methods. Since different deep learning models have different recognition characteristics, the user can input the desired recognition characteristics of the object detector and the text detector. Based on those characteristics, different machine learning or deep learning models can be selected. Also, a CNN (Convolution Neural Network) that recognizes characters and numbers can be used to recognize characters in an image. Furthermore, in the setting options, it is possible to select whether to group individual characters into words or whether to group individual words into n-grams. Additionally, by inputting words or tags in the setting options for recognition characteristics, it is also possible to limit recognition to specific types of objects input by the user. Also, buttons for setting the motion path of annotation elements are added to handle videos and embedded videos. Also, the technology described with images can also be implemented for individual frames of a video. Furthermore, the deep learning model can track the annotation subject over time and ensure that the bounding box of the annotation subject and the outline of the annotation subject move with the annotation subject. This can be done by using individual frame analysis at discrete points and an interpolation method between the discrete points. Thereby, the motion paths of the bounding box of the annotation subject and the annotation subject outline are obtained. Also, the contour movement of the annotation subject outline is calculated.The evolution settings of these motion paths and annotation outlines are available to the user through a panel, menu option, browser toolbar, or additional button on a floating toolbar that handles the motion path setting options for annotation elements. Machine learning and deep learning models for audio and speech recognition in videos for annotation purposes can be handled by another setting option in the panel, menu option, browser toolbar, or floating toolbar.

[0200] As used herein, the word "exemplary" means "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations, nor is it intended to be construed as a model that must be literally replicated. Accordingly, this specification is not limiting of the invention and is to be regarded as exemplary.

[0201] In particular, conditional language such as "can," "could," "might," or "may" generally implies that a particular embodiment includes, and other embodiments do not include, a particular feature, element, and / or step, unless otherwise specified or understood within the context in which it is used. Thus, such conditional language is not generally intended to mean that a feature, element, and / or step is in any way required for one or more embodiments, or that one or more embodiments necessarily include logic for determining, with user input or prompting, whether these features, elements, and / or steps are included in, or are to be performed in, any particular embodiment. The headings used herein are for the convenience of the reader and are not intended to limit the invention or the scope of the claims.

[0202] As used herein, the phrase "at least one" of a list of items refers to any combination of those items including a single member. By way of example, "at least one of A, B, or C" is intended to cover A, B, C, A and B, A and C, B and C, and A, B, and C. Connective language such as the phrase "at least one of X, Y, and Z" is generally understood within the context as being used to convey that an item, term, etc. may be at least one of X, Y, or Z, unless otherwise specified. Thus, such connective language is not intended to require that a particular embodiment have at least one of X, at least one of Y, and at least one of Z each present.

[0203] The flowcharts, GUIs, sequences, systems, components, and swimlane activity diagrams in the figures illustrate the usage examples, functionality, architecture, and operation of possible implementations of methods, computer program products, and systems according to various embodiments of the present invention. In this regard, each item in the flowcharts, GUIs, sequences, systems, components, and swimlane activity diagrams can represent a module, segment, or portion of instructions consisting of one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the items may occur out of the order noted in the figures. For example, two items shown in succession may in fact be executed substantially simultaneously, or the items may sometimes be executed in the reverse order depending on the functions involved. Also, it will be noted that each item, and combination of items, in the flowcharts, GUIs, sequences, systems, components, and swimlane activity diagrams that do not involve one or more users can be implemented by a special purpose hardware-based system that does not involve one or more users, which executes a particular function or action, or by a combination of special purpose hardware and computer instructions.

[0204] Aspects of the present invention are described herein with reference to the flowcharts, GUIs, sequences, systems, components, and swimlane activity diagrams of methods, computer program products, and systems according to embodiments of the present invention. It will be understood that each item of the flowcharts, GUIs, sequences, systems, components, and swimlane activity diagrams that does not involve one or more users, as well as combinations of items of the flowcharts, GUIs, sequences, systems, components, and swimlane activity diagrams that do not involve one or more users, can be implemented by computer-readable program instructions.

[0205] These computer-readable program instructions are provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to generate machine-executable instructions that implement functions / acts that do not involve one or more users, creating means for implementing functions / acts specified by one or more items of the flowcharts, GUIs, sequences, systems, components, and swimlane activity diagrams, where the one or more items do not involve one or more users. These computer-readable program instructions can also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other devices to function in a particular manner, and a computer-readable storage medium having instructions stored therein comprises a manufactured article that includes instructions for implementing the functional / act aspects specified in the item or items of the flowcharts, GUIs, sequences, systems, components, and swimlane activity diagrams, and the item and items may or may not include one or more users.

[0206] As will be understood by those skilled in the art, such code may be stored or adapted for storage on one or more tangible machine-readable media such as memory chips, local or remote hard disks, optical disks, or other media, which may be accessed by a processor-based system to execute the stored code. Note that the tangible medium may consist of paper or other suitable media with the instructions printed thereon. For example, the instructions may be electronically captured via optical scanning of paper or other media and then compiled, interpreted, or otherwise processed in a suitable manner and then stored in a computer memory. Further note that the tangible medium may constitute a version control repository that utilizes memory chips, local, or remote hard disks, optical disks, or other media for storage. For example, the instructions may be retrieved from the version management repository and then processed in a compiled, interpreted, or other suitable manner and then stored in a computer memory.

[0207] Computer-readable program instructions may be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device perform the functions / acts specified by items of a flowchart, GUI, sequence, system, component, swimlane activity diagram.

[0208] Furthermore, the methods, non-transitory computer-readable media, systems, and apparatuses described herein may be subject to various modifications and alternative forms, some specific examples of which are shown in the drawings and described in detail herein. However, the present invention is not limited to the specific forms or methods disclosed, and conversely, the present invention is intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the various embodiments described and the appended claims. Additionally, any disclosure in this specification of any specific feature, aspect, method, characteristic, quality, attribute, element, etc. related to an implementation or embodiment can be used in any and all other implementations or embodiments described herein. Any method disclosed herein need not be performed in the recited order. The methods disclosed herein can include specific actions performed by an operator, but the methods can also include, explicitly or implicitly, instructions by any third party for those actions. The scope disclosed herein also encompasses any and all overlaps, subranges, and combinations thereof.

[0209] Terms such as "at least" and "at a minimum" include the recited number. A number preceded by a term such as "about" or "approximately" should be interpreted as including the recited number and in the context (e.g., as accurately as reasonably possible in the context, e.g., -5%, 5%, 10%, 15%, etc.). For example, "about 3.5 mm" includes "3.5 mm". A phrase preceded by a term such as "substantially" should be interpreted as including the recited phrase and in the context (e.g., as reasonably possible in the context). For example, "substantially constant" includes "constant".

[0210] The detailed description is presented to enable those skilled in the art to make and use the invention and is provided in connection with the requirements for obtaining a patent. Various modifications to the exemplary embodiments will be readily apparent to those skilled in the art, and the general principles of the invention may be applied to other embodiments. Some features of the invention may be used without the corresponding use of other features. Accordingly, the invention is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features described herein.

Claims

1. 1. A method comprising: receiving creation instructions from the annotatable web browser for creating a notepad including a list of references to annotation objects already stored using one or more computer systems; Here, the annotation object is The Uniform Resource Locator (URL) of the document, and data representing a selected piece of text from a document, the selected piece of text representing an annotation; Information for locating the selected piece of text within the document, and A group of one or more deliverable screenshots, Deliverable screenshots correspond to the marked screenshots in the storage group, Deliverable screenshots are The marked screenshot itself, and URL to get the marked screenshot from CDN A predetermined selection from a group comprising a group of one or more deliverable screenshots, Equipped with The saved groups are one or more marked screenshots created from an initial marked screenshot including a portion of the document representing the selected piece of text, and the selected piece of text being visibly marked; It is characterized by: Creating a notepad using one or more computer systems; receiving, from the annotatable web browser, additional instructions to add, using one or more computer systems, a reference to the annotation object in a list of references to annotation objects in the notepad; and storing a notepad object corresponding to the notepad in an electronic database, the notepad object having a list of references to the annotation objects of the notepad; A method for providing the above.

2. The method of claim 1 , wherein a marked screenshot in a storage group is a thumbnail of the marked screenshot.

3. The method of claim 1 , wherein the list of references to annotation objects is initially empty.

4. 2. The method of claim 1 , receiving, with one or more computer systems, a deletion instruction to delete a reference in the list of references to the annotation object in the notepad; and updating the notepad object in the electronic database to reflect the changes caused by the deletion instructions; The method of claim 1 further comprising:

5. 2. The method of claim 1 , sending the notepad to the annotatable web browser based on a set of access control rules that determine whether a user account associated with a user of the annotatable web browser has permission to view the notepad; The method of claim 1 further comprising:

6. A non-transitory computer-readable medium, comprising: having data representing software executable by an annotatable web browser application running on at least one processor; The non-transitory computer readable medium comprises: Instructions for tracking a state of a notepad, comprising: The status of the notepad The presence of an active notepad, and No active notepad having a state selected from the group of states instructions for tracking a notepad, An indication of an active notepad setting, An active notepad exists when the notepad state has a state in which an active notepad exists, the active notepad is selected from a group of one or more notepads accessible to a user account of a user using an annotatable web browser; The notepad contains a list of references to annotation objects, and The annotation object is The Uniform Resource Locator (URL) of the document; data representing a selected piece of text from a document, the selected piece of text representing an annotation; Information for locating the selected text within a document, and A group of one or more deliverable screenshots, Deliverable screenshots correspond to the marked screenshots in the saved group, Deliverable screenshots The marked screenshot itself, and A URL to retrieve the marked screenshot from the CDN, a predetermined selection from a group including A group of one or more deliverable screenshots; Equipped with the storage group includes one or more marked screenshots created from an initial marked screenshot including a portion of the document showing the selected piece of text, and the selected piece of text is visibly marked; instructions to set up an active notepad that features instructions for displaying a screenshot bundle display set associated with an active notepad, the screenshot bundle display set includes a plurality of screenshot display bundles, each screenshot display bundle being associated with one annotation object; The screenshot display bundle is A thumbnail of the marked screenshots based on one deliverable screenshot, and A selected piece of text from the associated annotation object's document To have instructions to display a screenshot bundle display set associated with the active notepad; Equipped with The elements of the screenshot bundle display set are displayed as screenshot bundles displayed in an annotatable web browser display area; A non-transitory computer-readable medium comprising:

7. 7. The non-transitory computer readable medium of claim 6, the annotation object further includes an annotation comment; and the screenshot display bundle further includes annotation comments; The non-transitory computer readable medium of claim 6 .

8. 8. The non-transitory computer readable medium of claim 7, allowing a user of a notepad-enabled web browser to select to add the annotation object reference to a list of annotation objects in the active notepad; an annotation-enabled web browser with a notepad function sending a message to an annotation server to add an annotation object reference to the active notepad; further comprising instructions; 8. The non-transitory computer readable medium of claim 7.

9. 9. The non-transitory computer readable medium of claim 8, an indication that allows a user to select to add the annotation object reference to the active notepad's list of annotation objects is provided by adding a button to the notepad located on the context variant annotation control bar within the annotation comment box; The annotation comment box is Annotation comments, and a contextual variant annotation control bar that includes a group of buttons that expose to a user functionality including at least one of adding an annotation object reference to a list of references to annotation objects in the active notepad; Equipped with 9. The non-transitory computer-readable medium of claim 8.

10. 10. The non-transitory computer readable medium of claim 9, The non-transitory computer readable medium comprises: permitting a user of the notepad-enabled web browser to click on a screenshot bundle displayed in the display area; A web browser with notepad functionality and annotation capabilities Opening a tab to the URL of the annotation represented in the screenshot display bundle; Scrolling the document to the location of the selected piece of text, and Displaying annotations with highlighted text and annotation comment boxes; Executing an action having further comprising instructions; 10. The non-transitory computer readable medium of claim 9.

11. 8. The non-transitory computer readable medium of claim 7, the annotation object further includes data representative of the displayed annotation symbol; the displayed screenshot bundle further includes a displayed annotation symbol; the displayed annotation symbol includes a selection by the creator of the annotation from a group of semantic relationships of the annotation; and The group of semantic relations of annotations is a disagreement relationship indicating disagreement between the selected text fragment and the annotation comment; and a question relation indicating an annotation comment that asks a question about the selected piece of text; containing, 8. The non-transitory computer readable medium of claim 7.

12. 12. The non-transitory computer readable medium of claim 11, A group of semantic relations of annotations is an agreement relationship indicating an agreement between the selected text fragment and the annotation comment; and an exclamation relation indicating an annotation comment that specifies an exclamation remark that is a point of note or an interesting fact about a selected piece of text; [0033] 12. The non-transitory computer readable medium of claim 11.

13. The selected piece of text is highlighting the displayed annotation symbols with colors pre-assigned to the displayed annotation symbols according to the semantic relationship of the corresponding annotation; 12. The non-transitory computer readable medium of claim 11.

14. 12. The non-transitory computer readable medium of claim 11, the group of semantic relations of the annotation further includes an advertisement relation indicating that the annotation comment is an advertisement related to the selected piece of text; The creator of the annotation is an advertiser, and the displayed annotation symbol of the advertisement is a displayed annotation symbol corresponding to a relationship indicating that the annotation comment is an advertisement related to the selected piece of text; and Advertisements originate from an ad server; 12. The non-transitory computer readable medium of claim 11.

15. 1. A system comprising: a server operative to receive from the annotation-enabled web browser a creation instruction for creating a notepad including a list of references to annotation objects already stored using one or more computer systems; Here, the annotation object The Uniform Resource Locator (URL) of the document, and data representing a selected piece of text from a document, the selected piece of text representing an annotation; Information for locating the selected piece of text within the document, and A group of one or more deliverable screenshots, Deliverable screenshots correspond to the marked screenshots in the storage group, and Deliverable screenshots are The marked screenshot itself, and URL to get the marked screenshot from CDN A predetermined selection from a group comprising a group of one or more deliverable screenshots, Equipped with The saved groups are one or more marked screenshots created from an initial marked screenshot including a portion of the document showing the selected pieces of text, and the selected pieces of text being visibly marked; It is characterized by: a server operating to generate the notepad using one or more computer systems; a server operative to receive, from the annotation-enabled web browser, an additional instruction to add a reference to the annotation object in a list of references to annotation objects in the notepad using one or more computer systems; an electronic database operative to store a notepad object corresponding to the notepad; A system comprising:

16. One marked screenshot in the storage group is the thumbnail of the marked screenshot, The system of claim 15 .

17. The list of references to annotation objects is initially empty, The system of claim 15 .

18. 16. The system of claim 15, receiving, with one or more computer systems, a deletion instruction to delete a reference in the list of references to the annotation object in the notepad; and an annotation server operable to update the notepad object in the electronic database to reflect the changes caused by the deletion instruction; The system of claim 15 further comprising:

19. 16. The system of claim 15, an annotation server operative to transmit the notepad to the annotatable web browser based on a set of access control rules that determine whether a user account associated with a user of the annotatable web browser has permission to view the notepad; The system of claim 15 further comprising:

20. 1. A system comprising: a client electronic device having at least one processor and a memory; Web browser and 1. A web browser extension, comprising: Operable for modifying a web browser to create an annotatable web browser with a notepad function; In the state of a notepad, The status of the notepad The presence of an active notepad, and No active notepad having a state selected from a group of states having A notepad state featuring An active notepad, an active notepad exists when the notepad state has an active notepad exists state; the active notepad is selected from a group of notepads accessible to a user account of a user using a notepad-enabled, annotatable web browser; The notepad contains a list of references to annotation objects; and The annotation object is The Uniform Resource Locator (URL) of the document; data representing a selected piece of text from a document, the selected piece of text representing an annotation; Information for locating the selected text within a document, and A group of one or more deliverable screenshots, Deliverable screenshots correspond to the marked screenshots in the saved group, Deliverable screenshots The marked screenshot itself, and A URL to retrieve the marked screenshot from the CDN, including a predetermined selection from a group including A group of one or more deliverable screenshots; an annotation object comprising: the storage group includes one or more marked screenshots created from an initial marked screenshot including a portion of the document showing the selected piece of text, and the selected piece of text is visibly marked; An active notepad featuring 1. A screenshot bundle display set, comprising: having a plurality of screenshot display bundles with each screenshot display bundle associated with an annotation object; The screenshot display bundle is A thumbnail of a marked screenshot based on one deliverable screenshot, and A selected piece of text from the associated annotation object's document To have A screenshot bundle display set, Equipped with the elements of the screenshot bundle display set are displayed as screenshot bundles displayed in an annotatable web browser display area; A web browser extension that features A system comprising:

21. 21. The system of claim 20, the annotation object further includes an annotation comment; and The screenshot display bundle also includes annotation comments.

21. The system of claim 20.

22. 22. The system of claim 21, A web browser with notepad functionality and annotation capabilities allowing the user to select an annotation object reference to add to the active notepad's list of annotation objects; sending a message to an annotation server by the annotation-enabled web browser with notepad functionality to add an annotation object reference to the active notepad; 22. The system of claim 21 .

23. 23. The system of claim 22, An action for allowing a user to select an annotation object reference to add to a list of references to annotation objects in an active notepad, comprising: Contextual variants in annotation comment boxes are provided by the addition of a button to Notepad located on the annotation control bar, Annotation comment box Annotation comments, and a contextual variant annotation control bar that includes a group of buttons that expose to a user functionality including at least one of adding an annotation object reference to a list of references to annotation objects in the active notepad; Equipped 23. The system of claim 22.

24. 24. The system of claim 23, The annotatable web browser display area allows the user to click on the displayed screenshot bundle, A web browser with notepad functionality and annotation capabilities opening a tab for the URL of the annotation displayed by the screenshot display bundle; Scrolling the document to the location of the selected piece of text, and Displaying annotation highlight text and annotation comment boxes; operative to have a notepad function to perform an action having the following properties:

24. The system of claim 23.

25. 22. The system of claim 21, the annotation object further includes data representing the displayed annotation symbol; the displayed screenshot bundle further includes a displayed annotation symbol; the displayed annotation symbol includes a selection by the creator of the annotation from a group of semantic relationships of the annotation; and The group of semantic relations of annotations is a disagreement relationship indicating disagreement between the selected text fragment and the annotation comment; and a question relation indicating an annotation comment that asks a question about the selected piece of text; containing, 22. The system of claim 21 .

26. 26. The system of claim 25, A group of semantic relations of annotations is an agreement relationship indicating an agreement between the selected text fragment and the annotation comment; and an exclamation relation indicating an annotation comment that specifies an exclamation, which is a note or interesting fact about a selected piece of text; [0033] 26. The system of claim 25.

27. The selected piece of text is highlighting the displayed annotation symbols with colors pre-assigned to the displayed annotation symbols according to the semantic relationship of the corresponding annotation; 26. The system of claim 25,

28. 26. The system of claim 25, the group of semantic relations of the annotation further includes an advertisement relation indicating that the annotation comment is an advertisement related to the selected piece of text; The creator of the annotation is an advertiser, and the displayed annotation symbol of the advertisement is a displayed annotation symbol corresponding to a relationship indicating that the annotation comment is an advertisement related to the selected piece of text; and Advertisements originate from an ad server; 26. The system of claim 25.

Images