Method, computer-readable medium, and system for creating, organizing, displaying, and connecting annotations of a document within a web browser

Abstract

The present invention relates to a method, computer-readable medium (CRM), and system for creating, organizing, displaying, and connecting annotations of web documents within an annotation-enabled web browser. A method, CRM, and system are presented for an annotate-enabled web browser that transmits annotation-highlighted text, annotation comments, Uniform Resource Locators (URLs), Document Object Model (DOM) context, screenshots containing marked annotation-highlighted text, and annotation symbols to an annotation server. Additionally, a method, CRM, and system for organizing annotations within an annotation organization context area, visual cues for viewing and recognizing annotations within web documents, and a notepad for connecting annotations across web documents are presented. The present invention relates to taking notes within web documents, organizing those notes, discussing specific aspects of web documents with multiple people, and creating guided tours across multiple web documents as reminders for yourself or others.

Description

Background Art

[0001] The World Wide Web (WWW), web browsers, and browser extensions are well known. The WWW is a global information source where assets such as web pages, PDF documents, images, videos, etc. are identified by unique Uniform Resource Locators (URLs), and a web browser is a software application designed to access information on the WWW. Also, browser extensions are software created to extend the functionality of browsers. It is known that the functionality of a web browser can be changed by modifying the software codebase of the web browser or by providing a web browser extension for the web browser that supports adding functionality by such means.

[0002] As the popularity of the World Wide Web has grown, people, on average, are spending more time using the World Wide Web each day than in the past. Due to the increased average time spent using the World Wide Web, people have needs such as searching for information they previously saw on the WWW, sending snippets of text within web documents to friends and colleagues for discussion, referring to specific pieces of text or context within web documents, asking specific questions about parts of web documents, connecting information between web documents, finding and organizing content and snippets for later use, evaluating the value of crowd-sourced information, accessing other people's public comments on web documents and placing these comments in context, etc.

[0003] To address these use cases, people typically repeat searches to find previously accessed content, bookmark web documents, re-scan articles for snippets they remember seeing, copy & paste sections of text into email messages or other documents to share with friends and colleagues, reference sentences or paragraphs by number or web page location while discussing on the phone, ask questions in the comment section of articles, cut & paste within comments to reference desired text, download web pages, or copy snippets into personal word processing or spreadsheet documents. This is inefficient and time-consuming for many users. Additionally, there are no tools to identify inaccurate information and spread this knowledge to other users, allowing inaccurate information to persist longer than necessary. It would be advantageous to provide a method, computer-readable medium, and system for more efficiently handling these tasks.

Summary of the Invention

[0004] An object of the present invention is to provide a user with an annotatable web browser that can annotate web pages and documents so that a user located at another place on the Internet can identify questions, interesting points, statements of agreement, and statements of disagreement without reading user annotations, comments, or annotation-emphasized text using the same annotatable web browser. Such an arrangement improves the efficiency of users when sifting through the vast amount of information on the WWW.

[0005] A further object of the present invention is to provide a user with an annotatable web browser for annotating images, videos, and recordings uploaded or embedded in web pages so that users located at other places on the Internet can also view these image, video, and recording annotations using such an annotatable web browser.

[0006] Another object of the present invention is to ensure that after the annotation is added, the document display context is changed in a minimal way compared to the document display context without the annotation. By using a document viewing context that is changed in a minimal way, the user can consume the content without being distracted by the annotation, and at the same time, can find and utilize the annotation at an appropriate timing.

[0007] A further object of the present invention is to non-intrusively present advertisements to the user. One object is to show the user advertising content after these users have initiated an action caused by curiosity, which is caused by the presentation of an advertisement in a specific location or context within a web page. This location and context suggest the possibility that the advertisement is relevant to the user's current needs. Further, an advertisement presented without content can include some aspects of a complete advertisement, but the full details are not shown until the user initiates an action caused by curiosity planted by a contextual method that selectively and transparently targets the user based on the user's current needs.

[0008] Another object of the present invention is to provide an annotation viewing context area separate from the World Wide Web document viewing context area. This annotation viewing context area has a mechanism for presenting annotations to the user so that the user can understand familiar sites and the approximate location of the annotations, in addition to search and filtering. Further, other elements of the context of the annotation or highlighted text may become apparent without the user having to read the highlighted text or the user's annotation comments. Such a mechanism makes it easy to find a specific annotation among a large number of annotations.

[0009] A further object of the present invention is to provide an annotation compilation context area that enables a user to scan annotations and organize them into meaningful groups. The advantage of this object is to enable the user to first concentrate on creating annotations and then concentrate on organizing the annotations after they have been created. Further, the default compilation of the annotations need not be restored.

[0010] Another object of the present invention is to provide a user with an annotation display context area that enables the user to display a page of annotations, keep some annotations available for later review, and continue to search for other annotations. Further, it is a method for facilitating mental tracking by maintaining the annotations in a visual view while moving uninteresting annotations outside the visual view.

[0011] An additional object of the present invention is to always transparently convey privacy settings to the user. By conveying the privacy settings, it is possible to avoid accidentally sharing information that the user does not want to share.

[0012] A further object of the present invention is a method for creating a guided tour through a series of websites on the global Internet and at the same time drawing the user's attention to prominent points of interest on each site. This object further includes a method for modifying the tour and presenting the tour to the user.

[0013] Another object of the present invention is to provide a method for a group to manage access control permissions and communicate through messaging.

Brief Description of the Drawings

[0014] For a more complete understanding of the present invention, reference may be made to the following description and the accompanying drawings.

[0015] Figure 1 shows a flowchart in which an annotation is created, moved to two different collections, and shared with others via the collections.

[0016] Figure 2 shows a flowchart of an annotation viewed on two separate web pages on two separate days.

[0017] Figure 3 represents an annotatable web browser in an exemplary embodiment in which a web browser without the ability to create an annotation is extended with a web browser extension that provides annotation capabilities before the annotation process is started.

[0018] Figure 4 shows an annotatable web browser after the text has been highlighted.

[0019] Figure 5 shows an annotatable web browser before an annotation symbol is selected.

[0020] Figure 6 shows an annotatable web browser after an annotation symbol has been selected.

[0021] Figure 7 shows an annotatable web browser after an annotation has been created and the displayed annotation symbol is in an open state.

[0022] Figure 8 shows an annotatable web browser after an annotation has been made. Indicates that the created and displayed annotation symbol is in a closed state.

[0023] Figure 9 shows an annotation item box that provides the context of an annotation when it is presented not as an overlay on an existing web document adjacent to the highlighted annotation text but in a separate context area.

[0024] FIG. 10 shows an annotation composition context area presenting an annotation item box and a writable collection displayed within a context area separate from the original web document containing the annotation emphasized text.

[0025] FIG. 11 shows an annotation composition context area showing the process of moving and duplicating annotations to a collection that is a grouping of annotations and other items.

[0026] FIG. 12 shows an annotatable web browser displaying a web document with an annotation symbol shown in a closed state.

[0027] FIG. 13 shows an annotatable web browser displaying a web document with a displayed annotation symbol in a closed state with the corresponding displayed annotation symbol beacon.

[0028] FIG. 14 shows a sequence diagram showing the communication between an annotatable web server, a web server providing data of the visited URL, an annotation server, an electronic database, and a CDN.

[0029] FIG. 15 shows a sequence diagram showing the communication between a web browser for organizing annotations, an annotation server, an electronic database, and a CDN.

[0030] FIG. 16 shows a sequence diagram showing the communication between an annotatable web browser, a web server providing data of the visited URL, an annotation server, and an electronic database.

[0031] FIG. 17 shows a flowchart showing the reception and storage of a bundle of data elements.

[0032] FIG. 18 shows a flowchart showing the compilation and transmission of a bundle display set.

[0033] Figure 19 shows a flowchart illustrating the interaction among a user, a web browser, a web browser extension content script, and a web browser extension background script that enables the user to display a web page and create an annotation on the page.

[0034] Figure 20 shows a flowchart illustrating a way of interaction among a user, a web browser, a web browser extension background script, and a web browser extension content script that enables the user to view a web page with an annotation on the page.

[0035] Figure 21 shows a flowchart illustrating a method for a nurse to create a memo regarding the glycemic index for sharing with a patient using this exemplary embodiment.

[0036] Figure 22 shows a flowchart illustrating how a patient uses the memo provided by the nurse.

[0037] Figure 23 shows the user interface of the memo within the extension.

[0038] Figure 24 shows a sequence diagram illustrating the state where a memo is created and one annotation is created and added to the memo.

[0039] Figure 25 shows a sequence diagram illustrating the state where the memo is viewed, the first annotation is clicked, the browser navigates to its URL, scrolls to the appropriate location, and then displays the annotation.

[0040] Figure 26 shows a flowchart illustrating the state where a memo is created and an already created annotation is added to the memo.

[0041] Figure 27 shows a flowchart illustrating the state where the memo is viewed and the annotation of the memo is opened.

[0042] FIG. 28 shows an exemplary embodiment of a system diagram.

[0043] FIG. 29 shows another exemplary embodiment of a component diagram for an annotation server and an electronic database.

[0044] FIG. 30 shows an exemplary embodiment of a client electronic device for an annotation server.

[0045] FIG. 31 presents an exemplary embodiment of a collection item page for a collection of American history.

[0046] FIG. 32 shows an exemplary embodiment of a collection page.

[0047] FIG. 33 shows an exemplary embodiment of an inbox page showing system messages.

[0048] FIG. 34 shows an exemplary embodiment of an inbox page after a user has responded to a system message and other users have also sent messages.

[0049] FIG. 35 shows an image-annotation display and an image-annotation group creation interface.

[0050] FIG. 36 shows an image-annotation group creation interface in which one annotation group list item has been added to the annotation group candidate list.

[0051] FIG. 37 shows an image-annotation group creation interface having one annotation group list item to which a second non-colliding image annotation has not been added to the candidate annotation group list.

[0052] Figure 38 shows an image annotation group creation interface where two annotation group list items are added to the annotation group candidate list.

[0053] Figure 39 shows an image annotation group creation interface where one annotation group list item is added to the candidate annotation group list and the second collision image annotation is not added to the candidate annotation group list.

[0054] Figure 40 shows a video annotation browsing and video annotation group creation interface.

[0055] Figure 41 shows a video annotation display and video annotation group creation interface where one video annotation is added to the annotation group candidate list box having a motion path for the video annotation element.

[0056] Figure 42 shows a video annotation group creation interface where two annotation group list items are added to the candidate annotation group list having a motion path for the second video annotation element.

[0057] Figure 43 shows a video annotation group creation interface where one annotation group list item is added to the candidate annotation group list and the second collision video annotation is not added to the candidate annotation group list.

[0058] Figure 44 shows an audio annotation browsing and audio annotation group creation interface for timeline audio annotation and derived audio annotation creation interface.

[0059] Figure 45 shows a video annotation display and a video annotation group creation interface having a timeline video annotation and a derived video annotation creation interface.

[0060] Figure 46 shows an annotatable web browser that displays a video in which a video annotation beacon is embedded.

[0061] Figure 47 shows an annotatable web browser after an image annotation is created and the embedded and displayed image annotation is in an open state.

[0062] Figure 48 shows an annotatable web browser after a video annotation is created, indicating that the embedded and displayed video annotation is open during the video annotation playback range.

[0063] Figure 49 shows an annotatable web browser after an audio annotation is created and the embedded and displayed audio annotation is in an open state during the playback range of the audio annotation.

BEST MODE FOR CARRYING OUT THE INVENTION

[0064] A method, non-transitory computer-readable medium (CRM), and system are provided for adding annotations to web pages, documents, images, videos, and recordings within a web browser, organizing these annotations, viewing these annotations, and connecting these annotations to a useful digital memo book containing a series of annotations. A web document has a URL and includes web pages, pdf documents, and other document types that are displayed in a web browser. Further, these methods, non-transitory CRM, and systems provide facilities for content curation, sharing of content with other users, and management of access control. This specification includes flowcharts showing the basis for users to use the present system and non-transitory CRM. Further, graphical user interface (GUI) diagrams show the end results of the instructions of the computer-readable medium and what the system displays. Further, sequence diagrams show the interactions and data flows between various components, and flowcharts explain the methods. Further, swimlane diagrams show the operations of various components.

[0065] An annotation is a note, comment, or drawing of an explanation added to text, a figure, an image, a document, a video, or a recording. When describing the method, non-transitory CRM, and system, the term is introduced to describe annotations within a separate context, as well as data elements and structures representing annotations and objects added to the context of annotations within a web page, web document, document, image, video, and recording, as well as within another context area.

[0066] In FIG. 1, a flowchart of a teacher using an exemplary embodiment of an annotatable web browser is shown. This teacher has two separate goals. The first goal is to create a list of learning questions about the president for the students in the teacher's class. The teacher's class is small, and in this example within the exemplary embodiment depicted, the students can see each other's answers. The second goal is to find a house for sale near the school where the teacher teaches. As shown in FIG. 1, on the first day, the teacher [1] visits the Wikipedia page of Abraham Lincoln, creates question annotation [4], then visits the Wikipedia page of Woodrow Wilson and creates question annotation [5]. As shown in FIG. 1, on the second day [2], the teacher visits the Zillow page and creates a dagger annotation [6] of a house for sale near the workplace. As shown in FIG. 1, on the tenth day, the teacher [3] sets up two collections, the US President Learning Guide and Real Estate [7], adds students to the US President Learning Guide collection, adds roommates to the Real Estate collection [8], moves the question annotations of Abraham Lincoln and Woodrow Wilson to the US President Learning Guide [9], moves the dagger annotation regarding the house for sale near the workplace to the Real Estate collection

[10] , shares the US President Learning Guide collection with the students

[11] , and further adds roommates to the US President Learning Guide and Real Estate collections.

[0067] In FIG. 2, a flowchart of a student using an exemplary embodiment of an annotatable web browser is shown. The student's goal is to answer learning questions provided by the teacher. As shown in FIG. 2, on the 11th day, student

[12] visits the Wikipedia page of Abraham Lincoln

[14] , reads the Wikipedia page of Abraham Lincoln

[15] , sees the displayed annotation symbol for the teacher's closed question while reading the Wikipedia page of Abraham Lincoln

[16] , clicks on the displayed annotation symbol for the teacher's closed question, moves to the state where the annotation symbol for the question is open and displays the teacher's annotation comment

[17] , and responds with an answer to the teacher's question annotation comment.

[18] As shown in FIG. 2, on the 12th day, student

[13] visits the Wikipedia page of Woodrow Wilson

[19] , reads the Wikipedia page of Woodrow Wilson

[20] , sees the displayed annotation symbol for the teacher's closed question while reading the Wikipedia page of Woodrow Wilson

[21] , clicks on the displayed annotation symbol for the teacher's closed question, the annotation symbol for the question moves to the open state, displays the teacher's annotation comment

[22] , and responds with an answer to the teacher's question annotation comment.

[23]

[0068] In FIG. 3, an exemplary embodiment of an annotatable web browser

[24] is shown. The horizontally long and short rectangles displayed in the display area of the depicted web browser represent text. In the first paragraph, there are 11 rectangles representing 11 lines of text, and in the second paragraph, there are 5 rectangles representing 5 lines of text. In this exemplary embodiment, the web browser that supports the addition of web extensions is modified with an exemplary example of a web browser extension

[25] that adds an annotation function. The web browser extension

[25] is a software package typically present on a non-transitory computer-readable medium that can be loaded into memory by the web browser, supports the addition of web extensions, and the web browser application itself is a software application that is loaded into memory and executed on the processor of an electronic computing device. When the web browser application loads the web browser extension

[25] into memory, the instruction set of the web browser extension

[25] can be triggered by the instruction set of the web browser application at a specific point in the life cycle of the web browser. Some of the effects of the instruction set of the web browser extension

[25] on the user interface presented to the user are shown in FIGS. 5, 6, 7, 8, 12, 13, 23, 46, 47, 48, and 49.

[0069] In FIG. 4, an exemplary embodiment of an annotatable web browser

[24] shows the state before the instruction set of the web browser extension

[25] is triggered after the user selects text, which is called normal emphasized text

[26] .

[0070] In FIG. 5, an exemplary embodiment of an annotatable web browser

[24] is shown after the user selects text, the web browser extension

[25] instruction set is triggered, and the annotation symbol selection button bar

[28] is displayed. Further, the selected text will be called annotation emphasized text

[27] to distinguish it from normal emphasized text

[26] , as shown in the state of the exemplary embodiment depicted in FIG. 4. Normal emphasized text

[26] is the selected text before the web browser extension

[25] instruction set is triggered, or the selected text in a web browser that does not have an annotation function. The web browser extension

[25] instruction set processes the input after the user clicks any of the buttons shown on the annotation symbol selection button bar

[28] in FIG. 5.

[0071] In an exemplary embodiment, FIG. 6 depicts the effect of a user clicking on the question mark annotation symbol button in the annotation symbol selection button bar

[28] illustrated in FIG. 5. In an alternative exemplary embodiment, the annotation symbols shown in the annotation symbol selection button bar

[28] of FIG. 5 may be arranged one above the other in a vertical column as opposed to a horizontal row. In the exemplary embodiment depicted in FIG. 6, the displayed annotation symbol symbol

[29] of the open question and the annotation comment input box

[30] are shown. The displayed annotation symbol symbol

[29] of the open question can indicate that the user intends to enter a question regarding the annotation emphasis text

[27] into the annotation comment input box

[30] . The annotation comment input box

[30] is an input area for a user of the annotatable web browser

[24] to enter text that is stored in one or more computer systems. When the user finishes entering text and conveys the completion of the text entry operation to the annotatable web browser

[24] , the web browser extension

[25] instruction set causes the annotatable web browser

[24] to transition to the state depicted in FIG. 7.

[0072] In the exemplary embodiment depicted in FIG. 7, after an annotation is created and the displayed annotation symbol is visually displayed in an open state, the state of the annotatable web browser

[24] is displayed. The displayed annotation symbol in the open state displays screen elements such as annotation emphasis text

[27] , annotation comment

[32] , reputation selector

[33] , the displayed annotation symbol symbol

[29] of the opened question, and annotation comment box

[31] . The annotation comment

[32] is derived from the text input by the user as described above. The annotation comment box

[31] also has a context variant annotation control bar

[34] . With the context variant annotation control bar

[34] , the user can change privacy settings, share via social media or email, flag the annotation, and delete the annotation according to access control rules. With the reputation selector

[33] , the user can select an evaluation to assign to the annotation comment

[32] . For example, if the user thinks the comment is thoughtful, the user can assign a thoughtful evaluation to the annotation comment

[32] . By clicking outside the annotation emphasis text

[27] , the displayed annotation symbol

[29] of the opened question, and the annotation comment box

[31] , the user can transition the displayed annotation symbol to a closed state and close the annotation. In another embodiment, the reputation selector

[33] may be displayed inside the context variant annotation control bar

[34] as opposed to being arranged adjacent to the annotation comment

[32] .

[0073] In the exemplary embodiment depicted in FIG. 8, the state of the annotatable web browser

[24] after an annotation has been created and the displayed annotation symbol is shown in a closed state is shown. FIG. 8 shows the displayed annotation symbol

[35] of a closed question. The displayed annotation symbol

[35] of the closed question is visible but is designed so that the user is not distracted from the rest of the text.

[0074] After the user creates an annotation using an annotatable web browser

[24] , a bundle of data elements representing the annotation is sent to the server. This bundle of data elements can be used to construct an annotation item box

[36] in a different context area, as shown in the exemplary embodiment depicted in FIG. 9. In the exemplary embodiment depicted in FIG. 9, the annotation item box

[36] includes a screenshot

[37] with marked text, a separate context annotation emphasis text

[38] , a displayed annotation symbol

[39] for a separate context question, a separate context annotation comment

[40] , a separate context variant annotation control bar

[41] , and a separate context reputation selector

[42] . In this exemplary embodiment, the screenshot

[37] with marked text is a thumbnail-sized associated marked screenshot intended to enable the user to determine the web page where the annotation exists and the position of the annotation within the web page. FIGS. 7 and 9 can be viewed as two different contexts of the same exemplary embodiment or as two different exemplary embodiments. For the purpose of understanding the separate context annotation emphasis text

[38] , the displayed annotation symbol

[39] for a separate context question, and the separate context annotation comment

[40] , it is useful to view these two figures as two different contexts of the same exemplary embodiment. The separate context annotation emphasis text

[38] presents the annotation emphasis text

[27] in a separate context area of the annotation item box

[36] . The description of the separate context area in this situation indicates that the context is different from the original web page or web document where the annotation emphasis text

[27] occurred. The displayed annotation symbol

[39] for a separate context question presents a question annotation symbol in a separate context area of the annotation item box

[36] .Separate context annotation comments

[40] present annotation comments

[32] in a separate context area of the annotation item box

[36] . The separate context variant annotation control bar

[41] enables the user to change privacy settings, share on social media or by email, associate classification ratings with the annotation, or delete the annotation.

[0075] When the item box represents an annotation, it is called an annotation item box

[36] and is depicted in the exemplary embodiment of FIG. 9. In the exemplary embodiment, the item box can represent a web page, document, image, video, and recording, as well as annotations thereof. The item box may be a web page item box, document item box, image item box, video item box, recording item box, annotation item box, image-annotation item box, video-annotation item box, recording annotation item box, embedded image-annotation item box, embedded video-annotation item box, embedded recording item box, annotation group item box, derived annotation group item box, derived annotation-attached annotation group item box. A collection is a representative of a group of items including annotations, and a writable collection is a collection that the user can modify in some way. One form of modification is to add an item to a writable collection. Collections are used to organize items including annotations into meaningful groups. As an example, in FIG. 1, the teacher created two collections, one titled "US President Learning Guide" and the other titled "Real Estate".

[0076] In the exemplary embodiment depicted in FIG. 9, the annotation item box

[36] includes a separate context reputation selector

[42] in a separate context variant annotation control bar

[41] , enabling users who share annotations in the form of the annotation item box

[36] to vote on the reputation by selecting from a drop-down. This reputation is assigned to the annotation. This reputation has value within the social community. Annotations with reputations such as useful, interesting, or considerate, or annotations expressing opposition to the annotation emphasized text, can be selectively displayed based on various criteria selected by the user and determined to be useful by the annotation server

[61] shown in some other figures. Users can filter annotations based on this criterion, and the aggregated reputation of the user can also be connected to the reputation of the annotations created by the user. In contrast to star-based voting, detailed annotation-specific semantic information about why an annotation is valuable to other users within the online community can be collected from the user.

[0077] Furthermore, in addition to giving a reputation to an annotation, the annotation system functions better if users can reply to the annotation. Additionally, other users can also vote on the reputation of the reply. This system helps the community identify potentially inaccurate annotations as users provide feedback through replies and reputations.

[0078] Both the exemplary embodiment depicted in FIG. 7 and the exemplary embodiment depicted in FIG. 9 represent annotations. FIG. 7 represents an annotation embedded within the web document to which it is attached, while FIG. 9 represents an annotation outside of the web document. Since FIG. 9 has no context around the web document, a thumbnail-marked screenshot is provided to give the user viewing the annotation item box

[36] more context. FIGS. 7 and 9 can potentially depict two contexts of the same exemplary embodiment, or they can potentially depict two different exemplary embodiments. Both FIGS. 7 and 9 enable reputation voting. In FIG. 7, the reputation selector

[33] enables the user to assign a reputation to the annotation comment

[32] . In FIG. 9, another context reputation selector

[42] enables the user to assign a reputation to the annotation comment

[32] . This allows users sharing an annotation to evaluate the reputation of the annotation from different contexts.

[0079] In the exemplary embodiment depicted in FIG. 9, the annotation item box

[36] includes a separate context-variant annotation control bar

[41] that enables the user to change privacy settings. In the exemplary embodiment, dark gray indicates the most private setting where no one other than the creator of the annotation can view the annotation, light gray indicates that the annotation can be shared with a specific group of users, and white indicates that the annotation can be viewed by all users of the platform.

[0080] One of the separate context areas that can render the annotation item box

[36] depicted in FIG. 9 is the annotation composition context area

[43] of the exemplary embodiment depicted in FIG. 10. The annotation composition context area

[43] of FIG. 10 also includes an item bin

[44] for displaying the item box and a writable collection bin

[45] for displaying the writable collection. In FIG. 9, the annotation item box

[36] is depicted within the item bin

[44] , and the writable collection

[50] displayed within the writable collection bin

[45] is depicted. Further, the search filter

[46] can be used to change the item box displayed in the item bin

[44] . Further, the collection search filter

[47] can be used to change the display of the writable collection displayed in the writable collection bin

[45] . Further, the paging mechanism

[48] can be used to change the pages of the item box displayed in the item bin

[44] , and the collection paging mechanism

[49] can be used to change the collection pages of the displayed writable collection displayed in the writable collection bin

[45] .

[0081] In the exemplary embodiment depicted in FIG. 10, it can be seen that the item bin

[44] has a paging mechanism

[48] , and the writable collection bin

[45] has a collection paging mechanism

[49] . These paging mechanisms operate independently. The paging mechanism

[48] of the item bin

[44] moves forward and backward through the 6-item pages of the exemplary embodiment being displayed. The items shown in FIG. 10 represent annotations. All of these items are arranged sequentially. When the paging mechanism is moved forward, the next 6 items among the sequentially arranged items are displayed. When the paging mechanism is moved backward, the previous 6 items in the set of sequentially ordered items are shown. The collection paging mechanism

[49] within the writable collection bin

[45] moves forward and backward through the 3 collection pages in the exemplary embodiment being displayed. The collections shown in FIG. 10 represent groups of items that include annotations. Each collection represents a group of one or more items, and the items include web page items, document items, image items, video items, recording items, annotation items, image-annotation items, video-annotation items, recording annotation items, embedded image-annotation items, embedded video-annotation items, embedded recording items, annotation group items, derived annotation group items, annotation groups with derived annotation items. All of these collections are arranged sequentially. When the paging mechanism is moved forward, the next 3 items in the set of sequentially ordered collections are displayed. When the paging mechanism is moved backward, the previous 3 items in the sequentially arranged collections are shown. When the displayable area of the browser changes after a resize operation, the number of items displayed in the item bin

[44] and the number of writable collections displayed in the writable collection bin

[45] change. In an alternative exemplary embodiment, there may be no explicit writable permission.In this case, the writable collection bin

[45] will include the displayed collection. The paging mechanism

[48] and the collection paging mechanism

[49] take this shift into account and operate with the new page size.

[0082] In FIG. 10, in addition to the paging mechanism

[48] , the pins of all the annotation item boxes

[36] located in the item bin

[44] can also be seen. In the exemplary embodiment shown in FIG. 10, six items representing annotations are shown. Each of these six items can be in an unpinned or pinned state. As shown, the page size is 6, and when one of these items is pinned, the page size decreases to 5. As a result of the fixation, even if the paging mechanism

[48] is moved back and forth, the fixed item remains in the display area. When the paging mechanism

[48] is advanced, the fixed item remains in the viewing area and the remaining five items move out of the viewing area. Then, the next five items in sequential order replace the five removed items. When the paging mechanism moves backward, the five unpinned items are removed from the display area and the previous five items in the sequential order of the items replace the five items removed from the display area. When two items are pinned, the page size is reduced to 4. Pinning functions similarly for the collections within the writable collection bin

[45] .

[0083] Each time an item is pinned, the pinned item is excluded from the set of items that are sequentially ordered by an annotation server

[61] shown in some other figures. This can avoid the pinned item being displayed twice in the display area. When two items are fixed, the two fixed items are excluded from the set of items that are arranged in order by the annotation server

[61] . The exclusion from the sequential rearrangement is performed in addition to the page size change. The exclusion of the sequential order by the annotation server

[61] functions similarly for the collection of writable collection bins

[45] .

[0084] Filters such as the search filter

[46] shown in the exemplary embodiment of FIG. 10 are used to receive input from the user. In the case of the search filter

[46] of the item bin

[44] , it is a search phrase. These inputs are converted into parameter values that are sent to an annotation server

[61] shown in some other figures. These parameter values are used by the annotation server

[61] to input parameters into a request query used by an electronic database

[62] shown in some other figures. The request query includes parameter values and a page number. The request query with parameter values is used to specify a set of items and their order. The page number specifies a particular page. Further, the request query can include the number of items to be sent to a web browser running on a client electronic device. This request query includes parameters of filter values such as search phrases and an exclusion list of items to be excluded from the specified list of items, and the request query introduces a sequential ordering for the list of those items along with the parameters and the item exclusion list.

[0085] Filters such as the collection search filter

[47] shown in the exemplary embodiment of FIG. 10 are used to receive input from a user. In the case of the collection search filter

[47] of the writable collection bin

[45] , it is a collection search phrase. These inputs are converted into parameter values that are sent to the annotation server

[61] shown in some of the other figures. These parameter values are used by the annotation server

[61] to input parameters into the collection request query used by the electronic database

[62] shown in some of the other figures. The collection request query includes parameter values and a page number. The collection request query is used to specify a set of collections, along with the collection parameter values, and an ordering for those collections. The page number specifies a particular page. Further, the collection request query can include the number of collections to be specified that are to be sent to the web browser running on the client electronic device. This collection request query includes parameters of collection filter values such as collection search phrases, and an exclusion list of collections to be excluded from the list of the specified collections, and this collection request query introduces an ordering for the list of those collections, along with the parameters and the exclusion list of collections.

[0086] After the user creates an annotation, in the exemplary embodiment depicted in FIG. 10, the annotation is displayed in the item bin

[44] . Using a drag-and-drop process, the user can move or duplicate the annotation item box

[36] to a collection, as in the exemplary embodiment depicted in FIG. 11. In the exemplary embodiment depicted in FIG. 11, a moving annotation item box

[51] is depicted. As the moving annotation item box

[51] approaches the displayed writable collection

[50] , as in the state of the exemplary embodiment depicted in FIG. 10, the visual form of the displayed writable collection

[50] changes as depicted in FIG. 11, which shows screen elements including a move target

[52] and a duplicate target

[53] . In this exemplary embodiment, when the mouse pointer depicting the drag of the moving annotation item box

[51] is within the area indicated by the move target

[52] and released, the move action is triggered. In this exemplary embodiment, when the mouse pointer indicating the drag of the moving annotation item box

[51] is released in the area designated by the duplicate target, the duplication operation is initiated.

[0087] In the exemplary embodiment depicted in FIG. 12, the user is presented with a displayed annotation symbol

[35] for a closed question, a displayed annotation symbol

[54] for a closed exclamation, a displayed annotation symbol

[55] for a closed check, and a displayed annotation symbol

[56] for a closed incorrect answer. These four annotation symbols are examples of displayed annotation symbols. In the exemplary embodiment, the displayed annotation symbol has at least two possible states. These states include an open state and a closed state. In the closed state, the displayed annotation symbol is visible, and the displayed annotation symbol is designed to have low visibility in order to avoid distracting the user while the user is reading the web document. In the open state, additional information is displayed in the display area of the web browser. In the exemplary embodiment of FIG. 7, a displayed annotation symbol in the opened state is depicted. This displayed annotation symbol is the displayed annotation symbol for a question. In the exemplary embodiment of FIG. 7, the displayed annotation symbol

[29] for an opened question is shown together with an annotation emphasis text

[27] and an annotation comment box

[31] .

[0088] Annotation symbols are associated with elements of a group of semantic relationships of an annotation. Each semantic relationship of an annotation shows a relationship, typically based on user selection, between an annotation emphasis text

[27] and an annotation comment

[32] , as depicted in the exemplary embodiment of FIG. 7. Associated with each relationship within the group of semantic relationships of an annotation is a different annotation symbol. As an example, as depicted in the exemplary embodiment of FIG. 8, the displayed annotation symbol

[35] for a closed question indicates that the creator of the annotation has marked the annotation comment as a question regarding the annotation emphasis text. This displayed annotation symbol

[35] for a closed question enables a user familiar with how the exemplary embodiment functions to know that, before the user actually performs the action of clicking, as depicted in FIG. 7, clicking on the displayed annotation symbol

[35] for a closed question allows viewing of the annotation comment that is a question regarding the annotation emphasis text

[27] .

[0089] Once clicked, the displayed annotation symbol

[35] for a closed question becomes the displayed annotation symbol

[29] for an open question that shows the annotation emphasis text

[27] and the annotation comment box

[31] , like in the state of the exemplary embodiment depicted in FIG. 7, like in the state of the exemplary embodiment depicted in FIG. 8. Also, in the state of the exemplary embodiment depicted in FIG. 8, when the mouse hovers over the displayed annotation symbol

[35] for a closed question, the annotation emphasis text

[27] , the displayed annotation symbol

[29] for an open question, and the annotation comment box

[31] become visible and can be fixed by clicking. To make this process reflexive for the user, a color based on the annotation symbol can be assigned to the annotation emphasis text

[27] in FIG. 7. That is, the color of the annotation emphasis text

[27] is determined based on the semantic relationship of the annotation between the annotation emphasis text

[27] and the annotation comment.

[0090] To further illustrate the exemplary embodiment, observe the following. The displayed annotation symbol

[54] for a closed exclamation indicates that the annotation creator marked the annotation comment as an exclamation statement indicating a note or an interesting fact regarding the annotation emphasis text. The displayed annotation symbol

[55] for a closed check indicates that the annotation creator marked the annotation comment as a statement agreeing with the annotation emphasis text. Also, the closed incorrect annotation symbol

[56] indicates that the annotation creator marked the annotation comment as a statement not agreeing with the emphasized text of the annotation.

[0091] In addition to the annotation symbols discussed so far and the displayed annotation symbols depicted in FIG. 12, advertisement annotation symbols can be used to indicate another relationship within a group of semantic relationships of annotations. This relationship indicates that an annotation comment

[32] as in the state of the exemplary embodiment depicted in FIG. 7 is an advertisement related to the annotation emphasis text

[27] . In this situation, the displayed annotation symbol

[35] of a closed question will be replaced by the displayed annotation symbol of an advertisement, as in the state of the exemplary embodiment depicted in FIG. 8.

[0092] In the exemplary embodiment depicted in FIG. 12, several displayed annotation symbols can be seen. In this exemplary embodiment, corresponding to the question annotation symbol, the disagreement annotation symbol, the agreement annotation symbol, the exclamation annotation symbol, and the advertisement annotation symbol are five relationships in a group of semantic relationships of annotations. These semantic relationships of annotations are the question relationship, the disagreement relationship, the agreement relationship, the exclamation relationship, and the advertisement relationship.

[0093] In an exemplary embodiment, machine learning and deep learning models can be used to automatically determine the semantic relationships of annotations between annotation-emphasized text and annotation comments. Further, user data collected by the user determining the semantic relationships of annotations and writing annotation comments can be used to train deep learning models, and the by-products of these trained models can be used for other natural language processing purposes. For the use of data created by the user for the purpose of learning a deep learning-based natural language processing model, an opt-in or opt-out option can be provided to the user. To calculate sentence embeddings, a bidirectional long short-term memory (BiLSTM) model with a final max pooling layer can be used. Also, other machine learning and deep learning techniques can be used to create sentence embeddings. Sentence embeddings can be used to determine the similarity between two sentences. After calculating sentence embeddings for all sentences in the collection through the analysis of annotation-emphasized text, annotation comments, and other sentences occurring in items and replies, these sentence embedding vectors can be used to determine whether an item is similar to other items in the collection or to determine the similarity between two collections. Based on this similarity analysis, items can be proposed to the user for reading, discussion with other users, or addition to the collection. Also, based on the collections searched, browsed, or posted by the user, similar collections can be proposed to the user. Also, annotations on a web page can be linked to similar annotations and content in other web pages or documents through similarity analysis based on sentence embedding vectors. Also, in combination with a sentence similarity classifier, a convolutional neural network (CNN) model can be used to verify the semantic relationships of annotations specified by the user. The semantic relationship data of annotations provided by the user can be combined with the user reputation to verify the semantic relationships of annotations selected by the user and potentially identify inappropriate content generated and posted by the user.Bidirectional Encoder Representations from Transformers (BERT) can be used to identify named entities within web pages and web documents and automatically annotate items that a user may potentially be interested in. These named entities can be entities that exist in Wikipedia or products that are sold on online marketplaces. For moderation of user-generated annotations, pre-trained BERT fine-tuned for sentiment analysis can be used to flag inappropriate content generated by the user.

[0094] In the exemplary embodiment depicted in FIG. 13, a user can trigger a symbol beacon by pressing a symbol beacon trigger button

[57] to make a closed displayed annotation symbol more prominent. The symbol beacon can be in an untriggered state or a triggered state. When the symbol beacon trigger button

[57] is pressed, the symbol beacon transitions from an untriggered state to a triggered state. The trigger mechanism for the symbol beacon can also be a sequence of keyboard key presses or mouse button clicks. The trigger mechanism can also be placed in menu options within an annotatable web browser

[24] . It is the web browser extension

[25] instruction set that makes these triggers available, whether they are buttons available in the display area of the annotatable web browser

[24] , menu options of the annotatable web browser

[24] , combinations of keyboard key presses and mouse button clicks, input gestures on a touchscreen device, or input gestures via an input device, and triggers one displayed annotation symbol beacon

[58] for each displayed annotation symbol.

[0095] Next, a sequence diagram showing how messages are sent between different entities is shown. The messages include user input to an annotatable web browser

[24] or a normal web browser, and electronic messages between an electronic device and a computer system. The annotatable web browser

[24] is shown in the exemplary embodiment of FIG. 3. These sequence diagrams apply to the classes of the exemplary embodiments. After the sequence diagrams are presented, some message variations are presented. By more precisely identifying the messages, specific exemplary embodiments can be discussed.

[0096] Figure 14 is a sequence diagram applied to the classes of the exemplary embodiments. In Figure 14, the sequence diagram shows the interaction between user

[59] , an annotatable web browser

[24] , a URL server

[60] , an annotation server

[61] , an electronic database

[62] , and a content delivery network

[63] . When user

[59] visits a URL

[64] , the annotatable web browser

[24] requests a web document

[65] , and when the URL server

[60] returns the web document

[66] , the annotatable web browser

[24] displays the document to user

[59]

[67] . Next, user

[59] inputs an input for creating an annotation

[68] , and the annotatable web browser

[24] sends a bundle of data elements representing the annotation to the annotation server

[61]

[69] . The server sends an image for storage

[70] to the content delivery network

[63] and returns the URL of the sent image

[71] . Next, the annotation server

[61] instructs the electronic database

[62] to store the non-image bundle of data elements and the URL from the CDN as an annotation object

[72] , and the electronic database confirms the storage of the annotation object

[73] . Thereafter, the annotation server

[61] checks the received bundle of data elements and sends an annotation display bundle to the annotatable web browser

[24]

[74] , and displays the displayed annotation bundle indicating that the annotation has been successfully created to user

[59]

[75] . Reference is made in

[69] to a bundle of data elements representing the annotation. The bundle of data elements may vary depending on the details of the implementation of the exemplary embodiments and the type of annotation.

[0097] In FIG. 14, a sequence diagram applied to the classes of the exemplary embodiments is presented. Depending on the characteristics of the bundle of data elements representing the annotation or emphasized text, the annotation server

[61] records different data, and the annotatable web browser

[24] may receive different data and display different views due to the fact that it is selectable when generating the display representation of the annotation or emphasized text. Next, the screenshot bundle and the annotation symbol bundle that can serve the role of the bundle of data elements in FIG. 14 are discussed in the exemplary embodiments. Further, FIG. 14 may be applied to an exemplary embodiment where the annotation display bundle is the screenshot display bundle or the annotation symbol display bundle. Corresponding to the screenshot bundle is the screenshot display bundle, and corresponding to the annotation symbol bundle is the annotation symbol display bundle. These corresponding pairs are used when FIG. 14 uses the bundle of data elements and the annotation display bundle.

[0098] In an exemplary embodiment, a screenshot bundle includes a document's Uniform Resource Locator (URL), data of a selected text snippet from a document that includes a selected text snippet representing an annotation, and a marked screenshot of a portion of the document where the selected text snippet is visually marked to display the selected text snippet. The screenshot bundle can also include information for finding the selected text snippet within the document. This information for finding the portion of the selected text within the document typically includes Document Object Model (DOM) information. An annotation screenshot bundle is a screenshot bundle that also includes an annotation comment

[32] , as shown in the exemplary embodiment depicted in FIG. 7. The screenshot bundle can also include data for representing an annotation symbol. Also, a screenshot bundle that does not include the annotation comment

[32] can be used. In such a case, data representing emphasized text without the annotation comment

[32] is included in the screenshot display bundle, along with the relevant marked screenshot representing the emphasized text.

[0099] In an exemplary embodiment, an annotation symbol bundle is data for representing a Uniform Resource Locator (URL) of a document and a text snippet selected from the document, where the selected text snippet represents an annotation, an annotation comment

[32] , and an annotation symbol. The exemplary embodiment of FIG. 7 may be referred to. A screenshot bundle also includes information for finding a selected text snippet within a document. This information for finding a portion of the selected text within a document typically includes Document Object Model (DOM) information. The annotation symbol bundle may also include a marked screenshot of a portion of the document showing the selected text portion where the selected text portion is visually marked. Depending on the desired operation, the marked screenshot may or may not be required.

[0100] In an exemplary embodiment, the DOM information for finding a selected text portion within a document can include information for finding an element within the DOM that contains the first character of the selected text, an element within the DOM that contains the last character of the emphasized text, and information for finding the emphasized text itself. These two elements are referred to as the highlighting start element and the highlighting end element. The information recorded about the highlighting start element and the highlighting end element includes tags, IDs, classes, indexes, content, XPath, and parents. Additionally, the ancestor elements of the highlighted text start element are also recorded. The ancestor elements of the highlighting start element have tags, IDs, classes, indexes, and content recorded. To find the specified emphasized text at a later point in time, the DOM is searched using tags and IDs, XPath, tags and classes and indexes, tags and classes, tags and indexes, and tags only. Since the web page may be edited or the DOM structure of the web page may change as a result of the execution of JavaScript code that inserts elements into the DOM, a simple XPath search is not sufficient. In this exemplary embodiment, the goal is to find the selected text even when the web page has been edited. Multiple search methods are used to make it easier to find the selected text in different scenarios. Once the start element and the end element of the emphasized text are found, it becomes easier to find the selected emphasized text.

[0101] Another concern is that elements are inserted into the DOM of the visited web page by the JavaScript code of the web page. These inserted elements can be annotated. To facilitate this process, the browser's mutation observer can be used. When a change is detected, the DOM structure detection code can be re-executed. Once the start element and the end element of the emphasized text are found, the emphasized text can be inserted. In the case of annotated emphasized text, the displayed annotation symbol can also be inserted.

[0102] If the sent screenshot bundle or the screenshot marked with annotation symbol bundle is included, the image storage set is created from this marked screenshot by various operations including resizing, trimming, and image alteration. These operations are carried out to generate images that look good in different display areas on screens of different sizes. Once the storage set of this marked screenshot is created, it makes sense to decide whether to save these marked screenshots to a content delivery network (CDN). In the case of thumbnail-marked screenshots, there may be an advantage in saving the data together with all other data of the screenshot bundle or annotation symbol bundle. In the case of larger marked screenshots, it is usually better to save these images using a deliverable CDN. As shown in some figures, the annotation server

[61] usually decides which images in the storage set to store in the electronic database

[62] and which images to send to the CDN, as shown in some figures. The CDN typically returns a URL when an image is stored. These URLs are saved together with the rest of the screenshot bundle and annotation symbol bundle.

[0103] The deliverable screenshot is either the actual image stored in the electronic database

[62] or the URL, as shown in some figures. When sending data to the client device, the URL can be used to load the image from the CDN. Also, when sending the image directly, simply display the image. Finally, what is stored in the electronic database is the annotation object

[62] . The annotation object usually includes all the received data in the screenshot bundle or annotation symbol bundle and the deliverable screenshot of each marked screenshot in the storage set.

[0104] In order for an advertiser to reach customers with relevant ads related to the content that the user is currently engaged in, the ad screenshot bundle and the ad annotation symbol bundle can potentially be deployed together with their respective ad screenshot display bundles and ad annotation symbol display bundles. The displayed ad screenshot bundle is the visually displayed form of the data within the ad screenshot display bundle, and the displayed ad annotation symbol bundle is the visually displayed form of the data within the ad annotation symbol display bundle. Although Figure 7 can be used for illustration, the displayed annotation symbols are replaced with the displayed annotation symbols of the ad. In an exemplary embodiment, the ad screenshot bundle includes annotation comments

[32] created by the advertiser related to the annotation emphasis text

[27] . The ad annotation symbol bundle includes annotation symbols indicating the ad. The annotation symbol bundle includes annotation symbols, and the screenshot bundle may also include annotation symbols. While the screenshot bundle includes marked screenshots, the annotation symbol bundle may include marked screenshots. Although Figure 9 can be used for illustration, the displayed annotation symbols are replaced with the displayed annotation symbols of the ad. Further, the annotation comments are created by the advertiser together with other elements, and preferably, the overall form is similar to other annotation item boxes. When marked screenshots are included, creating a storage set and using a CDN to distribute these marked screenshots may be required to design a workable system. In an exemplary embodiment, these ad screenshot bundles and ad annotation symbol bundles are often stored and managed via an ad server that communicates with an annotation server. The ad server can be inserted although not explicitly shown in the sequence diagram.To incorporate one advertisement, the page size can be reduced by one. As shown in several figures, after the annotation server

[61] prepares a page of items representing the annotation, the annotation server

[61] can communicate with the advertisement server to obtain one advertisement.

[0105] The semantic web provides a semantic description of some web pages and documents available on the WWW through ontologies, which are implementations of a certain description logic. These ontologies need to be created for different domains and have the ability to represent concepts within those domains in logical form, as done in description logic. Collections of existing ontologies are published in projects such as KBPedia. Description logic is a type of logic defined in mathematical logic and has some characteristics of first-order logic. Depending on the characteristics of the logic family, the computational properties are different. Using the existing ontologies, logical statements can usually be created. In an exemplary embodiment, these logical statements can be translated into natural language, and some sentences in natural language can be translated into statements made within description logic using neural machine translation technology. Other deep learning techniques can also be used. Machine learning and deep learning models can translate some sentences in natural language into statements within a specific ontology. Within this exemplary embodiment, this methodology can be used for advertising purposes. Advertisers can express concepts related to a product or service within an ontology related to the product or service and write logical statements. These logical statements can then be input into an advertising platform. The advertising server can identify web pages and documents that contain natural language text that can be translated into statements within the ontology selected by the advertiser to represent concepts related to the product or service being advertised. Furthermore, the advertising server can determine which of these web pages and documents contain natural language text that represents statements similar to or logically related to those expressed by the advertiser. Once these web pages and documents are identified along with the corresponding locations within the documents, the advertiser can be given the opportunity to place an advertisement at a specific location that links to the web page, document, or that specific location within it. Using a logic analyzer, various results of the statements occurring in web pages and documents can be determined.Based on this information, web pages, documents, and locations within them can be narrowed down to find the most optimal locations for advertisers. Images, videos, and audio can be converted into text by means of image captioning, speech recognition, and textification. In some cases, this text can also be translated into statements in the same ontology used by the advertiser. After that, a logic analyzer is used to determine the most optimal candidates to show to the advertiser, which the advertiser can then consider. Based on the advertiser's selection, the advertising platform can generate the advertisements to be placed on the ad server. Further, in this exemplary embodiment, ontologies can be created for the purpose of representing facts about products and services sold in different domains such as insurance, financial services, legal services, healthcare, etc. These ontologies can then be used by various groups that advertise and sell services in these domains to create logically correct statements regarding their products and services for the purpose of searching media such as web pages, web documents, images, videos, and recordings. This media can then reach the audience that consumes this media, and the advertiser can know that there is a high likelihood that this audience is familiar with the statements made by the advertiser in the ontology. Further, in this exemplary embodiment, the media source itself has a logical relationship with the advertiser's statements, and the advertiser can use this logical relationship to provide more tailor-made products and more tailor-made targeting to meet the needs of customers.

[0106] To specify the details of an exemplary embodiment, as shown in FIG. 14, a bundle of data elements should be specified, along with details regarding the storage set and deliverable screenshots. The bundle of data elements can be a bundle of screenshots having their associated data elements. The bundle of data elements can also be a bundle of annotation symbols having their associated data elements. The storage set, size, resolution, cropping, and other characteristics of different types of marked screenshots should be determined. This is typically based on consideration of the type of display on which these marked screenshots will be presented, along with the size of the view being presented. In some cases, thumbnail-marked screenshots may be included in the storage set. If the size of these thumbnails is small, storing these images using a CDN may not be the most efficient approach. It may be more efficient to store the images in an electronic database

[62] , local file system, or other local location rather than in a content delivery network

[63] . Specifying whether the deliverable screenshots are actual images or URLs stored in an electronic database is another decision to consider when specifying implementation details. It will be apparent to those skilled in the art that various fine-grained variations can be introduced for image storage and retrieval. It will also be apparent to those skilled in the art that these variations are compatible and the differences are not substantial.

[0107] The electronic database

[62] in FIG. 14 and several other figures can take various forms, from relational databases to non-relational databases. Further, the electronic database

[62] may be a distributed electronic database system implemented on one or more computer systems. The most advantageous characteristics of the database can vary depending on the number of database clients and their usage patterns. In an exemplary embodiment, the annotation server

[61] is a database client. To handle the needs for consistency, availability, and durability, as well as numerous annotation servers with different latency requirements, the characteristics of the electronic database or distributed electronic database system may vary. Further, the distributed electronic database system can be implemented in various ways.

[0108] Furthermore, it should be noted that storing images in the electronic database

[62] may sometimes affect performance. In such cases, it is beneficial to store references in the database and use a different storage mechanism for specific images. In the case of a content delivery network

[63] , the reference is a URL. However, it is possible to use alternative mechanisms to store non-URL references and then use the references stored in the electronic database to store images in a searchable manner. In an exemplary embodiment, the images can be stored on the local file system, and the references in the database can correspond to the file paths. In such a case, the storage location with the electronic database should be considered a distributed electronic database that stores images in a more performant way. It will be apparent to those skilled in the art that various minor variations of this technique can be introduced for image storage and retrieval. It will also be apparent to those skilled in the art that these variations are interchangeable and the differences are not substantial.

[0109] Figure 15 is a sequence diagram applied to the classes of an exemplary embodiment. In order for Figure 15 to specify the details of the exemplary embodiment, an annotation display bundle and a displayed annotation bundle need to be specified. Further, Figure 15 may be applied to an exemplary embodiment in which the annotation display bundle is replaced with a screenshot display bundle and the displayed annotation bundle is replaced with a displayed screenshot bundle. In this case, it is necessary to specify a screenshot display bundle having data elements to be included, and it is also necessary to specify a mechanism for obtaining a deliverable screenshot. The user interface diagram, the data elements described for the screenshot bundle, and the considerations discussed above can be used to specify the annotation display bundle and the displayed annotation bundle. The displayed screenshot bundle is generally used when there is a screenshot marked with annotation emphasis without annotation comments. The displayed annotation screenshot bundle is used when there are marked screenshots, annotation emphasis, and annotation comments. Figure 15 is a sequence diagram showing the interaction between user

[59] , an annotatable web browser

[24] , an annotation server

[61] , an electronic database

[62] , and a content delivery network

[63] . When user

[59] visits the URL of the annotation server

[76] , the annotatable web browser

[24] requests the organize page of the annotation server from the annotation server

[61]

[77] , requests the annotation object required for page configuration from the electronic database

[62]

[78] , and an annotation object is returned to the annotation server

[61]

[79] . Next, the annotation server

[61] returns the organize page of the annotation server to the annotatable web browser

[24] together with the annotation display bundle

[80] , and requests an image of the annotation display bundle URL returned when the organize page of the annotation server is returned

[81] .Finally, the content delivery network

[63] returns the requested image to the annotatable web browser

[24] and displays an organized page with the annotation bundle presented to the user

[59] [82, 83]. Next, the user

[59] provides input to move the annotation to the annotatable web browser

[24]

[84] , and the annotation server

[61] requests that the annotation be moved by issuing a movement instruction for the annotation subject to the electronic database

[62] [85, 86]. In response, the electronic database

[62] approves the movement of the annotation object to the annotation server

[61]

[87] , the annotation server acknowledges that the annotation has moved

[88] , and the annotatable web browser

[24] causes the content with the annotation bundle presented at the new location to be presented to the user

[59] .

[0110] The presented annotation screenshot bundles and annotation item boxes are created using the data of the annotation screenshot display bundle sent from the annotation server

[61] , as shown in part of the figure. The annotation screenshot display bundle is created using the data of the annotation object obtained from the electronic database, as shown in some of the figures

[62] . As shown in the exemplary embodiment of FIG. 10, when displaying the annotation page in the screenshot bundle display area such as the item bin

[44] of the annotation editing context area

[43] of the organize web page, the data sent from the annotation server

[61] is the page where the thumbnail display bundle is displayed. These thumbnail display bundles include at least the deliverable screenshots for the thumbnail-marked screenshots within the storage group of the annotation objects stored in the electronic database

[62] . FIG. 9 can be used for illustration. In the exemplary embodiment, the thumbnail display bundle also includes data representing separate context-annotation emphasis text

[38] , separate context-annotation comments

[40] , and the displayed annotation symbols

[39] of separate context questions in a readable format that can be displayed by a web browser. Further, the separate context-annotation emphasis text

[38] and the displayed annotation symbols

[39] of separate context questions are color-coded. By all these data working together, the user can obtain more information at a glance. Also, the screenshot of the displayed annotation can be displayed on the annotation detail page, providing more detailed information about one annotation and many user comments and replies regarding that annotation.

[0111] FIG. 16 is a sequence diagram applied to the classes of the exemplary embodiments. In order for FIG. 16 to explicitly specify the exemplary embodiments, an annotation symbol display bundle having data elements to be included along with the visual appearance and the corresponding displayed annotation symbol bundle need to be specified. For this specification, the user interface diagram, the data elements described for the annotation symbol bundle, and the considerations described above can be used. In FIG. 16, it is a sequence diagram showing the interaction between user

[59] , annotatable web browser

[24] , URL server

[60] , annotation server

[61] , and electronic database

[62] . When user

[59] visits a URL

[90] , the annotatable web browser

[24] requests a web document from the URL server

[60]

[91] , and the server returns the web document to the annotatable web browser

[24]

[92] . Next, the annotatable web browser

[24] requests an annotation symbol display bundle representing an annotation from the annotation server

[61]

[93] , and the annotation server requests an annotation object from the electronic database

[62]

[94] . Thereafter, the electronic database

[62] returns the annotation object to the annotation server

[61]

[95] , and the annotation server returns an annotation symbol display bundle representing an annotation to the annotatable web browser

[24]

[96] , and displays the web document together with the closed displayed annotation symbol and an annotation that can be opened when moused over or clicked

[97] .

[0112] Figure 17 shows a flowchart applied to a class of exemplary embodiments. A particular exemplary embodiment can be specified if a bundle of data elements to be received and stored is accurately specified. For example, a screenshot bundle can be explicitly specified along with all of its data elements, or an annotation symbol bundle can be explicitly specified along with all of its elements. In this case, the screenshot bundle or the annotation symbol bundle becomes the bundle of data elements to be received. Further, if the bundle of data elements includes marked screenshots, it is necessary to specify the storage set and storage mechanism of the screenshots to be distributed. These details are useful for explaining the implementation. In an exemplary embodiment, the storage set can include marked screenshots for display on a large monitor, a notebook computer, a tablet, and a telephone, as well as thumbnail-marked screenshots for an item box. The exact size can be adjusted to the most common size currently in use for these displays. Further, the server receives a bundle of data elements representing an annotation

[98] in a client-server architecture and stores the received data elements and newly created elements using one or more computer systems to satisfy a request made by an annotatable web browser of the user's client device that creates an annotation or simple emphasis

[99] . In FIG. 17, a flowchart of a method implemented by one or more computer systems representing a server in a client-server architecture is shown. This method is implemented by the server to enable sharing of data that can be requested by the client user interface of the same client or different clients. The steps of this method are to receive a bundle of data elements representing an annotation

[98] and store the received data elements and newly created elements using one or more computer systems

[99] .

[0113] FIG. 18 shows a flowchart applied to a class of exemplary embodiments. When the data elements included in the bundle display set are specified exactly, a particular exemplary embodiment can be specified. This means specifying the data elements included in the display bundle. In an exemplary embodiment, the display bundle can be a screenshot display bundle or an annotation symbol display bundle. When the display bundle is a screenshot display bundle, the bundle display set becomes the screenshot display bundle set of FIG. 18. When the display bundle is an annotation symbol display bundle, the bundle display set will be the annotation symbol bundle display set of FIG. 18. Further, the server, in a client-server architecture, compiles a bundle display set including pages of the display bundle using one or more computer systems, and in order to satisfy a request made by an annotation-capable web browser running on a client device of a user who views pages of annotations within a web document or another context area, the server transmits the bundle display set in a format in which the display bundle can be displayed by the web browser running on the electronic device. The flowchart shown in FIG. 18 is implemented by one or more computer systems.

[0114] FIG. 19 is a flowchart applied to a class of exemplary embodiments. Certain exemplary embodiments can be specified such that a bundle of data elements is explicitly specified. For example, a bundle of screenshots having the data elements to be included can be explicitly specified, or a bundle of annotation symbols having the data elements to be included can be explicitly specified. In FIG. 19, in an exemplary embodiment, a flowchart of annotation creation is shown which is implemented by a set of instructions of a web browser extension that converts a web browser that supports the addition of a browser extension into an annotatable web browser

[24] . As shown in some of the figures, annotation creation is implemented by a browser extension. FIG. 19 shows the steps executed by the user

[59] , the web browser

[0102] , the web browser extension content script

[0103] , and the web browser extension background script

[0104] when creating an annotation. The user

[59] opens the browser

[0105] , and the web browser

[0102] loads the browser extension

[0106] . Next, the user

[59] visits a URL

[0107] , the web browser

[0102] reads the data sent from the accessed URL

[0108] , and injects the content script

[0109] . Here, the user

[59] browses the web page

[0110] and highlights the text

[0111] . At this time, as a result of the web browser extension content script

[0103] executing the instructions of the web browser extension, the annotation symbol selection button bar becomes available

[0112] . Next, the user

[59] presses a button corresponding to one of the annotation symbols on the annotation symbol selection button bar

[0113] , and the web browser extension content script

[0103] accordingly makes the annotation input box available

[0114] . Next, when the user

[59] enters text and presses the enter key

[0115] , the web browser extension content script

[0103] sends the bundle of data elements to the background script

[0116] At this time, the background script

[0104] of the web browser extension receives a bundle of data elements from the content script

[0117] , and sends the bundle of data elements to the annotation server

[0118] .

[0115] In FIG. 19, after the user

[59] enters text and presses Enter

[0115] , it is desirable for the web browser extension content script

[0103] to generate either an annotation symbol bundle or a screenshot bundle. An exemplary embodiment may require that the annotation symbol bundle includes a marked screenshot. If the marked screenshot is generated on the server side, the web page may have to be re-rendered or the screenshot may have to be marked. Doing so increases the computational load and makes errors more likely to occur. Generating a marked screenshot using the browser extension does not have these problems. In the browser extension, the user's privacy must be securely protected. The screenshot should not include the desktop or other browser tabs. Further, in an exemplary embodiment, it is desirable for the screenshot to be attached with annotation emphasized text

[27] and an annotation comment box

[31] . The annotation emphasized text

[27] and the annotation comment box

[31] are shown in FIG. 7 and may be useful for explanation in this case. In an exemplary embodiment, a simple delay can be implemented before the screenshot is taken. This delay should be sufficient to ensure that the annotation emphasized text

[27] and the annotation comment box

[31] are present on most devices. In an alternative exemplary embodiment, a more performant method can be implemented, and the web browser extension content script

[0103] searches for the presence of the annotation emphasized text

[27] and the annotation comment box

[31] in the DOM structure of the document and takes a screenshot if these elements are present. The search for the presence of the annotation emphasized text

[27] and the annotation comment box

[31] in the document DOM structure will occur at periodic intervals on a shorter time scale than the simple delay time described above.In an exemplary embodiment, the periodic interval can be one third of the length of a simple delay that would be selected.

[0116] Figure 20 is a flowchart applied to a class of exemplary embodiments. A specific exemplary embodiment can be specified if all of its data elements are explicitly specified in the displayed annotation symbol bundle. In Figure 20, in an exemplary embodiment, a flowchart of the display of an annotation showing steps performed by an instruction set of a web browser extension that converts a web browser supporting the addition of a browser extension into an annotatable web browser is presented. Figure 20 shows steps performed by user

[59] , web browser

[0102] , web browser extension content script

[0103] , and web browser extension background script

[0104] when viewing an annotation. User

[59] opens a browser

[0119] , and web browser

[0102] loads the browser extension

[0120] . Next, user

[59] accesses a URL

[0121] , and web browser

[0102] reads data sent from the accessed URL

[0122] and injects the content script

[0123] . Here, web browser extension background script

[0104] reads an annotation symbol display bundle representing an annotation for the URL

[0124] and sends the annotation symbol display bundle to the content script

[0125] . Now, web browser extension content script

[0103] receives the annotation symbol display bundle from the background script

[0126] and displays closed annotation symbols on the web page

[0127] . Now, user

[59] views the web page with the closed annotation symbols displayed

[0128] and clicks on the closed annotation symbols displayed

[0129] In response to this, the web browser extension content script

[0103] displays the annotation symbol in an open state.

[0130] Next, the user

[59] views the web page on which the selected annotation symbol is displayed in an open format.

[0131] .

[0117] In the exemplary embodiment depicted in FIG. 31, a collection item web page is shown. The collection item page context area

[0258] is the display area of the displayed item, the collection item paging mechanism

[0259] , the collection item search filter

[0260] , and the collection action menu trigger

[0261] . The collection action menu

[0262] displays the option to send a message to the collection

[0263] , the user invitation option

[0264] , and the user permission change option

[0265] . The collection action menu trigger

[0261] is used to trigger the collection action menu

[0262] . The collection item paging mechanism

[0259] may be used to change the page of the displayed collection displayed in the collection item page context area

[0258] . The option to send a message to the collection

[0263] creates a message chain with the first text message and sends it to the members of the collection who have more than read-only permissions for the collection. The message chain is displayed in the user's inbox. An exemplary embodiment of the user's inbox is depicted in FIG. 33. In the exemplary embodiment of FIG. 33, the American History collection message chain

[0281] is shown. Messages sent to the members of the collection using the option to send a message to the collection

[0263] are displayed in the recipient's inbox classified under the collection to which the message was sent. Among the members belonging to the collection, all members with more than read permissions can read the message, and all members with more than message permissions can reply. The user invitation option

[0264] is used to invite new users to the collection. The invitation is sent to the invited user's inbox. The user permission change option

[0265] can be used to change the permissions of the members of the collection.In the form of a message within a message chain, a notification of the change may be sent to the receiving tray of the user whose permission has been changed.

[0118] A collection item filter, such as the collection item search filter shown in the exemplary embodiment of FIG. 31

[0260] , is used to receive input from a user. In the case of the collection item page context area

[0258] and the collection item search filter

[0260] , it is a collection item search phrase. These inputs are converted into collection item parameter values that are sent to the annotation server

[61] , as shown in some of the figures. These collection item parameter values are used by the annotation server

[61] to input collection item parameters into a collection item request query used by the electronic database

[62] , as shown in some of the figures. The collection item request query includes collection item parameter values and a collection item page number. The collection item request query containing the collection item parameter values is used to specify a set of collection items and their sequential order. The collection item page number specifies a particular collection item page. Further, the collection item request query can include a specified number of collection items to be sent to a web browser running on a client electronic device. This collection item request query includes collection item parameters of collection item filter values such as a collection item search phrase, an exclusion list of collection items to be excluded from the list of specified collection items, and a sequential ordering for that list of collection items.

[0119] In the exemplary embodiment depicted in FIG. 32, a collection web page is shown. The collection page context area [

[0266] ] is a display area for the displayed collection, the collection page paging mechanism [

[0267] ], the collection page search filter [

[0268] ], and the collection page action menu trigger [

[0269] ]. The collection page action menu [

[0270] ] displays ownership filter options [

[0271] ], administratable filter options [

[0272] ], writable filter options [

[0273] ], messageable filter options [

[0274] ], viewable filter options [

[0275] ], and subscribe filter options [

[0276] ]. The collection page action menu trigger [

[0269] ] is used to trigger the action menu [

[0270] ] of the collection page. The paging mechanism [

[0267] ] of the collection page can be used to change the web page of the displayed collection shown in the collection page context area [

[0266] ]. The ownership filter options [

[0271] ], administratable filter options [

[0272] ], writable filter options [

[0273] ], messageable filter options [

[0274] ], viewable filter options [

[0275] ], and subscribe filter options [

[0276] ] filter the collection based on user permissions. The ownership permission is the highest level of permission, and the ownership permission is given to the creator and owner of the collection. The ownership permission enables other users to be given admin permissions for that collection and do anything permitted by the lower-level permissions. Admin permissions are lower-level permissions than ownership permissions and can elevate non-admin permissions, handle other administrative tasks, and execute anything permitted by the lower-level permissions.The writable permission is lower than the admin permission, and the user can add items to the collection and do anything permitted by the lower-level permissions. The messageable permission is lower than the writable permission, and the user can send messages to other members of the collection and do anything permitted by the viewable permission. The readable permission is lower than the messageable permission, and the user can view all items in the collection and read all collection messages for that collection. The subscribe permission gives the user readable permission for public collections. The subscribe permission is usually granted to the user who requested the permission without admin intervention because the messages associated with the collection are public. To enable the user to visualize the privacy settings, the background color is assigned based on color. Colors associated with shorter wavelengths mean greater permissions, and colors associated with longer wavelengths mean lesser permissions. That is, the colors of the rainbow are associated with permissions. Purple corresponds to the collections the user owns. Blue corresponds to the admirable collections. Green corresponds to the writable collections. Yellow corresponds to the messageable collections. Orange corresponds to the viewable collections. And red corresponds to the subscribe collections. For design purposes, variations of these colors in pastel colors may be used. Or, another set of backgrounds that mean a spectrum of values can be used. These backgrounds are applied to the collections shown in the exemplary embodiment of FIG. 32. For the exemplary embodiment of FIG. 31, the background of the web page is changed to correspond to the color associated with the permission of that collection.

[0120] In an exemplary embodiment, to facilitate scanning through a large number of items that contain annotations in a single collection, the annotations can be grouped into a stack set of displayed annotation items. Such a stack set of displayed annotation items appears as a single item with the visual appearance of stacked items. It is beneficial to place annotations from the same URL, up to nine annotations or another fixed configurable value, the displayed annotation items, into a stack set of displayed annotation items. To facilitate such a display, the paging mechanism needs to support the return of items in this format. As an example, imagine that 8 out of 10 items are annotation items that belong to the same URL. When these 8 items are grouped into a stack set of displayed annotation items, there will be 3 items on the collection item page. In order to ensure that there are 10 items even when a stack set of displayed annotation items is used, more items need to be returned by the annotation server

[61] . In an exemplary embodiment, a relational database is used and a complex structured query language (SQL) query using window functions may be required. Generally, an object-relational mapping (ORM) system is used by most up-to-date web frameworks to access a relational database . In an exemplary embodiment, the ORM system did not support window functions. To overcome this limitation, the underlying relational algebra library used by the ORM system was used. Using this relational algebra, complex SQL queries were generated using appropriate window functions. This made it possible to display a page of collection items with a set of displayed annotation items stacked on top of each other.

[0121] Collection page search filters such as the exemplary embodiment shown in FIG. 32 [

[0268] ] and other collection filters are used to receive input from a user. Collection page context area [

[0266] ] In the case of the collection page search filter [

[0268] ], it is a collection search phrase. These inputs are converted into collection parameter values that are sent to the annotation server

[61] , as shown in some of the figures. These collection parameter values are used, as shown in some of the figures, for the annotation server

[61] to input collection parameters into the collection request query used by the electronic database

[62] . The collection request query includes collection parameter values and a collection page number. The collection request query containing the collection parameter values is used to specify a set of collections and an ordering for those collections. The collection page number specifies a particular collection page. Further, the collection request query can include the number of collections to be sent to the web browser running on the client electronic device. This collection request query includes a collection parameter of collection filter values such as a collection search phrase, an exclusion list of collections to be excluded from the list of the specified collections, and a sequential ordering for the list of those collections.

[0122] In the exemplary embodiment depicted in FIG. 33, a received tray page is shown. The collection message chain section and title

[0277] includes an American history collection message chain section and title

[0279] and a car collection message chain section and title

[0280] . A collection message chain is a message chain related to a collection. These collection message chains appear in the "Collection Message" section under the name of the collection to which the collection message chain belongs. In this exemplary embodiment, the section and title

[0279] of the American history collection message chain includes the American history collection message chain

[0281] , and the section and title

[0280] of the car collection message chain lists the car collection message chain 1

[0282] and the car collection message chain 2

[0283] . A tagless message chain is a message chain not associated with any collection. A tagless message chain is displayed in the "Tagless Message" section. In the exemplary embodiment, the tagless message chain section and title

[0278] includes the tagless message chain 1

[0284] , the tagless message chain 2

[0285] , and the tagless message chain 3

[0286] . When the user clicks on a message chain, the message chain is displayed in the message chain context area

[0287] . In the message chain context area

[0287] , the first system message

[0288] of the American history message chain is displayed along with the system message acceptance button

[0289] and the system message rejection button

[0290] . The user can press the system message acceptance button

[0289] to accept an invitation to a collection or press the system message rejection button

[0290] to reject an invitation to a collection.

[0123] In the exemplary embodiment depicted in FIG. 34, after a user accepts an invitation to participate in a collection, a received tray page is displayed. The system message notification

[0291] indicates that the user has accepted an invitation to participate in a collection. The message chain message splitter

[0292] separates a certain user's message from other users' messages. Also, it should be apparent to those of ordinary skill in the art that the message chain message splitter

[0292] can be implemented using various visual design techniques to separate the message of the user in this context from other users' messages. Next, the first text message

[0293] of the American history message chain, the second text message

[0294] of the American history message chain, and the third text message

[0295] of the American history message chain are shown. And the message creation section and title

[0296] are displayed together with the message creation box

[0297] and the message send button

[0298] . This received tray has a connection to the permission management system and the collection. When permissions are granted to a message chain, all other messages within the message chain will be displayed even for users who originally did not have access rights when the message chain was created. This facilitates discussions, and access is controlled at the message chain level. Further, the system's message chains typically ensure that the collection administrator obtains information about the activities and permissions of group members. Additionally, users with message capabilities can send a message chain to a user within the collection or a selected group within the collection in one message. The ability to create a collection chain clarifies the subject of discussion and allows users to focus their attention on the collections and tasks they want to perform intensively.

[0124] The inbox is an effective means of inviting the user to a collection when the user is already on the site. Also, the inbox is effective for sharing items with others when the user is already on the site. To facilitate sharing of items and collections when the user is not on the site, links with cryptographic hashes are used. This link is impossible to guess, and the user with the link does not need to create an account to access the link. One can share just by possessing the link. Also, public items such as annotations and collections can be shared through social media.

[0125] In FIG. 21, as shown in some of the figures, a flowchart of a nurse using an exemplary embodiment of an annotatable web browser

[24] is shown. This nurse wants to teach the patient that rice is not good for diabetic patients and then show that the favorite dishes can be modified to be more friendly for diabetic patients. Once this memo is created, it can be shared with the patient. First day

[0132] The nurse visits the Wikipedia page for the glycemic index, creates a check annotation highlighting "supported by the American Diabetes Association" and adding the annotation comment "most physicians strongly believe that high glycemic index foods have an adverse effect on the health of diabetics"

[0134] , and adds the annotation to the memo

[0135] . Next, the nurse visits allrecipes.com, finds the recipe "Flavorful Spanish Rice and Beans", highlights "1 cup of raw white rice", creates an exclamation annotation with the annotation comment "the glycemic index of rice is very high, but the same recipe can be made with pearl barley"

[0136] , and adds the annotation to the memo

[0137] . Then, the nurse visits the Harvard Health website, finds the page on the glycemic index of over 100 foods, highlights "cooked white rice*", adds an exclamation annotation, stating "most people are not aware that white rice has such a high glycemic index. Note that it is 73, higher than most foods."

[0138] , and adds the annotation to the memo

[0139] . Next, the nurse visits the Harvard Health website again, finds the page on the glycemic index of over 100 foods, highlights "pearl barley" with a dagger annotation, stating "most people are not aware that pearl barley has such a low glycemic index. Note that it is 28, lower than most foods."

[0140] , and adds the annotation to the memo

[0141] . Then, the nurse visits the Target website, finds the product page for pearl barley, selects "Pearl Barley - 1lb - Market Pantry" with a dagger annotation, and writes "Pearl barley can be cooked in a rice cooker just like rice. Additionally, cooked pearl barley has a grain-like appearance and texture and is ideal as a substitute for rice. It can be used in recipes that call for rice instead of rice."

[0142] , and adds the annotation to the memo

[0143] . After creating the memo, on the second day

[0133] The nurse shares a memo with a patient who likes to make Spanish rice with diabetes

[0144] .

[0126] In FIG. 22, a flowchart of a patient using an exemplary embodiment of an annotatable web browser

[24] with a memo function is shown to view a memo created by a nurse. Patient

[0145] on the second day opens the memo link, clicks on the first displayed annotation screenshot bundle embedded in the memo

[0147] , reads the Wikipedia title of the glycemic index, reads the emphasized text "supported by the American Diabetes Association" after observing the checkmark-displayed annotation symbol, and reads aloud the annotation comment "Most doctors strongly believe that high glycemic index foods have an adverse effect on the health of diabetic patients"

[0148] . Next, the patient clicks on the second displayed annotation screenshot bundle embedded in the memo

[0149] , reads the title "Flavorful Spanish Rice and Beans" on allrecipes.com, sees the exclamation-mark-displayed annotation symbol, reads the emphasized text "1 cup of raw white rice", and then reads the annotation comment "The glycemic index of rice is very high, but the same recipe can be made with pearl barley"

[0150] . Thereafter, the patient clicks on the third displayed annotation screenshot bundle embedded in the memo

[0151] , reads the title "Glycemic Index of Over 100 Foods" on the Harvard Health website, reads the emphasized text "Boiled white rice*" after seeing the exclamation annotation symbol, and reads the annotation comment "Most people are not aware that the glycemic index of white rice is so high. Note that it is 73, higher than most foods."

[0152] . Next, the patient clicks on the fourth displayed annotation screenshot bundle embedded in the memo

[0153] , reads the title "Glycemic Index of Over 100 Foods" on the Harvard Health website, reads the emphasized text after seeing the dagger-displayed annotation symbol, reads "pearl barley", and "Many people are not aware that the glycemic index of pearl barley is so low.Note the annotation comment: "Note that it is lower than most foods at 28."

[0154] . The patient then clicks on the fifth annotated screenshot bundle embedded in the notepad

[0155] , reads the title of the target website, notices that it is the product page for pearl barley, reads the highlighted text "Pearl Barley - 1 lb - Market Pantry" after seeing the dagger-displayed annotation symbol, and reads the annotation comment stating that "Pearl barley can be cooked in a rice cooker just like rice. Additionally, cooked pearl barley has the appearance and texture of a grain and can substitute for rice. This can be used in recipes that call for rice instead of rice."

[0156] . After understanding the contents of the notepad, on the day of grocery shopping

[0146] , the patient goes to Target, buys pearl barley, and makes "Flavorful Spanish Barley and Beans"

[0157] .

[0127] In the exemplary embodiment depicted in FIG. 23, an annotatable web browser

[24] with a notepad function is presented. The annotatable web browser with a notepad function tracks whether an active notepad exists and enables the user to select an active notepad from a group of notepads accessible to the user. FIG. 23 shows an active notepad

[0159] and an annotation

[0160] embedded in the notepad. When an active notepad

[0159] exists, the annotatable web browser with a notepad function is in a state where an active notepad exists, and when an active notepad

[0159] does not exist, the annotatable web browser with a notepad function is in a state where an active notepad does not exist. The displayed notepad contains many notepad-embedded annotations

[0160] . The notepad-embedded annotation

[0160] is a container that includes a displayed annotation bundle indicating the data included in the annotation display bundle. In the exemplary embodiment, the notepad-embedded annotation

[0160] includes a displayed annotation screenshot bundle that also includes an annotation symbol. In an alternative embodiment, the notepad-embedded annotation

[0160] can include a displayed screenshot bundle that does not include an annotation symbol or an annotation comment. The user can add an annotation to the notepad by clicking on the add icon button

[0158] in the notepad depicted in FIG. 23. The notepad includes a list of references to annotation objects. As shown in some of the figures, the electronic database

[62] stores notepad objects along with a list of references to annotation objects. The annotation server

[61] , as shown in some of the figures, upon receiving a request from the annotatable web browser

[24] , retrieves the notepad object and the annotation object from the electronic database

[62] and sends the notepad to the annotatable web browser. It is displayed on the web browser

[24] . When the active notepad

[0159] is displayed, the user can navigate to the annotation by simply clicking on the notepad-embedded annotation

[0160] as depicted in the notepad-embedded annotation

[0160] of FIG. 23. In the classes of the exemplary embodiments, each notepad is associated with a bundle display set. In an exemplary embodiment, each notepad may be associated with a screenshot bundle display set. In another exemplary embodiment, each notepad may be associated with an annotation screenshot bundle display set.

[0128] The notepad contains rich information through links to other information sources along with annotation comments. This information may be reinforced by the description written by the creator of the notepad. Using these notepads, presentations can be created in various formats including PDF, Microsoft Word, and various other open source and proprietary document formats. These presentations are created by sequentially including, for each annotation, the relevant screenshot

[37] containing the marked text, the separate context-annotation highlighted text

[38] , the displayed annotation symbol

[39] of the separate context question, and the separate context-annotation comment

[40] as shown in FIG. 9. Further, this information can be reinforced by the text summary of the linked article generated using natural language processing technology along with image captioning technology. Further, the user can create explanatory texts to add to the specified locations, and these explanatory texts will be included in the output of the presentation.

[0129] In the exemplary embodiment depicted in FIG. 24, the interaction among a user

[59] for creating a memo, an annotatable web browser with a memo function

[0161] , a URL server

[60] , an annotation server

[61] , an electronic database

[62] , and a content delivery network

[63] is shown by a sequence diagram. First, the user

[59] visits the URL of the annotatable web browser with a memo function

[0161]

[0162] , requests a web document from the URL server

[60]

[0163] , the URL server returns the web document to the annotatable web browser with a memo function

[0161]

[0164] , and displays the web document to the user

[59]

[0165] . Next, the user

[59] makes an input for creating a memo of "flavorful Spanish barley and beans" in the annotatable web browser with a memo function

[0161]

[0166] , sends a request to create a memo of "flavorful Spanish barley and beans" to the memo server

[61]

[0167] , and sends an instruction to create a memo object to the electronic database

[62]

[0168] . Next, the electronic database

[62] permits the annotation server

[61] to create a memo object

[0169] , the annotation server permits the annotatable web browser with a memo function

[0161] to create a memo

[0170] , and displays the created memo to the user

[59]

[0171] . Next, the user

[59] makes an input for creating a blood glucose annotation

[0172] , sends an annotation screen bundle to the annotation server

[61]

[0173] , and the server sends an image for storage to the content delivery network

[63] as needed

[0174] . The content delivery network

[63] returns the URL of the stored image to the annotation server

[61]

[0175] , and sends an instruction to create a glycemic index annotation object with the CDN's URL to the electronic database

[62]

[0176] .Next, the electronic database

[62] permits the annotation server

[61] to create an annotation object. The annotation server sends the received annotation screenshot bundle to an annotatable web browser with a notepad function

[0161] and the displayed annotation screenshot bundle is presented to the user

[59] . Next, the user

[59] adds a blood glucose value annotation to the notepad of the annotatable web browser with a notepad function

[0161] . The annotation server

[61] requests to add a blood glucose value annotation object reference to the notepad, and an instruction to add a glycemic index annotation object reference to the notepad is sent to the electronic database

[62] . Then, the electronic database

[62] permits the annotation server

[61] to add an annotation object reference. The annotation server permits the annotatable web browser with a notepad function

[0161] to add a blood glucose value annotation object reference to the notepad, and a web document with a notepad to which the displayed annotation screenshot bundle is added is shown to the user

[59] .

[0130] In the exemplary embodiment depicted in FIG. 25, the sequence diagram shows the interactions among a user

[59] , an annotatable web browser with a notepad function

[0161] , a URL server

[60] , an annotation server

[61] , an electronic database

[62] , and a content delivery network

[63] for the purpose of viewing a notepad. First, the user

[59] accesses an arbitrary URL with an annotatable web browser with a notepad function

[0161]

[0186] , requests a notepad of "tasty Spanish barley and beans" from the annotation server

[61]

[0187] , and requests a notepad object from the electronic database

[62]

[0188] . Next, the electronic database

[62] returns the notepad object to the annotation server

[61]

[0189] , and the annotation server returns the notepad to the annotatable web browser having the notepad function

[0161]

[0190] . Next, the annotatable web browser with a notepad function

[0161] requests an annotation screen display bundle with blood glucose value annotation symbols from the annotation server

[61]

[0191] , requests a blood glucose value annotation object from the electronic database

[62]

[0192] , returns the annotation object with a CDN URL to the annotation server

[61]

[0193] , and returns the annotation screen display bundle with the CDN URL and annotation symbol data to the annotatable web browser with a notepad function

[0161]

[0194] . Next, the annotatable web browser with a notepad function

[0161] requests an image from the CDN using the CDN URL from the content delivery network

[63]

[0195] , returns the image from the CDN of the specified URL to the annotatable web browser with a notepad function

[0161]

[0196] , and displays the displayed notepad to the user

[59]

[0197] .Next, the user

[59] clicks on the first displayed annotation screenshot bundle in the annotation-enabled web browser with a notepad function

[0161] , requests the web document from the server for the URL

[60]

[0199] , the web document is returned to the annotation-enabled web browser with a notepad function

[0161]

[0200] , and the displayed annotation symbol in the open state is displayed on the web document

[0201] . Then, the annotation-enabled web browser with a notepad function

[0161] displays to the user

[59] the web document with the notepad placed horizontally with the displayed annotation symbol open and the browser scrolled to an appropriate position

[0202] .

[0131] In the exemplary embodiment depicted in FIG. 26, a flowchart of memo creation is presented that shows the steps executed by a set of instructions of a web browser extension that converts a web browser that supports the addition of a browser extension into an annotatable web browser with a notepad function. User

[59] opens the browser

[0203] , and then, the web browser

[0102] loads the browser extension

[0204] . Next, user

[59] visits a URL

[0205] . Thereafter, the web browser

[0102] reads the data sent from the visited URL

[0206] and injects a content script

[0207] . Thereafter, user

[59] browses the web page

[0208] and creates a memo

[0209] . As a result, the web browser extension content script

[0103] creates a memo

[0210] , and the web browser extension background script

[0104] sends a memo creation request to the annotation server

[0211] . Next, the web browser extension content script

[0103] displays the displayed memo

[0212] , and user

[59] browses the displayed memo and adds an already created annotation to the memo

[0213] . Thereafter, the web browser extension content script

[0103] adds the already created annotation to the memo

[0214] , and the web browser extension background script

[0104] sends a request to add a reference to an annotation object to the annotation server to the memo

[0215] .

[0132] In the exemplary embodiment depicted in FIG. 27, a flowchart of a notepad display showing the steps performed by a set of instructions of a web browser extension that converts a web browser that supports the addition of a browser extension into an annotatable web browser with a notepad function is presented. User

[59] opens the browser

[0216] , and the web browser

[0102] loads the browser extension

[0217] . Thereafter, user

[59] visits any URL

[0218] . Next, the web browser

[0102] reads the data sent from the visited URL

[0219] and injects a content script

[0220] . Thereafter, user

[59] opens the notepad

[0221] , and the web browser extension content script

[0103] requests the notepad

[0222] . Next, the web browser extension background script

[0104] requests from the annotation server a notepad of the annotation screenshot display bundle and annotation symbol data for all corresponding annotation objects in the reference list to the annotation object

[0223] , and the web browser extension content script

[0103] displays the displayed notepad

[0224] . Thereafter, user

[59] views the displayed notepad, clicks on the first displayed annotation screenshot bundle

[0225] , and the web browser extension content script

[0103] sends a message to the browser to visit the URL

[0226] . Then, the web browser

[0102] accesses the URL

[0227] and reads the data sent from the URL

[0228] . Next, the web browser extension background script

[0104] requests additional annotation object information such as a reply to the annotation comment from the annotation server

[0229] and obtains the additional annotation object information from the annotation server

[0230] . Next, the web browser extension content script

[0103] displays the displayed annotation symbol in an open state

[0231] and scrolls to the position of the displayed annotation symbol

[0232] .Thereafter, user

[59] views the page scrolled to the position of the annotation symbol displayed in the open state

[0233] .

[0133] In the exemplary embodiment depicted in FIG. 28, a system diagram is presented showing how a first user

[0234] , a second user

[0235] , a third user

[0236] , the browser of the first user

[0237] , the browser of the second user

[0238] , the browser of the third user

[0239] , an annotation server

[61] , a content delivery network

[63] , and an electronic database

[62] interact. The user communicates with the annotation server

[61] , and the annotation server pushes an image to the content delivery network

[63] . Also, the user communicates with the content delivery network

[63] to obtain images and other files. The user does not communicate directly with the electronic database

[62] . In this exemplary embodiment, the annotation server

[61] is placed on one computer system, and the electronic database

[62] is placed on a second computer system. By placing the electronic database

[62] on the same machine as the annotation server

[61] , the number of machines used can be reduced. The issue here is the number of concurrent users of the system. If this issue is resolved, it may be possible to determine the size of the system. The content delivery network

[63] in this exemplary embodiment is operated by a third party. The content delivery network (CDN) may be provided by a third party. Many third parties provide CDNs. These CDNs can be configured using a web interface, and files can be uploaded and accessed via an application programming interface (API). This API may be a REST (representational state transfer) API. It should be apparent to those skilled in the art how to configure and use such a third-party CDN based on the documentation of that third party.

[0134] In the exemplary embodiment depicted in FIG. 29, a component diagram is presented showing a way to build a more robust system that enables separating different components. Separating concerns can be advantageous for improving security and facilitating reproducible software deployment to a network of machines. The added security, separation of concerns, and reproducible deployment can ease the scaling of a system for handling a large number of users. FIG. 29 is a diagram showing a simplified version of an annotation server

[61] and an electronic database

[62] that communicate next to an alternative exemplary embodiment showing an annotation server

[61] including a load balancer

[0241] , a front-end web server

[0242] , an annotation API server

[0243] , a deep learning API

[0244] , and a message broker

[0245] . Also shown is an electronic database

[62] including a relational database

[0246] and a key-value store database

[0247] . In this exemplary embodiment, the load balancer

[0241] , the front-end web server

[0242] , the annotation API server

[0243] , the deep learning API

[0244] , the message broker

[0245] , the relational database

[0246] , and the key-value store database

[0247] are each being executed on a different machine having one or more central processing units (CPUs), memory, and a non-transitory computer-readable storage medium. Further, the deep learning API

[0244] comprises at least one of a graphical processing unit (GPU), a tensor processing unit (TPU), or a field programmable gate array (FPGA). In the exemplary embodiment depicted in FIG. 29, the load balancer

[0241] machine comprises one central processing unit (CPU) having 4 cores, 8 gb of random access memory (RAM), and a 250 gb solid state drive (SSD), and the front-end web server

[0242] machine comprises one CPU having 4 cores, 8 gb of RAM, and a 250 gb SSD.The machine of the Annotation API Server

[0243] has two CPUs with 32 cores each, for a total of 64 cores, 64 GB of RAM, and 2 TB of SSD. The Deep Learning API

[0244] machine has one 32-core CPU, 128 GB of RAM, two GPUs with 11 GB of RAM each, and 2 TB of SSD. The Message Broker

[0245] machine has one 4-core CPU, 32 GB of RAM, and 1 TB of SSD. The Relational Database

[0246] machine has one 32-core, 128 GB of RAM, and 2 TB of SSD, and the Key-Value Store Database

[0247] machine has one 4-core CPU with 128 GB of RAM and 1 TB of SSD. Machines that meet these hardware specifications enable this system to support many users with multiple concurrent users.

[0135] In the exemplary embodiment depicted in FIG. 29, the hardware specifications were described. Next, the theoretical basis for these specifications will be explained. The load balancer

[0241] simply routes requests to the appropriate location. As a result, the requirements for the non-transitory computer-readable storage area of this machine should be met by a 250 gb SSD. Compared to the calculations performed in other parts of the system, the load balancer

[0241] has a low computational load and a low memory load. As a result, 2 gb per core is sufficient, and considering the rest of the system, 4 cores are sufficient. As the system scales up, the number of instructions executed per second increases, and by increasing the CPU cores, it may be possible to prevent the load balancer

[0241] from becoming a bottleneck. However, based on the specifications given for the exemplary embodiment depicted in FIG. 29, the likelihood of the load balancer

[0241] becoming a bottleneck is low. The front-end web server

[0242] also does not require a large amount of non-transitory computer-readable storage area, a large number of CPU cores, or a large amount of memory. This machine does not store user data, and the front-end code it stores can be supported by the allocated storage space. Furthermore, the CPU cores of the front-end web server

[0242] machine simply send the requested resources to the user, and it is not common in normal use for the user to request front-end code such that the user's browser makes an api call to the annotation API server

[0243] . As a result, the CPU cores of the front-end web server

[0242] machine have a lower computational load and memory requirements than the annotation API server

[0243] machine. The annotation API server

[0243] has a high computational load because API requests are made from the user's web browser when creating, organizing, and viewing annotations. The total computing power of the annotation API server

[0243] machine may become a bottleneck when there are a large number of concurrent users. In the exemplary embodiment, this machine is equipped with 64 CPU cores and 64 gb of RAM.If the code of the Annotation API Server

[0243] is written in a thread-safe manner, 1 GB of RAM per CPU core is sufficient. If the code is not thread-safe or the web framework consumes a large amount of memory, it may be desirable to use 2 GB of RAM per CPU core for the Annotation API Server

[0243] . To scale the Annotation API Server

[0243] to support more users, it is more cost-effective to add machines rather than providing more CPU cores and more RAM per machine. The size of the hard disk of the Annotation API Server

[0243] is mainly for enabling the storage of log files. For the Deep Learning API

[0244] , it is advisable to have two GPUs with sufficient GPU RAM to run the deep learning model. The CPU cores and RAM are used to process the data processing steps that concentrate on the CPU and to load the data consumed by the GPU into memory. A 2tb SSD can store large-scale deep learning models and other data for these models. The Message Broker

[0245] should be operational at all times when the system is running and should not face a shortage of computing power. 32 GB of RAM exists to ensure sufficient RAM for storing large messages that may include images. Since these images will not be stored for a long time, 32 GB of RAM is sufficient. Additionally, the Message Broker

[0245] requires 4 cores. The Message Broker

[0245] has lower computational processing requirements in terms of the number of instructions per second than the Annotation API Server

[0243] in this exemplary embodiment. The Key-Value Store Database

[0247] uses 1 core in this exemplary embodiment. However, the Key-Value Store Database

[0247] uses RAM to store the reputations of various item objects such as annotation objects and user reputation votes for these item objects.Furthermore, disk space is used for logging and for temporarily storing backups of the key-value store database in memory. The relational database [

[0246] ] has 128 gb of RAM to be able to store all data in the database in RAM during operation, has 2 tb of SSD to ensure sufficient storage space for logging, and is equipped with 32 cores to provide the computing power to process queries sent from the annotation API server [

[0243] ].

[0136] To scale up the system depicted in the exemplary embodiment of FIG. 29, the following steps can be taken. The new exemplary embodiment will have one or more load balancers [

[0241] ], one or more front-end web servers [

[0242] ], one or more annotation API servers [

[0243] ], and one or more deep learning APIs [

[0244] ]. The load balancer, front-end web server, annotation API server, and deep learning API server can be added to improve the overall performance of the system. In many cases, the message broker can be organized into a message broker cluster. The message broker cluster can be used to scale up the new exemplary embodiment. Most key-value store database [

[0247] ] software also has the ability to create a key-value store database cluster.

[0137] In a new exemplary embodiment, to enable a relational database [

[0246] ] to handle the most users, it is desirable to use a relational database [

[0246] ] that uses threads as opposed to processes. In some cases, while a thread takes about 256 kb of memory per thread, a process takes 10 mb of memory per process. For this reason, a relational database that relies on threads can have more than 20 times as many open connections as a relational database that relies on processes. Even if the connection pooling tool used by the process-dependent relational database is used, the style of connection made by the annotation API server is suitable for thread-based connections [

[0243] ]. The annotation API server [

[0243] ] is transaction-dependent and requires performance. As a result, it is desirable to use SQL transaction pooling rather than SQL session pooling or SQL statement pooling. SQL session pooling has low performance, and SQL statement pooling makes transaction implementation more difficult than using SQL transaction pooling. When using SQL transaction pooling, thread-based connections provide the best performance when the computer's instructions need to be executed on a processor not being used by the relational database [

[0246] ]. This is because the connection remains open while work is being done on an external processor. As a result, the relational database [

[0246] ] may be given fewer SQL statements to execute than the relational database [

[0246] ] can handle. This results in an overall throughput decrease.

[0138] To further enhance the capabilities of a relational database [

[0246] ], it is advantageous to perform read scaling on the relational database [

[0246] ] by creating read replicas. By reducing the write load on the relational database [

[0246] ], the system can be further scaled up. The write load can be reduced in a new exemplary embodiment by performing reputation voting through a key-value store database [

[0247] ]. To further expand the capacity of the relational database, in a new exemplary embodiment, the number of CPU cores and RAM of the relational database [

[0246] ] machine can be increased. Another step that can be taken is to use an in-memory relational database [

[0246] ] that implements SQL without full durability guarantees for most relational databases. Such a system can use an in-memory database with checkpointing and write-ahead logging to prevent loss of database data other than the data that arrived at the in-memory database within the last few seconds during a power outage. Through these steps, the relational database can be further expanded. In an HPC cluster, remote direct memory access (RDMA) via InfiniBand networking enables a relational database to use more memory than is locally available on a single machine. Since this HPC cluster can be scaled up to hundreds or thousands of machines, the corresponding relational database [

[0246] ] can support a large number of concurrent users.

[0139] In the exemplary embodiment depicted in FIG. 30, different types of client electronic devices

[0248] that can communicate with an annotation server

[61] and a content delivery network

[63] are shown. The devices shown are a laptop computer

[0249] , a desktop

[0250] , an iPad

[0251] , an Android tablet

[0252] , an iPhone

[0253] , an Android mobile phone

[0254] , an Apple TV via Air-Web

[0255] , an Android TV

[0256] , and a Raspberry Pi

[0257] . For the laptop computer

[0249] , the desktop

[0250] , the Android tablet

[0252] , the Android mobile phone

[0254] , and the Raspberry Pi

[0257] , the introduction of a web browser and a web browser extension supports an annotation-enabled web browser with a notepad function. Also, for the iPad

[0251] and the iPhone

[0253] , an annotation-enabled web browser is supported by installing an extension function of the web browser "Safari" and an app. Apple TV

[0255] and Android TV

[0256] via Air Web may not currently be suitable for installing an annotation-enabled web browser that includes a web browser and a web browser extension. Furthermore, the form factor may not be ideal for creating annotations. However, it is possible to create a progressive web application that communicates with the annotation server for these devices. Then, just as a digital photo frame displays an image, annotated materials can be displayed. Then, these devices will be able to display rich annotations created using other devices shown in FIG. 30.

[0140] Next, the annotation of images, videos, and recordings will be described. The GUI diagrams in the exemplary embodiments will be presented as what the system displays and as the end result of non-transitory computer-readable media instructions executed on a web browser running on a processor. In an exemplary embodiment, the non-transitory computer-readable media instructions may be for a web browser executed on a processor, or the non-transitory computer-readable media instructions may be incorporated into an annotatable web browser. As described above, the annotatable web browser in the exemplary embodiment is a web browser having a web browser extension. When using an annotatable web browser, images, videos, and audio recordings embedded within any web page can be annotated. The web site to be annotated does not need to include computer-readable media instructions for annotating the embedded images, videos, and recordings. When using a normal web browser, if the instructions of the non-transitory computer-readable media are delivered via a specific web site and the annotation of the content is performed on that specific web site, images, videos, and recordings can be annotated.

[0141] In addition to the annotatable web browser for annotating documents and media on the World Wide Web, software applications for mobile devices and Internet TVs can also use non-transitory computer-readable media instructions implementing the methods, techniques, and interfaces described in this document to annotate images, videos, and recordings that are available to a group of users on the Internet or a local intranet.

[0142] After describing the GUI, the bundle of data created to transfer data between components will be described. When the relevant data bundles are discussed, the methods and system architectures are discussed in the context of the system architectures and components already discussed.

[0143] In the exemplary embodiment depicted in FIG. 35, an image [

[0299] ] is shown along with an image-annotation subject [

[0300] ], a bounding box of the image-annotation subject [

[0301] ], an image-annotation arrow [

[0302] ], and an image-annotation comment within the box [

[0303] ]. The image-annotation subject [

[0300] ] is an entity of interest included in the image. The image-annotation subject [

[0300] ] can be a person, a vehicle, a soccer ball, a tennis ball, or other object distinguishable within the image by a human or a computer system. The bounding box of the image-annotation subject [

[0301] ] is a bounding box drawn around the image-annotation subject [

[0300] ] by a user or a computer system. The image-annotation comment within the box [

[0303] ] is created by the user, and the annotation comment needs to be related to the image-annotation subject [

[0300] ]. The image-annotation arrow [

[0302] ] is a line, sometimes with arrowheads on both sides, that connects the bounding box of the image-annotation subject [

[0301] ] and the image-annotation comment within the box [

[0303] ]. The bounding box of the image-annotation subject [

[0301] ], the image-annotation arrow [

[0302] ], and the image-annotation comment within the box [

[0303] ] are created by the user and are objects included in the image-annotation. The user can select a rectangle tool from a panel or a floating toolbar on the web page and draw the bounding box [

[0301] ] of the image-annotation subject. Further, the user can input the position of the text and the image-annotation comment within the box [

[0303] ]. The bounding box [

[0301] ] of the image-annotation subject and the image-annotation comment within the box [

[0303] ] are connected by the image-annotation arrow [

[0302] ]. The objects of the image-annotation are not always displayed simultaneously.For example, only the bounding box

[0301] of the image-annotation subject is displayed, and other objects of the image-annotation can be made to be displayed when the mouse is moved over the image-annotation subject

[0300] or the bounding box

[0301] of the image-annotation subject. The visible element of the image-annotation is an element that is visualized without the user moving the mouse over other elements. The mouse-over element of the image-annotation is an element that becomes visible when the user moves the mouse over the visible element of the image-annotation.

[0144] Below the image [

[0299] ], a control panel including an image previous annotation arrow button [

[0304] ], an image next annotation arrow button [

[0305] ], a button including image collision [

[0306] ], an add button to the image candidate annotation group [

[0307] ], an image candidate annotation group list box [

[0308] ], and an image-annotation group creation button [

[0309] ] is displayed. The label of the add button to the image candidate annotation group [

[0307] ] is a short name "Add to Annotation Group". However, the image annotation is first added to the image candidate annotation group, and then, when the user clicks the image creation annotation group button [

[0309] ], it becomes an image-annotation group. The image annotation includes objects such as a bounding box [

[0301] ] of the image-annotation subject, an image-annotation arrow [

[0302] ], and an image-annotation comment [

[0303] ] within the box. One image annotation may include one or more annotation subjects [

[0300] ]. In the electronic database

[62] , the image annotation is stored as an image-annotation object. This image-annotation object is sequentially ordered for the image annotation by the electronic database

[62] . This ordering of the image annotation gives meaning to the concepts of previous and next image annotations. The image previous annotation arrow button [

[0304] ] deletes the objects included in the current image annotation from the displayed view assuming that the current image annotation is not an image annotation candidate, and adds the objects of the previous image annotation to the displayed view. The image next annotation arrow button [

[0305] ] deletes the objects included in the current image annotation from the displayed view if the current image annotation is not an image annotation candidate, and adds the objects of the next image annotation to the displayed view. An image-annotation group includes a plurality of image annotations.Before creating an image-annotation group, it is necessary to select an image-annotation. The selected image-annotation becomes a candidate image-annotation. The candidate image-annotations are listed in the image candidate annotation group list box

[0308] . Figure 35 depicts a scenario where the image-annotation has not yet been added to the image candidate annotation group list box

[0308] . Therefore, the message in the image candidate annotation group list box

[0308] is the image candidate annotation group list empty message

[0310] .

[0145] The button

[0306] including image collision determines the display properties of the image-annotation that collides with the image-annotation candidates listed in the image candidate annotation group list box

[0308] . The possible states are excluding the collision and including the collision. When the button

[0306] including image collision is clicked, the image-annotation that collides with the candidate image-annotations listed in the image candidate annotation group list box

[0308] is displayed. In the exemplary embodiment depicted in Figure 35, the label of the button is changed to "Exclude Collision". When this button is clicked again, the image-annotation that collides with the image-annotation candidates listed in the image candidate annotation group list box

[0308] is no longer displayed. When the image-annotation group creation button

[0309] is clicked, an item is created that displays an image-annotation group including all the image-annotations listed in the image annotation group candidate list box

[0308] at the time the image-annotation group creation button

[0309] is clicked.

[0146] In an alternative embodiment, the bounding box

[0301] of the image-annotation subject and the image-annotation arrow

[0302] may not be displayed. In an alternative embodiment, only the bounding box

[0301] of the image-annotation subject and the image-annotation comment

[0303] within the box exist. Alternatively, in another alternative embodiment, only the image-annotation comment

[0303] within the box exists near the image-annotation subject

[0300] .

[0147] In the exemplary embodiment depicted in FIG. 36, an image

[0299] with a first image candidate annotation group list item

[0311] added to the image candidate annotation group list box

[0308] is displayed. This state is generated by clicking the add button

[0307] for the image-annotation candidate group in the state of the exemplary embodiment depicted in FIG. 35. After the image-annotation is added to the image-annotation candidate group list, in the state of the exemplary embodiment depicted in FIG. 36, by clicking the next image-annotation arrow button

[0305] , the state depicted in FIG. 37 is created.

[0148] In the exemplary embodiment depicted in FIG. 37, a second image-annotation subject

[0312] is displayed together with the bounding box

[0313] of the second image-annotation subject created by the user, the second image-annotation arrow

[0314] , and the second image-annotation comment

[0315] within the box. When the add button

[0307] for the image-annotation candidate group is clicked in the state of the exemplary embodiment shown in FIG. 37, the state of the exemplary embodiment shown in FIG. 38 is generated.

[0149] In the exemplary embodiment depicted in FIG. 38, a second image candidate annotation group list item

[0316] is shown. When the image creation annotation group button

[0309] is pressed, an annotation group including two annotations depicted in the image candidate annotation group list box

[0308] of FIG. 38 will be created.

[0150] In the exemplary embodiment depicted in FIG. 39, the user previously clicked the image collision button

[0306] of FIG. 36 that enables the display of colliding image annotations. After pressing the image next annotation arrow button

[0305] from the state depicted in FIG. 36, the state depicted in FIG. 39 can be brought about. In FIG. 39, a collision image annotation arrow

[0317] and a collision image annotation comment within the box

[0318] are displayed.

[0151] In the exemplary embodiment depicted in FIG. 40, video

[0319] is shown along with video annotation playback

[0320] . The video

[0319] shows a video annotation subject

[0321] , a bounding box of the video annotation subject

[0322] , a video annotation arrow

[0323] , and a video annotation comment within the box

[0324] . The annotation subject

[0321] is an entity of interest in the video. The video annotation subject

[0321] can be a person, a vehicle, a soccer ball, a tennis ball, or other objects distinguishable within the video by a human or a computer system. The bounding box of the video annotation subject

[0322] is a bounding box drawn around the video annotation subject

[0321] by a user or a computer system. The video annotation comment box

[0324] is created by the user, and the annotation comment needs to be related to the video annotation subject

[0321] . The video annotation arrow

[0323] is a line that can have arrowheads on both sides connecting the bounding box of the video annotation subject

[0322] and the video annotation comment in the box

[0324] . The bounding box of the video annotation subject

[0322] , the video annotation arrow

[0323] , and the video annotation comment input box

[0324] are objects created by the user and included in the video annotation. The user can select the rectangle tool from a panel or a floating toolbar on the web page and draw the first video annotation subject bounding box

[0322] . Further, the user can input text, place the video annotation comment in the box

[0324] , and connect them with the video annotation arrow

[0323] . When the video

[0319] is played, the video annotation playback

[0320] is also performed.During the playback of video annotation playback

[0320] , the user creating the video annotation can move the bounding box

[0322] of the subject of the video annotation using the mouse and confirm that the bounding box

[0322] of the video annotation subject is properly placed across the entire video. The user can move the video annotation comment within the box

[0324] using the mouse and confirm that the video annotation comment within the box

[0324] is properly placed across the entire video. If the motion paths of the bounding box

[0322] of the video annotation subject and the video annotation comment in the box

[0324] are separated, the motion path of the video annotation arrow

[0323] can be calculated together with the temporal rotation, temporal contraction, temporal expansion, and temporal position to correspond to the motion paths of the bounding box

[0322] of the video annotation subject and the video annotation comment in the box

[0324] . Also, the user can pause the video and resize the bounding box

[0322] of the video annotation subject and the video annotation comment within the box

[0324] . Not all objects of the video annotation are always displayed at the same time. For example, after the user mouses over the video annotation subject

[0321] or the bounding box

[0322] of the video annotation subject, other objects of the video annotation can be displayed, and only the bounding box

[0322] of the video annotation subject can be displayed. The visible elements of the video annotation are elements that are visualized without mousing over other elements. The mouse-over elements of the video annotation are elements that become visible when the user mouses over the visible elements of the video annotation.

[0152] Below the video

[0319] , a control panel including a video pre-annotation arrow button

[0325] , a video next annotation arrow button

[0326] , a collision button including the video

[0327] , a button to add to the video candidate annotation group

[0328] , a video candidate annotation group list box

[0329] , and a video annotation group creation button

[0330] is displayed. The label of the button to add to the video annotation group candidate

[0328] has a short name "Add Annotation Group". However, video annotations are first added to the video candidate annotation group and become a video annotation group after the user clicks the video creation annotation group button

[0330] . Video annotations include objects such as a bounding box

[0322] of the video annotation subject, a video annotation arrow

[0323] , and a video annotation comment input box

[0324] . Video annotations can include multiple video annotation subjects

[0321] . In the electronic database

[62] , video annotations are stored as video annotation objects. And these video annotation objects are sequentially ordered by the electronic database

[62] for video annotations. This ordering of video annotations gives meaning to the concept of previous and next video annotations. The video pre-annotation arrow button

[0325] deletes the objects included in the current video annotation from the displayed view assuming that the current video annotation is not a video annotation candidate and adds the objects of the previous video annotation to the displayed view. The video next annotation arrow button

[0326] deletes the objects included in the current video annotation from the displayed view assuming that the current video annotation is not a candidate video annotation and adds the objects of the next video annotation to the displayed view.A video annotation group includes a plurality of video annotations. Before a video annotation group is created, it is necessary to select video annotations. In the selection process, the video annotations become candidate video annotations. The candidate video annotations are listed in the video candidate annotation group list box

[0329] . FIG. 40 depicts a scenario where the video annotation has not yet been added to the video annotation candidate group list box

[0329] . Therefore, the message in the video annotation candidate group list box

[0329] is the video annotation candidate group list empty message

[0331] .

[0153] The collision button

[0327] including the video determines the display properties of the video annotation that collides with the candidate video annotations listed in the video candidate annotation group list box

[0329] . In the case of video annotations, a collision means that two video annotations collide at a certain point. The collision may be displayed in red on the playback line of the video annotation

[0320] . When the collision button

[0327] including the video is clicked, the video annotation that collides with the candidate video annotations listed in the video candidate annotation group list box

[0329] is displayed. In the exemplary embodiment depicted in FIG. 40, the button label is changed to "Exclude collision". When this button is clicked again, the video annotation that collides with the video annotation candidates listed in the video candidate annotation group list box

[0329] is no longer displayed. When the video annotation group creation button

[0330] is clicked, the following items are created. When the video creation annotation group button

[0330] is clicked, a video annotation group is generated that includes all the video annotations listed in the video candidate annotation group list box

[0329] . When the video addition candidate annotation group button

[0328] is clicked in the state depicted in the exemplary embodiment of FIG. 40, the state depicted in the exemplary embodiment of FIG. 41 is created with the first video candidate annotation group list item

[0335] added to the video candidate annotation group list box

[0329] .

[0154] In an alternative embodiment, the bounding box

[0322] of the video annotation subject and the video annotation arrow

[0323] may not be displayed. In an alternative embodiment, only the bounding box

[0322] of the video annotation subject and the video annotation comment within the box

[0324] exist. Or, in another alternative embodiment, only the video annotation comment within the box

[0324] exists near the video annotation subject

[0321] .

[0155] In the exemplary embodiment depicted in FIG. 40, the video

[0319] player controls are separate from the video annotation playback

[0320] . In an alternative embodiment, it is possible to integrate both controls into a single playback control. This may result in a more complex input interface, but it can reduce the number of elements displayed on the screen. This comment regarding the integration of the two playback controls also applies to the exemplary embodiments depicted in FIGS. 41, 42, 43, 44, and 45.

[0156] In the exemplary embodiment depicted in FIG. 41, the bounding box motion path

[0332] of the video annotation subject, the video annotation arrow motion path

[0333] , and the video annotation comment input box motion path

[0334] are shown. The bounding box motion path

[0332] of the video annotation subject shows the motion path of the bounding box

[0322] when the video annotation subject

[0321] moves within the video, and is supposed to function as the bounding box of the video annotation subject

[0321] . The video annotation arrow motion path

[0333] shows the motion path of the video annotation arrow

[0323] when the video annotation arrow

[0323] moves as the video

[0319] progresses in time. It can be seen that the video annotation arrow

[0323] is translated, rotated, expanded, and contracted parallel in time from the bounding box

[0322] of the video annotation subject and the video annotation comment

[0319] . The box

[0324] moves in time. The video annotation comment input box motion path

[0334] shows the motion path when the video annotation comment within the box

[0324] moves as the video

[0319] progresses in time.

[0157] In the exemplary embodiment depicted in the figure, 42 represents the second video annotation subject

[0336] , the bounding box of the second video annotation subject

[0337] , the second video annotation arrow

[0338] , the second video annotation comment within the box

[0339] , the bounding box motion path of the second video annotation subject

[0340] , the motion path 1 of the second video annotation arrow

[0341] , the motion path 2 of the second video annotation arrow

[0342] , the bounding box position of the final second video annotation subject

[0343] , and the position of the final second video annotation arrow

[0344] . The bounding box

[0337] of the second video annotation subject follows the second video annotation subject

[0336] along the bounding box motion path

[0340] of the second video annotation subject to the bounding box position

[0343] of the final second video annotation subject. The second video annotation arrow

[0338] is transformed to the final second video annotation arrow position

[0344] over time. The transformation including movement, rotation, and contraction is depicted by the motion path 1

[0341] of the second video annotation arrow and the motion path 2

[0342] of the second video annotation arrow. The second video annotation comment within the box

[0339] remains fixed in time. The second video annotation candidate group list item

[0345] is displayed in the video annotation candidate group list box

[0329] that occurs when the user clicks the add button

[0328] to the video annotation candidate group.

[0158] In the exemplary embodiment depicted in FIG. 43, the user may have previously clicked on the collision button

[0327] including video in the state depicted in the exemplary embodiment of FIG. 41. As a result, video annotations having a collision with the video annotation in the candidate annotation group list are displayed. After clicking on the video next annotation arrow button

[0326] in the state depicted in the exemplary embodiment of FIG. 41 and then clicking on the collision button

[0327] including video, the collision video annotation arrow

[0346] and the collision video annotation comment in the box

[0347] may be displayed. In the exemplary embodiment of FIG. 43, a collision is depicted. However, the collision may occur at a point not depicted in a specific video frame. In such a case, the collision may be depicted in red on the timeline of the video annotation playback

[0320] within the exemplary embodiment.

[0159] In the exemplary embodiment depicted in FIG. 44, recording playback

[0348] and recording annotation playback

[0349] are shown. Below the recording annotation playback

[0349] , a recording annotation arrow

[0350] , a recording annotation comment box

[0351] , a recording annotation connection arrow

[0352] , and a recording-derived annotation comment box

[0353] are shown. Below the recording annotation playback

[0349] , there is a control panel such as a pre-recording annotation arrow button

[0354] , a next recording annotation arrow button

[0355] , a recording candidate annotation group addition button

[0356] , a recording-derived annotation creation button

[0357] , a recording candidate annotation group list box

[0358] , a recording-derived annotation group creation button

[0359] , and a recording-derived annotation comment input box

[0360] . The label of the addition button

[0356] to the recording candidate annotation group is a short name "Add Annotation Group". However, the recording annotation is first added to the recording candidate annotation group and becomes a recording annotation group after the user clicks the recording annotation group creation button

[0359] .

[0160] In order to create a recording annotation in the exemplary embodiment depicted in FIG. 44, the user uses a mouse to select a short segment of the recording annotation playback

[0349] . After the selection is made, a recording annotation arrow

[0350] and a recording annotation comment box

[0351] appear centered on the selection, and the user can create a recording annotation comment in the recording annotation comment box

[0351] . The recording annotation includes objects such as the recording annotation arrow

[0350] and the recording annotation comment box

[0351] . The short segment of the recording annotation playback

[0349] can be marked as shown by the embedded recording annotation playback range

[0393] in FIG. 49. FIG. 49 shows an embedded recording, and the marked short segment is on the playback bar for the recording playback

[0348] , but the same style of the marked segment can be used for the recording annotation playback

[0349] . Also, it is possible to leave the segment unmarked, or to display the marked segment when mousing over the selected segment, the recording annotation arrow

[0350] , or the recording annotation comment box

[0351] . Furthermore, a question sound annotation symbol that behaves similarly to the question embedded sound annotation symbol

[0394] in FIG. 49 can be introduced.

[0161] The recording annotation can include one or more selected segments of sound. In the electronic database

[62] , the recording annotation is stored as a recording annotation object. These recording annotation objects are then sequentially ordered by the electronic database

[62] for the recording annotation. This ordering of the recording annotation gives meaning to the concepts of the previous recording annotation and the next recording annotation. The pre-recording annotation arrow button

[0354] deletes the objects included in the current recording annotation from the displayed view on the assumption that the current recording annotation is not a recording annotation candidate, and adds the objects included in the previous recording annotation to the displayed view. The next recording annotation arrow button

[0355] deletes the objects included in the current recording annotation from the displayed view on the assumption that the current recording annotation is not a recording annotation candidate, and adds the objects of the next recording annotation to the view being displayed. A recording annotation group is a group of recording annotations that the user wants to group. These groupings are displayed as one item, and derivative recording annotations can be created that use one or more recording annotations from the recording annotation group and associate these one or more recording annotations. Further, the derivative recording annotation and the recording annotation on which it depends can be displayed as one item. The recording annotation can be added to the recording annotation group by clicking the add button

[0356] to the recording annotation candidate group. As a result, a new recording annotation is displayed in the recording annotation candidate group list box <358>. When the recording annotation is added to the recording annotation group, the user can create an annotation derived from the recording. A recording derivative annotation is an annotation that links other recording annotations and adds a comment to the recording derivative annotation comment box

[0353] connected to a plurality of recording annotation comment boxes.The recorded sound-derived annotation comment box

[0353] is connected to each sound source annotation comment box

[0351] using the recorded sound annotation connection arrow

[0352] . The recorded sound-derived annotation is created by selecting a desired recorded sound annotation and clicking the recorded sound-derived annotation creation button

[0357] . Next, enter the recorded sound-derived annotation comment

[0361] in the recorded sound-derived annotation comment input box

[0360] and press Enter. The recorded sound candidate annotation group list box

[0358] contains the recorded sound candidate annotation group list item

[0362] . This list includes both the recorded sound annotation and the recorded sound-derived annotation. The first item of the recorded sound candidate annotation group list item

[0362] is the first recorded sound candidate annotation group list item

[0363] .

[0162] In the exemplary embodiment depicted in FIG. 45, video [

[0319] ] and video annotation playback [

[0320] ] are shown. Below the video annotation playback [

[0320] ], a video annotation arrow [

[0364] ], a video annotation comment box [

[0365] ], a video annotation connection arrow [

[0366] ], and a video-derived annotation comment box [

[0367] ] are shown. Below the video annotation playback [

[0320] ], a control panel including a previous video annotation arrow button [

[0325] ], a next video annotation arrow button [

[0326] ], a collision button including video [

[0327] ], an add button to video candidate annotation group [

[0328] ], a video-derived annotation creation button [

[0368] ], a video candidate annotation group list box [

[0329] ], a created video annotation group button [

[0330] ], and a video-derived annotation comment input box [

[0369] ] is shown. The label of the video add button [

[0328] ] has a short name of "Add Annotation Group". However, the video annotation is first added to the video candidate annotation group and becomes a video annotation group after the user clicks the video annotation group creation button [

[0330] ]. To create a video annotation in the exemplary embodiment, the user uses the mouse to select a short segment of the video annotation playback [

[0320] ]. After the selection is made, the video annotation arrow [

[0364] ] and the video annotation comment box [

[0365] ] appear starting from the center of the selection, and the user can create a video annotation comment in the video annotation comment box [

[0365] ]. The short segment of the video annotation playback [

[0320] ] can be marked as shown by the embedded video annotation playback range [

[0375] ] in FIG. 46. FIG. 46 shows an embedded video, and the marked short segment is on the playback bar of the video [

[0319] ], but the marked segments of the same style can be used in the video annotation playback [

[0320] ].It is possible not to mark the segment, or to display the marked segment when mousing over the selected segment, the arrow of the video annotation

[0364] , or the comment box of the video annotation

[0365] . Furthermore, a question video annotation symbol that behaves similarly to the question-embedded video annotation symbol

[0376] in Fig. 46 can be introduced.

[0163] As described above for another exemplary embodiment, video annotation includes objects such as a bounding box

[0322] of a video annotation subject, a video annotation arrow

[0323] , and an in-box video annotation comment

[0324] . Video annotation can include a plurality of video annotation subjects

[0321] . In an electronic database

[62] , video annotation is stored as a video annotation object. And these video annotation objects are sequentially ordered by the electronic database

[62] for video annotation. This ordering of video annotation gives meaning to the concept of previous and next video annotation. The previous video annotation arrow button

[0325] deletes the objects included in the current video annotation from the displayed view on the assumption that the current video annotation is not a video annotation candidate, and adds the objects of the previous video annotation to the displayed view. The next video annotation arrow button

[0326] deletes the objects included in the current video annotation from the displayed view on the assumption that the current video annotation is not a video annotation candidate, and adds the objects of the next video annotation to the displayed view. A video annotation group is a collection of video annotations that a user wants to group. These groupings are displayed as one item, and derivative video annotations can be created that use one or more video annotations from the video annotation group and associate these one or more video annotations. Further, the derived video annotation can be displayed as one item together with the video annotation on which it depends. To add a video annotation to a video annotation candidate group, click the add button

[0328] to the video annotation candidate group. As a result, a new video annotation is displayed in the video annotation candidate group list box <329>.When a video annotation is added to a video candidate annotation group, the user can create an annotation derived from the video. A video-derived annotation is an annotation that links other video annotations and adds a comment to a video-derived annotation comment box

[0367] connected to multiple video annotation comment boxes. The video-derived annotation comment box

[0367] is connected to each video annotation comment box

[0365] using a video annotation connection arrow

[0366] . A video-derived annotation is created by selecting a desired video annotation and clicking a video-derived annotation creation button

[0368] . Next, enter a video-derived annotation comment

[0370] in a video-derived annotation comment input box

[0369] and press Enter. A video candidate annotation group list box

[0329] displays a video candidate annotation group list item

[0371] . This list includes both video annotations and video-derived annotations. The first item of the video candidate annotation group list item

[0371] is the first video candidate annotation group list item

[0372] . Clicking a video creation annotation group button

[0330] enables creation of an annotation group that includes the video candidate annotations in the video candidate annotation group list box

[0329] . A video inclusion collision button

[0327] operates in the same manner as the video inclusion collision button

[0327] in the exemplary embodiments depicted in FIGS. 40, 41, 42, and 43.

[0164] In the exemplary embodiment depicted in FIG. 46, embedded video [

[0373] ] and embedded video playback [

[0374] ] are shown. Embedded video [

[0373] ] and embedded video playback [

[0374] ] exist within a web page being viewed in an annotatable web browser. In the exemplary embodiment depicted in FIG. 46, an embedded video - annotation playback range [

[0375] ], a question - embedded video - annotation symbol [

[0376] ], a disagreement - embedded video - annotation symbol [

[0377] ], an embedded video - annotation beacon [

[0378] ], an embedded video - annotation beacon question symbol [

[0379] ], an embedded video - annotation beacon disagreement symbol [

[0380] ], and an embedded video - annotation beacon playback time [

[0381] ] are shown. The symbol - beacon - trigger button

[57] triggers the display of one fully - displayed video - annotation beacon for each video - annotation of the embedded video [

[0373] ], provided that the embedded video [

[0373] ] does not contain more than three video - annotations. If there are more than three video - annotations, an alternative display methodology enables the user to better visualize the viewable video - annotations. The fully - displayed video - annotation beacon includes the embedded video - annotation playback range [

[0375] ], the embedded video - annotation symbol, the embedded video - annotation beacon [

[0378] ], the embedded video - annotation beacon symbol, and the embedded video - annotation beacon playback time [

[0381] ]. The embedded video - annotation playback range [

[0375] ] highlights the time range that the user considers relevant to the video - annotation. Embedded video - annotation symbols, such as the question - embedded video - annotation symbol [

[0376] ] and the disagreement - embedded video - annotation symbol [

[0377] ], convey the semantic relationship of the annotation between the selected video segment and the video - annotation comment. The embedded video - annotation beacon [

[0378] ] serves to draw the user's attention to the presence of the video - annotation and some of its characteristics.The embedded video annotation beacon symbol is placed in the vicinity of the embedded video annotation beacon

[0378] for the semantic relationship of the annotation between the selected video segment and the video annotation comment. Two examples of the embedded video annotation beacon symbol are the embedded video annotation beacon question symbol

[0379] and the embedded video annotation beacon disagreement symbol

[0380] . The embedded video annotation beacon playback time

[0381] placed in the vicinity of the embedded video annotation beacon

[0378] conveys the length of the video segment related to the video annotation comment.

[0165] In the exemplary embodiment depicted in FIG. 46, the playback range

[0375] of the embedded video annotation and the question embedded video annotation symbol

[0376] are shown in the context of the embedded video

[0373] and the embedded video playback

[0374] . However, the video annotation playback range and the video annotation symbol can also be used on the video annotation playback

[0320] depicted in FIG. 45.

[0166] In the exemplary embodiment depicted in FIG. 47, an embedded image [

[0382] ] is shown. The embedded image [

[0382] ] exists within a web page, and the embedded image - annotation subject [

[0383] ] occurs within the embedded image [

[0382] ]. In the exemplary embodiment depicted in FIG. 47, the bounding box [

[0384] ] of the embedded image - annotation subject, the embedded image - annotation arrow [

[0385] ], and the embedded image - annotation comment within the box [

[0386] ] are shown. The embedded image - annotation subject [

[0383] ] is the object of interest within the image. The embedded image - annotation subject [

[0383] ] can be a person, a car, a soccer ball, a tennis ball, or any other object distinguishable within the image by a human or a computer system. The bounding box [

[0384] ] of the embedded image - annotation subject is a bounding box drawn around the embedded image - annotation subject [

[0383] ] by a user or a computer system. The user can select a rectangle tool from some panel or floating toolbar on the web page and draw the bounding box [

[0384] ] of the embedded image - annotation subject. Further, the user can input text and place the embedded image - annotation comment within the box [

[0386] ]. The embedded image - annotation comment within the box [

[0386] ] is created by the user, and the annotation comment needs to be related to the embedded image - annotation subject [

[0383] ]. The embedded image - annotation arrow [

[0385] ] is a line that connects the bounding box [

[0384] ] of the embedded image - annotation subject and the embedded image - annotation comment within the box [

[0386] ] and can have arrowheads on both sides. The subject boundary box [

[0384] ] of the embedded image - annotation, the embedded image - annotation arrow [

[0385] ], and the image - annotation comment within the embedded box [

[0386] ] are embedded objects created by the user and included in the image - annotation.

[0167] Embedded objects of images and annotations may not all be displayed at the same time. As an example, only the bounding box

[0384] of the embedded image and annotation sub-object is visible together with other embedded objects of the image and annotation that are visualized after the user hovers over the embedded image and annotation sub-object

[0383] or the bounding box

[0384] of the embedded image and annotation sub-object. Visible elements of the image and annotation are embedded elements that are visualized without the user hovering over other elements. Mouse-over embedded elements of the image and annotation are embedded elements that become visible when the user hovers over the visible embedded elements of the image and annotation. When the user hovers over an embedded object of the image and annotation, a context-variant annotation control bar box

[0387] is displayed. The user can use the context-variant annotation control bar

[34] embedded in the context-variant annotation control bar box

[0387] to change privacy settings, share by email, share by social media, flag the image and annotation as inappropriate, delete the image and annotation object from an electronic database

[62] on the premise of sufficient permission, or edit the embedded object consisting of the image and annotation. Furthermore, the user can vote on the reputation of the image and annotation using the reputation selector

[33] .

[0168] In the exemplary embodiment depicted in FIG. 48, an embedded video

[0373] is shown. The embedded video

[0373] and the embedded video playback

[0374] exist within a web page, and the embedded video annotation subject

[0388] occurs within the embedded video

[0373] . In the exemplary embodiment depicted in FIG. 48, the bounding box

[0389] of the embedded video annotation subject, the embedded video annotation arrow

[0390] , and the embedded video annotation comment within the box

[0391] are shown. The embedded video annotation subject

[0388] is an entity of interest within the video. The embedded video annotation subject

[0388] can be a person, a vehicle, a soccer ball, a tennis ball, or other object distinguishable within the video by a human or a computer system. The bounding box

[0389] of the embedded video annotation subject is a bounding box drawn around the embedded video annotation subject

[0388] by a user or a computer system. As the embedded video annotation subject

[0388] moves within the video, it is desirable for the bounding box

[0389] of the embedded video annotation subject to continue to function as the bounding box of the embedded video annotation subject

[0388] . The embedded video annotation comment within the box

[0391] is created by a user, and the annotation comment needs to be related to the embedded video annotation subject

[0388] . The embedded video annotation arrow

[0390] is a line that can have arrowheads on both sides connecting the bounding box

[0389] of the embedded video annotation subject and the embedded video annotation comment within the box

[0391] .The bounding box of the embedded video annotation subject

[0389] , the embedded video annotation arrow

[0390] , and the embedded video annotation comment inbox

[0391] are embedded objects created by the user and included in the video annotation.

[0169] The user can select a rectangular tool from somewhere on the web page or the floating toolbar and draw the bounding box [

[0389] ] of the first embedded video annotation subject. Further, the user can input text, place the embedded video annotation comment in the box [

[0391] ], and connect them with the embedded video annotation arrow [

[0390] ]. In order to ensure that the objects are properly placed during video playback, either the user or the computer system needs to determine the motion path of the bounding box [

[0389] ] of the embedded video annotation subject. The user creating the video annotation has the option to trace the path of the bounding box [

[0389] ] of the embedded video annotation subject as the video is played back in time. During the playback of the embedded video playback [

[0374] ], the user creating the video annotation can move the bounding box [

[0389] ] of the embedded video annotation subject with the mouse to confirm that the bounding box [

[0389] ] of the embedded video annotation subject is properly placed throughout the embedded video. The user can move the embedded video annotation comment in the box [

[0391] ] with the mouse to confirm that the embedded video annotation comment in the box [

[0391] ] is properly placed throughout the video. Either the annotation-capable web browser, the extension of the web browser, or the code of the web page records the motion path when the user drags the mouse. Next, when the video is played back, the paths of the bounding box [

[0389] ] of the embedded video annotation subject and the embedded video annotation comment in the box [

[0391] ] evolve along their respective motion paths.If the motion paths of the bounding box of the embedded video annotation subject

[0389] and the embedded video annotation comment in box

[0391] are known, the motion path, its temporal rotation, temporal contraction, temporal expansion, and temporal position of the embedded video annotation arrow

[0390] can be calculated corresponding to the motion paths of the bounding box of the embedded video annotation subject

[0389] and the embedded video annotation comment in box

[0391] . Also, when the desired size of the bounding box of the embedded video annotation subject

[0388] changes as the video is played, the user can pause the video and resize the bounding box of the embedded video annotation subject

[0389] and the embedded video annotation comment in the box

[0391] .

[0170] Video annotation embedded objects are not all visualized simultaneously. For example, it is possible that only the bounding box

[0389] of the embedded video annotation subject

[0388] or the embedded video annotation subject is visible, and other embedded objects of the video annotation become visible after the user mouses over the embedded video annotation subject

[0388] or the bounding box

[0389] of the embedded video annotation subject. The visible elements of the video annotation are embedded elements that are visualized without mousing over other elements. The mouse-over embedded elements of the video annotation are embedded elements that are visualized when the user mouses over the visible embedded elements of the video annotation. When mousing over an embedded object of the video annotation, the context-variant annotation control bar box

[0387] is displayed. The user can use the context-variant annotation control bar

[34] embedded in the context-variant annotation control bar box

[0387] to change privacy settings, share by email, share by social media, flag the video annotation as inappropriate, delete the video annotation object from an electronic database

[62] assuming sufficient permission, or edit the embedded object consisting of the video annotation. Furthermore, the user can vote on the reputation of the video annotation using the reputation selector

[33] . Associated with each relationship within the semantic relationship group of the annotation are embedded symbols of different video annotations. As an example, the question-embedded video annotation symbol

[0376] indicates that the creator of the video annotation has specified the semantic relationship of the annotation between the selected part of the video and the video annotation comment as a question relationship.The question-embedded video annotation symbol

[0376] enables a user who is familiar with how the exemplary embodiment functions to click on the question-embedded video annotation symbol

[0376] before actually performing the click operation, so as to know that video annotation comments, which are questions about the embedded video annotation playback range

[0375] , can be viewed. The five relationships in the group of semantic relationships of annotations corresponding to the question annotation symbol, the disagreement annotation symbol, the agreement annotation symbol, the exclamation annotation symbol, and the advertisement annotation symbol are the semantic relationships of annotations. These semantic relationships of annotations are the question relationship, the disagreement relationship, the agreement relationship, the exclamation relationship, and the advertisement relationship. These video annotation symbols and the semantic relationships of video annotations are applied to the relationship between the embedded video annotation playback range

[0375] and the comment input field

[0391] of the embedded video annotation.

[0171] In the exemplary embodiment depicted in FIG. 49, the embedded sound recording [

[0392] ] is shown together with an embedded sound recording annotation playback range [

[0393] ] and a question embedded sound annotation symbol [

[0394] ]. The embedded recording annotation playback range [

[0393] ] specifies a selected portion of the recording corresponding to the sound annotation. The embedded sound annotation symbol is associated with an element of a group of semantic relationships of the annotation. The semantic relationship of each annotation indicates the relationship between the embedded recording annotation playback range [

[0393] ] and the sound annotation comment. Different embedded sound annotation symbols are associated with each relationship of the group of semantic relationships of the annotation. As an example, the question embedded sound annotation symbol [

[0394] ] indicates that the creator of the sound annotation has specified the semantic relationship of the annotation between the selected portion of the sound of the recording and the sound annotation comment as a question relationship. The question embedded sound annotation symbol [

[0394] ] enables a user familiar with how the exemplary embodiment functions to know that before the user actually performs the click operation, by clicking on the question embedded sound annotation symbol [

[0394] ], the user can view the sound annotation comment, which is a question regarding the embedded recording annotation playback range [

[0393] ]. The annotation comment box

[31] is displayed when the mouse is placed over the embedded recording annotation playback range [

[0393] ]. The user can use the context - variant annotation control bar

[34] embedded in the annotation comment box

[31] to change privacy settings, share by email, share by social media, flag the sound annotation as inappropriate, delete the sound annotation object from the electronic database

[62] subject to sufficient permission, or edit the embedded object consisting of the sound annotation. Further, the user can vote on the reputation of the sound annotation using the reputation selector

[33] .The question annotation symbol, the disagreement annotation symbol, the agreement annotation symbol, the exclamation annotation symbol, and the advertisement annotation symbol correspond to five relationships in the group of semantic relationships of annotations. These semantic relationships of annotations are the question relationship, the disagreement relationship, the agreement relationship, the exclamation relationship, and the advertisement relationship. These sound annotation symbols and the semantic relationships of the sound annotations are applied to the relationship between the embedded recording annotation playback range

[0393] and the sound annotation comment

[32] .

[0172] Aspects of the present invention are as described in FIGS. 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, and 49, and it will be understood that each item of these GUI figures can be implemented by computer-readable program instructions. Next, methods and systems related to the features and capabilities depicted in the exemplary embodiments of FIGS. 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, and 49 will be described and associated with the previously presented materials. Describing these methods and systems will include describing a bundle of data elements used to receive and store data, and a display bundle used to send and display visual elements to a client device. When the bundle of display elements is described, various item boxes are described. After describing these item boxes, the relationship with other presented elements including the user interface will be described. The bundle of data elements, the display bundle, and the item boxes to be described hereinafter are to be explained and interpreted within the context of the exemplary embodiments.

[0173] In the context of an exemplary embodiment, a bundle of data elements used to receive data from a client device and store this data using one or more computer systems will now be described. The bundle of data elements includes an image-annotation bundle, a video-annotation bundle, an audio-recording annotation bundle, an embedded-image annotation bundle, an embedded-video annotation bundle, an embedded-audio-recording annotation bundle, an annotation-group bundle, a derived-annotation-group bundle, and an annotation-group bundle with derived annotations.

[0174] In the context of an exemplary embodiment, an image-annotation bundle includes one or more image-annotation atoms. An image-annotation atom is a unit that can be used meaningfully by itself to annotate an image. For example, an image-annotation atom can include a bounding box

[0301] of an image-annotation subject, an image-annotation comment

[0303] within the box, and an image-annotation arrow

[0302] , as shown in the exemplary embodiment depicted in FIG. 36. However, the image-annotation comment

[0303] within the box can also be used alone near the image-annotation subject

[0300] . Thus, the image-annotation comment

[0303] within the box can itself become an image-annotation atom. In addition to image-annotation atoms, the image-annotation bundle includes frame information of the image to be annotated. The frame information includes the image size specified by the number of pixels on the x-axis and the number of pixels on the y-axis. When the image size is specified, based on the frame information, relative coordinates are assigned to the bounding box

[0301] of the image-annotation subject, which is an element constituting the image-annotation atom, the image-annotation comment

[0303] within the box, and the image-annotation arrow

[0302] . When the bounding box

[0301] of the image-annotation subject is specified by the X and Y coordinates of its center, width, and length, coordinates based on the lower left corner of the image are specified. The image-annotation comment

[0303] within the box can be specified in a similar manner together with the markup information of the annotation comment included in the image-annotation comment

[0303] within the box. This markup information includes the font size and formatting information of the text. Also, information regarding the mechanism for creating the text is included in the image-annotation bundle. The creation mechanism can be what the user wrote, an image caption using an artificial intelligence (AI) or deep learning model, another annotation, or a combination of multiple techniques.Furthermore, at least one of the user name, machine learning model, and other annotation data is recorded. When a combination of technologies is used, that information is also included in the image-annotation bundle. The image-annotation arrow [

[0302] ] can be specified by the X and Y coordinates of its start and end points and information specifying whether the arrowhead is at the start point or the end point. In this case as well, the coordinates are specified with reference to the lower left corner of the image. Each element included in the annotation atom specifies, in addition to the relative coordinate information, the generation mechanism. The generation mechanism specifies whether the element was generated by the user, a machine learning model, or a simple algorithm. Also, at least one of the user name, machine learning model name, and algorithm name is recorded. Furthermore, ovals can also be used. The ellipse is specified by the X and Y coordinates of the center relative to the lower left corner of the image, the width, height, and rotation of the ellipse. Furthermore, the generation mechanism is recorded together with at least one of the user name, machine learning model name, and algorithm name.

[0175] In the context of an exemplary embodiment, the embedded image-annotation bundle includes not only all the information within the image-annotation bundle but also the URL of the web page or document where the image is located. The description written about the image-annotation bundle is desirably interpreted by replacing the embedded image-annotation subject [

[0383] ] with the image-annotation subject [

[0300] ], the bounding box of the embedded image-annotation subject [

[0384] ] with the bounding box of the image-annotation subject [

[0301] ], the embedded image-annotation arrow [

[0385] ] with the image-annotation arrow [

[0302] ], and the embedded image-annotation comment [

[0386] ] within the box with the image-annotation comment within the box. FIGS. FIG. 36 and FIG. 47 need to be examined in detail. Also, when the document is a pdf or a complex web page, information for finding the image, such as the page of the pdf document or the DOM locator of the image, is also stored.

[0176] In the context of an exemplary embodiment, a video annotation bundle includes one or more video annotation atoms. A video annotation atom is a unit that can be meaningfully used by itself to annotate a video. For example, a video annotation atom can include a bounding box of a video annotation subject

[0322] , a bounding box motion path of a video annotation subject

[0332] , a video annotation comment input box

[0324] , a video annotation comment input box motion path

[0334] , a video annotation arrow

[0323] , and a video annotation arrow motion path

[0333] as shown in the exemplary embodiment depicted in FIG. 41. However, during video playback, it is also possible to place the video annotation comment in box

[0324] near the video annotation subject

[0321] and use box

[0324] and the video annotation comment input box motion path

[0334] alone. Therefore, the video annotation comment in box

[0324] and the motion path of the video annotation comment input box

[0334] can be video annotation atoms by themselves. The video annotation bundle includes, in addition to the video annotation atoms, frame information regarding the video to be annotated. The frame information includes a video size specified by the number of pixels in the x-axis direction and the number of pixels in the y-axis direction. This includes the area of the video

[0319] and should not include the playback area of the video annotation

[0320] .When the video size is specified, based on the frame information, video annotation atom composition elements such as the bounding box of the video annotation subject [

[0322] ], the bounding box motion path of the video annotation subject [

[0332] ], the video annotation comment input box [

[0324] ], the video annotation comment input box motion path [

[0334] ], the video annotation arrow [

[0323] ], and the video annotation arrow motion path [

[0333] ] are specified in relative coordinates. When the bounding box of the video annotation subject [

[0322] ] is specified by the X and Y coordinates of its center, width, and length, the coordinates are specified as relative coordinates from the lower left corner of the video. The bounding box motion path of the video annotation subject [

[0332] ] needs to specify the temporal evolution of the bounding box of the video annotation subject [

[0322] ]. This specifies the temporal changes in the X and Y coordinates of its center, its width, and its length. This specification is done by specifying the X and Y coordinates based on the lower left corner of the video at discrete time points within the video annotation playback range together with an interpolation mechanism between these discrete time steps. Since the relative x coordinate, relative y coordinate, width, and length are real numbers, remembering the values at discrete time points and specifying the interpolation method is sufficient for all four values. Interpolation methods such as linear interpolation, spline curves, and Bézier curves can be considered. The video annotation comment of the box [

[0324] ] and the motion path of the video annotation comment input box [

[0334] ] can also be specified in a similar way, and the markup information of the annotation comment included in the video annotation comment of the box [

[0324] ] can also be specified. This markup information includes font size and formatting information of the text. Furthermore, information regarding the text creation mechanism is included in the video annotation bundle. The creation mechanism can be what the user wrote, video captioning using AI or deep learning models, another annotation, or a combination of multiple methods. Furthermore, at least one of the user name, machine learning model, and other annotation data is recorded.When a combination of technologies is used, that information is also included in the video annotation bundle. The video annotation arrow

[0323] and the video annotation arrow motion path

[0333] can be specified by the relative X and Y coordinates of their start and end points, and information specifying whether the head of the arrow is at the start point or the end point. The specification of the video annotation arrow motion path

[0333] is done by specifying the relative X and Y coordinates of the start and end points at discrete time points within the video annotation playback range, and the interpolation mechanism between these discrete time steps. Here too, the coordinates are specified with reference to the lower left corner of the video. Each element included in the annotation atom specifies, in addition to the relative coordinate information, the generation mechanism. The generation mechanism specifies whether the element was created by the user, an AI or deep learning model, another annotation, or a combination of technologies. Furthermore, at least one of the user name, AI or deep learning model name, and algorithm name is recorded. Furthermore, an oval can also be used. The ellipse is specified by the X and Y coordinates of the center relative to the lower left corner of the video, the width and height of the ellipse, and the rotation. Furthermore, the generation mechanism is recorded together with at least one of the user name, AI or deep learning model name, and algorithm name. The video annotation symbol information can also be included in the video annotation bundle.

[0177] In the context of an exemplary embodiment, an embedded video annotation bundle includes not only all the information within the video annotation bundle, but also the URL of the web page or document where the video is located. The description written for the video annotation bundle should be interpreted as substituting, respectively, the embedded video annotation subject

[0388] for the video annotation subject

[0321] , the bounding box of the embedded video annotation subject

[0389] for the bounding box of the video annotation subject

[0322] , the embedded video annotation arrow

[0390] for the video annotation arrow

[0323] , the video annotation comment

[0391] within the embedded box for the video annotation comment

[0324] within the box, the bounding box motion path of the embedded video annotation subject for the bounding box motion path of the video annotation subject

[0332] , the motion path of the embedded video annotation arrow for the motion path of the video annotation arrow

[0333] , the motion path of the video annotation comment within the embedded box for the motion path of the video annotation comment input box

[0334] , and the embedded video annotation symbol for the video annotation symbol. Details need to be examined with reference to the form of the figure and Figure 48. If the document is a pdf or a complex web page, information for finding the video, such as the page of the pdf document or the DOM locator of the video, is also stored.

[0178] In the context of an exemplary embodiment, a recording annotation bundle includes one or more recording annotation atoms. A recording annotation atom is a unit that can be meaningfully used to annotate a recording by itself. For example, a recording annotation atom can include a recording annotation playback range, a recording annotation symbol, and a recording annotation comment. However, it is also possible to use only the recording annotation playback range and the recording annotation comment. Therefore, the recording annotation playback range and the recording annotation comment can also constitute a recording annotation atom by themselves. A recording annotation comment can be associated with markup information of the recording annotation comment. This markup information includes, in addition to text format information, font size. Furthermore, information regarding the creation mechanism of the text is in the recording annotation bundle. The creation mechanism can be what the user wrote, speech recognition by an AI or a deep learning model, copying of other annotations, or a combination of multiple methods. Furthermore, the creation mechanism is recorded together with at least one of a username, an AI or deep learning model name, and other annotation references. When a combination of technologies is used, one or more types of source names are recorded. Also, the playback range of the recording annotation can be determined based on a sound search. Therefore, it is desirable that the generation mechanism of the recording playback range is also recorded together with at least one of a username, an AI or deep learning model name, and other annotation comments. When a plurality of methods are combined and used, a plurality of generation mechanisms and sound source information are recorded. Furthermore, recording frame information is recorded to facilitate placing the recording annotation playback range in the recording annotation playback control.

[0179] In the context of the exemplary embodiments, an embedded recording annotation bundle includes not only all the information within the recording annotation bundle but also the URL of the web page or document where the recording is located. The description written about the recording annotation bundle needs to be interpreted by replacing the embedded recording annotation playback range

[0393] with the recording playback range, the embedded recording annotation symbol with the recording annotation symbol, and the embedded recording annotation comment with the recording annotation comment. Further, in order to make it easier to place the embedded recording playback range

[0393] in the embedded recording playback control, it should be understood that the embedded recording frame information is recorded in the embedded recording annotation bundle. Details need to be examined with reference to the forms of FIGS. 44 and 49. When the document is a complex web page, information for identifying the position of the recorded sound, such as the DOM locator of the recorded sound, is also stored.

[0180] In the context of the exemplary embodiments, an annotation group bundle includes references to all the annotations included in that particular annotation group. Further, information regarding whether the annotations conflict is stored together with metadata indicating when the annotation group was created and metadata indicating whether the conflict detection was performed by the user or AI or a deep learning model. At least one of the user and AI or the deep learning model that performed the conflict detection is also stored in the annotation group bundle.

[0181] In the context of an exemplary embodiment, a derived annotation group bundle includes references to all annotations included in that particular derived annotation group. Further, information regarding whether the annotations conflict is stored together with metadata indicating when the annotation group was created and whether the conflict detection was performed by a user or an AI or a deep learning model. At least one of the user and the AI or the deep learning model that performed the conflict detection is also stored in the derived annotation group bundle. The derived annotation group bundle includes one derived annotation along with references to all annotations included in that particular annotation group. The annotation connection arrow can be determined from the annotation references present within the derived annotation group bundle. In this exemplary embodiment, only one derived annotation exists, which provides all the information for drawing the annotation connection arrow. Further, the derived annotation comment is stored in the derived annotation group bundle. Markup information for the derived annotation comment is also stored. This markup information includes the font size and formatting information of the text. Further, information regarding the mechanism for creating the text is stored in the derived annotation group bundle. The creation mechanism can be something written by the user, something generated by an AI or a deep learning model, another annotation that was copied, or a combination of multiple techniques. Further, at least one of the user name, the machine learning model, and other annotation data is recorded. If a combination of techniques was used, that information is also included in the derived annotation group bundle.

[0182] In the context of an exemplary embodiment, an annotation group bundle with derived annotations includes all the derived annotations and references to annotations that exist in that particular annotation group with derived annotations. Further, information regarding whether an annotation conflicts is stored along with metadata indicating when the annotation group with derived annotations was created, and whether the conflict detection was performed by a user or an AI or a deep learning model. At least one of the user and the AI or the deep learning model that performed the conflict detection is also stored in the annotation group bundle.

[0183] Next, several types of display bundles and item boxes will be described. These descriptions are to be construed as being in the context of exemplary embodiments. In the context of the exemplary embodiments, the display bundles described are an image-annotation display bundle, a video-annotation display bundle, a recording annotation display bundle, an embedded image-annotation display bundle, an embedded video-annotation display bundle, an embedded recording annotation display bundle, an annotation-group display bundle, a derived annotation-group display bundle, and an annotation-group display bundle with derived annotations. In the context of the exemplary embodiments, the item boxes described are an image-annotation item box, a video-annotation item box, a recording annotation item box, an embedded image-annotation item box, an embedded video-annotation item box, an embedded recording item box, an annotation-group item box, a derived annotation-group item box, and an annotation-group item box with derived annotations. One purpose of the display bundles in the exemplary embodiments is for the web browser to provide data for displaying the corresponding item box associated with that type of annotation or media item. Thus, the item boxes are described in relation to the annotation item box

[36] presented in the exemplary embodiment depicted in FIG. 9. In addition to the numbered items within the annotation item box, the annotation item box also includes item box annotation emphasis text, item box annotation symbols, item box annotation comments, and item box URLs. When various item boxes are described in the context of the exemplary embodiments, data structures corresponding to the various display bundles can be illustrated. Further, referring to the displayed bundles can be useful for purposes of explanation. The displayed bundles are the visual output created using the display bundles. This visual output is included within the item boxes.In a context like a notepad, the displayed bundle can exist even if the entire item box does not exist.

[0184] In an exemplary embodiment, the image-annotation item box, as shown in the exemplary embodiment of FIG. 9, will include a separate context-variant annotation control bar

[41] and a separate context reputation selector

[42] , like the annotation item box

[36] . Instead of the screenshot

[37] with marked text, the image-annotation item box includes a thumbnail of the image along with the image-annotation displayed on top of the thumbnail. Instead of the separate context-annotation emphasized text

[38] , the displayed annotation symbol

[39] for the separate context question, and the separate context-annotation comment

[40] , the image-annotation item box displays an image-annotation comment. The URL is not displayed in the image-annotation item box but is displayed in the embedded image-annotation item box. The image-annotation display bundle needs to appropriately resize and overlay objects such as the subject boundary box of the image-annotation

[0301] , the image-annotation comment in the box

[0303] , the image-annotation arrow

[0302] , etc., and the marked thumbnail image of the image and the image displayed thereon. Further, the image-annotation display bundle needs to include the image-annotation comment, the user name, and the reputation information.

[0185] In an exemplary embodiment, the embedded image-annotation item box will include a separate context-variant annotation control bar

[41] and a separate context reputation selector

[42] , such as the annotation item box

[36] . Instead of a screenshot containing marked text, the embedded image-annotation item box includes a thumbnail of the embedded image and image-annotations displayed on top of the thumbnail image. Instead of the highlighted text

[38] of a separate context annotation, the displayed annotation symbol

[39] of a separate context question, and the separate context annotation comment

[40] , the embedded image-annotation item box has an embedded image-annotation comment. Its URL corresponds to the URL containing the embedded image to which the image-annotation was applied. The embedded image-annotation display bundle needs to include a marked thumbnail image of the embedded image with the image-annotation displayed, where objects such as the bounding box of the embedded image-annotation subject

[0384] , the embedded image-annotation comment

[0386] within the box, and the embedded image-annotation arrow

[0385] are appropriately resized and overlaid so that a web browser can display these embedded image-annotation item boxes. Further, the embedded image-annotation display bundle needs to include the embedded image-annotation comment, the user name, and reputation information.

[0186] In an exemplary embodiment, the video annotation item box will include a separate context variant annotation control bar

[41] and a separate context reputation selector

[42] , such as the annotation item box

[36] shown in the exemplary embodiment of FIG. 9. Instead of a screenshot

[37] with marked text, the video annotation item box includes a thumbnail of a video frame at a selected time point, along with a video annotation displayed on the thumbnail of the video frame at the same selected time point. Further, the play button can play the thumbnail video and a thumbnail video showing a part of the video annotation. Instead of the highlighted text

[38] of a separate context annotation, video sound summarization text, which is machine-generated text summarizing the sound in the video, will be displayed. Next, a video annotation comment is displayed in the video annotation item box, and a video annotation symbol that determines the semantic relationship of the annotation between the video sound summary text and the video annotation comment is also displayed in the video annotation item box. The URL does not exist in the video annotation item box but exists in the embedded video annotation item box. In order for a web browser to be able to display these video annotation item boxes, the video annotation display bundle needs to include a marked thumbnail image of the video frame and the selected-time video on it, with objects such as the bounding box of the video annotation subject

[0322] , the video annotation comment

[0324] in the box, and the video annotation arrow

[0323] appropriately resized and overlaid. When the user presses the play button, the video annotation display bundle needs to include a thumbnail video with the video annotation overlaid.Furthermore, the video annotation display bundle must include video audio summary text, video annotation symbols, video annotation comments, user names, and reputation information. Furthermore, the user will be able to modify mechanically generated video annotations.

[0187] In an exemplary embodiment, the embedded video annotation item box, as shown in the exemplary embodiment of FIG. 9, will include a separate context-variant annotation control bar

[41] and a separate context reputation selector

[42] , such as the annotation item box

[36] . Instead of a screenshot

[37] with marked text, the embedded video annotation item box includes a thumbnail of the embedded video frame at a selected point in time, along with the embedded video annotation displayed on the thumbnail of the embedded video frame at the same selected point in time. Further, the play button can be enabled to play the thumbnail video and the thumbnail video showing a portion of the video annotation. Instead of a separate context annotation emphasis text

[38] , it becomes an embedded video sound summary text that is part of the machine-generated text summarizing the sound within the video. Next, the embedded video annotation item box has an embedded video annotation comment, and the embedded video annotation item box also has an embedded video annotation symbol that determines the semantic relationship of the annotation between the embedded video sound summary text and the embedded video annotation comment. The URL of the web page in which the video is embedded is displayed in the item box of the embedded video annotation. In order for a web browser to be able to display these embedded video annotation item boxes, the embedded video annotation display bundle needs to include a marked thumbnail image of the frame of the embedded video at a selected point in time, with objects such as the bounding box of the embedded video annotation subject

[0389] , the embedded video annotation comment

[0391] within the box, and the embedded video annotation arrow

[0390] appropriately resized and overlaid, and the embedded video annotation displayed thereon.When the user presses the play button, the embedded video annotation display bundle should include a thumbnail video with the embedded video annotation overlaid. Further, the embedded video annotation display bundle should include the embedded video audio summary text, the embedded video annotation symbol, the embedded video annotation comment, the user name, and the reputation information. Further, the user should be able to modify aspects of the mechanically generated video annotation.

[0188] In an exemplary embodiment, the recorded annotation item box will include a separate context - variant annotation control bar

[41] and a separate context reputation selector

[42] , like the annotation item box

[36] shown in the exemplary embodiment of FIG. 9. Instead of a screenshot

[37] with marked text, the recorded annotation item box will include a thumbnail representing the recording, called a thumbnail related recording image. This is done first by creating a recorded summary text that converts the recording into a text representation. This is done using speech understanding techniques such as speech recognition. Once the text representation is available, an AI or deep - learning model is used to find an image related to the recorded summary text. This image will be called the related recording image. The thumbnail representation of the related recording image is the thumbnail related recording image. Instead of the separate context annotation highlighted text

[38] , it will be the recorded summary text. Next, the recorded annotation item box will have a recorded annotation comment, and the recorded annotation item box will also have a recorded annotation symbol that determines the semantic relationship of the annotation between the recorded summary text and the recorded annotation comment. The URL does not exist in the recorded annotation item box but exists in the embedded recorded annotation item box. The recorded annotation display bundle needs to include the recording image associated with the thumbnail, the recorded summary text, the recorded annotation comment, the recorded annotation symbol, the user name, and the levitation information so that a web browser can display these recorded annotation item boxes. Further, the user will be able to modify the mechanically - generated parts of the recorded annotation.

[0189] In an exemplary embodiment, the embedded recording annotation item box will include a separate context - variant annotation control bar

[41] and a separate context reputation selector

[42] , such as the annotation item box

[36] . Instead of a screenshot

[37] containing marked text, the embedded recording annotation item box will include a thumbnail representing the embedded recording, called a thumbnail - associated embedded recording image. This is done by first creating an embedded recording summary text that converts the embedded recording into a text representation. This is done using speech understanding techniques such as speech recognition. Once the text representation is available, AI or deep - learning models can be used to find an image related to the embedded recording summary text. This image will be called the associated embedded recording image. The thumbnail representation of the associated embedded recording image will be the thumbnail - associated embedded recording image. Instead of the separate context - annotation highlighted text

[38] , it will be the embedded recording summary text. Next, the embedded recording annotation item box will have an embedded recording annotation comment, and the embedded recording annotation item box will also have an embedded recording annotation symbol that determines the semantic relationship of the annotation between the embedded recording summary text and the embedded recording annotation comment. The URL of the website where the embedded recording was found will be displayed in the embedded recording annotation item box. The embedded recording annotation display bundle needs to include the embedded recording image associated with the thumbnail, the embedded recording summary text, the embedded recording annotation comment, the embedded recording annotation symbol, the user name, and the reputation information so that a web browser can display these embedded recording annotation item boxes. Further, the user will be able to modify the mechanically - generated parts of the recording annotation.

[0190] In an exemplary embodiment, the annotation group item box will include a separate context variant annotation control bar

[41] and a separate context reputation selector

[42] , such as the annotation item box

[36] shown in the exemplary embodiment of FIG. 9. Instead of a screenshot

[37] with marked text, the annotation group item box will include a thumbnail representing the annotation group. In the case of image annotation, this means a thumbnail image of an image having all the image annotations included in the annotation group. The image annotation and the annotation group include annotations of the embedded image. In the case of video annotation, it means a thumbnail image of the video at a certain point in time and all the video annotations included in the annotation group at the same point in time. Further, in the thumbnail video, a thumbnail video in which all the video annotations of the annotation group are displayed for the entire thumbnail video will be included. The video annotation and the annotation group include video annotations of the embedded video. For audio annotations, it means a thumbnail image representing the sound for all the audio playback ranges for all the audio annotations within the annotation group. Such an image will be based on the audio summary text for all the audio playback ranges. Instead of the separate context annotation highlighted text

[38] , a list of the annotations within that particular annotation group is displayed. Instead of the separate context annotation comment,

[40] will be the user-generated comment regarding that particular annotation group. Further, if the machine-generated image and the default text are not appropriate, these elements can be changed by the user. In another embodiment, the list of annotations of the annotation group can be replaced by a more appropriate summary text generated by a human.A URL does not exist in an annotation group item box without embedded annotations, but will exist in an annotation group item box that contains embedded annotations. The embedded annotation group annotation symbol can be used to indicate the semantic relationship between the source material and the annotations within the annotation group. The number of annotations within the annotation group is also displayed in the annotation group item box. The annotation group display bundle should include the related thumbnail image or thumbnail video, a list of references to the annotation objects, the related summary text, the related annotation comments, the related annotation group annotation symbol, the user name, and the reputation information so that a web browser can display these annotation group annotation item boxes. Further, the user will be able to modify the mechanically generated aspects of the annotation group.

[0191] In an exemplary embodiment, a derived annotation group item box will include a separate context variant annotation control bar

[41] and a separate context reputation selector

[42] , such as annotation item box

[36] , as shown in the exemplary embodiment of FIG. 9. Instead of a screenshot

[37] with marked text, the derived annotation group item box will include a thumbnail representing the derived annotation group. In the case of image annotation, this means a thumbnail image of an image having all the image annotations included in the derived annotation group. The image annotation and the derived annotation group include image annotations of embedded images. In the case of video annotation, it means a thumbnail image of the video at a certain point in time and all the video annotations included in the derived annotation group at the same point in time. Further, the thumbnail video will include a thumbnail video in which all the video annotations included in the derived annotation group are displayed throughout the thumbnail video. The video annotation and the derived annotation group include video annotations of embedded videos. In the case of recorded audio annotation, it means a thumbnail image representing the sound for all the recorded playback ranges of all the recorded audio annotations of the derived annotation group. Such an image will be based on the recorded summary text for all the recorded playback ranges. Instead of the highlighted text

[38] of a separate context annotation, a list of annotations within that particular derived annotation group is displayed. Instead of a separate context annotation comment,

[40] will be a user-generated comment regarding that particular derived annotation group. Further, if the machine-generated images and default text are not appropriate, these elements can be modified by the user. In an alternative embodiment, the list of annotations of the derived annotation group can be replaced with a more appropriate summary text generated by a human.The URL does not exist in a derived annotation group item box without embedded annotations, but exists in a derived annotation group item box that contains embedded annotations. The derived annotation group annotation symbol may be used to indicate the semantic relationship between the source material and the annotations within the annotation group. The annotation group display bundle should include related thumbnail images or thumbnail videos, a reference list to the annotation objects, related summary text, related derived annotation comments, related derived annotation group annotation symbols, user names, and reputation information so that a web browser can display these annotation group annotation item boxes. Additionally, the user will be able to modify the mechanically generated parts of the derived annotation group.

[0192] At this point, the annotation group and the derived annotation group may seem similar within the context of the exemplary embodiments described so far. However, the derived annotation group, the derived annotation group item box, and the derived annotation group display bundle, when used, will be treated differently from the annotation group, the annotation group item box, and the annotation group display bundle. The annotation group shows the links between annotations more closely, and the derived annotation group can be incorporated into the annotation group. The impact of these differences can be seen by considering FIGS. 44 and 45.

[0193] In an exemplary embodiment, an annotation group with derived annotations includes a separate context - variant annotation control bar

[41] and a separate context reputation selector

[42] , such as an annotation item box

[36] , as shown in the exemplary embodiment of FIG. 9. The annotation group item box with derived annotations and the corresponding annotation group display bundle with derived annotations are similar to the annotation group item box and the annotation group display bundle, except that information regarding derived annotations, such as its constituent annotations and derived annotation comments, exists for each derived annotation group. The existence of this information enables the display of derived annotations as depicted in FIGS. 44 and 45. Further, the item box of the annotation group with derived annotations displays the number of annotations within the annotation group with derived annotations and the number of derived annotation groups. All of this information is reflected in the annotation group display bundle with derived annotations. The item box of the annotation group with derived annotations displays a separate context - variant annotation control bar

[41] and a separate context reputation selector

[42] , similar to the annotation item box

[36] . Instead of a screenshot containing marked text, the item box of the annotation group with derived annotations displays a representative thumbnail of the annotation group with derived annotations

[37] . In the case of image - annotations, it means a thumbnail image of an image that includes all of the image - annotations included in the annotation group of the derived annotation. Since the derived annotation group changes the image by the annotations included in the derived annotation group, what is important is the set of all the annotations included in the derived annotation group. Image - annotations and annotation groups include annotations of embedded images.In the case of video annotation, it means all video annotations included in an annotation group that has a thumbnail image of the video at a certain point in time and the derived annotation at the same point in time. Furthermore, the thumbnail video will include a thumbnail video in which all video annotations included in the annotation group with the derived annotation are displayed throughout the thumbnail video. Again, what is important is the set of all annotations included in the annotation group with the derived annotation. Video annotations and annotation groups include the video annotations of the embedded video. In the case of audio recording annotation, it means a thumbnail image representing the sound for all audio recording playback ranges of all audio recording annotations included in an annotation group with the derived annotation. Such an image will be based on the audio recording summary text for all audio recording playback ranges in the set of all annotations in the annotation group with the derived annotation. Instead of the highlighted text of the separate context annotation

[38] , the list of annotations of that specific annotation group becomes the derived annotation. Instead of the separate context annotation, the comment

[40] becomes the user-generated comment regarding that specific annotation group. Furthermore, if the machine-generated images and default text are not appropriate, these elements can be modified by the user. In an alternative embodiment, the list of annotations of the annotation group with the derived annotation can be replaced by a more appropriate summary text generated by a human. The URL does not exist in the annotation group item box with the derived annotation without the embedded annotation, but will exist in the annotation group item box with the derived annotation including the embedded annotation. An embedded annotation group including the derived annotation symbol can be used to show the semantic relationship between the source material and the annotations in the annotation group including the derived annotation.In the item box of the annotation group with derived annotations, the number of annotations in the annotation group with derived annotations and the number of derived annotation groups are displayed. In order for a web browser to display the item boxes of these annotation groups with derived annotations, the annotation group display bundle with derived annotations should include related thumbnail images or thumbnail videos, a reference list to annotation objects, related summary text, related annotation comments, related derived annotation group symbols, user names, and reputation information. Furthermore, users will be able to modify the mechanically generated aspects of the annotation groups with derived annotations.

[0194] We have described various types of data element bundles, display bundles, and item boxes within the context of the exemplary embodiments, and the changes that need to be made to FIGS. 14, 15, 16, 17, and 18 will be described. These changes are to adapt these sequence diagrams and flowcharts to the methods of implementation and systems related to the features and capabilities shown and described in the described explanations as well as in the exemplary embodiments depicted in FIGS. 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, and 49. The sequence diagram of FIG. 14 is applied with one of the data element bundles, the corresponding display bundle, and the corresponding displayed bundle previously described for different types of annotations when different types of annotations are implemented. The display bundle occurs when FIG. 14 uses the annotation display bundle, and the displayed bundle occurs when FIG. 14 uses the displayed annotation bundle. The sequence diagram of FIG. 15 is also applied with the display bundle and the displayed bundle previously described. These displayed ones occur where the displayed bundle of annotations occurs in FIG. 15, and the displayed bundle occurs where FIG. 15 uses the displayed annotation bundle. The display bundle has data for displaying either its corresponding item box or its corresponding displayed bundle. Depending on the exemplary embodiment, either the item box or the displayed bundle can be displayed on the user interface. Typically, the item box is a bundle displayed with a separate context-variant annotation control bar

[41] and a separate context reputation selector

[42] . The separate context-variant annotation control bar

[41] and the separate context reputation selector

[42] are shown for the annotation item box

[36] in the exemplary embodiment shown in FIG. 9.The sequence diagram of FIG. 16 is applied by replacing the annotation symbol display bundle with an appropriate embedded annotation display bundle and replacing the closed displayed annotation symbol with a visible element of the appropriate embedded annotation. The embedded annotation display bundle and the visible element of the embedded annotation are as described above. FIGS. 17 and 18 are applied as they are when the bundle of data elements and the display bundle are selected from the types previously defined in this specification.

[0195] In an exemplary embodiment, the item bin

[44] within the annotation composition context area

[43] of FIG. 10 can include an annotation item box, an embedded image - annotation item box, an embedded video - annotation item box, an embedded recording item box, an annotation - group item box, a derived - annotation - group item box, a derived - annotation - group item box, and a memo pad. The memo pad can also be displayed as a memo - item box. All of these types of item boxes are provided with a separate context - variant - annotation - control bar

[41] as depicted for the annotation item box

[36] . While other elements of the annotation item box

[36] change for other types of item boxes, the separate context - variant annotation - control bar

[41] and the displayed form factor of the item box do not change. This allows these new types of item boxes to be integrated into the item bin

[44] . All of these item types are created on an external site and are displayed in the item bin as soon as the user creates them

[44] . This is easily achieved by using a web socket that pushes information to all connected web browsers from an annotation server

[61] that should receive a display bundle based on access control and use cases. The item box can be moved from the item bin

[44] to a writable collection represented by the displayed writable collection

[50] as depicted in the exemplary embodiment of FIG. 11. Further, the annotation item box, the embedded image - annotation item box, the embedded video - annotation item box, and the embedded sound recording item box can be added to the active notepad

[0159] using the add icon

[0158] button in the notepad in the context - variant - annotation - control bar

[34] as depicted in the exemplary embodiment of FIG. 23.The icon to add to the notepad

[0158] button is not displayed in the context-variant annotation control bar

[34] labeled in all figures. However, such a button can be added to the context-variant annotation control bar

[34] within an exemplary embodiment. Other item types can also be added to the active notepad

[0159] using a trigger for adding to the notepad. In all cases, clicking on an item box within the active notepad

[0159] should open the relevant item in a browser tab, scroll to the position of the annotation, and display the annotation. All different types of item boxes can be displayed within a collection. The collection shown in the exemplary embodiment depicted in Figure 31 shows only annotation item boxes, but all types of item boxes can be present within the collection. All comments regarding the stack set of displayed annotation items apply to stack sets of various types of items. Additionally, comments regarding window functionality also apply. The set of stacked items can be grouped based on a URL or based on an annotated image item, video item, recording item, or document item. Further, the type of item included in the collection does not change other properties of the collection. The collection represented by the displayed collection can be displayed on a page similar to that shown in the exemplary embodiment of Figure 32.

[0196] As described above, a filter such as the search filter shown in the exemplary embodiment of FIG. 10 is used to receive input from a user. In the case of the search filter

[46] of the item bin

[44] , it is a search phrase. These inputs are converted into parameter values that are sent to the annotation server

[61] . These parameter values are used at the annotation server

[61] to enter parameters into the request query used in the electronic database

[62] . The request query includes parameter values and page numbers. The request query containing parameter values is used to specify a set of items and the order of those items. These items include web page annotations, embedded image - annotations, embedded video - annotations, embedded recording annotations, image - annotations, video - annotations, recording annotations, annotation - groups, derived annotation - groups, annotation - groups with derived annotations, PDF document annotations, PDF documents, images, videos, links, normal comments, and the like. The filter can also be restricted to a specific item type. The page number specifies a particular page. Further, the request query can include the number of items to be sent to the web browser running on the client electronic device. This request query includes parameters of filter values such as search phrases, an exclusion list of items to be excluded from the list of specified items, and a sequential ordering for the list of those items. As explained earlier, this sequential ordering provides information for paging in the exemplary embodiments depicted in FIGS. 10 and 31. Collection - filters and collection paging operate as previously explained for the description of the exemplary embodiments depicted in FIGS. 10 and 32. The same applies to the pinning function.

[0197] The inbox described in the exemplary embodiment depicted in FIG. 33 would function in a similar manner if the collection included different item types. The difference would be that the user would be able to discuss more item type details within the collection message. Additionally, functions can be built into message chains specialized for each type of item. And functions can be implemented to change the objects that are annotation components, approve annotation changes, or fork annotations added to the current collection. Also, when discussing image annotations, the discussion can proceed smoothly by using selectors for each object of the image annotation. Furthermore, selectors for other item types, such as video annotations and recording annotations, can be incorporated into the message chain function. With such functions, the subject bounding box

[0322] of the video annotation, the video annotation comment

[0324] within the box, or the annotation arrow

[0323] can be changed, and also the motion path

[0332] of the subject bounding box of the video annotation, the video annotation arrow motion path

[0333] , and the video annotation comment input box motion path

[0334] can be changed. After a change is proposed in the message chain, the owner of the item can approve the change, and the admin can approve forking the item into a new item in the context of the current collection.

[0198] The exemplary embodiments depicted in FIGS. 28 and 29 are also applicable when there are multiple item types. In this situation, the annotation server

[61] and the electronic database

[62] will process these different types of items. In an exemplary embodiment, the annotation server

[61] may have a REST API for processing different item types, and the electronic database

[62] has a schema for processing different data elements within various types of bundles. The hardware requirements are affected as follows. If the video

[0319] is not stored locally and is referenced via a URL, the presence of the video

[0319] will not increase the storage requirements as much as if the video

[0319] were stored locally. If stored locally or via a content delivery network

[63] , the video

[0319] will have a significant impact on the storage requirements. The presence of the recorded playback

[0348] may create a need to use a deep learning model to process audio and speech recognition. The presence of the video

[0319] will increase the system's computational processing requirements because it generally has greater computational processing requirements than other media types.

[0199] In an exemplary embodiment, to provide additional annotation functionality to a user, machine learning and deep learning models can be deployed and configured via a web browser extension of an annotatable web browser. If the annotatable web browser is not constructed through the use of a web browser extension, it is also possible to directly process the deployment and configuration through the annotatable web browser. This functionality can be made available to the user through a panel or floating toolbar anywhere on the web page. This functionality can also be exposed through menu options or a fixed browser toolbar. The floating toolbar can be attached to the page assignment or placed within the side panel of the web browser. In the case of images and embedded images, this panel or floating toolbar can be made to utilize buttons for tasks such as the recognition of objects within the image, the recognition of text within the image, the setting of recognition characteristics for object and text detectors, and the setting of characteristics for annotation elements. Annotation elements include a bounding box for the annotation subject, an annotation arrow, and an annotation comment box. When the object recognition button is pressed, a dashed bounding box of the annotation subject candidate is drawn around the annotation subject candidate. The user can select this bounding box and write an annotation comment in the annotation comment input box. Depending on the settings, an annotation arrow may be automatically created. The user can also be provided with the option to select the bounding boxes of multiple annotation subjects, as a result of which one annotation arrow is created for each bounding box of an annotation subject, with one end of each annotation arrow pointing to the bounding box of the annotation subject and the other end pointing to the annotation comment input box or the annotation comment box depending on whether an annotation comment has been written. The placement of the arrow is a configurable property.Other configurable characteristics are the visibility characteristics of the annotation subject's bounding box, annotation arrow, and annotation comment box. The visibility characteristics include visible, mouse over, and invisible. Visible is for annotation elements that are always displayed. Mouse over corresponds to annotation elements that are visualized on mouse over of the target area. And invisible corresponds to annotation elements that are always invisible. The mouse over target area can be at least one of the visible annotation subject or annotation element. In an alternative embodiment, an annotation subject outline is used that encloses a closed area including the pixels of the annotation subject. In contrast to the bounding box of the annotation subject, these annotation subject outlines can be used. Further, the object outline can be a mouse over element as opposed to a visible element, and this visible characteristic can be controlled through configuration settings. It is also possible to have configuration settings that allow the user to select whether to use the bounding box of the annotation subject or the annotation subject outline. Determining the bounding box of the annotation subject or the annotation subject outline depends on a machine learning model or deep learning model that can recognize an object and determine the bounding box of the annotation subject or the outline of the annotation subject. The deep learning model can be executed on a web browser operating on a client electronic device by using the CPU or GPU of the client electronic device. The GPU can be used through the use of the WebGL interface exposed by modern web browsers, and as a result, the GPU of the client electronic device can be used.The bounding box of the annotation subject and the outline of the annotation subject can then be determined through the use of an annotation object detector using a region-based convolutional neural network (R-CNN), or through object segmentation using a region-based convolutional neural network (R-CNN). In addition to the R-CNN-based model, alternative embodiments can employ other machine learning or deep learning methods. Since different deep learning models have different recognition characteristics, the user can input the desired recognition characteristics of the object detector and the text detector. Based on those characteristics, different machine learning or deep learning models can be selected. Also, a CNN (Convolution Neural Network) that recognizes characters and numbers can be used to recognize characters in an image. Furthermore, in the setting options, it is possible to select whether to group individual characters into words and whether to group individual words into n-grams. Additionally, by inputting words or tags in the setting options for recognition characteristics, it is also possible to limit recognition to specific types of objects input by the user. Also, to handle videos and embedded videos, buttons for setting the motion path of the annotation elements are added. Also, the technology described with the image can also be implemented for individual frames of the video. Furthermore, the deep learning model can track the annotation subject over time and ensure that the bounding box of the annotation subject and the outline of the annotation subject move with the annotation subject. This can be done by using individual frame analysis at discrete points and an interpolation method between the discrete points. Thereby, the motion paths of the bounding box of the annotation subject and the annotation subject outline are obtained. Also, the contour movement of the annotation subject outline is calculated.The evolution settings for these motion paths and annotation outlines are available to the user via a panel, menu option, browser toolbar, or additional button on a floating toolbar that handles the motion path setting options for annotation elements. Machine learning and deep learning models for audio and speech recognition in videos for annotation purposes can be handled by another setting option in a panel, menu option, browser toolbar, or floating toolbar.

[0200] As used herein, the word "exemplary" means "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations, nor is it intended to be construed as a model that must be literally replicated. Accordingly, this specification is not intended to limit the invention and is to be regarded as exemplary.

[0201] Conditional language, particularly the use of words such as "can," "could," "might," or "may," is generally intended to convey that a particular embodiment includes, and that other embodiments do not include, a particular feature, element, and / or step, unless specifically stated otherwise or understood within the context in which it is used. Thus, such conditional language is not generally intended to mean that a feature, element, and / or step is in any way required for one or more embodiments, or that one or more embodiments necessarily include logic for determining, with user input or prompting, whether these features, elements, and / or steps are included in, or are to be performed in, any particular embodiment. Headings used herein are for the convenience of the reader and are not intended to limit the invention or the scope of the claims.

[0202] As used herein, the phrase "at least one" of a list of items refers to any combination of those items including a single member. By way of example, "at least one of A, B, or C" is intended to cover A, B, C, A and B, A and C, B and C, and A, B, and C. Connective language such as the phrase "at least one of X, Y, and Z" is generally understood within the context as being used to convey that an item, term, etc. may be at least one of X, Y, or Z, unless otherwise specified. Thus, such connective language is not intended to require that a particular embodiment have at least one of X, at least one of Y, and at least one of Z each present.

[0203] The flowcharts, GUIs, sequences, systems, components, and swimlane activity diagrams in the figures illustrate the usage examples, functionality, architecture, and operation of possible implementations of methods, computer program products, and systems according to various embodiments of the present invention. In this regard, each item in the flowcharts, GUIs, sequences, systems, components, and swimlane activity diagrams can represent a module, segment, or portion of instructions consisting of one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the item may occur out of the order noted in the figures. For example, two items shown in succession may in fact be executed substantially simultaneously, or the items may sometimes be executed in the reverse order depending on the functions involved. Also, it will be noted that each item and combination of items in the flowcharts, GUIs, sequences, systems, components, and swimlane activity diagrams that do not involve one or more users can be implemented by a special-purpose hardware-based system that does not involve one or more users, which performs a particular function or act, or by a combination of special-purpose hardware and computer instructions.

[0204] Aspects of the present invention are described herein with reference to the flowcharts, GUIs, sequences, systems, components, and swimlane activity diagrams of methods, computer program products, and systems according to embodiments of the present invention. It will be understood that each item of the flowcharts, GUIs, sequences, systems, components, and swimlane activity diagrams that does not involve one or more users, as well as combinations of items of the flowcharts, GUIs, sequences, systems, components, and swimlane activity diagrams that do not involve one or more users, can be implemented by computer-readable program instructions.

[0205] These computer-readable program instructions are provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to generate machine-executable instructions that execute via the computer's processor or other programmable data processing apparatus, creating means for implementing functions / acts that do not involve one or more users, specified by one or more items of the flowcharts, GUIs, sequences, systems, components, and swimlane activity diagrams, where the one or more items do not involve one or more users. These computer-readable program instructions can also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other devices to function in a particular manner, and the computer-readable storage medium having instructions stored therein comprises a manufactured article that includes instructions for implementing aspects of the functions / acts specified in the item or items of the flowcharts, GUIs, sequences, systems, components, and swimlane activity diagrams, and the item and items may or may not include one or more users.

[0206] As will be understood by those skilled in the art, such code may be stored or adapted for storage on one or more tangible machine-readable media such as memory chips, local or remote hard disks, optical disks, or other media, which may be accessed by a processor-based system and execute the stored code. It should be noted that the tangible medium may be composed of paper or other suitable media on which the instructions are printed. For example, the instructions may be electronically captured via optical scanning of paper or other media and then compiled, interpreted, or otherwise processed in a suitable manner and then stored in a computer memory. Further, it should be noted that the tangible medium may constitute a version control repository that utilizes memory chips, local, or remote hard disks, optical disks, or other media for storage. For example, the instructions may be retrieved from the version management repository and then processed in a compiled, interpreted, or other suitable manner and then stored in a computer memory.

[0207] Computer-readable program instructions may be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to generate a computer-implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified by items of a flowchart, GUI, sequence, system, component, swimlane activity diagram.

[0208] Furthermore, the methods, non-transitory computer-readable media, systems, and apparatuses described herein may be subject to various modifications and alternative forms, and specific examples thereof are shown in the drawings and described in detail herein. However, the present invention is not limited to the specific forms or methods disclosed, and conversely, the present invention is intended to cover all modifications, equivalents, and alternatives that fall within the spirit and scope of the various embodiments described and the appended claims. Additionally, any disclosure herein of any specific feature, aspect, method, characteristic, feature, quality, attribute, element, etc. related to an implementation or embodiment can be used in any of the other implementations or embodiments described herein. Any method disclosed herein need not be performed in the order recited. The methods disclosed herein can include specific actions performed by an operator, but the methods can also include, explicitly or implicitly, any third-party instructions for those actions. The scope disclosed herein also encompasses any and all overlaps, sub-ranges, and combinations thereof.

[0209] Words such as "at least" and "at a minimum" include the recited number. A number preceded by a term such as "about" or "approximately" should be interpreted in light of the circumstances to include the recited number (e.g., as accurately as reasonably possible under the circumstances, e.g., -5%, 5%, 10%, 15%, etc.). For example, "about 3.5 mm" includes "3.5 mm". A phrase preceded by a term such as "substantially" should be interpreted in light of the circumstances to include the recited phrase (e.g., as reasonably possible under the circumstances). For example, "substantially constant" includes "constant".

[0210] The detailed description is presented to enable a person skilled in the art to make and use the invention and is provided in connection with the requirements for obtaining a patent. Various modifications to the exemplary embodiments will be readily apparent to those skilled in the art, and the general principles of the invention may be applied to other embodiments. Some features of the invention may be used without the corresponding use of other features. Accordingly, the invention is not intended to be limited to the embodiments shown, but to give the broadest scope consistent with the principles and features described herein.

Claims

1. A non-transitory computer-readable medium having data stored therein representing software executable by a web browser application operating on at least one processor, the non-transitory computer-readable medium comprising instructions for displaying a plurality of displayed annotation symbols embedded within the text of a web document, the displayed annotation symbols having a state selected from a group of states, and the group of states comprising a closed state, and an open state , wherein an annotation symbol displayed in the closed state displays the annotation symbol on the web document while minimizing visual changes to the web document, and an annotation symbol displayed in the open state displays screen elements, the screen elements comprising an annotation that highlights a selected text piece from the document, where the selected text piece represents an annotation, an annotation comment, and a displayed annotation symbol that designates a selection by the creator of the annotation from a group of semantic relationships of the annotation , wherein the group of semantic relationships of the annotation comprises a disagreement relationship indicating a disagreement between the selected text piece and the annotation comment, and a question relationship indicating an annotation comment that asks a question about the selected text piece , characterized in that it is a non-transitory computer-readable medium.

2. The non-transitory computer-readable medium according to claim 1, wherein the group of semantic relationships of the annotation further comprises a consent relationship indicating consent between the selected text piece and the annotation comment, and an exclamation relationship indicating an annotation comment that designates an exclamation that is a point of attention or an interesting fact regarding the selected text piece , characterized in that it is the non-transitory computer-readable medium according to claim 1.

3. The non-transitory computer-readable medium according to claim 1, wherein the selected text piece is highlighted in a color pre-assigned to the displayed annotation symbol according to the semantic relationship of the annotation to which the displayed annotation symbol corresponds , characterized in that it is the non-transitory computer-readable medium according to claim 1. The non-transitory computer-readable medium according to claim 1, wherein annotation comments, and a context variant annotation control bar including controls for a user to control privacy settings selected from a group of privacy states, and different background colors are assigned to each privacy state within the group of privacy states, and the background color of the annotation comment box indicating the privacy setting, and an annotation symbol in an open state that displays an annotation comment box including the non-transitory computer-readable medium according to claim 1.

5. The non-transitory computer-readable medium according to claim 1, wherein it further comprises an instruction for displaying a symbol beacon, the symbol beacon has a triggered state, and a non-triggered state, and has a state selected from a group of states, the symbol beacon is provided with a trigger mechanism for transitioning the symbol beacon from a non-triggered state to a triggered state, and when the symbol beacon is in a triggered state, one displayed annotation symbol beacon is displayed for each displayed annotation symbol, and all displayed annotation symbols are made more prominent within the viewable area of the user of the annotatable web browser. The non-transitory computer-readable medium according to claim 1, characterized in that.

6. The non-transitory computer-readable medium according to claim 5, wherein the displayed annotation symbol beacon and the corresponding displayed annotation symbol are assigned the same pre-assigned color according to the semantic relationship of the corresponding annotation of the displayed annotation symbol. The non-transitory computer-readable medium according to claim 5, characterized in that.

7. The non-transitory computer-readable medium according to claim 5, wherein the trigger mechanism is a button displayed in the viewable area of the annotatable web browser. The non-transitory computer-readable medium according to claim 5, characterized in that.

8. The non-transitory computer-readable medium according to claim 5, wherein the trigger mechanism is pressing a key on the keyboard, and pressing a button on the mouse and is a pre-defined combination of inputs selected from a group including. The non - transitory computer - readable medium according to claim 5, characterized in that...

9. The non - transitory computer - readable medium according to claim 6, characterized in that the trigger mechanism is a menu option in an annotatable web browser.

10. The non - transitory computer - readable medium according to claim 1, wherein... The group of semantic relationships of the annotation further includes an advertising relationship indicating that the annotation comment is an advertisement related to the selected text piece, and The creator of the annotation is the advertiser, and The displayed annotation symbol of the advertisement corresponds to the displayed annotation symbol of the relationship indicating that the annotation comment is an advertisement related to the selected text piece, The non - transitory computer - readable medium according to claim 1, characterized in that...

11. An annotation server operable to receive an annotation symbol bundle representing an annotation from an annotatable web browser, wherein the annotation symbol bundle has a document's Uniform Resource Locator (URL), and data representing a text piece selected from the document, where the selected text piece represents an annotation, and information for finding the selected text piece within the document, and an annotation comment, and and data representing an annotation symbol that designates a user selection from a group of semantic relationships of the annotation, and the group of semantic relationships of the annotation has a disagreement relationship indicating a disagreement between the selected text piece and the annotation comment, and a question relationship indicating an annotation comment that asks a question about the selected text piece, characterized in that, and an electronic database functioning to store the annotation symbol bundle, and the annotation server is operable to receive from the annotatable browser a marked screenshot including the annotation symbol bundle representing the annotation and a part of the document indicating the selected text piece, and the selected text piece is visually marked, A system characterized in that...

12. The system according to claim 11, wherein the marked screenshot is a thumbnail of the marked screenshot.

13. An annotation server that operates to receive an annotation symbol bundle representing an annotation from an annotatable web browser, wherein the annotation symbol bundle a uniform resource locator (URL) of the document, data for representing a selected text piece from the document, where the selected text piece represents an annotation, information for finding the selected text piece within the document, an annotation comment, and data representing an annotation symbol that specifies a user selection from a group of semantic relationships of the annotation having, wherein the group of semantic relationships of the annotation a disagreement relationship indicating a disagreement between the selected text piece and the annotation comment, and a question relationship indicating an annotation comment that asks a question about the selected text piece characterized by having, and an electronic database that functions to store the annotation symbol bundle, a system comprising, The annotation server further comprises a function for creating an annotation symbol display bundle from the annotation symbol bundle, The annotation symbol display bundle data of a text piece representing an annotation selected from the document, where the selected text piece is a representative of the annotation, information for finding the text piece selected from within the document, an annotation comment, and data representing an annotation symbol, characterized by a system.

14. The system according to claim 13, wherein the annotation server The function for compilation includes receiving a request from an annotatable web browser operating on an electronic device and obtaining a symbol bundle display set including a plurality of annotation symbol display bundles using one or more computer systems, and The function for transmission includes transmitting the symbol bundle display set in a form that displays a plurality of annotation symbol display bundles by an annotatable web browser operating on an electronic device, further comprising The system according to claim 13, characterized in that **Claim 15** The system according to claim 14, wherein the function for compilation includes obtaining an advertising annotation symbol display bundle, the advertising annotation symbol display bundle is an annotation symbol display bundle that is also an advertisement, the advertising annotation symbol display bundle comes from an advertising server, the function for transmission includes transmitting the advertising annotation symbol display bundle in a form that is displayed by a web browser operating on an electronic device, The system according to claim 14, characterized in that **Claim 16** A system comprising a client electronic device having at least one processor and a memory, a web browser, and a web browser extension that functions to create a modified graphical user interface (GUI) display of the web browser, The modified GUI display comprises a plurality of displayed annotation symbols, received from an annotation server, embedded in the text of a web document, the displayed annotation symbols having a state selected from a group of states, and the group of states a closed state, and an open state, characterizing the modified GUI display, the annotation symbols displayed in the closed state display annotation symbols on the web document with minimal visual change to the web document, and the annotation symbols displayed in the open state received from the annotation server, highlighting annotations of selected text pieces selected from the document, where the selected text pieces represent annotations, annotation comments, and displayed annotation symbols that specify a selection by the creator of the annotation from a group of semantic relationships of the annotation, displaying elements comprising the group of semantic relationships of the annotation has a disagreement relationship indicating disagreement between the selected text piece and the annotation comment, and a question relationship indicating an annotation comment that asks a question about the selected text piece, characterizing the system. **Claim 17** The system according to claim 16, wherein ​ A group of semantic relationships of the annotation indicates a consent relationship indicating consent between the selected text piece and the annotation comment, and an exclamation relationship indicating an annotation comment that designates an exclamation that is a point of attention or an interesting fact regarding the selected text piece, further includes The system according to claim 16, characterized in that

18. The system according to claim 16, wherein the selected text piece is emphasized in a color pre-assigned to the displayed annotation symbol according to the semantic relationship of the annotation to which the displayed annotation symbol corresponds The system according to claim 16, characterized in that

19. The system according to claim 16, wherein the annotation comment, a control bar that displays a control for the user to control a privacy setting selected from a group of privacy states, and the background color of an annotation comment box indicating a privacy setting assigned to each privacy state in the group of privacy states with different background colors, further includes an element that displays an annotation comment box having the displayed annotation symbol in an open state, The system according to claim 16.

20. The system according to claim 16, wherein the modified GUI display further includes a triggered state and, and a non-triggered state a symbol beacon having a state selected from the group consisting of, the symbol beacon includes a trigger mechanism for transitioning the symbol beacon from a non-triggered state to a triggered state, and when the symbol beacon is in a triggered state, a screen element including the displayed annotation symbol beacon is displayed, making the position of the displayed annotation symbol more prominent for each displayed annotation symbol displayed in the user-viewable area of the annotatable web browser The system according to claim 16, characterized in that

21. The system according to claim 20, wherein the displayed annotation symbol beacon and the corresponding displayed annotation symbol are assigned the same pre-assigned color according to the semantic relationship of the annotation to which the displayed annotation symbol corresponds, The system according to claim 20, characterized in that

22. The system according to claim 20, wherein the trigger mechanism is a button displayed in a viewable area of an annotatable web browser The system according to claim 20, characterized in that

23. The system according to claim 20, wherein pressing a key on the keyboard, and pressing a button on the mouse a trigger mechanism that is a combination of predefined inputs selected from a group including The system according to claim 20, characterized in that

24. The system according to claim 20, characterized in that the trigger mechanism is a menu option in an annotatable web browser

25. The system according to claim 16, wherein the group of semantic relationships of the annotation further includes an advertising relationship indicating that the annotation comment is an advertisement related to the selected text piece, the creator of the annotation is the advertiser, the displayed annotation symbol corresponds to the relationship indicating that the annotation comment is an advertisement related to the selected text piece, and is the displayed annotation symbol, and the advertisement is transmitted from an advertisement server The system according to claim 16, characterized in that

Images