Refining mode in tablet computing device

Abstract

Embodiments of the present invention provide a computerized system and method in an electronic paper tablet device to provide a user with two modes for interacting with the electronic paper tablet device. In the authoring mode, the user can input text, make annotations, draw and take other authoring actions. In the refinement mode, the user can edit texts, annotations and drawings and perform other refinement on actions taken by himself / herself in the creation mode. Both hardware and software of the electronic paper tablet support both modes, including special hardware keys on a keyboard associated with the electronic paper tablet that enable a user to switch between the two modes. Thus, embodiments of the present invention enable a user to closely process text, edit and comment text using a stylus device (or even an input mechanism such as a finger), while maintaining integrity with underlying text and its display.

Description

Technical Field

[0001] This disclosure generally relates to hybrid graphic and text editing in devices suitable for receiving graphic and text input from multiple sources, such as keyboards and styluses. Specifically, embodiments of the invention enable users of tablet devices to switch from author input mode to edit or refine mode on their tablet devices to make various changes to documents displayed on the tablet devices. Background Technology

[0002] Many tablet computers, some mobile phones, and some PCs come equipped with both a keyboard and a touch pen or stylus input system. One type of tablet computer (e-paper tablet) typically includes both a keyboard and some form of stylus. The interactive display on these devices combines a screen such as an LCD, OLED, plasma, or electrophoretic display (EPD) with its input system. The input system recognizes the presence of an input object, such as a stylus touching or approaching the display, which can generate new lines or drawings to be displayed on the device.

[0003] Multiple input devices enhance the capabilities of these electronic devices for users. For example, a user can add text and drawings (e.g., annotations) to a document simultaneously. Text characters can be generated in response to user interaction with alphanumeric input devices (e.g., keyboards and / or touchscreen keyboards) and can be used to update text documents and / or create new text documents. Example annotations include sketches, drawings, markings, and doodles corresponding to gestures made by the user (e.g., using input mechanisms). Users can even collaborate with others to develop documents.

[0004] Similarly, users can read text on an e-paper tablet screen and use input devices (such as styluses) to annotate and add other comments (e.g., edit) to the text. The tablet device can generate annotations in response to free-form gestures provided by the user near the displayed text (e.g., to a touchscreen).

[0005] Overall, this behavior in these e-paper tablets makes the device more like traditional paper, a significant advantage some users find. Of course, offering users so many input options in an electronic device can complicate its use. While such a device may offer greater flexibility than traditional pen and paper (e.g., adding drawings in the middle of text without erasing and rewriting), the controls for making such changes may not be immediately accessible. Similarly, many users may not tolerate a lengthy training period to learn how to use their own devices. Therefore, these controls should be easily accessible to users.

[0006] Despite significant progress in improving the usability of tablet devices in recent years, further improvements are still needed. Furthermore, the specific use cases of collaboration with annotations on such devices appear to require additional functionality not available on traditional devices. Summary of the Invention

[0007] Embodiments of the present invention provide a computerized annotation system including a computer memory and a display screen that displays a document having text characters, wherein these text characters have an underlying document representation stored in the computer memory such that each text character in the document has a unique identifier. The computerized annotation system may further include an input detector module that receives a first annotation on the document displayed on the display screen, wherein the first annotation includes graphical data generated by interaction between the display screen and an input mechanism, wherein the input detector module initiates an update to the display screen to display both the document and the first annotation. The computerized annotation system may further include a refinement module that determines whether the computerized annotation system is in authoring mode or refinement mode, wherein if the computerized annotation system is in authoring mode, the refinement module initiates an update to store the document including the first annotation in the computer memory, and if the computerized annotation system is in refinement mode, the refinement module initiates refinement of the document.

[0008] Embodiments of the present invention also provide a computerized method in a computerized annotation system, the computerized method comprising: retrieving a document having text characters from a computer memory for display on a screen of the computerized annotation system, wherein the text characters have an underlying document representation stored in the computer memory such that each text character in the document has a unique identifier. The computerized method may receive a first annotation on the document displayed on the screen, wherein the first annotation includes graphical data generated by interaction between the screen and an input mechanism. The computerized method updates the screen to show both the document and the first annotation. The computerized method determines whether the computerized annotation system is in author mode or refinement mode. If it is determined that the computerized annotation system is in author mode, the first annotation is stored in the computer memory, thereby modifying the document. If it is determined that the computerized annotation system is in refinement mode, refinement of the document is initiated. Attached Figure Description

[0009] The disclosed embodiments have advantages and features that will become more apparent from the specific embodiments, the appended claims, and the accompanying drawings (the drawings). A brief description of the drawings follows.

[0010] Figure 1The diagram illustrates the system architecture of an electronic paper tablet device 110, which receives input from the input mechanism 120 when a user makes a gesture using the input mechanism 120.

[0011] Figure 2 This is a block diagram of the system architecture of an electronic paper tablet device 110 according to an embodiment of the present invention. The electronic paper tablet device includes an input detector module 210, an input digitizer 220, a display system 230, and a graphics generator 240.

[0012] Figure 3 The illustration shows an embodiment of the invention with a design for... Figures 1 to 2 The front and right perspective views of the electronic paper tablet 300, which is described in the electronic paper tablet device 110.

[0013] Figure 4 The illustration shows the hardware components of an example electrophoretic display (EPD) according to an embodiment of the present invention.

[0014] Figure 5 The illustration shows a representative display stack 500 according to an embodiment of the present invention, which consists of various display component layers associated with a contact-sensitive display on an electronic paper flat panel device 110.

[0015] Figure 6 The illustration shows a block diagram of components of an example machine that is capable of reading instructions from a machine-readable medium and executing those instructions in a processor (or controller) according to an embodiment of the present invention.

[0016] Figure 7 The illustration shows a rear view of an electronic paper tablet 300 according to an embodiment of the present invention, showing volcano feet 701a-701d, spring pads 703, and antenna area 705.

[0017] Figure 8 The illustration shows a top view of an electronic paper tablet device 300 according to an embodiment of the present invention, showing volcano feet 701a, 701d and a power button 801.

[0018] Figure 9 The illustration shows a bottom view of an electronic paper tablet device 300 according to an embodiment of the present invention, showing volcano feet 701b, 701d and a USB-C connector 307.

[0019] Figure 10 The illustration shows a right view of an electronic paper tablet device 300 according to an embodiment of the present invention, showing volcano feet 701b, 701a and a charging area 304 for recharging the input device 120 when the input device is an active handwriting stylus.

[0020] Figure 11The illustration shows a left view of an electronic paper tablet device 300 according to an embodiment of the present invention, showing volcano feet 701c, 701d.

[0021] Figure 12 An electronic paper tablet device 300 according to an embodiment of the present invention is illustrated. The electronic paper tablet device is connected to a detachable keyboard 1201 having special keys 1205, and the electronic paper tablet device 300 is held in an folio 1203.

[0022] Figure 13 An active stylus 1300 according to an embodiment of the present invention is illustrated. The active stylus includes a core component 1302, which itself includes one or more antennas configured to communicate with a tablet device (such as an electronic paper tablet device 110).

[0023] Figure 14 The illustration shows the outer casing 1401 of a stylus 1400 designed to fit the user's hand according to an embodiment of the present invention.

[0024] Figure 15 An anchoring module according to an embodiment of the present invention is illustrated, which provides a numeric identifier (e.g., anchoring ID "1234567890" 1503) for a portion of text 1501 having a specific character (e.g., "A").

[0025] Figure 16 The illustration shows the annotation "On" represented as a series of dots (e.g., dot 1604) according to an embodiment of the present invention.

[0026] Figure 17 The illustration shows a note 1701 (e.g., a strikethrough mark) according to an embodiment of the invention, which is anchored to a text character (e.g., "A") via an anchoring module 270 using a bounding box 1703.

[0027] Figure 18 The illustration shows an embodiment of the invention applied to two annotations 1805 and 1809, which are “borderless” embodiments.

[0028] Figures 19A to 19D The illustration shows the use of annotation anchoring by anchoring module 270 according to an embodiment of the present invention, and the effect of anchoring on text displayed in a part of the interface (e.g., a display screen).

[0029] Figure 20 The illustration shows the display 2005 of the electronic paper tablet device 110, which is connected to the keyboard 1201 and can be used by the user with a stylus 1400.

[0030] Figure 21The illustration shows a display 2005 of an electronic paper tablet device 110 in a refined mode according to an embodiment of the present invention.

[0031] Figures 22 to 24 The illustration shows a series of actions that a user can take in refinement mode to select a segment of text, handwritten notes, and drawings on the display 2005, according to an embodiment of the present invention.

[0032] Figures 25 to 27 The illustration shows a series of actions that a user can take in refinement mode to select a segment of text, handwritten notes, and drawings on the display 2005, according to another embodiment of the invention.

[0033] Figure 28 and Figure 29 An embodiment of a refinement mode for editing a document on a display 2005 according to an embodiment of the present invention is illustrated.

[0034] Figure 30 and Figure 31 An embodiment of the present invention is illustrated for creating a refinement mode with additional whitespace in a document on a display 2005.

[0035] Figure 32 and Figure 33 The illustration shows a series of steps that a user can take to move and / or rearrange portions of a document displayed on a display 2005, according to an embodiment of the present invention.

[0036] Figure 34 and Figure 35 The illustration depicts a series of steps related to a refinement mode according to an embodiment of the present invention, which allows a user to edit a given document displayed on a display 2005 by placing caret characters in the text.

[0037] Figure 36 The illustration shows additional gestures that a user can make using a stylus 1400 in finer mode for editing a document, according to an embodiment of the present invention.

[0038] The accompanying drawings illustrate various embodiments of the invention for illustrative purposes only. Those skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods shown herein may be employed without departing from the principles described herein. Detailed Implementation

[0039] The accompanying drawings (Figures) and the following description refer to preferred embodiments only by way of illustration. It should be noted that, from the following discussion, alternative embodiments of the structures and methods disclosed herein will be readily considered as feasible alternatives that can be employed without departing from the claimed principles.

[0040] Reference will now be made in detail to several embodiments, examples of which are illustrated in the accompanying drawings. It is worth noting that similar or identical reference numerals may be used in the drawings whenever feasible, and similar or identical reference numerals may indicate similar or identical functions. The drawings depict embodiments of the disclosed system (or method) for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods shown herein may be employed without departing from the principles described herein.

[0041] Overview

[0042] A system and related processes are disclosed to assist users in inputting data into their tablet devices (particularly e-paper tablets) using various input mechanisms (such as keyboards and styluses), and to help users navigate various modes of using their tablet devices (e.g., entering new information into documents and editing documents). Furthermore, the provided operational controls are designed to be relatively intuitive for the user and serve as an aid to improving the efficiency of using such tablet devices. The description of this particular invention begins from... Figure 20 The invention begins by describing related and associated hardware devices for implementing embodiments of the invention. Therefore, before describing the invention in more detail, the electronic paper tablet, its associated keyboard, its stylus, and the interaction of these hardware components will be described. Following this description, embodiments of the invention will be described.

[0043] System and device configuration for main tablet devices

[0044] like Figure 1As shown, an electronic paper tablet device 110, operating as a computerized annotation system, receives input from an input mechanism 120. For example, when a user makes a gesture using the input mechanism 120, the input mechanism 120 makes physical contact with a touch-sensitive surface (e.g., a touch-sensitive screen) on the electronic paper tablet device 110. The input mechanism 120 can be a finger, a stylus, or a marker. The tablet device 110 referred to here is called an "electronic paper tablet," a device that allows the user to mimic the feeling of writing with a regular pen and paper. Such devices are also referred to as "electronic paper" and "electronic ink." Based on the nature of the contact, the electronic paper tablet device 110 generates and executes instructions for updating the content displayed on the touch-sensitive screen to reflect the gesture input. For example, in response to a gesture that translates a spoken message (e.g., written text or drawing), the electronic paper tablet device 110 updates the touch-sensitive screen to display the translated message. As another example, in response to a gesture that selects a navigation option, the electronic paper tablet device 110 updates the screen to display a new page associated with the navigation option. The electronic paper tablet device 110 (described below, which performs as a computerized annotation system) may include a computer system configured to receive touch input (e.g., detect handwriting or gestures on a display screen) and process the gestures into instructions for updating the user interface (e.g., the display screen) to provide a response corresponding to the gesture (e.g., displaying the result of the gesture) on the electronic paper tablet device 110 for display, and to store such gestures in a computer memory (e.g., ...). Figure 6 The memory 604 shown herein contains the gestures. One particular form of gesture described herein includes “annotations” made by the user to text displayed on the electronic paper tablet device 110. The electronic paper tablet device 110 may also include a keyboard for typing input. Although embodiments of the invention are designed for electronic paper systems, embodiments of the invention are also applicable to other forms of computing devices capable of receiving and processing input from a handwriting stylus device.

[0045] Input mechanism 120 can refer to any device or object compatible with the touch-sensitive screen of electronic paper tablet device 110, particularly a stylus device, such as a so-called active pen device with its own power supply or a static pen that receives its power when engaged with the touch-sensitive screen on electronic paper tablet device 110. In one embodiment, input mechanism 120 can work with an electronic ink (e.g., E-ink) touch-sensitive screen. For example, input mechanism 120 can refer to any device or object that can be contacted with a screen and from which the screen can detect a touch or contact of input mechanism 120. Once a touch or contact is detected, electronics associated with the screen generate a signal that electronic paper tablet device 110 can process into a gesture that can be provided for display on the screen. When a gesture is detected by input mechanism 120, electronics within the touch-sensitive screen generate a signal that encodes instructions for displaying content or updating content previously displayed on the screen of electronic paper tablet device 110 based on movement on the screen of the detected gesture. For example, when processed by the electronic paper tablet device 110, the encoded signal can cause a representation of the detected gesture, such as doodles, to be displayed on the screen of the electronic paper tablet device 110. As mentioned, the input mechanism 120 can be a stylus or other type of pointing device, including a part of the user's body, such as a finger.

[0046] In one embodiment, the input mechanism 120 is a coil of magnetic material. When brought near the screen of the electronic paper tablet device 110, the magnetic coil helps generate a magnetic field that encodes signals conveying instructions, which are processed by the electronic paper tablet device 110 to provide a representation of a gesture for display on the screen, for example, as a marker. The input mechanism 120 can be force (e.g., pressure) and tilt sensitive, allowing the system to make a natural visual response to both pressure and tilt applied by the user. Furthermore, the interaction between the input mechanism and the touch-sensitive screen of the electronic paper tablet device 110 can generate different encoded signals for processing, for example, to display representations of gestures with different characteristics on the screen, such as thicker line markers. In an alternative embodiment, the input mechanism 120 includes a power source (e.g., a battery) that can generate an electric field using the touch-sensitive surface. It is worth noting that the encoded signal is a signal that is generated and can be conveyed. The encoded signal can have signal patterns that can be used for further analog or digital analysis (or interpretation).

[0047] In one embodiment, the touch-sensitive screen is a capacitive touchscreen. The screen may be designed using glass or polymer materials coated with a conductive material. Electrodes or electrical components carrying alternating current are arranged along the screen coating (e.g., with diamond-shaped crosshairs) to maintain a constant level of current flow across the entire screen. A second set of electrodes is arranged horizontally. The matrix of vertically active electrodes and horizontally inactive electrodes generates an electrostatic field at each point on the screen. When an input mechanism 120 with conductive properties (e.g., a coiled magnetic coil, a human finger, or something else that triggers a capacitive effect) comes into contact with an area of ​​the screen of the electronic paper tablet device 110, current flows through the horizontally arranged electrodes, thereby disrupting the electrostatic field at the point of contact on the screen. The disruption of the electrostatic field at each point covered by the gesture can be measured, for example, as a change in capacitance and encoded as an analog or digital signal.

[0048] In an alternative embodiment, the touch-sensitive screen is a resistive touchscreen. A resistive touchscreen comprises two metal layers: a first metal layer with striped electrodes located on a substrate (such as glass or plastic); and a second metal layer with transparent electrodes positioned therein. When a contact from an input mechanism (e.g., a stylus, finger, or palm) is made on the surface of the touchscreen, the two layers are pressed together. Upon contact, a voltage gradient is applied to the first layer and measured as a distance through the second layer to determine the horizontal coordinates of the touch point on the screen. A subsequent voltage gradient is applied to the second layer to determine the vertical coordinates of the touch point on the screen. The combination of the horizontal and vertical coordinates records the exact location of the touch point on the touch-sensitive screen. Unlike capacitive touchscreens that rely on conductive input mechanisms, resistive touchscreens are configured to sense contact from virtually any input mechanism. Although some embodiments of the electronic paper tablet device 110 are described herein with reference to capacitive touchscreens, those skilled in the art will recognize that resistive touchscreens can also be implemented.

[0049] In yet another alternative embodiment, the touch-sensitive screen is an inductive touchscreen. An inductive touchscreen includes a metal front layer configured to detect deflection when contact is made on the screen by an input mechanism. Accordingly, the inductive touchscreen is configured to sense contact from virtually any input mechanism. Although some embodiments of the electronic paper tablet device 110 have been described herein with reference to capacitive touchscreens, those skilled in the art will recognize that alternative touchscreen technologies can be implemented, such as inductive touchscreens.

[0050] According to some embodiments of the present invention, cloud server 130 is configured to receive information from electronic paper tablet device 110 and / or transmit instructions and / or documents to electronic paper tablet device 110. For example... Figure 1As shown, cloud server 130 may include cloud data processor 150 and data storage device 160. Data recorded and stored by ePaper tablet device 110 can be transmitted to cloud server 130 via network 140 for storage in data storage device 160. Similarly, ePaper tablet device 110 can receive data from cloud server 130, including documents simultaneously worked on by a user of ePaper tablet device 110 and by a user on another computing device simultaneously editing the same document. In addition, data storage device 160 can store documents, images, or other types of content generated or recorded by the user through ePaper tablet device 110. In some embodiments, cloud data processor 150 monitors the activity and usage of ePaper tablet device 110, transmits processing instructions to ePaper tablet device 110, and can also communicate with other devices. For example, cloud data processor 150 can adjust the synchronization protocol between data stored in data storage device 160 and ePaper tablet device 110, and possibly with other devices.

[0051] Interaction between the ePaper Tablet Device 110 and the Cloud Server 130 typically occurs via a network 140, which enables communication between the ePaper Tablet Device 110 and the Cloud Server 130. In one embodiment, the network 140 uses standard communication technologies and / or protocols, including but not limited to links using technologies such as Ethernet, 802.11, WiMAX, 3G, 4G, LTE, Digital Subscriber Line (DSL), Asynchronous Transfer Mode (ATM), InfiniBand, and PCI Fast Advanced Switching. The network 140 may also utilize dedicated, custom, or private communication links. The network 140 may include any combination of local area networks and / or wide area networks, using both wired and wireless communication systems. The cloud server 130 may be implemented alternatively and, in some embodiments, may be replaced by hardware and software that provide similar functionality but may not be considered a conventional cloud server.

[0052] Figure 2 This is a block diagram of the system architecture of an electronic paper tablet device 110 operating as a computerized annotation system according to an example embodiment. Figure 2 In the illustrated embodiment, the electronic paper flat panel device 110 includes an input detector module 210, an input digitizer 220, a display system 230, and a graphics generator 240.

[0053] The input detector module 210 recognizes gestures that have been made or are being made on the screen of the electronic paper tablet device 110. The input detector module 210 refers to electronic devices integrated into the screen of the electronic paper tablet device 110, configured to interpret coded signals generated by contact between the input mechanism 120 and the screen as recognizable gestures. To this end, the input detector module 210 can evaluate the properties of the coded signal to determine whether the signal represents an intentional or unintentional gesture by the user. When the electronic paper tablet device 110 has a keyboard or keyboard functionality, the input detector module 210 can also assist in processing text input.

[0054] The input digitizer 220 can be configured to convert analog signals encoded by contact between the input mechanism 120 and the screen into a digital instruction set. The converted digital instruction set can be processed by the electronic paper tablet device 110 to generate or update the user interface displayed on the screen to reflect intentional gestures.

[0055] Display system 230 may include physical and firmware (or software) components to display (e.g., render) a user interface on a screen. The user interface may correspond to any type of visual representation that can be presented to or viewed by a user of the electronic paper tablet device 110.

[0056] Based on the digital signal generated by the input digitizer 220, the graphics generator 240 can be configured to generate or update graphics of the user interface to be displayed on the screen of the electronic paper tablet device 110. The display system 230 can be configured to present these graphics of the user interface to the user using electronics integrated into the screen.

[0057] When the input mechanism 120 contacts the touch-sensitive screen of the electronic paper tablet device 110, the input detector module 210 recognizes a gesture made through the screen. This gesture can be recognized as part of an encoded signal generated by a force (e.g., pressure) sensor in the input mechanism 120 and / or by corresponding electronics of the screen of the display system 230. The encoded signal is transmitted to the input detector module 210, which evaluates the properties of the encoded signal according to at least one gesture rule to determine whether the gesture was intentionally made by the user. If the input detector module 210 determines that the gesture was intentional, it transmits the encoded signal to the digitizer output. The encoded signal is an analog representation of the gesture received by a sensor matrix embedded in the screen of the device 110.

[0058] In one example embodiment, the input digitizer 220 converts physical points on the screen touched by the input mechanism 120 into a set of instructions for updating content provided for display on the screen; this set of instructions may also be stored in computer memory (e.g., ...). Figure 6(See memory 604). For example, if the input detector module 210 detects an intentional gesture to slide from the first page to the second page, the input digitizer 220 receives an analog signal generated by the input mechanism 120 when it performs the swipe gesture. The input digitizer 220 generates a digital signal for the swipe gesture, which instructs the display system 230 of the electronic paper tablet device 110 to update the user interface of the screen to transition from, for example, the current page (or the first page) to the next page (or the second page, which may be before or after the first page).

[0059] In one example embodiment, the graphics generator 240 receives digital instruction signals (e.g., swipe gestures indicating page transitions, such as flipping or turning pages) generated by the input digitizer 220. The graphics generator 240 generates graphics or updates previously displayed user interface graphics based on the received signals. The generated or updated user interface graphics are provided to be displayed on the screen of the electronic paper tablet device 110 by the display system 230, for example, showing the user a transition from the current page to the next page.

[0060] exist Figure 2 In the illustrated embodiment, the graphics generator 240 includes a rasterizer module 250, a depixelator module 260, an anchoring module 270, and a thinning module 280. User input gestures drawn on a touch-sensitive surface are received as vector graphics and input to the rasterizer module 250. The rasterizer module 250 converts the input vector graphics into raster graphics, which can be displayed (or provided for display) on the touch-sensitive surface. The depixelator module 260 can apply image processing techniques to convert the displayed raster graphics back into vector graphics, for example, to improve the processing power of the e-paper tablet device 110 and save memory on the e-paper tablet device 110. In at least one embodiment, when the content displayed on the screen is exported to a different format or exported to a different system, the depixelator module 260 can convert the displayed raster graphics back into vector graphics.

[0061] The graphics generator 240 may include an anchoring module 270 that anchors user gestures (e.g., lines forming annotations) associated with text and other information on the display to anchor these annotations to the corresponding text or other information, such that these annotations remain anchored to the corresponding text or other information even when displayed on other systems or taking into account further changes to the underlying text (e.g., adding or deleting words or numbers) or changes to other annotations. The anchoring module 270 may include appropriate electronic hardware for performing its necessary functions and / or may include features transmitted via a processor (e.g., ...). Figure 6 The operation of the processor 602 shown.

[0062] The graphics generator 240 may include a refinement module 280 that receives and interprets user input commands received during refinement modes, as discussed below, interacts with the anchoring module 270 to determine which parts of the document displayed on the electronic paper tablet device 110 are relevant to the user-received commands (typically via a stylus pen), and participates in other functions on the electronic paper tablet device 110 to execute and complete the received commands. The refinement module 280 may include appropriate electronic hardware for performing its necessary functions and / or may include components via a processor (e.g., ...). Figure 6 The operation of the processor 602 shown.

[0063] Further details regarding the structure and function of the electronic paper tablet and its graphic display can be found in USP 11,158,097 by Martin Sandsmark and Gunnar Sletta entitled “Generating vector graphics by processing raster graphics” and USP 10,824,274 by Sondre HoffDyvik, Martin Sandsmark, and Magnus Haug Wanberg entitled “Interactive displays”, both of which are incorporated herein by reference.

[0064] Figure 3 The illustration shows a targeted Figures 1 to 2 The electronic paper tablet device 110 is described in front and right perspective views of the electronic paper tablet 300 operating as a computerized annotation system. In addition, the electronic paper tablet 300 includes a touch-sensitive display 303. The display 303 has been processed to provide a paper-like feel when the user of the device interacts with the device using the input device 120. Figure 3 A charging area 304 for recharging the input device 120 when the input device is an active stylus, according to an embodiment of the present invention, is also shown. A set of magnets may be located inside the electronic paper tablet 300 near the charging area 304 to hold the input device 120 in place while it is recharging. Figure 3 A USB-C connector 307 is also shown, which can be used to provide power to the electronic paper tablet 300 and to transfer various types of data to or from the electronic paper tablet 300. The electronic paper tablet 300 also includes [further details to be provided below]. Figures 5 to 10 Several actuators and other features are shown in the diagram.

[0065] Figure 4The illustration shows hardware components of an example electrophoretic display (EPD) according to a disclosed embodiment. As discussed, various display technologies can be employed, including EPD, LCD, and reflective LCD (rLCD). The specific display device deployed may be... Figure 2 The electronic paper flat panel device 110 shown is part of the display system 230 and generates... Figure 3 The image shown is displayed on the display 303 of the electronic paper tablet 300. The EPD includes a gate driver 409, a source driver 411, a shift register 423 with data and clock signal lines, a latch 425, a voltage selector 427, and rows constituting the display 405. The EPD industry borrows some components and concepts from the LCD industry; however, there are also some fundamental differences between the two devices. Of particular relevance here is the persistence of pixels in an EPD display. Unlike LCD displays, EPD displays do not require the frequent refreshes required by LCD displays. In an EPD display, for example, once a neutral voltage is set for a pixel, that pixel will not change and will persist for a long time, especially compared to an LCD display.

[0066] As mentioned, the electrophoretic display (EPD) 405 utilizes many aspects of LCD manufacturing infrastructure and driving mechanisms. The driving electronics typically consist of a gate driver (GD) 409 and a source driver (SD) 411. The display 405 has multiple rows of pixels. The pixel values ​​within a row can vary; for example, a logic high voltage might indicate a "black" pixel, while a logic low voltage or "ground" might indicate a colorless pixel. The pixels in the EPD 405 function similarly to small capacitors that operate over long time intervals. An EPD pixel contains a large number of charged particles suspended in a liquid. If a charge is applied, the particles move to the surface, where they become visible. White and black particles carry opposite charges, allowing the pixel's display to change from white to black by applying opposite charges. Therefore, the waveform applied to the EPD comprises long strings of voltages to change from black to white or vice versa. EPD technology is also known to be capable of applying variable voltage levels that mix white and black particles to produce a variety of grayscale shades. Voltage levels within a pixel can also be layered to provide shades between colorless and black (e.g., grayscale levels). The groups of pixels surrounding each other can form areas that provide the user with some visible characteristics, such as images on the screen, like the images on the screen of the display system 230 of the electronic paper tablet device 110.

[0067] To change pixel values ​​in an area, the display 405 typically scans from the top row (e.g., row 0, 421) and applies voltages to update pixels in a specific row where the pixels need to be changed to correspond to the displayed image. In this example, a start pulse (GDSP) 403 can be used to reset driver 411 to row 0, 421, and a direction (DIR) 404 can be used to reset the direction. Row-by-row selection is performed by selecting a row (e.g., active row 413) by driving driver gate 409. All pixels in a row are addressed simultaneously using data transmitted to the display. Latch 425 receives the next set of voltages to be applied to the pixel row from shift register 423. When the scan of the active row is complete and the pixels are changed or updated as necessary, a clock pulse (GDCLK) 415 is sent to driver gate 409 to move to the next row 417 for scanning.

[0068] As mentioned, those skilled in the art will recognize that similar functionality can be achieved using standard LCD, OLED, MicroLED, or other types of displays, and the description of EPD technology provided herein is merely to illustrate one embodiment of the invention.

[0069] The source driver 411 is used to set the target voltage for each pixel / column in the selected row. This source driver consists of a shift register 423 for storing voltage data, a latch circuit 425 for enabling pixel data transfer during the previous row exposure, and a voltage selector (multiplexer) 427 for converting the latched voltage selection into the actual voltage. For all rows to be updated, all voltage values ​​must be shifted into register 423 and latched to make the voltage available.

[0070] Figure 5 The illustration depicts a representative display stack 500, which comprises various display component layers associated with a touch-sensitive display on an electronic paper tablet device 110. From bottom to top, these layers include an EPD display 507, an optically clear adhesive layer 506, a light guide sheet 504, an optically clear adhesive layer 505, a touch sensor 503, an optically clear adhesive layer 502, and a front panel 501. The EPD display 507 is located above the device's writing technology 508 and may include components related to… Figure 4 The component is similar to the EPD display 405 described herein. The writing technology 508 may include components similar to... Figure 2Similar to the graphic generator 240 disclosed herein. Optically transparent adhesive layers 506, 504, and 502 are used to attach components from other parts of the stack 500 together, but in a manner that does not alter or obstruct light transmission through the stack 500 to the user. A touch sensor 503 operates as part of the touchscreen, as already described above. The stack 500 is typically covered by a front panel 501 that the user can touch during device operation. The front panel 501 is typically made of glass, but may also be made of other materials. For example, U.S. Application No. 18 / 114,896, filed February 27, 2023, describes a front panel suitable for implementation in the stack 500 entitled “CoverLens for Consumer Electronics Device,” inventored by HesselVink and Lars. And Thong Xuan Nguyen, whose application is incorporated herein by reference.

[0071] exist Figure 5 The light guide plate 504 shown can be an ultra-thin light guide plate. Consumer demand is driving display stacks (such as display stack 500) to become thinner and thinner, allowing the entire device (e.g., electronic paper tablet 300) to become thinner and to reduce parallax effects caused by the separation between the EPD and the writing surface.

[0072] It should be noted that although the discussion herein takes place in the context of an electronic paper tablet device 110, the principles described can be applied to other computer systems, including, for example, personal computers, smartphones, and other tablet devices (e.g., Apple iPad, Samsung Galaxy Tab, Amazon Fire). The computer system will refer to... Figure 6 Further description.

[0073] Figure 6 This is a block diagram illustrating components of an example machine according to one embodiment, capable of reading instructions from a machine-readable medium and executing those instructions in a processor (or controller). In this example, Figure 6 A computer system 600 is shown (e.g., Figure 1 The illustrated electronic paper tablet device 110 is a schematic representation of a machine in the form of an example of a computing portion, within which program code (e.g., software) can be executed to cause the machine to perform any or more methods discussed herein. The electronic paper tablet device 110 may include some or all of the components of the computer system 600. The program code may include instructions 624 executable by one or more processors 602. In the electronic paper tablet system 110, the instructions may correspond to, for example... Figure 1 , Figure 2 and Figure 4 The functional components described herein. Figure 6 This is an example of a processing system in which some or all of the components described herein can be utilized by the modules described herein for execution.

[0074] Although the embodiments described herein are within the context of an electronic paper tablet system 110, it should be noted that these principles can be applied to other touch-sensitive devices. In these contexts, Figure 6 The machine can be a server computer, client computer, personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), cellular phone, smartphone, web device, network router, Internet of Things (IoT) device, switch or bridge, or any machine capable of (sequentially or otherwise) executing instructions 624 specifying actions to be taken by that machine. Furthermore, although only a single machine is illustrated, the term "machine" should also be understood to include any collection of machines that individually or jointly execute instructions 624 to perform any one or more methods discussed herein.

[0075] Example computer system 600 includes one or more processors 602 (e.g., a central processing unit (CPU), one or more graphics processing units (GPUs), one or more digital signal processors (DSPs), one or more application-specific integrated circuits (ASICs), one or more radio frequency integrated circuits (RFICs), or any combination thereof), main memory 604, and static memory 606, which are configured to communicate with each other via bus 608. Computer system 600 may further include a visual display interface 610. The visual interface may include software drivers that enable the display of a user interface on a screen (or monitor). The visual interface may display the user interface directly (e.g., on a screen) or indirectly (e.g., via a visual projection unit) on a surface, window, etc. For ease of discussion, the visual interface may be described as a screen or display. Visual interface 610 may include, for example, a touch-enabled screen of electronic paper tablet system 110 or may be connected to it and may be associated with display system 230. The computer system 600 may also include an input device 612 (e.g., a stylus, keyboard, and / or touchscreen keyboard), a cursor control device 614 (e.g., a mouse, trackball, joystick, motion sensor, or other pointing instrument), a storage unit 616, a signal generation device 618 (e.g., a speaker), and a network interface device 620, which are also configured to communicate via a bus 608.

[0076] Storage unit 616 includes a machine-readable medium 622 on which instructions 624 (e.g., software) embodying any one or more methods or functions described herein are stored (or encoded). The instructions 624 (e.g., software) may also reside wholly or at least partially within main memory 604 or processor 602 (e.g., within the processor's cache memory) during execution by computer system 600, main memory 604, and processor 602, which also constitutes a machine-readable medium. The instructions 624 (e.g., software) may be transmitted or received via network 426 through network interface device 620.

[0077] Although machine-readable medium 622 is shown as a single medium in the example embodiment, the term "machine-readable medium" should be considered to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) capable of storing instructions (e.g., instruction 624). The term "machine-readable medium" should also be considered to include any medium capable of storing instructions (e.g., instruction 624) that are executed by a machine and cause the machine to perform any one or more methods disclosed herein. The term "machine-readable medium" includes, but is not limited to, data repositories in the form of solid-state memory, optical media, and magnetic media.

[0078] The computer system 600 may also include one or more sensors 625. It should also be noted that the computing device may include only... Figure 6 A subset of the components shown and described. For example, an IoT device may include only a processor 602, a small storage unit 616, a main memory 604, a visual interface 610, a network interface device 620, and a sensor 625.

[0079] Representative electronic paper tablet

[0080] Figure 3 Provided similar Figures 1 to 2 A representative view of the electronic paper 110 and the electronic paper tablet 300 operating as a computerized annotation system. Figure 7 The illustration shows a rear view of an electronic paper tablet 300 according to an embodiment of the present invention, illustrating volcano feet 701a-701d, spring pads 703, and antenna region 705. Antenna region 705 is located on the main antenna of the electronic paper tablet 300 (e.g., on the electronic paper tablet device 110, adjacent to...). Figure 1 The cloud server 130 shown communicates with the antenna (which can generate beacon signals for communication with a stylus pen) located outside and above, allowing the e-paper tablet device 300 to connect to, for example, the internet. The spring pad 703 allows the e-paper tablet device 300 to connect to other devices, such as folio devices with keyboards, like... Figure 12 The hardware configuration shown.

[0081] Figure 8 The illustration shows a top view of an electronic paper tablet device 300 according to an embodiment of the present invention, showing volcano feet 701a, 701d and a power button 801.

[0082] Figure 9 The illustration shows a bottom view of an electronic paper tablet device 300 according to an embodiment of the present invention, showing volcano feet 701b, 701d and a USB-C connector 307.

[0083] Figure 10 The illustration shows a right view of an electronic paper tablet device 300 according to an embodiment of the present invention, illustrating volcano feet 701b, 701a and a charging region 304 for recharging the input device 120 when the input device is an active stylus. A set of magnets may be located inside the electronic paper tablet 300 near the charging region 304 to hold the input device 120 in place while it is recharging.

[0084] Figure 11 The illustration shows a left view of an electronic paper tablet device 300 according to an embodiment of the present invention, showing volcano feet 701c, 701d.

[0085] Figure 12An electronic paper tablet device 300 according to an embodiment of the present invention is illustrated. This electronic paper tablet device is connected to a detachable keyboard 1201 and held within a folio 1203. According to an embodiment of the present invention, the keyboard 1201 may have one or more special keys 1205, the execution of which will be discussed below. The keyboard 1201 and its operation of providing text input to the electronic paper tablet device 300 can be performed in a conventional manner. Further details regarding embodiments of folio 1203 and keyboard 1201 can be found in the following documents: U.S. Patent Application 17 / 985,755, filed November 11, 2022, entitled “Electric Terminals Electrically Connecting a Device Mount to a Computing Device,” by Gaute Wiig Nordby, Borge Strand-Bergesen, Erik Andre Bengtsson, and Vegard Bakke Svendsen; and U.S. Patent Application 17 / 985,754, filed November 11, 2022, entitled “Device Mount For Supporting A Computing Device,” both of which are incorporated herein by reference in their entirety.

[0086] Example of an active handwriting stylus

[0087] An active stylus pen (or more commonly, an "active pen") is a stylus input device that allows a user to write, sketch, or draw on the display of a computing device (e.g., an e-paper tablet device 110), assisting the e-paper tablet device 110 in operating as a computerized annotation system. An active stylus pen includes digital components and / or circuitry that communicate with the computing device (e.g., an e-paper tablet). This communication enables advanced features such as force (e.g., pressure) sensitivity, tilt detection, programmable buttons, palm detection, eraser tip, memory settings, and write data transmission. More broadly, communication between the computing device and the active stylus pen allows for the placement of various peripheral sensors within the active stylus pen, with the resulting data reported to the computing device, such as the e-paper tablet. Such sensors placed within an active stylus pen can range from simple buttons to enhanced artificial intelligence features.

[0088] The electronic components of an active pen typically include a power source that enables the device's electronics to provide lower latency and higher fidelity than other types of pens (e.g., passive pens). Active pens offer many advantages over passive pens, including hover latency; for example, an active pen can typically be activated only by being near a display (e.g., a display associated with the electronic paper tablet device 110).

[0089] Once a stylus touches or contacts the display screen of a device such as electronic paper tablet device 110, the electronics associated with the display screen generate a signal, which the electronic paper tablet (e.g., electronic paper tablet device 110) can process as a gesture made by the user. When a handwriting stylus gesture is detected, the electronics within the touch-sensitive screen generate a signal that encodes instructions for displaying content or updating content previously displayed on the screen of the electronic paper tablet device based on the movement of the detected gesture on the screen.

[0090] Compared to active pens, passive pens typically have no internal power supply. A passive pen remains inactive until it touches the device's screen (e.g., a tablet screen), causing a signal to travel from the device through the passive pen and back. The electronics associated with a passive pen can be integrated into the stylus device or even housed in a small case placed inside a pen-shaped cap designed to be more ergonomic than the smaller case containing the electronics and other components.

[0091] Figure 13The illustration shows an active stylus 1300, which includes a core component 1302. This core component itself includes one or more antennas configured to communicate with a tablet device, such as an electronic paper tablet device 110 operating as a computerized annotation system. The active stylus 1300 may include one or more force sensing systems 1304 that detect force, such as force applied by a user to the display of the electronic paper tablet device 110. Additionally, the active stylus 1300 can also help mimic human interaction with conventional writing instruments such as pencils and paper, for example, providing a "feeling of a pencil and paper stack." The active stylus 1300 also includes a power source, such as a battery 1306. In addition, the battery 1306 allows the active stylus 1300 to support a "hover" function, which allows the active stylus 1300 to enter a sleep state to conserve battery power when it is not actively engaged with the display of the e-paper tablet device 110, and to wake up from the sleep state when the core component 1302 detects proximity to the display of the e-paper tablet device 110. According to an embodiment of the invention, the active stylus 1300 may not actually draw lines on the display of the e-paper tablet device 110 until the tip of the active stylus 1300 physically touches the display of the e-paper tablet device 110.

[0092] The active pen 1300 typically includes a PCBA 1305, which includes the electronics required to drive the signal lines associated with the core component 1302. The PCBA 1305 may alternatively be configured as a flexible printed circuit (FPC). The PCBA 1305 may include a suitable active pen PCBA or IC / ASIC / MCU that processes data received from the core component 1304 to sense force or displacement pressure with high resolution during interaction between the active pen 1300 and an object (e.g., the surface of a computing device such as electronic paper tablet device 110).

[0093] The active pen 1300 can act as Figure 1 The input mechanism 120 described herein provides data input to the electronic paper tablet device 110, which can cause drawing to appear on the display of the electronic paper tablet device 110. The active pen 1300 may also include an erasing system that receives user commands related to the display portion of the screen of a computing device (e.g., the electronic paper tablet device 110) associated with erasing. In a sense, the erasing system operates exactly the same as an active handwriting stylus system, except that one is for drawing and the other for erasing.

[0094] During operation, when the marker tip engages with the display on a tablet device (e.g., electronic paper tablet device 110), the force sensing system 1304 receives the force applied to the marker tip of the active pen 1300 (e.g., ...). Figure 13 The physical force of the marker tip 1302 (as shown) is received and converted into an electronic signal, which is transmitted to a PCBA 1305. The electronic components of the PCBA can perform various processes on the received signal. The PCBA 1305 can then transmit the signal back to a flat panel device (e.g., an electronic paper flat panel device 110) for further actions (e.g., drawing lines).

[0095] In an embodiment where the secondary antenna system in the active pen 1300 implements the tail eraser function, the active pen 1300 also includes a second antenna system 1303 that enables the active pen 1300 to transmit and receive signals with a computing device (e.g., an electronic paper tablet device 110) via the tail eraser portion of the active pen 1300 near the cap 1308. In this embodiment, the active pen 1300 also includes a second force (e.g., pressure) sensing system 1307 that controls the force (e.g., pressure) applied to the display of the computing device from the erasing activity. Similarly, the PCBA 1305 may include a suitable active pen PCBA or IC / ASIC / MCU that processes the data received from the second antenna system 1303 associated with the erasing function.

[0096] The active pen 1300 typically also includes an outer housing 1301 as a stylus holder, which is typically cylindrical in shape and made of a non-metallic material (such as plastic). The stylus holder contains internal electronics within the housing 1301. A cap 1308 may be provided at the top of the housing 1301.

[0097] Figure 14 The illustration shows the outer housing 1401 of a stylus 1400 designed to fit a user's hand according to an embodiment of the present invention. The stylus 1400 includes a marker tip 1402 and an eraser 1403. The marker tip 1402 serves as... Figure 13 The core component 1302 shown operates as a part of the core component 1302, while the eraser 1403 acts as a part of the core component 1302. Figure 13 The second antenna system 1303 is shown in part of its operation. When an active stylus (e.g., stylus 1300 and / or stylus 1400) includes multiple antennas as part of a core (e.g., core 1302), the stylus can provide additional capabilities.

[0098] Further details regarding handwriting styluses interoperable with embodiments of the present invention can be found in the following documents: U.S. Patent Application Serial No. 18 / 779,151, filed July 22, 2024, entitled "Replaceable Conductive Marker Tip"; U.S. Patent Application Serial No. 18 / 779,154, filed July 22, 2024, entitled "Advanced Paper Emulation"; U.S. Patent Application Serial No. 18 / 779,158, filed July 22, 2024, entitled "Marker Writing System"; U.S. Patent Application Serial No. 18 / 779,164, filed July 22, 2024, entitled "Marker Writing System"; and U.S. Patent Application Serial No. 18 / 779,164, filed July 22, 2024, entitled "Captive Object". U.S. Patent Application Serial No. 18 / 779,170 entitled “FlexureMechanism”; and U.S. Patent Application Serial No. 18 / 779,149 entitled “Active Pen-Stylus Precise Eraser”, filed July 22, 2024, are owned by the applicant of this application and are incorporated herein by reference in their entirety.

[0099] Anchored annotations

[0100] Electronic paper flat panel device 110 (e.g.) Figure 12 The configuration shown can accept input from the keyboard (e.g., ...). Figure 12 The keyboard 1201 shown) and input mechanisms (such as...) Figure 13 The handwriting stylus 1301 shown can be used for input from both. Input can also be obtained from memory storage units (e.g., ...). Figure 1 The cloud server 130 shown uploads documents containing text and graphics, which are then edited on the electronic paper tablet device 110, which operates as a computerized annotation system. In all these applications, the electronic paper tablet device user can draw with a stylus or type with a keyboard.

[0101] When a user interacts with the document (e.g., using both the keyboard 1201 and the stylus 1301), the e-paper tablet device 110 stores the changes to the document in memory (e.g., local memory and / or remote memory). Figure 6The document content in the memory (604) shown can be represented using a data structure that facilitates operation in a distributed computing environment, such as a conflict-free replicated data type (CRDT) data structure. CRDT data structures can be replicated across multiple computers in a network and typically have the following characteristics: 1) Applications can independently and concurrently update any copy without coordinating with other copies; 2) Algorithms (which are themselves part of the CRDT data type) automatically resolve any inconsistencies that may occur; and 3) Although copies may have different states at any given point in time, they will eventually converge to a document.

[0102] Data structures such as CRDTs may be useful in other applications, such as coordinating and arranging document input in mixed-modal scenarios, for example... Figure 12 As shown, users can input information from multiple devices (e.g., keyboards, styluses, and possibly other input mechanisms) and / or even receive contributions to the document from a remote document creator. Therefore, CRDT ensures document consistency regardless of whether a single participant interacts with a tablet device (e.g., e-paper tablet device 110) using multiple input methods (e.g., keyboards and styluses) or with collaborators.

[0103] For example, a user can generate annotations for a document (e.g., free-form gestures that may include sketches, drawings, markings, underlines, and doodles, for example, made by the user using an input mechanism such as a stylus). When the user applies a marking input device (e.g., a stylus) to the physical screen, these annotations are recorded point-by-point at hardware-specific sampling intervals. Each sequence of the stylus input device touching the screen, moving to record points, and up to and including lifting can be considered a line. This line is then stored in the physical memory of the electronic paper tablet device 110 (e.g., ...). Figure 6 The memory shown is 604. For each line created, Figure 2 The anchoring module 270 shown can create a new group (e.g., a note) for the line or update an existing group of lines (e.g., another note). The anchoring module 270 can be implemented in hardware and / or software to operate with the electronic paper tablet device 110.

[0104] Comments in a document should generally be displayed relative to any other element to which they belong (e.g., text and / or drawing), even if that element is later changed or the document is displayed on a different computer system. Comments are typically not associated (anchored) to their corresponding text, such that changes to the text and / or to the comment itself and / or the display of the text on different devices can cause the comment's position relative to its corresponding text to become misaligned in the device display, making it impossible for the user to determine which part of the text corresponds to a given comment.

[0105] Annotation anchoring allows text, drawings, and annotations in a document to maintain their original (e.g., spatial) relationships, regardless of how the document is modified or by which computer system renders (displays) it. Further details about anchored annotations can be found in U.S. Patent Application No. 18 / 380,265, filed October 16, 2023, entitled "Anchoring annotations," the entire contents of which are incorporated herein by reference.

[0106] Figure 2 The anchoring module 270 shown associates anchor groups (e.g., sets of lines, glyphs, circles, etc.) with a unique identifier called an anchor ID. If the digital text has not yet arrived at the electronic paper tablet device 110 which already has a suitable CRDT format (e.g., with anchor IDs), the anchoring module 270 can assign a unique identifier (e.g., an anchor ID) to each character in the digital text.

[0107] Figure 15 An illustration of a portion of text 1501 is provided, where a specific character (e.g., “A”) has received a numeric identifier (e.g., anchor ID “1234567890” 1503). Comments can be anchored to this numeric identifier. For example, if the document does not already have such a unique identifier, the anchoring module 270 can assign a unique identifier (e.g., anchor ID) to each character in the document in the CRDT. Using these unique identifiers, the anchoring module 270 effectively knows the position of a character relative to nearby characters (e.g., adjacent characters or characters within a threshold number of characters).

[0108] The CRDT identifier for a character can take many forms and does not need to include the complete sequence of numbers. When a user edits digital text (with CRDT representation) using the virtual or physical keyboard of the e-paper tablet device 110, the digital text in memory is edited, but the unique identifier for each character (e.g., a CDRT identifier) ​​remains the same. Even if deleted characters are no longer visible to the user, these characters can retain their anchor IDs and relative positions in the digital text. Therefore, the anchoring module 270 can continuously derive the physical position associated with the display screen for each anchor ID, regardless of how the user edits the document. When due to... Figure 1 When the cloud storage unit 130 receives an updated version of the digital text and updates the digital text, the anchoring operation of the anchoring module 270 can also be performed.

[0109] When a user inserts or adds a comment to a document, the anchoring module 270 can associate the comment with a unique identifier (e.g., an anchor ID) of the text (e.g., a single character, word, or paragraph) in the document (referred to as an "anchor"). This process positions the comment in the document relative to the position / location of the associated anchor in the document. For example, if the text around the anchor changes, the anchoring module 270 or another element of the electronic paper tablet device 110 can move the anchor to a different location in the document and update the position of the comment in the document so that it moves according to the anchor displayed on the screen. Thus, the comment moves with the anchor. Accordingly, the movement of the comment is the same as the movement of the anchor.

[0110] If the anchor point position moves, the display of the anchored group will move accordingly based on the rules outlined in the display algorithm associated with display system 230. After anchoring module 270 assigns a unique identifier to an existing character, if a new character is inserted into the document, anchoring module 270 assigns a unique identifier (e.g., anchor ID) to the new character. By employing this system, the unique identifier of a character can remain the same even if a text character is inserted or removed before or after it. Anchoring module 270 can use these unique identifiers to determine the geometric position of the anchor point across editing and across devices or clients.

[0111] Anchoring module 270 can select the anchor point of the annotation based on several different parameters. Example parameters include: 1) geometry (e.g., the initial position of the annotation in the document, or the shape or size of the annotation); and 2) time (e.g., the time when the user generated the annotation relative to other annotations or text). For example, after the user generates the annotation, anchoring module 270 can anchor the annotation to the nearest character. In another example, anchoring module 270 anchors the annotation to the character closest to the center of the annotation.

[0112] Anchoring module 270 allows multiple annotations to be grouped together to share a single anchor point. For example, bracket, circular, and triangular annotations (together forming an exclamation mark) can be grouped together by anchoring module 270 so that these elements share a single anchor point. According to embodiments of the invention, anchoring module 270 can perform such grouping if an annotation is within a threshold distance of another annotation or if the user generates another annotation within a threshold time of an existing annotation. Such grouping can help preserve drawings or sketches that include multiple annotations (e.g., lines including the handwritten exclamation mark described above). In this way, grouping can capture and preserve the user's annotation intent.

[0113] Anchoring can be implemented in a variety of ways, such as Figure 16 The method shown. Figure 16An annotation “On”, represented as a series of points (e.g., point 1604), is shown according to an embodiment of the invention. Anchoring module 270 can determine the location of other objects (e.g., annotations), such as points displayed at a distance from the annotation “on”, such as point 1602. Anchoring module 270 can apply constraints to determine when these other objects are within the range associated with the annotation “on”. For example, in… Figure 16 In the process, anchoring module 270 applies an elliptical shape 1603 around point 1604 when determining which other objects are within the scope of the "on" annotation. Anchoring module 270 will similarly use elliptical shapes around other points within the "on" annotation to make the determination. (Note:) Figure 16 The dots shown are for illustrative purposes only. In reality, the dots for annotations such as "on" are more numerous than those for other purposes. Figure 16 (There are many more shown in the figure.) According to an embodiment of the invention, the anchoring module 270 can apply other shapes when performing the determination, and the anchoring module 270 can apply test areas of different sizes.

[0114] Figure 16 The embodiments shown do not involve using bounding boxes around annotations, while other embodiments may involve bounding boxes. According to embodiments of the invention, a bounding box is a rectangular box containing one or more objects (e.g., annotations). In embodiments using bounding boxes, anchoring module 270 compares the bounding boxes around the annotation to find other lines (e.g., annotations) to group and form larger compound annotations. Figure 16 As shown, if the anchoring module 270 instead examines all the individual points that make up an annotation to determine if these points are close to other points (which form another annotation), and then determines whether the two annotations should be linked together to form a compound annotation, the use of a bounding box can be avoided. Therefore, in such an embodiment, the bounding box check of the anchoring module 270 is replaced by a point-by-point check, in which each annotation comprises a plurality of individual points that form a multi-segment line (e.g., a letter or shape) comprising the annotation. Thus, the anchoring module 270 performs a check to compare the distance between each point in the new line (e.g., an annotation) and other points in existing lines / groups of lines (e.g., annotations) that may have some overlap. If none of the point-to-point distances in one annotation are within the margin used by another annotation, the anchoring module 270 does not group the new line (e.g., an annotation) with other annotations.

[0115] Figure 17The illustration shows a note 1701 (e.g., a strikethrough marker), which is anchored to a text character (e.g., "A") via an anchoring module 270 using a bounding box 1703. In this example, the anchoring module 270 selects the text character "A" as the anchor point for note 1701 because it is the character closest to the center of the bounding box 1703.

[0116] Figure 18 The illustration shows an example of a "boundaryless" design applied to two annotations, 1805 and 1809. (As shown...) Figure 18 As shown, anchoring module 270 has explored the polygonal regions surrounding the annotations "so" and "cool!", and anchoring module 270 has determined the targets above these two annotations. Figure 16 The discussion methods overlap. Therefore, anchoring module 270 forms a compound annotation 1807 consisting of both annotation 1805 and annotation 1809.

[0117] In the pseudocode, the logic executed by anchor module 270 is as follows:

[0118] for ptA in groupA(eg,"So")

[0119] for ptB in groupB(eg,"Cool")

[0120] if ellipseAround(ptA).contains(ptB)->it is a match

[0121] In embodiments using bounding boxes, anchoring module 270 can use the center of the bounding box to find the nearest point in the text to be used as an anchor. However, in embodiments without bounding boxes, anchoring module 270 can adjust the center by examining all points in a group (e.g., a note) and calculating an average position, where the anchoring module assigns exponential weights to points in the note that are closer to the text. Therefore, if anchoring module 270 encounters a note that includes an arrow pointing to the text, anchoring module 270 is likely to anchor the note closer to the tip of the arrow, even if the arrow is at the top of the entire note.

[0122] According to an embodiment of the invention, the anchoring module 270 can determine whether one or more (e.g., any) points in the new annotation intersect with existing text or annotations. If yes, the anchoring module 270 marks the annotation as "vertically and horizontally anchored." If no, the anchoring module marks the annotation as "vertically anchored." The determination may vary depending on whether the new annotation is above the text.

[0123] a) For the annotation “vertical and horizontal anchoring”, the anchoring module 270 (e.g., only) checks against other annotations that are already marked as “vertical and horizontal anchoring”.

[0124] (b) For lines that are “vertically anchored,” the anchoring module 270 (e.g., only) checks annotations that are already marked as “vertically anchored.” Additionally, the anchoring module 270 may add extra margin around the bounding box of the new annotation to more easily form groups outside the text. In other words, the anchoring module 270 may enlarge the search area or bounding box. For embodiments employing bounding boxes, this may result in the search area or bounding box intersecting with more annotations.

[0125] Anchoring module 270 identifies anchor points and selects anchor points for annotations. For embodiments without bounding boxes, anchoring module 270 similarly groups annotations and selects anchor points for them.

[0126] If the character to which a comment is anchored moves vertically within the document, the comment's display on the screen of the electronic paper tablet device 110 can move vertically by the same distance or substantially the same distance (e.g., + / - 10%). For example, if moving the comment vertically by the same distance would cause it to overlap with another comment, the anchoring module 270 interoperates with the display system 230 to adjust the movement so that these comments do not overlap. If the character moves only horizontally within the document, the comment may maintain its position. In other words, the position of the anchored comment can remain unchanged. Thus, for example, if a paragraph moves, a comment on one side of that paragraph (e.g., in the margin) will follow the paragraph.

[0127] If the character to which a comment is anchored moves within the document, the display of the comment may move the same distance vertically and horizontally, or substantially the same distance in both directions (e.g., + / - 10%). Thus, for example, if a word moves, a circular comment surrounding that word will follow. As mentioned above, in some embodiments, the comment movement does not perfectly match the anchor movement (e.g., to avoid creating overlapping comments).

[0128] Figures 19A to 19D The illustration shows the use of annotation anchoring by anchoring module 270 according to some embodiments and the effect of anchoring on text displayed in a part of the interface (e.g., a display screen).

[0129] Figure 19A The illustration shows a portion of an interface displayed by a computer system (e.g., electronic paper tablet device 110). This interface includes text paragraphs and annotations anchored by anchoring module 270. Figure 19B In the text, the first comment is added by the user below the second paragraph.

[0130] exist Figure 19C In the middle, the second paragraph moves downwards due to the user inserting a small image. Due to the early anchoring of anchor module 270, the first annotation moves downwards along with the second paragraph. The user adds the second annotation to the newly generated space between the first and second paragraphs.

[0131] exist Figure 19D In the example, the user creates a third paragraph between the first paragraph and the second comment. Because the anchoring module 270 anchors in advance, the second comment, along with the second paragraph and the first comment, moves downwards. Additionally, the anchoring module 270 adds a third comment between the first and third paragraphs.

[0132] Therefore, the electronic paper tablet device 110 can be configured to draw below and between text, with corresponding anchoring matching the changes made. Thus, the drawing below and between text will remain in its natural position.

[0133] Refined modes with and without function keys

[0134] The inventors have developed a method to assist users in seamlessly switching between creative input modes (e.g., creation mode) and editing / refinement modes (e.g., refinement mode) to complement the regular creation mode. Refinement mode serves as an aid to editing tasks, where typed notes and handwritten notes are creatively combined. At a higher level, refinement mode bridges the gap between handwriting, drawing, and typing experiences, enabling users of the e-paper tablet device 110 to fully explore and develop their ideas into breakthroughs. Therefore, refinement mode includes a "state" where the user transitions from creating content to reviewing / editing content, combining typed notes with handwritten notes and drawings, with the aim of providing a smooth hybrid input experience so that the most suitable tool is always used for work. The anchoring discussed above supports and to some extent implements refinement mode. Refinement mode can be implemented without anchoring, but it would be much more complex.

[0135] like Figure 12 The electronic paper tablet device 110, which operates as a computerized annotation system, may have a corresponding keyboard 1201. The keyboard 1201 may include special function keys 1205 that can be used to switch between creation mode and refinement mode. For example, executing special function key 1205 engages refinement mode, while releasing (e.g., pressing / clicking) special function key 1205 returns the user to creation mode. Special function key 1205 may include an indicator light that signals to the user that refinement mode has been engaged. When special function key 1205 is executed (e.g., pressed / clicked) a second time, the electronic paper tablet device 110 returns to creation mode, and the indicator light turns off to indicate the mode change to the user.

[0136] Figure 20 The illustration shows the display 2005 of the electronic paper tablet device 110, which is connected to the keyboard 1201 and can be used by the user with a stylus 1400. Figure 20 The hardware configuration shown is Figure 12 This is an abstract version of the hardware configuration shown. In authoring mode, the user can access menus and commands 2001 suitable for authoring mode, where they can draw new illustrations, write new comments, type new text, and take other actions suitable for authoring mode. Anchoring module 270 takes appropriate actions to link the new authoring inputs together, as described above.

[0137] If a user wishes to switch from creative mode to refined mode, the user can simply perform (e.g., press / click) the special function key 1205 (as above). Figure 12 The user can switch modes by pressing special function keys (on and off). Users can quickly switch between creation mode and refinement mode by pressing special function keys (on and off). Some users may be particularly adept at switching between modes. If the e-paper tablet device 110 is not attached to a physical keyboard, the user can still access the functions associated with refinement mode via the selection tools or menus displayed on the e-paper tablet device 110. In other words, the same functions are available to the user, but there may not be a clear distinction between creation mode and refinement mode. Therefore, for example, according to an embodiment of the invention, the user can also access refinement functions by making appropriate selections on menu 2001 in display 2005.

[0138] When a user switches from author mode to refinement mode, the e-paper tablet 110 and the stylus 1400 continue to operate in the same way—however, the e-paper tablet 110 may now interpret the user's input differently. Therefore, in a sense, the existence of refinement mode expands the possibilities for user creation and engagement with the e-paper tablet 110. For example, a set of gestures made in author mode might result in drawing a line on a paragraph, while the same gesture made in refinement mode might be interpreted differently by the e-paper tablet 110, leading to refinement (or editing) of the document presented on the display 2005. For instance, a line drawn on a paragraph in refinement mode might cause the paragraph to be highlighted or selected for subsequent actions such as cutting and pasting. The highlighted or selected area could include both text notes and handwritten notes, as well as drawings.

[0139] Similar to the creation mode, the refinement mode is implemented and supported by the anchoring module 270, which uses anchoring information to determine which part of the display the user's gesture has selected. As mentioned above, this selection can include text notes, handwritten notes, and drawings. According to an embodiment of the invention, the anchoring module 270 can interact with the refinement module 280.

[0140] When "Refinement Mode" is active, the user can use the stylus 1400 to select, move, rearrange, and edit elements displayed on the e-paper tablet device 110. Since many of these actions begin with selection, both "Refinement Mode" and the stylus 1400 effectively share certain functionalities supported by the anchoring module 270 and the refinement module 280. Embodiments of Refinement Mode cover a broader range of functions, extending beyond the capabilities of the stylus 1400 operating alone.

[0141] Figure 21 The illustration shows a display 2005 in refinement mode according to an embodiment of the present invention. When refinement mode is active, a status icon 2101 may appear on the display screen 300 to indicate that the electronic paper tablet device 110 is in refinement mode rather than creation mode.

[0142] As described above, users can operate the stylus 1400 to perform various tasks enabled by the refinement mode. The refinement mode allows users to comfortably and seamlessly refine the content that combines typed and handwritten text. For example:

[0143] • The format / function shortcuts between the creation mode and refinement mode allow users to easily create structures and organize their ideas in documents. For example, when a user is interested in creating, the creation mode on the e-paper tablet device 110 supports the necessary creation activities, and when a user is interested in refining the creation output, the refinement mode on the e-paper tablet device 110 supports the requested refinement activities.

[0144] The Refinement Mode aims to more seamlessly integrate the experiences of handwriting, typing, and line creation. For example, handwriting input via the stylus 1400 "Digital Power" on the e-paper tablet device 110 can even include features allowing users to search within their own handwritten content and complete input fields (such as notebook names, search fields, etc.) using handwritten text. Providing line flexibility for typed text allows for exploring the possibilities of adjusting and freely placing text boxes. In Refinement Mode, the stylus 1400 can be used to select and rearrange text, move handwritten notes, and draw. Refinement Mode allows users to convert selected handwritten text into typed text while preserving its layout. Refinement Mode also supports cutting content, such as text or images, from PDF documents.

[0145] The refinement mode enables an essentially unlimited number of user commands and actions. The basic idea is that user input using the stylus 1400 shifts from drawing lines, text, and graphics on the display 2005 to selecting, rearranging, changing, and editing content such as text and / or handwritten notes. These operations may have previously been available to the user through the use of similar menus on the display 2005, but refinement mode allows these selections to be performed directly and naturally, greatly reducing the need for intermediate steps involving menus.

[0146] The granular mode presents the user's "state" from creating content to reviewing / editing content. As discussed above, the switching between these states can be triggered by special function keys 1205 on the keyboard 1201 associated with the e-paper tablet device 110 and / or by the on-screen menu key (e.g., when the keyboard 1201 is not present).

[0147] Figures 22 to 35 The illustration shows some actions a user can take using the refinement mode according to an embodiment of the invention. However, additional functions can be operated within the spirit of the refinement mode, provided these actions are within the hardware capabilities of the electronic paper tablet device 110 and the capabilities of the anchoring module 270 and the refinement module 280, for example, effectively understanding the user's intention to make gestures using the stylus 1400. For example, Figure 36 A list of gestures that can be implemented in granular mode is provided to further expand its capabilities.

[0148] Figures 22 to 24 The illustration shows a series of actions that a user can take in refinement mode to select a segment of text, handwritten notes, and drawings on the display 2005, according to an embodiment of the present invention.

[0149] Figure 22 The illustration depicts the action of a user using a stylus 1400 to begin selecting text according to an embodiment of the present invention. The user draws lines 2203, such as... Figure 22 As shown. If the user draws line 2203 in author mode, then the line will become part of the document displayed on monitor 2005. When the user draws line 2203 in refinement mode, then the drawn line will not become part of the document displayed on monitor 2005. Line 2203 may be displayed on monitor 2005 in a different color (e.g., blue) than the document text displayed on monitor 2005. Anchoring module 270 helps determine the position of line 2203 within the document displayed on monitor 2005.

[0150] As mentioned above, the e-paper tablet device 110 may not be connected to a keyboard. In this case, if the user uses the selection tool in the menu presented on the e-paper tablet device 110, its behavior will be the same as the refinement mode discussed immediately below. In short, according to an embodiment of the invention, when a keyboard is not present, the user can use the stylus 1400 to erase, erase a segment, select, etc.

[0151] like Figure 23 As shown, according to an embodiment of the present invention, after a user draws a line 2203 by holding a stylus 1400 on the screen, a mark 2305 consisting of upward marks (e.g., triangles) and downward marks (e.g., triangles) can appear on the display 2005. Again, because the user has selected a thinning mode, the mark 2305 appears on the display 2005, but does not become part of the document displayed on the display 2005. The anchoring module 270 can determine the position of the mark 2305 within the document displayed on the display 2005.

[0152] like Figure 24 As shown, according to an embodiment of the present invention, the user uses a stylus 1400 to instruct... Figure 23 The position 2403 below the indicated mark 2305 allows the electronic paper tablet device 110 to select the document portion 2401 below mark 2305 on the display 2005. The electronic paper tablet device 110 can be configured to select all portions of the document below position 2403, or select all portions of the document below position 2403 as shown on the display 2005. According to embodiments of the invention, portion 2401 may be highlighted or displayed with some other similar indication.

[0153] The user's selection of portion 2401 can also cause the refinement mode to display a selection menu 2405. The selection menu may include commands such as cut, copy, paste, bold, and italic. If the user presses one of these commands using the stylus 1400, the necessary actions will be taken on the selected portion 2401 (e.g., the selected portion can be cut, copied, pasted, made bold, or italicized). The anchoring module 270 works in conjunction with the refinement module 280 to enable the highlighting of portion 2401 and / or the actions taken using the selection menu 2405. In other words, the anchoring module 270 determines the document portion affected by the refinement module 280, and other software on the electronic paper tablet device 110 can perform any selected action, the results of which are reflected in the underlying document and / or displayed on the display 2005.

[0154] Figures 25 to 27The illustration shows a series of actions that a user can take in refinement mode to select a segment of text, handwritten notes, and drawings on the display 2005, according to another embodiment of the invention.

[0155] Figure 25 The illustration depicts the action of a user using a stylus 1400 to begin selecting text according to an embodiment of the present invention. The user draws short lines 2501 with the stylus 1400, such as... Figure 25 As shown. If the user draws line 2501 in author mode, the line will become part of the text on monitor 2005. When the user draws line 2501 in refinement mode, the drawn line will not become part of the document displayed on monitor 2005. Line 2501 can be displayed in a different color than the document color displayed on monitor 2005 (e.g., blue). Anchoring module 270 helps determine the position of line 2501 within the document displayed on monitor 2005.

[0156] like Figure 26 As shown, according to an embodiment of the present invention, after the user draws line 2501, a mark 2601 consisting of upward marks (e.g., triangles) and downward marks (e.g., triangles) can appear on display 2005. Again, because the user selects the thinning mode, mark 2601 appears on display 2005, but does not become part of the document displayed on display 2005. Anchoring module 270 can determine the position of mark 2601 within the document displayed on display 2005.

[0157] like Figure 27 As shown, the thinning mode via the thinning module 280 has extended another line 2701 beyond the endpoint of the line 2501 where the mark 2601 appears. If the user then uses the stylus 1400 to select above or below the line formed by line segments 2501 and 2701, it will appear in a similar manner to... Figure 24 Select the document portion above or below the line as shown.

[0158] Figures 22 to 24 and Figures 25 to 27 Both embodiments of the invention shown allow users to bypass certain parts of a document on the display 2005 when making their own selections; for example, a user can select text but not related comments, or vice versa.

[0159] Figure 28 and Figure 29 An embodiment of a refinement mode for editing a document on a display 2005 according to an embodiment of the present invention is illustrated.

[0160] like Figure 28As shown, a user draws a line 2801 on a piece of text displayed on a monitor 2005 using a stylus 1400. If the user draws line 2801 in author mode, then line 2801 becomes part of the document displayed on the monitor 2005. However, since line 2801 represents a command in refinement mode, line 2801 does not become part of the document on the monitor 2005. Anchoring module 270, in cooperation with refinement module 280, determines which parts of the document displayed on the monitor 2005 are affected by line 2801. According to an embodiment of the invention, line 2801 may be displayed in a different color than the text displayed on the monitor 2005.

[0161] Figure 29 The illustration shows an embodiment of the invention, and Figure 28 The document portion 2901 associated with line 2801 shown is selected on display 2005 due to the thinning mode. The user's selection of portion 2901 can also cause the thinning mode to display a selection menu 2903. The selection menu may include commands such as cut, copy, paste, bold, and italic. If the user presses one of these commands using the stylus 1400, an action will be taken on the selected portion 2901 (e.g., the selected portion can be cut, copied, pasted, made bold, or italicized) and the document displayed on display 2005 will change. Anchor module 270 cooperates with thinning module 280 to enable the highlighting of portion 2901 and / or the actions taken using selection menu 2903. In other words, anchor module 270 determines the document portion affected by thinning module 280, and other software on electronic paper tablet device 110 can perform any selected action within the document and on display 2005.

[0162] Figure 30 and Figure 31 An embodiment of the present invention is illustrated for creating a refinement mode for adding additional whitespace in a document on a display 2005. A similar process can also be used to remove whitespace from a document.

[0163] like Figure 30 As shown, the user taps a portion of a document displayed on the monitor 2005 using the stylus 1400, then holds the stylus on the monitor 2005, and then, for example, moves the stylus 1400 downwards to create a space between two paragraphs. Figure 26 Similarly, this action causes the "refinement mode" to display marker 3001. Marker 3001 may include an upper portion (e.g., a triangle) and a lower portion (e.g., a triangle). The user can then use the stylus 1400 to perform an action (e.g., a tap) above or below marker 3001.

[0164] Figure 31 The illustration shows the user using the 1400 stylus pen. Figure 30 An action (e.g., a click) is taken below the marker 3001 to add additional blank space 3101 to the document shown on display 2005. The amount of blank space added by this action can be predetermined and / or can be a variable setting within the refinement module 280 on the electronic paper tablet device 110. Thus, a user can switch to refinement mode to create blank space between paragraphs and then switch back to authoring mode to continue writing. Alternatively, according to embodiments of the invention, authoring mode may also include additional commands, wherein the user can use a stylus to “stretch” the document above or below the marker 3001 by drawing an appropriate distance.

[0165] As mentioned above, a similar process can be used to remove whitespace from a document.

[0166] Figure 32 and Figure 33 The illustration shows a series of steps that a user can take to move and / or rearrange portions of a document displayed on a display 2005, according to an embodiment of the present invention.

[0167] like Figure 32 As shown, the user has previously used the refinement mode to select a specific segment 3201 of the document displayed on monitor 2005. The selected specific segment 3201 of the document includes a portion of text and handwritten notes. The refinement mode has already caused this portion to be highlighted, but the specific segment 3201 can be indicated in other ways.

[0168] The user's selection of a specific segment 3201 can also cause the refinement mode to display a selection menu 3203. The selection menu can be commands such as cut, copy, paste, bold, and italic. If the user presses one of these commands using the stylus 1400, an action will be taken on the specific segment 3201 (e.g., the selected portion can be cut, copied, pasted, made bold, or italicized). The anchoring module 270 collaborates with the refinement module 270 to enable the highlighting of the specific segment 3201 and / or the actions taken using the selection menu 3203. In other words, the anchoring module 270 determines the portion of the document affected by the refinement module 270, and other software on the electronic paper tablet device 110 can perform any selected action.

[0169] like Figure 33As shown, once the user begins dragging the selected fragment 3201 to another location on the display 2005, a line 3301 appears on the display 2005. The line 3301 can be drawn automatically to provide feedback to the user about where the selected fragment 3201 will be moved. Because the electronic paper tablet device 110 is in thinning mode, the line 3301 can be displayed on the display 2005, but it is not part of the document displayed on the display 2005. In this embodiment of thinning mode, the user can then take an action, such as clicking within the selection 3201, which will cause the thinning mode to move the selection 3201 to the position of the line 3301, as shown both on the display 2005 and on the document itself. For example, the tip of the stylus 1400 might need to be positioned above the horizontal line below the line “why this myth is destructive,” as... Figure 33 As shown. According to an embodiment of the present invention, the anchoring module 270 and the refinement module 280 cooperate to achieve this change to the document.

[0170] Figure 34 and Figure 35 The illustration depicts a series of steps related to a refinement mode according to an embodiment of the present invention, which allows a user to edit a given document displayed on a display 2005 by placing caret characters in the text.

[0171] like Figure 34 As shown, in refinement mode, the user uses the stylus 1400 to make a small mark 3401 in the text displayed on the monitor 2005. If the mark 3401 is made in authoring mode, then the mark is drawn on the monitor 2005 and added to the document on the monitor 2005. Because the electronic paper tablet device 110 is in refinement mode, the mark 3401 appears on the monitor 2005, but it does not cause any changes to the document on the monitor 2005. Refinement mode can also cause the cursor to appear along with the mark 3401.

[0172] Figure 35 Further illustration shows that when the user switches to author mode and begins typing on keyboard 1201 to add new text 3501 to the document displayed on monitor 2005, Figure 34 The result is that an insertion character is placed next to the small mark 3401 in the document. New text 3501 is added to the document displayed on monitor 2005. Through the collaboration of anchoring module 270 and refinement module 280, the position of the new text is linked to the appropriate position in the document.

[0173] Figure 36The illustration further illustrates gestures that a user can make using a stylus 1400 in finer mode for editing a document, according to an embodiment of the present invention.

[0174] In the first gesture 3601 of the refinement mode, the user erases a section of text and then holds the stylus in place. Refinement mode can use gesture 3601 to perform tasks such as deleting erased text from the underlying document and display 2005.

[0175] In the second gesture 3603, in refinement mode, the user draws a circle around a text element and holds the stylus in place. Refinement mode allows the user to select text in a document using gesture 3603 for purposes such as highlighting, cutting, copying, pasting, moving, dragging, and even creating hyperlinks. Refinement mode ensures that such changes are made to the underlying document and on the monitor 2005.

[0176] In Gesture 3605, the user draws a box around a text element and holds the stylus in place. Fine-tuning mode can use Gesture 3605 to perform similar tasks to Gesture 3603. The advantage of gestures like Gesture 3605 is that users don't need to interact with the toolbar or select tasks like highlighting from a menu.

[0177] In the fourth gesture 3607, the user adds a V-shaped symbol or caret between two text elements and holds the stylus in that position. The refinement mode can use gesture 3607 for tasks such as inserting words into a document or opening an input field to write one or two words into the document, possibly using the stylus 1400 to input between existing text segments in the document.

[0178] In the fifth gesture 3609, users make a point with a stylus and then draw a line to another point, allowing them to drag the text that is currently being drawn. The refinement mode allows for various tasks using the gesture 3609, such as precisely moving the caret to apply further refinement.

[0179] In the sixth gesture 3611, the user makes a dot, then draws a line below and above a piece of text, and then another line on top of the text. This allows the now-enclosed text to be effectively lifted by the handle drawn by the user. The refinement mode can use gesture 3611 for various tasks, such as precisely adjusting selections within a document for further refinement.

[0180] The refinement mode allows users to enjoy new gestures implemented over time using the same hardware set up in the e-paper tablet device 110. Refinement mode may also include features such as spell check. As described above, using the e-paper tablet device 110 in refinement mode, users can seamlessly switch between typed and handwritten notes and creatively combine them. Refinement mode provides users with the flexibility to organize, construct, and better understand their own ideas. The e-paper tablet device 110 thus becomes a device that further supports the user's own way of thinking.

[0181] Additional considerations

[0182] It should be understood that the accompanying drawings and descriptions of this disclosure have been simplified to illustrate elements relevant to a clear understanding of this disclosure, while many other elements present in typical systems have been omitted for clarity. Those skilled in the art will recognize that other elements and / or steps are desirable and / or necessary when implementing this disclosure. However, because such elements and steps are well-known in the art and because they are detrimental to a better understanding of this disclosure, discussion of such elements and steps is not provided herein. The disclosure herein relates to all such variations and modifications of these elements and methods known to those skilled in the art.

[0183] Some portions of this disclosure describe embodiments with respect to the algorithms and symbolic representations of information operations. These algorithmic descriptions and representations are commonly used by those skilled in the art of data processing to effectively communicate the substance of their work to others skilled in the art. While these operations are described functionally, computationally, or logically, they should be understood as being implemented via computer programs or equivalent circuits, microcode, etc. Furthermore, it has sometimes proven convenient, without loss of generality, to refer to these operations as an engine. The described operations and their associated engines can be embodied in software, firmware, hardware, or any combination thereof.

[0184] As used herein, any reference to "an embodiment" or "embodiment" means that a particular element, feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment. Throughout this specification, the phrase "in an embodiment" does not necessarily refer to the same embodiment in all instances.

[0185] When a value is described as “about” or “basically” (or its derivatives), it should be interpreted as accurate to + / - 10%, unless the context reveals otherwise. As an example, “about ten” should be understood as meaning “in the range of nine to eleven”.

[0186] As used herein, the terms “comprising,” “including,” “having,” or any other variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, article, or apparatus that includes a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Furthermore, unless expressly stated to the contrary, “or” means inclusive or rather, not exclusive. For example, condition A or B is satisfied in any of the following cases: A is true (or exists) and B is false (or does not exist); A is false (or does not exist) and B is true (or exists); and both A and B are true (or exist).

[0187] Additionally, the terms "a" or "an" are used to describe elements and components in the embodiments described herein. This is merely for convenience and to give the general meaning of the invention. This description should be understood to include one or at least one, and the singular includes the plural, unless it is obvious that it has a different meaning.

[0188] While specific embodiments and applications have been illustrated and described, it should be understood that the disclosed embodiments are not limited to the precise constructions and components disclosed herein. Various modifications, alterations, and variations that will be apparent to those skilled in the art may be made to the arrangement, operation, and details of the methods and apparatus disclosed herein without departing from the spirit and scope of this disclosure.

Claims

1. A computerized annotation system comprising: a computer memory; a display screen displaying a document having textual characters, wherein the textual characters have an underlying document representation stored in the computer memory such that each textual character in the document has a unique identifier; an input detector module receiving a first annotation to the document displayed on the display screen, wherein the first annotation comprises graphical data resulting from an interaction between the display screen and an input mechanism, wherein the input detector module initiates an update to the display screen to display both the document and the first annotation; and a refinement module determining whether the computerized annotation system is in a composition mode or a refinement mode, wherein if the computerized annotation system is in the composition mode, the refinement module initiates storing an update to the document including the first annotation in the computer memory, and if the computerized annotation system is in the refinement mode, the refinement module initiates a refinement to the document.

2. The computerized annotation system of claim 1, wherein, The refinement module determines whether the computerized annotation system is in the composition mode or the refinement mode based on a state of a special key on a keyboard electronically connected to the computerized annotation system.

3. The computerized annotation system of claim 2, wherein, A user of the computerized annotation system engages the special key to enter the refinement mode and disengages the special key to enter the composition mode.

4. The computerized annotation system of claim 1, wherein, The first annotation indicates a refinement to the document.

5. The computerized annotation system of claim 4, wherein, The input detector module receives additional annotations in the refinement mode, and wherein the refinement module determines a refinement to be made to the document based on the first annotation and at least one of the additional annotations.

6. The computerized annotation system of claim 5, wherein, The refinement module determines that the refinement to be made to the document in the refinement mode includes one of moving content in the document to another location in the document, editing text in the document, deleting text in the document, creating additional whitespace in the document, removing whitespace in the document, and changing a formatting feature of the document.

7. The computerized annotation system of claim 6, wherein, The first annotation includes a line drawn on the document from a first side to a second side, wherein the refinement module instructs the display screen to project an indicator proximately positioned adjacent to the line on the display screen, wherein the input detector module receives a second annotation comprising a click on the display screen from the input mechanism, wherein the refinement mode instructs the display screen to select content of the document displayed on the display screen based on the click location.

8. The computerized annotation system of claim 7, wherein, The refinement mode instructs the display screen to project a selection menu proximate to the selected content of the document.

9. The computerized annotation system of claim 8, wherein, The selection menu displays graphical commands for at least one of cutting the selected content of the document, copying the selected content of the document, cutting the selected content of the document, rendering text in the selected content of the document in bold, and rendering text in the selected content of the document in italics.

10. The computerized annotation system of claim 1, wherein, The input mechanism is a stylus electronically connected to the computerized annotation system.

11. The computerized annotation system of claim 1, further comprising an anchoring module that: calculates a center of the first annotation; determining a text character in the document that is closest to the determined first annotation center on the display screen; and storing in the computer memory a link between the determined unique identifier of the text character and the first annotation, wherein the link is an anchor point for the first annotation and the document such that when the document is displayed in the future, the first annotation remains linked to the display of the determined text character.

12. The computerized annotation system of claim 11, wherein, the anchoring module computing the center of the first annotation includes computing a weighted center of the first annotation such that portions of the first annotation that are closer to the document on the display screen are given a higher weight in determining the center of the first annotation.

13. The computerized annotation system of claim 11, wherein, the input detector module is further configured to: change the location of one or more text characters in the document on the display screen to accommodate the edit, the determined text character being one of the one or more text characters that is changed in location on the display screen; and change the location of the annotation based on the change in location of the determined text character.

14. The computerized annotation system of claim 1, wherein, the document displayed on the display screen is encoded in the computer memory in a conflict resolution data type (CRDT) data structure.

15. A computerized method in a computerized annotation system, comprising: retrieving a document having text characters from a computer memory for display on a display screen of the computerized annotation system, wherein the text characters have an underlying document representation stored in the computer memory such that each text character in the document has a unique identifier; receiving a first annotation to the document displayed on the display screen, wherein the first annotation includes graphical data resulting from an interaction between the display screen and an input mechanism; updating the display screen to show both the document and the first annotation; determining whether the computerized annotation system is in a create mode or a refine mode; if it is determined that the computerized annotation system is in the create mode, storing the first annotation in the computer memory and altering the document; and if it is determined that the computerized annotation system is in the refine mode, initiating a refinement to the document.

16. The computerized method of claim 15, further comprising: receiving an electronic input from a special key on a keyboard that is electronically connected to the computerized annotation system, wherein the electronic input determines whether the computerized annotation system is in a create mode or a refine mode.

17. The computerized method of claim 15, wherein, determining that the computerized annotation system is in the refine mode, the method further comprising: receiving a second annotation to the document displayed on the display screen, wherein the second annotation includes graphical data resulting from an interaction between the display screen and an input mechanism; determining a refinement to the document based on the first annotation and the second annotation.

18. The computerized method of claim 15, wherein, the first annotation includes a line drawn on the document from a first side to a second side, the method further comprising: displaying an indicator on the display screen that is positioned proximate to the line; receiving a second annotation that includes a click on the display screen from an input mechanism; and selecting content of the document displayed on the display screen based on a location of the second annotation on the display screen.

19. The computerized method of claim 18, further comprising: A selection menu is projected on the display proximate the selected content of the document, wherein the selection menu displays graphical commands for at least one of: cutting the selected content of the document, copying the selected content of the document, pasting the selected content of the document, presenting text in the selected content of the document in bold, and presenting text in the selected content of the document in italics.

20. The computerized method of claim 15, wherein, The document displayed on the display is encoded in the computer memory in a conflict-free replicated data type (CRDT) data structure.

Images